Sep 10 21:59:17.890289 ip-10-0-134-144 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 10 21:59:17.890301 ip-10-0-134-144 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 10 21:59:17.890308 ip-10-0-134-144 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 10 21:59:17.890615 ip-10-0-134-144 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 10 21:59:27.356628 ip-10-0-134-144 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 10 21:59:27.356644 ip-10-0-134-144 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 3746888c654547bf966993ad05c2d87c --
Sep 10 22:01:17.256999 ip-10-0-134-144 systemd[1]: Starting Kubernetes Kubelet...
Sep 10 22:01:17.635614 ip-10-0-134-144 kubenswrapper[2582]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 10 22:01:17.635614 ip-10-0-134-144 kubenswrapper[2582]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 10 22:01:17.635614 ip-10-0-134-144 kubenswrapper[2582]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 10 22:01:17.635614 ip-10-0-134-144 kubenswrapper[2582]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 10 22:01:17.635614 ip-10-0-134-144 kubenswrapper[2582]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 10 22:01:17.636467 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.636380    2582 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 10 22:01:17.640037 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.640021    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 22:01:17.640037 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.640037    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 22:01:17.640101 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.640040    2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 22:01:17.640101 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.640044    2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 22:01:17.640101 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.640047    2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 22:01:17.640101 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.640049    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 22:01:17.640101 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.640052    2582 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 22:01:17.640101 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.640055    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 22:01:17.640101 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.640058    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 22:01:17.640101 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.640060    2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 22:01:17.640101 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.640064    2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 22:01:17.640101 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.640066    2582 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 22:01:17.640101 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.640069    2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 22:01:17.640101 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.640072    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 22:01:17.640101 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.640075    2582 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 22:01:17.640101 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.640077    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 22:01:17.640101 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.640080    2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 22:01:17.640101 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.640083    2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 22:01:17.640101 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.640086    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 22:01:17.640101 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.640089    2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 22:01:17.640101 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.640098    2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 22:01:17.640101 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.640101    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 22:01:17.640588 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.640104    2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 22:01:17.640588 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.640107    2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 22:01:17.640588 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.640112    2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 22:01:17.640588 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.640116    2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 22:01:17.640588 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.640119    2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 22:01:17.640588 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.640122    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 22:01:17.640588 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.640125    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 22:01:17.640588 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.640128    2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 22:01:17.640588 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.640130    2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 22:01:17.640588 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.640133    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 22:01:17.640588 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.640136    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 22:01:17.640588 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.640139    2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 22:01:17.640588 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.640142    2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 22:01:17.640588 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.640145    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 22:01:17.640588 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.640147    2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 22:01:17.640588 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.640150    2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 22:01:17.640588 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.640153    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 22:01:17.640588 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.640156    2582 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 22:01:17.640588 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.640158    2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 22:01:17.641074 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.640161    2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 22:01:17.641074 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.640164    2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 22:01:17.641074 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.640167    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 22:01:17.641074 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.640169    2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 22:01:17.641074 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.640172    2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 22:01:17.641074 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.640175    2582 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 22:01:17.641074 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.640178    2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 22:01:17.641074 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.640181    2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 22:01:17.641074 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.640184    2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 22:01:17.641074 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.640187    2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 22:01:17.641074 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.640190    2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 22:01:17.641074 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.640192    2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 22:01:17.641074 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.640195    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 22:01:17.641074 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.640197    2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 22:01:17.641074 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.640201    2582 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 22:01:17.641074 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.640204    2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 22:01:17.641074 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.640206    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 22:01:17.641074 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.640209    2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 22:01:17.641074 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.640213    2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 22:01:17.641541 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.640217    2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 22:01:17.641541 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.640221    2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 22:01:17.641541 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.640224    2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 22:01:17.641541 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.640227    2582 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 22:01:17.641541 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.640230    2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 22:01:17.641541 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.640232    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 22:01:17.641541 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.640235    2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 22:01:17.641541 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.640237    2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 22:01:17.641541 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.640240    2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 22:01:17.641541 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.640243    2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 22:01:17.641541 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.640245    2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 22:01:17.641541 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.640248    2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 22:01:17.641541 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.640251    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 22:01:17.641541 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.640254    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 22:01:17.641541 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.640256    2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 22:01:17.641541 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.640259    2582 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 22:01:17.641541 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.640262    2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 22:01:17.641541 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.640264    2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 22:01:17.641541 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.640267    2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 22:01:17.642060 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.640270    2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 22:01:17.642060 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.640273    2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 22:01:17.642060 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.640275    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 22:01:17.642060 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.640278    2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 22:01:17.642060 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.640281    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 22:01:17.642060 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.640283    2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 22:01:17.642060 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.640286    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 22:01:17.642060 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.640289    2582 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 22:01:17.642060 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.640291    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 22:01:17.642060 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.640294    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 22:01:17.642060 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.640296    2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 22:01:17.642060 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.640299    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 22:01:17.642060 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.640301    2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 22:01:17.642060 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.640304    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 22:01:17.642060 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.640306    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 22:01:17.642060 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.640308    2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 22:01:17.642060 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.640311    2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 22:01:17.642060 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.640313    2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 22:01:17.642060 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.640316    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 22:01:17.642060 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.640318    2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 22:01:17.642541 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641018    2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 22:01:17.642541 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641026    2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 22:01:17.642541 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641030    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 22:01:17.642541 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641033    2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 22:01:17.642541 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641036    2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 22:01:17.642541 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641039    2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 22:01:17.642541 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641041    2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 22:01:17.642541 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641044    2582 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 22:01:17.642541 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641048    2582 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 22:01:17.642541 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641051    2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 22:01:17.642541 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641053    2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 22:01:17.642541 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641058    2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 22:01:17.642541 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641061    2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 22:01:17.642541 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641064    2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 22:01:17.642541 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641067    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 22:01:17.642541 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641070    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 22:01:17.642541 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641073    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 22:01:17.642541 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641076    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 22:01:17.642541 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641079    2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 22:01:17.643027 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641082    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 22:01:17.643027 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641085    2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 22:01:17.643027 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641088    2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 22:01:17.643027 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641091    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 22:01:17.643027 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641093    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 22:01:17.643027 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641096    2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 22:01:17.643027 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641098    2582 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 22:01:17.643027 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641101    2582 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 22:01:17.643027 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641104    2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 22:01:17.643027 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641106    2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 22:01:17.643027 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641109    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 22:01:17.643027 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641111    2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 22:01:17.643027 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641114    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 22:01:17.643027 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641117    2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 22:01:17.643027 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641119    2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 22:01:17.643027 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641122    2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 22:01:17.643027 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641125    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 22:01:17.643027 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641128    2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 22:01:17.643027 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641130    2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 22:01:17.643027 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641134    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 22:01:17.643512 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641136    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 22:01:17.643512 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641141    2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 22:01:17.643512 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641144    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 22:01:17.643512 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641147    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 22:01:17.643512 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641150    2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 22:01:17.643512 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641153    2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 22:01:17.643512 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641155    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 22:01:17.643512 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641158    2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 22:01:17.643512 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641161    2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 22:01:17.643512 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641164    2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 22:01:17.643512 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641167    2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 22:01:17.643512 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641169    2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 22:01:17.643512 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641171    2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 22:01:17.643512 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641174    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 22:01:17.643512 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641177    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 22:01:17.643512 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641179    2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 22:01:17.643512 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641182    2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 22:01:17.643512 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641184    2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 22:01:17.643512 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641187    2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 22:01:17.643995 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641189    2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 22:01:17.643995 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641192    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 22:01:17.643995 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641194    2582 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 22:01:17.643995 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641197    2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 22:01:17.643995 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641200    2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 22:01:17.643995 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641202    2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 22:01:17.643995 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641205    2582 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 22:01:17.643995 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641207    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 22:01:17.643995 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641210    2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 22:01:17.643995 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641213    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 22:01:17.643995 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641216    2582 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 22:01:17.643995 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641218    2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 22:01:17.643995 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641221    2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 22:01:17.643995 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641224    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 22:01:17.643995 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641227    2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 22:01:17.643995 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641229    2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 22:01:17.643995 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641232    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 22:01:17.643995 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641234    2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 22:01:17.643995 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641237    2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 22:01:17.643995 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641240    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 22:01:17.644479 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641243    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 22:01:17.644479 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641246    2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 22:01:17.644479 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641248    2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 22:01:17.644479 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641251    2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 22:01:17.644479 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641253    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 22:01:17.644479 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641256    2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 22:01:17.644479 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641258    2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 22:01:17.644479 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641261    2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 22:01:17.644479 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641263    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 22:01:17.644479 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641266    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 22:01:17.644479 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641268    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 22:01:17.644479 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641271    2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 22:01:17.644479 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641273    2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 22:01:17.644479 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641276    2582 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 22:01:17.644479 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641279    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 22:01:17.644479 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641281    2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 22:01:17.644479 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641284    2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 22:01:17.644479 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641286    2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 22:01:17.644479 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641289    2582 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 22:01:17.644479 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641291    2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 22:01:17.644479 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641294    2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 22:01:17.645004 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641667    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 22:01:17.645004 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641671    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 22:01:17.645004 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641674    2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 22:01:17.645004 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641677    2582 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 22:01:17.645004 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641680    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 22:01:17.645004 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641684    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 22:01:17.645004 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641686    2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 22:01:17.645004 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641689    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 22:01:17.645004 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641692    2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 22:01:17.645004 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641694    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 22:01:17.645004 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641697    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 22:01:17.645004 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641700    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 22:01:17.645004 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641702    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 22:01:17.645004 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641705    2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 22:01:17.645004 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641708    2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 22:01:17.645004 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641710    2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 22:01:17.645004 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641713    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 22:01:17.645004 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641732    2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 22:01:17.645004 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641734    2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 22:01:17.645004 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641737    2582 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 22:01:17.645490 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641740    2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 22:01:17.645490 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641742    2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 22:01:17.645490 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641745    2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 22:01:17.645490 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641748    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 22:01:17.645490 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641751    2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 22:01:17.645490 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641753    2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 22:01:17.645490 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641756    2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 22:01:17.645490 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641759    2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 22:01:17.645490 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641761    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 22:01:17.645490 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641764    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 22:01:17.645490 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641767    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 22:01:17.645490 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641771    2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 22:01:17.645490 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641775    2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 22:01:17.645490 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641778    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 22:01:17.645490 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641781    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 22:01:17.645490 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641783    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 22:01:17.645490 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641786    2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 22:01:17.645490 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641789    2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 22:01:17.645490 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641791    2582 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 22:01:17.645490 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641794    2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 22:01:17.646007 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641797    2582 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 22:01:17.646007 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641799    2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 22:01:17.646007 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641802    2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 22:01:17.646007 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641804    2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 22:01:17.646007 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641808    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 22:01:17.646007 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641810    2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 22:01:17.646007 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641813    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 22:01:17.646007 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641817    2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 22:01:17.646007 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641821    2582 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 22:01:17.646007 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641824    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 22:01:17.646007 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641827    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 22:01:17.646007 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641830    2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 22:01:17.646007 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641833    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 22:01:17.646007 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641835    2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 22:01:17.646007 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641838    2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 22:01:17.646007 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641840    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 22:01:17.646007 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641843    2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 22:01:17.646007 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641845    2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 22:01:17.646007 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641848    2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 22:01:17.646479 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641851    2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 22:01:17.646479 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641853    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 22:01:17.646479 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641855    2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 22:01:17.646479 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641858    2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 22:01:17.646479 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641862    2582 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 22:01:17.646479 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641865    2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 22:01:17.646479 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641868    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 22:01:17.646479 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641870    2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 22:01:17.646479 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641873    2582 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 22:01:17.646479 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641875    2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 22:01:17.646479 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641878    2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 22:01:17.646479 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641881    2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 22:01:17.646479 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641883    2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 22:01:17.646479 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641886    2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 22:01:17.646479 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641889    2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 22:01:17.646479 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641891    2582 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 22:01:17.646479 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641894    2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 22:01:17.646479 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641896    2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 22:01:17.646479 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641899    2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 22:01:17.646479 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641902    2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 22:01:17.647018 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641908    2582 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 22:01:17.647018 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641910    2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 22:01:17.647018 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641912    2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 22:01:17.647018 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641915    2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 22:01:17.647018 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641918    2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 22:01:17.647018 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641920    2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 22:01:17.647018 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641922    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 22:01:17.647018 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641925    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 22:01:17.647018 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641928    2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 22:01:17.647018 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641930    2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 22:01:17.647018 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641933    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 22:01:17.647018 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641937    2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 22:01:17.647018 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641940    2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 22:01:17.647018 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641943    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 22:01:17.647018 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641946    2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 22:01:17.647018 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641948    2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 22:01:17.647018 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641953    2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 22:01:17.647018 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641955    2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 22:01:17.647018 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641958    2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 22:01:17.647474 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.641961    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 22:01:17.647474 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.642029    2582 flags.go:64] FLAG: --address="0.0.0.0"
Sep 10 22:01:17.647474 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.642074    2582 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 10 22:01:17.647474 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.642082    2582 flags.go:64] FLAG: --anonymous-auth="true"
Sep 10 22:01:17.647474 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.642086    2582 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 10 22:01:17.647474 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.642091    2582 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 10 22:01:17.647474 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.642094    2582 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 10 22:01:17.647474 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.642098    2582 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 10 22:01:17.647474 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.642102    2582 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 10 22:01:17.647474 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.642105    2582 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 10 22:01:17.647474 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.642108    2582 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 10 22:01:17.647474 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.642112    2582 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 10 22:01:17.647474 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.642115    2582 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 10 22:01:17.647474 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.642120    2582 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 10 22:01:17.647474 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.642123    2582 flags.go:64] FLAG: --cgroup-root=""
Sep 10 22:01:17.647474 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.642126    2582 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 10 22:01:17.647474 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.642130    2582 flags.go:64] FLAG: --client-ca-file=""
Sep 10 22:01:17.647474 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.642133    2582 flags.go:64] FLAG: --cloud-provider="external"
Sep 10 22:01:17.647474 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.642136    2582 flags.go:64] FLAG: --cluster-dns="[]"
Sep 10 22:01:17.647474 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.642140    2582 flags.go:64] FLAG: --cluster-domain=""
Sep 10 22:01:17.647474 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.642143    2582 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 10 22:01:17.647474 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.642146    2582 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 10 22:01:17.647474 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.642149    2582 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 10 22:01:17.647474 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.642153    2582 flags.go:64] FLAG: --container-log-max-files="5"
Sep 10 22:01:17.648054 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.642156    2582 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 10 22:01:17.648054 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.642160    2582 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 10 22:01:17.648054 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.642163    2582 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 10 22:01:17.648054 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.642166    2582 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 10 22:01:17.648054 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.642169    2582 flags.go:64] FLAG: --contention-profiling="false"
Sep 10 22:01:17.648054 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.642173    2582 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 10 22:01:17.648054 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.642176    2582 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 10 22:01:17.648054 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.642180    2582 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 10 22:01:17.648054 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.642183    2582 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 10 22:01:17.648054 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.642187    2582 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 10 22:01:17.648054 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.642190    2582 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 10 22:01:17.648054 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.642193    2582 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 10 22:01:17.648054 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.642195    2582 flags.go:64] FLAG: --enable-load-reader="false"
Sep 10 22:01:17.648054 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.642198    2582 flags.go:64] FLAG: --enable-server="true"
Sep 10 22:01:17.648054 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.642202    2582 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 10 22:01:17.648054 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.642206    2582 flags.go:64] FLAG: --event-burst="100"
Sep 10 22:01:17.648054 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.642209    2582 flags.go:64] FLAG: --event-qps="50"
Sep 10 22:01:17.648054 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.642212    2582 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 10 22:01:17.648054 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.642215    2582 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 10 22:01:17.648054 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.642218    2582 flags.go:64] FLAG: --eviction-hard=""
Sep 10 22:01:17.648054 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.642222    2582 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 10 22:01:17.648054 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.642226    2582 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 10 22:01:17.648054 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.642229    2582 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 10 22:01:17.648054 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.642233    2582 flags.go:64] FLAG: --eviction-soft=""
Sep 10 22:01:17.648054 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.642235    2582 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 10 22:01:17.648671 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.642238    2582 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 10 22:01:17.648671 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.642241    2582 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 10 22:01:17.648671 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.642244    2582 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 10 22:01:17.648671 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.642247    2582 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 10 22:01:17.648671 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.642250    2582 flags.go:64] FLAG: --fail-swap-on="true"
Sep 10 22:01:17.648671 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.642253    2582 flags.go:64] FLAG: --feature-gates=""
Sep 10 22:01:17.648671 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.642257    2582 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 10 22:01:17.648671 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.642260    2582 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 10 22:01:17.648671 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.642263    2582 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 10 22:01:17.648671 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.642266    2582 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 10 22:01:17.648671 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.642269    2582 flags.go:64] FLAG: --healthz-port="10248"
Sep 10 22:01:17.648671 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.642272    2582 flags.go:64] FLAG: --help="false"
Sep 10 22:01:17.648671 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.642276    2582 flags.go:64] FLAG: --hostname-override="ip-10-0-134-144.ec2.internal"
Sep 10 22:01:17.648671 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.642279    2582 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 10 22:01:17.648671 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.642282    2582 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 10 22:01:17.648671 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.642285    2582 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 10 22:01:17.648671 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.642288    2582 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 10 22:01:17.648671 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.642292    2582 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 10 22:01:17.648671 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.642295    2582 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 10 22:01:17.648671 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.642297    2582 flags.go:64] FLAG: --image-service-endpoint=""
Sep 10 22:01:17.648671 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.642300    2582 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 10 22:01:17.648671 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.642303    2582 flags.go:64] FLAG: --kube-api-burst="100"
Sep 10 22:01:17.648671 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.642307    2582 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 10 22:01:17.649317 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.642310    2582 flags.go:64] FLAG: --kube-api-qps="50"
Sep 10 22:01:17.649317 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.642313    2582 flags.go:64] FLAG: --kube-reserved=""
Sep 10 22:01:17.649317 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.642316    2582 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 10 22:01:17.649317 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.642319    2582 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 10 22:01:17.649317 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.642322    2582 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 10 22:01:17.649317 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.642326    2582 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 10 22:01:17.649317 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.642329    2582 flags.go:64] FLAG: --lock-file=""
Sep 10 22:01:17.649317 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.642332    2582 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 10 22:01:17.649317 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.642334    2582 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 10 22:01:17.649317 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.642338    2582 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 10 22:01:17.649317 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.642343    2582 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 10 22:01:17.649317 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.642346    2582 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 10 22:01:17.649317 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.642349    2582 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 10 22:01:17.649317 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.642352    2582 flags.go:64] FLAG: --logging-format="text"
Sep 10 22:01:17.649317 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.642354    2582 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 10 22:01:17.649317 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.642357    2582 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 10 22:01:17.649317 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.642360    2582 flags.go:64] FLAG: --manifest-url=""
Sep 10 22:01:17.649317 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.642363    2582 flags.go:64] FLAG: --manifest-url-header=""
Sep 10 22:01:17.649317 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.642367    2582 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 10 22:01:17.649317 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.642370    2582 flags.go:64] FLAG: --max-open-files="1000000"
Sep 10 22:01:17.649317 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.642374    2582 flags.go:64] FLAG: --max-pods="110"
Sep 10 22:01:17.649317 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.642378    2582 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 10 22:01:17.649317 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.642381    2582 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 10 22:01:17.649317 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.642384    2582 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 10 22:01:17.649317 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.642387    2582 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 10 22:01:17.649997 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.642390    2582 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 10 22:01:17.649997 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.642393    2582 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 10 22:01:17.649997 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.642396    2582 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 10 22:01:17.649997 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.642403    2582 flags.go:64] FLAG: --node-status-max-images="50"
Sep 10 22:01:17.649997 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.642406    2582 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 10 22:01:17.649997 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.642410    2582 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 10 22:01:17.649997 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.642413    2582 flags.go:64] FLAG: --pod-cidr=""
Sep 10 22:01:17.649997 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.642417    2582 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 10 22:01:17.649997 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.642422    2582 flags.go:64] FLAG: --pod-manifest-path=""
Sep 10 22:01:17.649997 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.642425    2582 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 10 22:01:17.649997 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.642428    2582 flags.go:64] FLAG: --pods-per-core="0"
Sep 10 22:01:17.649997 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.642431    2582 flags.go:64] FLAG: --port="10250"
Sep 10 22:01:17.649997 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.642435    2582 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 10 22:01:17.649997 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.642438    2582 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0da292d6ed59b840a"
Sep 10 22:01:17.649997 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.642441    2582 flags.go:64] FLAG: --qos-reserved=""
Sep 10 22:01:17.649997 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.642444    2582 flags.go:64] FLAG: --read-only-port="10255"
Sep 10 22:01:17.649997 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.642447    2582 flags.go:64] FLAG: --register-node="true"
Sep 10 22:01:17.649997 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.642450    2582 flags.go:64] FLAG: --register-with-taints=""
Sep 10 22:01:17.649997 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.642454    2582 flags.go:64] FLAG: --registry-burst="10"
Sep 10 22:01:17.649997 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.642456    2582 flags.go:64] FLAG: --registry-qps="5"
Sep 10 22:01:17.649997 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.642459    2582 flags.go:64] FLAG: --reserved-cpus=""
Sep 10 22:01:17.649997 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.642462    2582 flags.go:64] FLAG: --reserved-memory=""
Sep 10 22:01:17.649997 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.642466    2582 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 10 22:01:17.649997 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.642469    2582 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 10 22:01:17.649997 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.642472    2582 flags.go:64] FLAG: --rotate-certificates="false"
Sep 10 22:01:17.650642 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.642475    2582 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 10 22:01:17.650642 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.642478    2582 flags.go:64] FLAG: --runonce="false"
Sep 10 22:01:17.650642 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.642481    2582 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 10 22:01:17.650642 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.642484    2582 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 10 22:01:17.650642 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.642489    2582 flags.go:64] FLAG: --seccomp-default="false"
Sep 10 22:01:17.650642 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.642492    2582 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 10 22:01:17.650642 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.642495    2582 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 10 22:01:17.650642 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.642499    2582 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 10 22:01:17.650642 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.642502    2582 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 10 22:01:17.650642 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.642505    2582 flags.go:64] FLAG: --storage-driver-password="root"
Sep 10 22:01:17.650642 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.642508    2582 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 10 22:01:17.650642 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.642511    2582 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 10 22:01:17.650642 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.642514    2582 flags.go:64] FLAG: --storage-driver-user="root"
Sep 10 22:01:17.650642 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.642517    2582 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 10 22:01:17.650642 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.642520    2582 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 10 22:01:17.650642 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.642523    2582 flags.go:64] FLAG: --system-cgroups=""
Sep 10 22:01:17.650642 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.642526    2582 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 10 22:01:17.650642 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.642532    2582 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 10 22:01:17.650642 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.642535    2582 flags.go:64] FLAG: --tls-cert-file=""
Sep 10 22:01:17.650642 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.642539    2582 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 10 22:01:17.650642 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.642543    2582 flags.go:64] FLAG: --tls-min-version=""
Sep 10 22:01:17.650642 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.642546    2582 flags.go:64] FLAG: --tls-private-key-file=""
Sep 10 22:01:17.650642 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.642550    2582 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 10 22:01:17.650642 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.642553    2582 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 10 22:01:17.650642 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.642556    2582 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 10 22:01:17.651260 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.642559    2582 flags.go:64] FLAG: --v="2"
Sep 10 22:01:17.651260 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.642563    2582 flags.go:64] FLAG: --version="false"
Sep 10 22:01:17.651260 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.642567    2582 flags.go:64] FLAG: --vmodule=""
Sep 10 22:01:17.651260 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.642570    2582 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 10 22:01:17.651260 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.642574    2582 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 10 22:01:17.651260 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.642692    2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 22:01:17.651260 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.642696    2582 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 22:01:17.651260 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.642699    2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 22:01:17.651260 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.642702    2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 22:01:17.651260 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.642705    2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 22:01:17.651260 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.642708    2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 22:01:17.651260 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.642712    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 22:01:17.651260 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.642733    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 22:01:17.651260 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.642737    2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 22:01:17.651260 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.642740    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 22:01:17.651260 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.642743    2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 22:01:17.651260 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.642746    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 22:01:17.651260 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.642748    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 22:01:17.651260 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.642751    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 22:01:17.651260 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.642754    2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 22:01:17.651260 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.642757    2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 22:01:17.651797 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.642759    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 22:01:17.651797 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.642762    2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 22:01:17.651797 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.642765    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 22:01:17.651797 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.642768    2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 22:01:17.651797 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.642771    2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 22:01:17.651797 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.642777    2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 22:01:17.651797 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.642781    2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 22:01:17.651797 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.642785    2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 22:01:17.651797 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.642788    2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 22:01:17.651797 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.642790    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 22:01:17.651797 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.642793    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 22:01:17.651797 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.642795    2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 22:01:17.651797 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.642798    2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 22:01:17.651797 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.642801    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 22:01:17.651797 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.642803    2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 22:01:17.651797 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.642806    2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 22:01:17.651797 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.642809    2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 22:01:17.651797 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.642812    2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 22:01:17.651797 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.642814    2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 22:01:17.651797 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.642817    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 22:01:17.652300 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.642819    2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 22:01:17.652300 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.642822    2582 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 22:01:17.652300 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.642827    2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 22:01:17.652300 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.642829    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 22:01:17.652300 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.642832    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 22:01:17.652300 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.642835    2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 22:01:17.652300 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.642838    2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 22:01:17.652300 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.642840    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 22:01:17.652300 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.642843    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 22:01:17.652300 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.642846    2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 22:01:17.652300 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.642848    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 22:01:17.652300 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.642851    2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 22:01:17.652300 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.642854    2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 22:01:17.652300 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.642857    2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 22:01:17.652300 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.642859    2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 22:01:17.652300 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.642863    2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 22:01:17.652300 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.642865    2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 22:01:17.652300 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.642872    2582 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 22:01:17.652300 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.642875    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 22:01:17.652300 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.642877    2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 22:01:17.652810 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.642880    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 22:01:17.652810 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.642883    2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 22:01:17.652810 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.642885    2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 22:01:17.652810 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.642888    2582 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 22:01:17.652810 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.642891    2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 22:01:17.652810 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.642893    2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 22:01:17.652810 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.642896    2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 22:01:17.652810 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.642898    2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 22:01:17.652810 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.642901    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 22:01:17.652810 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.642904    2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 22:01:17.652810 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.642906    2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 22:01:17.652810 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.642909    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 22:01:17.652810 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.642911    2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 22:01:17.652810 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.642914    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 22:01:17.652810 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.642920    2582 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 22:01:17.652810 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.642922    2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 22:01:17.652810 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.642925    2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 22:01:17.652810 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.642928    2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 22:01:17.652810 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.642930    2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 22:01:17.652810 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.642933    2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 22:01:17.653305 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.642936    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 22:01:17.653305 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.642939    2582 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 22:01:17.653305 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.642941    2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 22:01:17.653305 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.642944    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 22:01:17.653305 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.642946    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 22:01:17.653305 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.642949    2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 22:01:17.653305 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.642951    2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 22:01:17.653305 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.642954    2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 22:01:17.653305 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.642958    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 22:01:17.653305 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.642963    2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 22:01:17.653305 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.642967    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 22:01:17.653305 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.642970    2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 22:01:17.653305 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.642973    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 22:01:17.653305 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.642976    2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 22:01:17.653305 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.642979    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 22:01:17.653305 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.642983    2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 22:01:17.653305 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.642986    2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 22:01:17.653305 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.642989    2582 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 22:01:17.653305 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.642992    2582 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 22:01:17.653887 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.642995    2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 22:01:17.653887 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.642998    2582 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 22:01:17.653887 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.643001    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 22:01:17.653887 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.643004    2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 22:01:17.653887 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.643451    2582 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 10 22:01:17.653887 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.650563    2582 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 10 22:01:17.653887 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.650579    2582 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 10 22:01:17.653887 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.651246    2582 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 10 22:01:17.653887 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.651255    2582 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 10 22:01:17.653887 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651312    2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 22:01:17.653887 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651317    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 22:01:17.653887 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651321    2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 22:01:17.653887 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651325    2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 22:01:17.653887 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651328    2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 22:01:17.653887 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651330    2582 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 22:01:17.654260 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651333    2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 22:01:17.654260 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651335    2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 22:01:17.654260 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651338    2582 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 22:01:17.654260 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651341    2582 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 22:01:17.654260 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651344    2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 22:01:17.654260 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651347    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 22:01:17.654260 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651349    2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 22:01:17.654260 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651352    2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 22:01:17.654260 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651355    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 22:01:17.654260 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651358    2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 22:01:17.654260 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651361    2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 22:01:17.654260 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651363    2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 22:01:17.654260 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651366    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 22:01:17.654260 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651369    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 22:01:17.654260 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651373    2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 22:01:17.654260 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651376    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 22:01:17.654260 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651379    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 22:01:17.654260 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651381    2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 22:01:17.654260 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651384    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 22:01:17.654753 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651387    2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 22:01:17.654753 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651390    2582 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 22:01:17.654753 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651394    2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 22:01:17.654753 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651397    2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 22:01:17.654753 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651402    2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 22:01:17.654753 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651405    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 22:01:17.654753 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651408    2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 22:01:17.654753 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651411    2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 22:01:17.654753 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651414    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 22:01:17.654753 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651417    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 22:01:17.654753 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651419    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 22:01:17.654753 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651422    2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 22:01:17.654753 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651425    2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 22:01:17.654753 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651427    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 22:01:17.654753 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651430    2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 22:01:17.654753 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651433    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 22:01:17.654753 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651435    2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 22:01:17.654753 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651438    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 22:01:17.654753 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651440    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 22:01:17.655212 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651443    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 22:01:17.655212 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651445    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 22:01:17.655212 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651448    2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 22:01:17.655212 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651451    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 22:01:17.655212 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651453    2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 22:01:17.655212 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651457    2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 22:01:17.655212 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651459    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 22:01:17.655212 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651462    2582 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 22:01:17.655212 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651464    2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 22:01:17.655212 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651467    2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 22:01:17.655212 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651469    2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 22:01:17.655212 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651472    2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 22:01:17.655212 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651474    2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 22:01:17.655212 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651478    2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 22:01:17.655212 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651480    2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 22:01:17.655212 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651483    2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 22:01:17.655212 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651485    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 22:01:17.655212 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651489    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 22:01:17.655212 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651493    2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 22:01:17.655212 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651496    2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 22:01:17.655812 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651498    2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 22:01:17.655812 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651501    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 22:01:17.655812 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651504    2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 22:01:17.655812 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651506    2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 22:01:17.655812 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651511    2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 22:01:17.655812 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651514    2582 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 22:01:17.655812 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651517    2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 22:01:17.655812 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651520    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 22:01:17.655812 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651523    2582 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 22:01:17.655812 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651526    2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 22:01:17.655812 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651529    2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 22:01:17.655812 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651532    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 22:01:17.655812 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651534    2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 22:01:17.655812 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651537    2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 22:01:17.655812 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651540    2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 22:01:17.655812 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651543    2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 22:01:17.655812 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651546    2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 22:01:17.655812 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651549    2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 22:01:17.655812 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651552    2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 22:01:17.655812 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651554    2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 22:01:17.656296 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651557    2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 22:01:17.656296 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651560    2582 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 22:01:17.656296 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651562    2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 22:01:17.656296 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651565    2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 22:01:17.656296 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651567    2582 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 22:01:17.656296 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651570    2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 22:01:17.656296 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651573    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 22:01:17.656296 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651575    2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 22:01:17.656296 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651578    2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 22:01:17.656296 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651582    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 22:01:17.656296 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651584    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 22:01:17.656296 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651587    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 22:01:17.656296 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651590    2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 22:01:17.656296 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651593    2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 22:01:17.656296 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651595    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 22:01:17.656664 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.651600    2582 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 10 22:01:17.656664 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651725    2582 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 22:01:17.656664 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651731    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 22:01:17.656664 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651734    2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 22:01:17.656664 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651737    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 22:01:17.656664 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651739    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 22:01:17.656664 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651742    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 22:01:17.656664 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651745    2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 22:01:17.656664 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651747    2582 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 22:01:17.656664 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651750    2582 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 22:01:17.656664 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651753    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 22:01:17.656664 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651756    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 22:01:17.656664 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651759    2582 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 22:01:17.656664 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651762    2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 22:01:17.656664 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651764    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 22:01:17.656664 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651767    2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 22:01:17.657072 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651770    2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 22:01:17.657072 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651772    2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 22:01:17.657072 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651775    2582 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 22:01:17.657072 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651777    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 22:01:17.657072 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651780    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 22:01:17.657072 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651783    2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 22:01:17.657072 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651785    2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 22:01:17.657072 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651788    2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 22:01:17.657072 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651790    2582 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 22:01:17.657072 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651793    2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 22:01:17.657072 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651795    2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 22:01:17.657072 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651798    2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 22:01:17.657072 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651801    2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 22:01:17.657072 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651805    2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 22:01:17.657072 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651808    2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 22:01:17.657072 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651810    2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 22:01:17.657072 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651813    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 22:01:17.657072 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651817    2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 22:01:17.657072 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651821    2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 22:01:17.657534 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651824    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 22:01:17.657534 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651827    2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 22:01:17.657534 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651829    2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 22:01:17.657534 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651832    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 22:01:17.657534 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651835    2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 22:01:17.657534 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651838    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 22:01:17.657534 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651841    2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 22:01:17.657534 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651845    2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 22:01:17.657534 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651848    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 22:01:17.657534 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651851    2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 22:01:17.657534 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651854    2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 22:01:17.657534 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651857    2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 22:01:17.657534 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651860    2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 22:01:17.657534 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651862    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 22:01:17.657534 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651865    2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 22:01:17.657534 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651868    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 22:01:17.657534 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651870    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 22:01:17.657534 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651873    2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 22:01:17.657534 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651875    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 22:01:17.658084 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651878    2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 22:01:17.658084 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651880    2582 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 22:01:17.658084 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651883    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 22:01:17.658084 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651885    2582 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 22:01:17.658084 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651888    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 22:01:17.658084 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651891    2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 22:01:17.658084 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651893    2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 22:01:17.658084 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651896    2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 22:01:17.658084 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651899    2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 22:01:17.658084 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651902    2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 22:01:17.658084 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651904    2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 22:01:17.658084 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651907    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 22:01:17.658084 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651909    2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 22:01:17.658084 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651912    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 22:01:17.658084 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651914    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 22:01:17.658084 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651917    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 22:01:17.658084 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651919    2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 22:01:17.658084 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651922    2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 22:01:17.658084 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651924    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 22:01:17.658084 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651927    2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 22:01:17.658562 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651930    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 22:01:17.658562 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651932    2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 22:01:17.658562 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651935    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 22:01:17.658562 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651937    2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 22:01:17.658562 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651940    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 22:01:17.658562 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651943    2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 22:01:17.658562 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651946    2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 22:01:17.658562 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651948    2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 22:01:17.658562 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651951    2582 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 22:01:17.658562 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651953    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 22:01:17.658562 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651956    2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 22:01:17.658562 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651958    2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 22:01:17.658562 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651961    2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 22:01:17.658562 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651963    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 22:01:17.658562 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651966    2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 22:01:17.658562 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651968    2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 22:01:17.658562 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651971    2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 22:01:17.658562 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651974    2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 22:01:17.658562 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651976    2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 22:01:17.659030 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651980    2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 22:01:17.659030 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651983    2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 22:01:17.659030 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651987    2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 22:01:17.659030 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651989    2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 22:01:17.659030 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651992    2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 22:01:17.659030 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651995    2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 22:01:17.659030 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:17.651997    2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 22:01:17.659030 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.652003    2582 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 10 22:01:17.659030 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.652117    2582 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 10 22:01:17.659030 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.654377    2582 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 10 22:01:17.659030 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.655322    2582 server.go:1019] "Starting client certificate rotation"
Sep 10 22:01:17.659030 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.655416    2582 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 10 22:01:17.659030 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.655478    2582 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 10 22:01:17.659385 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.657813    2582 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 10 22:01:17.661733 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.661667    2582 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 10 22:01:17.671435 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.671416    2582 log.go:25] "Validated CRI v1 runtime API"
Sep 10 22:01:17.676172 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.676155    2582 log.go:25] "Validated CRI v1 image API"
Sep 10 22:01:17.678325 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.678310    2582 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 10 22:01:17.681088 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.681063    2582 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 ee8f40c8-0543-4b23-a508-804dd30665ac:/dev/nvme0n1p4 fd693b99-0251-4c35-a598-e8ad7af1d538:/dev/nvme0n1p3]
Sep 10 22:01:17.681170 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.681086    2582 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 10 22:01:17.685902 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.685883    2582 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 10 22:01:17.686495 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.686384    2582 manager.go:216] Machine: {Timestamp:2026-09-10 22:01:17.68513102 +0000 UTC m=+0.313330731 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3119766 MemoryCapacity:33164476416 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec29fbb964beaf225c65761fdf1f2ee4 SystemUUID:ec29fbb9-64be-af22-5c65-761fdf1f2ee4 BootID:3746888c-6545-47bf-9669-93ad05c2d87c Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:0c:d7:1d:42:cd Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:0c:d7:1d:42:cd Speed:0 Mtu:9001} {Name:ovs-system MacAddress:26:30:af:97:10:92 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164476416 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 10 22:01:17.686495 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.686493    2582 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 10 22:01:17.686609 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.686575    2582 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 10 22:01:17.689823 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.689799    2582 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 10 22:01:17.689989 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.689827    2582 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-144.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 10 22:01:17.690658 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.690647    2582 topology_manager.go:138] "Creating topology manager with none policy"
Sep 10 22:01:17.690735 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.690665    2582 container_manager_linux.go:309] "Creating device plugin manager"
Sep 10 22:01:17.690735 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.690676    2582 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 10 22:01:17.691383 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.691372    2582 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 10 22:01:17.692622 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.692611    2582 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 10 22:01:17.696870 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.696857    2582 state_mem.go:36] "Initialized new in-memory state store"
Sep 10 22:01:17.697003 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.696992    2582 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 10 22:01:17.697075 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.697032    2582 kubelet.go:486] "Attempting to sync node with API server"
Sep 10 22:01:17.697075 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.697055    2582 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 10 22:01:17.697075 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.697071    2582 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 10 22:01:17.697199 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.697083    2582 kubelet.go:398] "Adding apiserver pod source"
Sep 10 22:01:17.697199 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.697098    2582 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 10 22:01:17.697460 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.697446    2582 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 10 22:01:17.697527 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.697469    2582 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 10 22:01:17.699734 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.699697    2582 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 10 22:01:17.702426 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.702413    2582 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 10 22:01:17.702471 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.702443    2582 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 10 22:01:17.704749 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.704737    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 10 22:01:17.704791 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.704755    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 10 22:01:17.704791 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.704764    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 10 22:01:17.704791 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.704769    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 10 22:01:17.704791 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.704778    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 10 22:01:17.704791 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.704783    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 10 22:01:17.704922 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.704797    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 10 22:01:17.704922 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.704802    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 10 22:01:17.704922 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.704810    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 10 22:01:17.704922 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.704816    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 10 22:01:17.704922 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.704832    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 10 22:01:17.704922 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.704838    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 10 22:01:17.706165 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.706155    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 10 22:01:17.706165 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.706165    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 10 22:01:17.708393 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:01:17.708372    2582 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 10 22:01:17.708439 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:01:17.708376    2582 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-134-144.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 10 22:01:17.709973 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.709962    2582 server.go:1268] "Started kubelet"
Sep 10 22:01:17.710081 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.710044    2582 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 10 22:01:17.712121 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.712091    2582 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 10 22:01:17.712255 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.710077    2582 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 10 22:01:17.712837 ip-10-0-134-144 systemd[1]: Started Kubernetes Kubelet.
Sep 10 22:01:17.714338 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.713376    2582 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-x25n9"
Sep 10 22:01:17.714766 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.714748    2582 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 10 22:01:17.717520 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.717507    2582 server.go:310] "Adding debug handlers to kubelet server"
Sep 10 22:01:17.719067 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:01:17.719051    2582 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 10 22:01:17.720621 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.720606    2582 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-x25n9"
Sep 10 22:01:17.723189 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:01:17.722281    2582 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-144.ec2.internal.18d414ab3f7b3e86  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-144.ec2.internal,UID:ip-10-0-134-144.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-134-144.ec2.internal,},FirstTimestamp:2026-09-10 22:01:17.709934214 +0000 UTC m=+0.338133925,LastTimestamp:2026-09-10 22:01:17.709934214 +0000 UTC m=+0.338133925,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-144.ec2.internal,}"
Sep 10 22:01:17.724454 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.724435    2582 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 10 22:01:17.725061 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.725046    2582 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 10 22:01:17.725198 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:01:17.725177    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-144.ec2.internal\" not found"
Sep 10 22:01:17.725491 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.725473    2582 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-134-144.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 22:01:17.725854 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.725840    2582 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 10 22:01:17.726171 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.725994    2582 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 10 22:01:17.726271 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.726174    2582 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 10 22:01:17.726271 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.726077    2582 factory.go:55] Registering systemd factory
Sep 10 22:01:17.726271 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.726210    2582 factory.go:223] Registration of the systemd container factory successfully
Sep 10 22:01:17.726271 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.726253    2582 reconstruct.go:97] "Volume reconstruction finished"
Sep 10 22:01:17.726271 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.726263    2582 reconciler.go:29] "Reconciler: start to sync state"
Sep 10 22:01:17.726460 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.726448    2582 factory.go:153] Registering CRI-O factory
Sep 10 22:01:17.726514 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.726465    2582 factory.go:223] Registration of the crio container factory successfully
Sep 10 22:01:17.726561 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.726522    2582 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 10 22:01:17.726561 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.726541    2582 factory.go:103] Registering Raw factory
Sep 10 22:01:17.726561 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.726557    2582 manager.go:1196] Started watching for new ooms in manager
Sep 10 22:01:17.727021 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.727007    2582 manager.go:349] Starting recovery of all containers
Sep 10 22:01:17.733827 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.733806    2582 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 22:01:17.737382 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:01:17.737357    2582 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-134-144.ec2.internal\" not found" node="ip-10-0-134-144.ec2.internal"
Sep 10 22:01:17.740266 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.740252    2582 manager.go:354] Recovery completed
Sep 10 22:01:17.748035 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.748018    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 22:01:17.750447 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.750423    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-144.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 22:01:17.750520 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.750465    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-144.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 22:01:17.750520 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.750479    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-144.ec2.internal" event="NodeHasSufficientPID"
Sep 10 22:01:17.751131 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.751117    2582 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 10 22:01:17.751131 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.751129    2582 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 10 22:01:17.751253 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.751148    2582 state_mem.go:36] "Initialized new in-memory state store"
Sep 10 22:01:17.751253 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.751164    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 22:01:17.752011 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.751992    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-144.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 22:01:17.752086 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.752024    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-144.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 22:01:17.752086 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.752039    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-144.ec2.internal" event="NodeHasSufficientPID"
Sep 10 22:01:17.752302 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.752284    2582 policy_none.go:49] "None policy: Start"
Sep 10 22:01:17.752337 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.752306    2582 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 10 22:01:17.752337 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.752317    2582 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 10 22:01:17.753487 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.753475    2582 policy_none.go:47] "Start"
Sep 10 22:01:17.805841 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.805827    2582 manager.go:337] "Starting Device Plugin manager"
Sep 10 22:01:17.805958 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:01:17.805865    2582 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 10 22:01:17.805958 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.805879    2582 server.go:87] "Starting device plugin registration server"
Sep 10 22:01:17.806185 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.806172    2582 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 10 22:01:17.806255 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.806186    2582 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 10 22:01:17.806304 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.806295    2582 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 10 22:01:17.806390 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.806365    2582 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 10 22:01:17.806390 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.806376    2582 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 10 22:01:17.806880 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:01:17.806862    2582 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 10 22:01:17.806956 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:01:17.806904    2582 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-134-144.ec2.internal\" not found"
Sep 10 22:01:17.818300 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.818278    2582 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 10 22:01:17.819621 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.819598    2582 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 10 22:01:17.819764 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.819640    2582 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 10 22:01:17.819764 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.819660    2582 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 10 22:01:17.819764 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:01:17.819700    2582 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 10 22:01:17.822283 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.822265    2582 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 22:01:17.906563 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.906501    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 22:01:17.907519 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.907503    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-144.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 22:01:17.907589 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.907533    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-144.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 22:01:17.907589 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.907548    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-144.ec2.internal" event="NodeHasSufficientPID"
Sep 10 22:01:17.907589 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.907571    2582 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-134-144.ec2.internal"
Sep 10 22:01:17.915966 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.915951    2582 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-134-144.ec2.internal"
Sep 10 22:01:17.916023 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:01:17.915972    2582 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-134-144.ec2.internal\": node \"ip-10-0-134-144.ec2.internal\" not found"
Sep 10 22:01:17.920023 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.920006    2582 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-144.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-134-144.ec2.internal"]
Sep 10 22:01:17.920092 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.920083    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 22:01:17.920981 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.920965    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-144.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 22:01:17.921054 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.920998    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-144.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 22:01:17.921054 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.921012    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-144.ec2.internal" event="NodeHasSufficientPID"
Sep 10 22:01:17.922425 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.922413    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 22:01:17.922552 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.922539    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-144.ec2.internal"
Sep 10 22:01:17.922594 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.922568    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 22:01:17.923150 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.923133    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-144.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 22:01:17.923237 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.923164    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-144.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 22:01:17.923237 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.923179    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-144.ec2.internal" event="NodeHasSufficientPID"
Sep 10 22:01:17.923237 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.923135    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-144.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 22:01:17.923237 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.923204    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-144.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 22:01:17.923237 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.923217    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-144.ec2.internal" event="NodeHasSufficientPID"
Sep 10 22:01:17.924402 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.924387    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-144.ec2.internal"
Sep 10 22:01:17.924445 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.924435    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 22:01:17.925149 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.925132    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-144.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 22:01:17.925224 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.925158    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-144.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 22:01:17.925224 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.925168    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-144.ec2.internal" event="NodeHasSufficientPID"
Sep 10 22:01:17.926500 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.926481    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f31b7187b87e2b509b6e74131f844880-config\") pod \"kube-apiserver-proxy-ip-10-0-134-144.ec2.internal\" (UID: \"f31b7187b87e2b509b6e74131f844880\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-144.ec2.internal"
Sep 10 22:01:17.926572 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.926515    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6ea49b038141fb7080f5d505e03c9c21-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-144.ec2.internal\" (UID: \"6ea49b038141fb7080f5d505e03c9c21\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-144.ec2.internal"
Sep 10 22:01:17.926572 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:17.926542    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6ea49b038141fb7080f5d505e03c9c21-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-144.ec2.internal\" (UID: \"6ea49b038141fb7080f5d505e03c9c21\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-144.ec2.internal"
Sep 10 22:01:17.932743 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:01:17.932710    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-144.ec2.internal\" not found"
Sep 10 22:01:17.948399 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:01:17.948377    2582 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-144.ec2.internal\" not found" node="ip-10-0-134-144.ec2.internal"
Sep 10 22:01:17.952748 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:01:17.952731    2582 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-144.ec2.internal\" not found" node="ip-10-0-134-144.ec2.internal"
Sep 10 22:01:18.026756 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:18.026736    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6ea49b038141fb7080f5d505e03c9c21-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-144.ec2.internal\" (UID: \"6ea49b038141fb7080f5d505e03c9c21\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-144.ec2.internal"
Sep 10 22:01:18.026841 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:18.026773    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6ea49b038141fb7080f5d505e03c9c21-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-144.ec2.internal\" (UID: \"6ea49b038141fb7080f5d505e03c9c21\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-144.ec2.internal"
Sep 10 22:01:18.026841 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:18.026792    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f31b7187b87e2b509b6e74131f844880-config\") pod \"kube-apiserver-proxy-ip-10-0-134-144.ec2.internal\" (UID: \"f31b7187b87e2b509b6e74131f844880\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-144.ec2.internal"
Sep 10 22:01:18.026908 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:18.026847    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f31b7187b87e2b509b6e74131f844880-config\") pod \"kube-apiserver-proxy-ip-10-0-134-144.ec2.internal\" (UID: \"f31b7187b87e2b509b6e74131f844880\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-144.ec2.internal"
Sep 10 22:01:18.026908 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:18.026860    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6ea49b038141fb7080f5d505e03c9c21-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-144.ec2.internal\" (UID: \"6ea49b038141fb7080f5d505e03c9c21\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-144.ec2.internal"
Sep 10 22:01:18.026908 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:18.026847    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6ea49b038141fb7080f5d505e03c9c21-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-144.ec2.internal\" (UID: \"6ea49b038141fb7080f5d505e03c9c21\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-144.ec2.internal"
Sep 10 22:01:18.032794 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:01:18.032762    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-144.ec2.internal\" not found"
Sep 10 22:01:18.133147 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:01:18.133121    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-144.ec2.internal\" not found"
Sep 10 22:01:18.233983 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:01:18.233910    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-144.ec2.internal\" not found"
Sep 10 22:01:18.252114 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:18.252078    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-144.ec2.internal"
Sep 10 22:01:18.256323 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:18.256223    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-144.ec2.internal"
Sep 10 22:01:18.334808 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:01:18.334764    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-144.ec2.internal\" not found"
Sep 10 22:01:18.435268 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:01:18.435243    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-144.ec2.internal\" not found"
Sep 10 22:01:20.208044 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:01:20.207957    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-144.ec2.internal\" not found"
Sep 10 22:01:20.308477 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:01:20.308438    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-144.ec2.internal\" not found"
Sep 10 22:01:20.327629 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:20.327593    2582 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 10 22:01:20.327782 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:20.327761    2582 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 10 22:01:20.327829 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:20.327790    2582 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 10 22:01:20.372318 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:20.372286    2582 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 22:01:20.396955 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:20.396923    2582 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 10 22:01:20.397387 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:20.397368    2582 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-144.ec2.internal"
Sep 10 22:01:20.398456 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:20.398421    2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-09 21:56:19 +0000 UTC" deadline="2028-06-14 05:41:12.18389307 +0000 UTC"
Sep 10 22:01:20.398507 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:20.398456    2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15415h39m51.785439678s"
Sep 10 22:01:20.407397 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:20.407371    2582 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 10 22:01:20.409074 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:20.409047    2582 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-144.ec2.internal"
Sep 10 22:01:20.409164 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:20.409088    2582 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 10 22:01:20.417719 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:20.417702    2582 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 10 22:01:20.433734 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:20.433712    2582 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-swlbb"
Sep 10 22:01:20.442040 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:20.442018    2582 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-swlbb"
Sep 10 22:01:20.472585 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:20.472530    2582 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 22:01:20.632731 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:20.632701    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf31b7187b87e2b509b6e74131f844880.slice/crio-e2d0762684bfdf4c31031d02342191806696517cd8631eac4bf01822efdcfcb6 WatchSource:0}: Error finding container e2d0762684bfdf4c31031d02342191806696517cd8631eac4bf01822efdcfcb6: Status 404 returned error can't find the container with id e2d0762684bfdf4c31031d02342191806696517cd8631eac4bf01822efdcfcb6
Sep 10 22:01:20.636610 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:20.636593    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 22:01:20.687780 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:20.687745    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ea49b038141fb7080f5d505e03c9c21.slice/crio-32292f4db980e7e3278e7b8102f250666a5044ff0aa6a664a7fd8dad2872ae4e WatchSource:0}: Error finding container 32292f4db980e7e3278e7b8102f250666a5044ff0aa6a664a7fd8dad2872ae4e: Status 404 returned error can't find the container with id 32292f4db980e7e3278e7b8102f250666a5044ff0aa6a664a7fd8dad2872ae4e
Sep 10 22:01:21.290946 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.290907    2582 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 22:01:21.370458 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.370408    2582 apiserver.go:52] "Watching apiserver"
Sep 10 22:01:21.377573 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.377543    2582 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 10 22:01:21.378626 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.378594    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-144.ec2.internal","openshift-multus/multus-8rwdz","openshift-network-diagnostics/network-check-target-rrm25","openshift-network-operator/iptables-alerter-p8f5t","kube-system/kube-apiserver-proxy-ip-10-0-134-144.ec2.internal","openshift-dns/node-resolver-ncrzz","openshift-image-registry/node-ca-74bb6","openshift-multus/multus-additional-cni-plugins-dt25t","openshift-multus/network-metrics-daemon-xw7pd","openshift-ovn-kubernetes/ovnkube-node-zbjsk","kube-system/global-pull-secret-syncer-n9r9x","kube-system/konnectivity-agent-cjdvg","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x4xzs","openshift-cluster-node-tuning-operator/tuned-dth7z"]
Sep 10 22:01:21.381181 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.381160    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-74bb6"
Sep 10 22:01:21.383487 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.383448    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n9r9x"
Sep 10 22:01:21.383594 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:01:21.383523    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n9r9x" podUID="0d6f601b-532d-4f1f-9580-977997448ea9"
Sep 10 22:01:21.383656 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.383634    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-69g6j\""
Sep 10 22:01:21.383710 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.383649    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 10 22:01:21.383938 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.383920    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 10 22:01:21.384138 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.384120    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 10 22:01:21.386076 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.386055    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rrm25"
Sep 10 22:01:21.386172 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:01:21.386133    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rrm25" podUID="064a91af-7e3d-4e71-b743-04d68d63174e"
Sep 10 22:01:21.388537 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.388499    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-p8f5t"
Sep 10 22:01:21.390703 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.390685    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-dth7z"
Sep 10 22:01:21.393133 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.393112    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-ncrzz"
Sep 10 22:01:21.395087 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.394928    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 10 22:01:21.395087 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.394963    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 10 22:01:21.395087 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.394978    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-nngt6\""
Sep 10 22:01:21.395295 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.395099    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-sc9dj\""
Sep 10 22:01:21.395546 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.395523    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xw7pd"
Sep 10 22:01:21.395610 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:01:21.395585    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xw7pd" podUID="a6c7caf6-cae8-4a81-98c5-e4db1671eff5"
Sep 10 22:01:21.397365 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.397345    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 10 22:01:21.397609 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.397362    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 10 22:01:21.397609 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.397441    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 10 22:01:21.397609 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.397486    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 10 22:01:21.397773 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.397722    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-9dgfc\""
Sep 10 22:01:21.397773 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.397753    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 10 22:01:21.399436 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.399144    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x4xzs"
Sep 10 22:01:21.401685 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.401665    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-8rwdz"
Sep 10 22:01:21.403103 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.403063    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 10 22:01:21.403278 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.403257    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 10 22:01:21.403393 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.403321    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-qw9d6\""
Sep 10 22:01:21.404369 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.404344    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-zbjsk"
Sep 10 22:01:21.404495 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.404397    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 10 22:01:21.404592 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.404558    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-jg8vm\""
Sep 10 22:01:21.404758 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.404737    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 10 22:01:21.405328 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.405305    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 10 22:01:21.405780 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.405580    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 10 22:01:21.405780 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.405596    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 10 22:01:21.407212 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.407183    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-dt25t"
Sep 10 22:01:21.409681 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.409664    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 10 22:01:21.409959 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.409944    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 10 22:01:21.410052 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.410014    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 10 22:01:21.410525 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.410196    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 10 22:01:21.410525 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.410396    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-c7ksq\""
Sep 10 22:01:21.410525 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.410481    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-cjdvg"
Sep 10 22:01:21.410729 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.410595    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 10 22:01:21.411073 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.411051    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 10 22:01:21.411169 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.411101    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 10 22:01:21.411169 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.411120    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 10 22:01:21.412682 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.409172    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0d6f601b-532d-4f1f-9580-977997448ea9-original-pull-secret\") pod \"global-pull-secret-syncer-n9r9x\" (UID: \"0d6f601b-532d-4f1f-9580-977997448ea9\") " pod="kube-system/global-pull-secret-syncer-n9r9x"
Sep 10 22:01:21.412682 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.411442    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7821d1b4-78c1-4500-84e4-1b1fc03ad836-host-slash\") pod \"iptables-alerter-p8f5t\" (UID: \"7821d1b4-78c1-4500-84e4-1b1fc03ad836\") " pod="openshift-network-operator/iptables-alerter-p8f5t"
Sep 10 22:01:21.412682 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.411475    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vzmd\" (UniqueName: \"kubernetes.io/projected/b73372d8-bb63-4e98-82e3-05c929fad192-kube-api-access-9vzmd\") pod \"tuned-dth7z\" (UID: \"b73372d8-bb63-4e98-82e3-05c929fad192\") " pod="openshift-cluster-node-tuning-operator/tuned-dth7z"
Sep 10 22:01:21.412682 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.411544    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fe63ec98-ecdb-4555-8799-a1f3e9558da3-etc-selinux\") pod \"aws-ebs-csi-driver-node-x4xzs\" (UID: \"fe63ec98-ecdb-4555-8799-a1f3e9558da3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x4xzs"
Sep 10 22:01:21.412682 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.411574    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvt5v\" (UniqueName: \"kubernetes.io/projected/a6c7caf6-cae8-4a81-98c5-e4db1671eff5-kube-api-access-rvt5v\") pod \"network-metrics-daemon-xw7pd\" (UID: \"a6c7caf6-cae8-4a81-98c5-e4db1671eff5\") " pod="openshift-multus/network-metrics-daemon-xw7pd"
Sep 10 22:01:21.412682 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.411600    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwfrj\" (UniqueName: \"kubernetes.io/projected/064a91af-7e3d-4e71-b743-04d68d63174e-kube-api-access-zwfrj\") pod \"network-check-target-rrm25\" (UID: \"064a91af-7e3d-4e71-b743-04d68d63174e\") " pod="openshift-network-diagnostics/network-check-target-rrm25"
Sep 10 22:01:21.412682 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.411629    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b73372d8-bb63-4e98-82e3-05c929fad192-etc-sysctl-conf\") pod \"tuned-dth7z\" (UID: \"b73372d8-bb63-4e98-82e3-05c929fad192\") " pod="openshift-cluster-node-tuning-operator/tuned-dth7z"
Sep 10 22:01:21.412682 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.411679    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b73372d8-bb63-4e98-82e3-05c929fad192-run\") pod \"tuned-dth7z\" (UID: \"b73372d8-bb63-4e98-82e3-05c929fad192\") " pod="openshift-cluster-node-tuning-operator/tuned-dth7z"
Sep 10 22:01:21.412682 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.411710    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-k6l75\""
Sep 10 22:01:21.412682 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.411711    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fe63ec98-ecdb-4555-8799-a1f3e9558da3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-x4xzs\" (UID: \"fe63ec98-ecdb-4555-8799-a1f3e9558da3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x4xzs"
Sep 10 22:01:21.412682 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.411814    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fe63ec98-ecdb-4555-8799-a1f3e9558da3-device-dir\") pod \"aws-ebs-csi-driver-node-x4xzs\" (UID: \"fe63ec98-ecdb-4555-8799-a1f3e9558da3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x4xzs"
Sep 10 22:01:21.412682 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.411880    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fe63ec98-ecdb-4555-8799-a1f3e9558da3-sys-fs\") pod \"aws-ebs-csi-driver-node-x4xzs\" (UID: \"fe63ec98-ecdb-4555-8799-a1f3e9558da3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x4xzs"
Sep 10 22:01:21.412682 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.411909    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chqgf\" (UniqueName: \"kubernetes.io/projected/fe63ec98-ecdb-4555-8799-a1f3e9558da3-kube-api-access-chqgf\") pod \"aws-ebs-csi-driver-node-x4xzs\" (UID: \"fe63ec98-ecdb-4555-8799-a1f3e9558da3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x4xzs"
Sep 10 22:01:21.412682 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.411941    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f430e7fc-0f5e-48e2-9ea6-f97c277cdb06-host\") pod \"node-ca-74bb6\" (UID: \"f430e7fc-0f5e-48e2-9ea6-f97c277cdb06\") " pod="openshift-image-registry/node-ca-74bb6"
Sep 10 22:01:21.412682 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.412026    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2k6gp\" (UniqueName: \"kubernetes.io/projected/7821d1b4-78c1-4500-84e4-1b1fc03ad836-kube-api-access-2k6gp\") pod \"iptables-alerter-p8f5t\" (UID: \"7821d1b4-78c1-4500-84e4-1b1fc03ad836\") " pod="openshift-network-operator/iptables-alerter-p8f5t"
Sep 10 22:01:21.412682 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.412095    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b73372d8-bb63-4e98-82e3-05c929fad192-etc-systemd\") pod \"tuned-dth7z\" (UID: \"b73372d8-bb63-4e98-82e3-05c929fad192\") " pod="openshift-cluster-node-tuning-operator/tuned-dth7z"
Sep 10 22:01:21.413440 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.412153    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b73372d8-bb63-4e98-82e3-05c929fad192-lib-modules\") pod \"tuned-dth7z\" (UID: \"b73372d8-bb63-4e98-82e3-05c929fad192\") " pod="openshift-cluster-node-tuning-operator/tuned-dth7z"
Sep 10 22:01:21.413440 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.412181    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fe63ec98-ecdb-4555-8799-a1f3e9558da3-registration-dir\") pod \"aws-ebs-csi-driver-node-x4xzs\" (UID: \"fe63ec98-ecdb-4555-8799-a1f3e9558da3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x4xzs"
Sep 10 22:01:21.413440 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.412212    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7821d1b4-78c1-4500-84e4-1b1fc03ad836-iptables-alerter-script\") pod \"iptables-alerter-p8f5t\" (UID: \"7821d1b4-78c1-4500-84e4-1b1fc03ad836\") " pod="openshift-network-operator/iptables-alerter-p8f5t"
Sep 10 22:01:21.413440 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.412263    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b73372d8-bb63-4e98-82e3-05c929fad192-etc-sysconfig\") pod \"tuned-dth7z\" (UID: \"b73372d8-bb63-4e98-82e3-05c929fad192\") " pod="openshift-cluster-node-tuning-operator/tuned-dth7z"
Sep 10 22:01:21.413440 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.412301    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/16caf96d-d15f-45e5-b003-1aec5786ea58-hosts-file\") pod \"node-resolver-ncrzz\" (UID: \"16caf96d-d15f-45e5-b003-1aec5786ea58\") " pod="openshift-dns/node-resolver-ncrzz"
Sep 10 22:01:21.413440 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.412325    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b73372d8-bb63-4e98-82e3-05c929fad192-sys\") pod \"tuned-dth7z\" (UID: \"b73372d8-bb63-4e98-82e3-05c929fad192\") " pod="openshift-cluster-node-tuning-operator/tuned-dth7z"
Sep 10 22:01:21.413440 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.412355    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7w6k\" (UniqueName: \"kubernetes.io/projected/16caf96d-d15f-45e5-b003-1aec5786ea58-kube-api-access-x7w6k\") pod \"node-resolver-ncrzz\" (UID: \"16caf96d-d15f-45e5-b003-1aec5786ea58\") " pod="openshift-dns/node-resolver-ncrzz"
Sep 10 22:01:21.413440 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.412386    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6c7caf6-cae8-4a81-98c5-e4db1671eff5-metrics-certs\") pod \"network-metrics-daemon-xw7pd\" (UID: \"a6c7caf6-cae8-4a81-98c5-e4db1671eff5\") " pod="openshift-multus/network-metrics-daemon-xw7pd"
Sep 10 22:01:21.413440 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.412391    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 10 22:01:21.413440 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.412415    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f430e7fc-0f5e-48e2-9ea6-f97c277cdb06-serviceca\") pod \"node-ca-74bb6\" (UID: \"f430e7fc-0f5e-48e2-9ea6-f97c277cdb06\") " pod="openshift-image-registry/node-ca-74bb6"
Sep 10 22:01:21.413440 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.412457    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzhtv\" (UniqueName: \"kubernetes.io/projected/f430e7fc-0f5e-48e2-9ea6-f97c277cdb06-kube-api-access-vzhtv\") pod \"node-ca-74bb6\" (UID: \"f430e7fc-0f5e-48e2-9ea6-f97c277cdb06\") " pod="openshift-image-registry/node-ca-74bb6"
Sep 10 22:01:21.413440 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.412500    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0d6f601b-532d-4f1f-9580-977997448ea9-dbus\") pod \"global-pull-secret-syncer-n9r9x\" (UID: \"0d6f601b-532d-4f1f-9580-977997448ea9\") " pod="kube-system/global-pull-secret-syncer-n9r9x"
Sep 10 22:01:21.413440 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.412548    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b73372d8-bb63-4e98-82e3-05c929fad192-etc-sysctl-d\") pod \"tuned-dth7z\" (UID: \"b73372d8-bb63-4e98-82e3-05c929fad192\") " pod="openshift-cluster-node-tuning-operator/tuned-dth7z"
Sep 10 22:01:21.413440 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.412598    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b73372d8-bb63-4e98-82e3-05c929fad192-tmp\") pod \"tuned-dth7z\" (UID: \"b73372d8-bb63-4e98-82e3-05c929fad192\") " pod="openshift-cluster-node-tuning-operator/tuned-dth7z"
Sep 10 22:01:21.413440 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.412630    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b73372d8-bb63-4e98-82e3-05c929fad192-host\") pod \"tuned-dth7z\" (UID: \"b73372d8-bb63-4e98-82e3-05c929fad192\") " pod="openshift-cluster-node-tuning-operator/tuned-dth7z"
Sep 10 22:01:21.413440 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.412664    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b73372d8-bb63-4e98-82e3-05c929fad192-etc-modprobe-d\") pod \"tuned-dth7z\" (UID: \"b73372d8-bb63-4e98-82e3-05c929fad192\") " pod="openshift-cluster-node-tuning-operator/tuned-dth7z"
Sep 10 22:01:21.413440 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.412715    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b73372d8-bb63-4e98-82e3-05c929fad192-etc-kubernetes\") pod \"tuned-dth7z\" (UID: \"b73372d8-bb63-4e98-82e3-05c929fad192\") " pod="openshift-cluster-node-tuning-operator/tuned-dth7z"
Sep 10 22:01:21.414240 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.412751    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b73372d8-bb63-4e98-82e3-05c929fad192-var-lib-kubelet\") pod \"tuned-dth7z\" (UID: \"b73372d8-bb63-4e98-82e3-05c929fad192\") " pod="openshift-cluster-node-tuning-operator/tuned-dth7z"
Sep 10 22:01:21.414240 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.412809    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0d6f601b-532d-4f1f-9580-977997448ea9-kubelet-config\") pod \"global-pull-secret-syncer-n9r9x\" (UID: \"0d6f601b-532d-4f1f-9580-977997448ea9\") " pod="kube-system/global-pull-secret-syncer-n9r9x"
Sep 10 22:01:21.414240 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.412873    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b73372d8-bb63-4e98-82e3-05c929fad192-etc-tuned\") pod \"tuned-dth7z\" (UID: \"b73372d8-bb63-4e98-82e3-05c929fad192\") " pod="openshift-cluster-node-tuning-operator/tuned-dth7z"
Sep 10 22:01:21.414240 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.412930    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16caf96d-d15f-45e5-b003-1aec5786ea58-tmp-dir\") pod \"node-resolver-ncrzz\" (UID: \"16caf96d-d15f-45e5-b003-1aec5786ea58\") " pod="openshift-dns/node-resolver-ncrzz"
Sep 10 22:01:21.414240 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.412968    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fe63ec98-ecdb-4555-8799-a1f3e9558da3-socket-dir\") pod \"aws-ebs-csi-driver-node-x4xzs\" (UID: \"fe63ec98-ecdb-4555-8799-a1f3e9558da3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x4xzs"
Sep 10 22:01:21.414240 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.413013    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-9nd2t\""
Sep 10 22:01:21.414240 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.413896    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 10 22:01:21.443718 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.443676    2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-09 21:56:20 +0000 UTC" deadline="2028-02-20 21:27:34.128649874 +0000 UTC"
Sep 10 22:01:21.443718 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.443719    2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12671h26m12.684934813s"
Sep 10 22:01:21.497712 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.497668    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-144.ec2.internal" event={"ID":"6ea49b038141fb7080f5d505e03c9c21","Type":"ContainerStarted","Data":"32292f4db980e7e3278e7b8102f250666a5044ff0aa6a664a7fd8dad2872ae4e"}
Sep 10 22:01:21.498984 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.498831    2582 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 10 22:01:21.499463 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.499410    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-144.ec2.internal" event={"ID":"f31b7187b87e2b509b6e74131f844880","Type":"ContainerStarted","Data":"e2d0762684bfdf4c31031d02342191806696517cd8631eac4bf01822efdcfcb6"}
Sep 10 22:01:21.513817 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.513788    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqsbj\" (UniqueName: \"kubernetes.io/projected/3cf12f53-0edf-43cc-913f-50b1669152b2-kube-api-access-fqsbj\") pod \"multus-additional-cni-plugins-dt25t\" (UID: \"3cf12f53-0edf-43cc-913f-50b1669152b2\") " pod="openshift-multus/multus-additional-cni-plugins-dt25t"
Sep 10 22:01:21.513956 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.513827    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7w6k\" (UniqueName: \"kubernetes.io/projected/16caf96d-d15f-45e5-b003-1aec5786ea58-kube-api-access-x7w6k\") pod \"node-resolver-ncrzz\" (UID: \"16caf96d-d15f-45e5-b003-1aec5786ea58\") " pod="openshift-dns/node-resolver-ncrzz"
Sep 10 22:01:21.513956 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.513845    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3cf12f53-0edf-43cc-913f-50b1669152b2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dt25t\" (UID: \"3cf12f53-0edf-43cc-913f-50b1669152b2\") " pod="openshift-multus/multus-additional-cni-plugins-dt25t"
Sep 10 22:01:21.513956 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.513884    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f430e7fc-0f5e-48e2-9ea6-f97c277cdb06-serviceca\") pod \"node-ca-74bb6\" (UID: \"f430e7fc-0f5e-48e2-9ea6-f97c277cdb06\") " pod="openshift-image-registry/node-ca-74bb6"
Sep 10 22:01:21.513956 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.513924    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0d6f601b-532d-4f1f-9580-977997448ea9-dbus\") pod \"global-pull-secret-syncer-n9r9x\" (UID: \"0d6f601b-532d-4f1f-9580-977997448ea9\") " pod="kube-system/global-pull-secret-syncer-n9r9x"
Sep 10 22:01:21.514176 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.514002    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0d6f601b-532d-4f1f-9580-977997448ea9-dbus\") pod \"global-pull-secret-syncer-n9r9x\" (UID: \"0d6f601b-532d-4f1f-9580-977997448ea9\") " pod="kube-system/global-pull-secret-syncer-n9r9x"
Sep 10 22:01:21.514176 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.514038    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b73372d8-bb63-4e98-82e3-05c929fad192-tmp\") pod \"tuned-dth7z\" (UID: \"b73372d8-bb63-4e98-82e3-05c929fad192\") " pod="openshift-cluster-node-tuning-operator/tuned-dth7z"
Sep 10 22:01:21.514176 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.514142    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1731f853-3630-4b20-b78c-5243c08f20e4-host-var-lib-cni-bin\") pod \"multus-8rwdz\" (UID: \"1731f853-3630-4b20-b78c-5243c08f20e4\") " pod="openshift-multus/multus-8rwdz"
Sep 10 22:01:21.514176 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.514170    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8750bcd5-12c7-45a5-9157-84ed58d80d24-host-run-netns\") pod \"ovnkube-node-zbjsk\" (UID: \"8750bcd5-12c7-45a5-9157-84ed58d80d24\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbjsk"
Sep 10 22:01:21.514362 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.514193    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8750bcd5-12c7-45a5-9157-84ed58d80d24-env-overrides\") pod \"ovnkube-node-zbjsk\" (UID: \"8750bcd5-12c7-45a5-9157-84ed58d80d24\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbjsk"
Sep 10 22:01:21.514362 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.514218    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b73372d8-bb63-4e98-82e3-05c929fad192-host\") pod \"tuned-dth7z\" (UID: \"b73372d8-bb63-4e98-82e3-05c929fad192\") " pod="openshift-cluster-node-tuning-operator/tuned-dth7z"
Sep 10 22:01:21.514362 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.514246    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b73372d8-bb63-4e98-82e3-05c929fad192-etc-modprobe-d\") pod \"tuned-dth7z\" (UID: \"b73372d8-bb63-4e98-82e3-05c929fad192\") " pod="openshift-cluster-node-tuning-operator/tuned-dth7z"
Sep 10 22:01:21.514362 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.514271    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b73372d8-bb63-4e98-82e3-05c929fad192-etc-kubernetes\") pod \"tuned-dth7z\" (UID: \"b73372d8-bb63-4e98-82e3-05c929fad192\") " pod="openshift-cluster-node-tuning-operator/tuned-dth7z"
Sep 10 22:01:21.514362 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.514291    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8750bcd5-12c7-45a5-9157-84ed58d80d24-systemd-units\") pod \"ovnkube-node-zbjsk\" (UID: \"8750bcd5-12c7-45a5-9157-84ed58d80d24\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbjsk"
Sep 10 22:01:21.514362 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.514311    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8750bcd5-12c7-45a5-9157-84ed58d80d24-var-lib-openvswitch\") pod \"ovnkube-node-zbjsk\" (UID: \"8750bcd5-12c7-45a5-9157-84ed58d80d24\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbjsk"
Sep 10 22:01:21.514362 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.514360    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f430e7fc-0f5e-48e2-9ea6-f97c277cdb06-serviceca\") pod \"node-ca-74bb6\" (UID: \"f430e7fc-0f5e-48e2-9ea6-f97c277cdb06\") " pod="openshift-image-registry/node-ca-74bb6"
Sep 10 22:01:21.514691 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.514368    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b73372d8-bb63-4e98-82e3-05c929fad192-host\") pod \"tuned-dth7z\" (UID: \"b73372d8-bb63-4e98-82e3-05c929fad192\") " pod="openshift-cluster-node-tuning-operator/tuned-dth7z"
Sep 10 22:01:21.514691 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.514345    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8750bcd5-12c7-45a5-9157-84ed58d80d24-host-run-ovn-kubernetes\") pod \"ovnkube-node-zbjsk\" (UID: \"8750bcd5-12c7-45a5-9157-84ed58d80d24\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbjsk"
Sep 10 22:01:21.514691 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.514438    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8750bcd5-12c7-45a5-9157-84ed58d80d24-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zbjsk\" (UID: \"8750bcd5-12c7-45a5-9157-84ed58d80d24\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbjsk"
Sep 10 22:01:21.514691 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.514461    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f0c8381d-52b6-4511-90be-59aebfef283a-konnectivity-ca\") pod \"konnectivity-agent-cjdvg\" (UID: \"f0c8381d-52b6-4511-90be-59aebfef283a\") " pod="kube-system/konnectivity-agent-cjdvg"
Sep 10 22:01:21.514691 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.514480    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b73372d8-bb63-4e98-82e3-05c929fad192-etc-modprobe-d\") pod \"tuned-dth7z\" (UID: \"b73372d8-bb63-4e98-82e3-05c929fad192\") " pod="openshift-cluster-node-tuning-operator/tuned-dth7z"
Sep 10 22:01:21.514691 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.514483    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0d6f601b-532d-4f1f-9580-977997448ea9-kubelet-config\") pod \"global-pull-secret-syncer-n9r9x\" (UID: \"0d6f601b-532d-4f1f-9580-977997448ea9\") " pod="kube-system/global-pull-secret-syncer-n9r9x"
Sep 10 22:01:21.514691 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.514533    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fe63ec98-ecdb-4555-8799-a1f3e9558da3-socket-dir\") pod \"aws-ebs-csi-driver-node-x4xzs\" (UID: \"fe63ec98-ecdb-4555-8799-a1f3e9558da3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x4xzs"
Sep 10 22:01:21.514691 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.514536    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0d6f601b-532d-4f1f-9580-977997448ea9-kubelet-config\") pod \"global-pull-secret-syncer-n9r9x\" (UID: \"0d6f601b-532d-4f1f-9580-977997448ea9\") " pod="kube-system/global-pull-secret-syncer-n9r9x"
Sep 10 22:01:21.514691 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.514568    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1731f853-3630-4b20-b78c-5243c08f20e4-host-run-k8s-cni-cncf-io\") pod \"multus-8rwdz\" (UID: \"1731f853-3630-4b20-b78c-5243c08f20e4\") " pod="openshift-multus/multus-8rwdz"
Sep 10 22:01:21.514691 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.514599    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b73372d8-bb63-4e98-82e3-05c929fad192-etc-kubernetes\") pod \"tuned-dth7z\" (UID: \"b73372d8-bb63-4e98-82e3-05c929fad192\") " pod="openshift-cluster-node-tuning-operator/tuned-dth7z"
Sep 10 22:01:21.515191 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.514609    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1731f853-3630-4b20-b78c-5243c08f20e4-host-run-multus-certs\") pod \"multus-8rwdz\" (UID: \"1731f853-3630-4b20-b78c-5243c08f20e4\") " pod="openshift-multus/multus-8rwdz"
Sep 10 22:01:21.515191 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.514903    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0d6f601b-532d-4f1f-9580-977997448ea9-original-pull-secret\") pod \"global-pull-secret-syncer-n9r9x\" (UID: \"0d6f601b-532d-4f1f-9580-977997448ea9\") " pod="kube-system/global-pull-secret-syncer-n9r9x"
Sep 10 22:01:21.515191 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.514930    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvt5v\" (UniqueName: \"kubernetes.io/projected/a6c7caf6-cae8-4a81-98c5-e4db1671eff5-kube-api-access-rvt5v\") pod \"network-metrics-daemon-xw7pd\" (UID: \"a6c7caf6-cae8-4a81-98c5-e4db1671eff5\") " pod="openshift-multus/network-metrics-daemon-xw7pd"
Sep 10 22:01:21.515191 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.514951    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1731f853-3630-4b20-b78c-5243c08f20e4-system-cni-dir\") pod \"multus-8rwdz\" (UID: \"1731f853-3630-4b20-b78c-5243c08f20e4\") " pod="openshift-multus/multus-8rwdz"
Sep 10 22:01:21.515191 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.514996    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1731f853-3630-4b20-b78c-5243c08f20e4-hostroot\") pod \"multus-8rwdz\" (UID: \"1731f853-3630-4b20-b78c-5243c08f20e4\") " pod="openshift-multus/multus-8rwdz"
Sep 10 22:01:21.515191 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.515014    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1731f853-3630-4b20-b78c-5243c08f20e4-etc-kubernetes\") pod \"multus-8rwdz\" (UID: \"1731f853-3630-4b20-b78c-5243c08f20e4\") " pod="openshift-multus/multus-8rwdz"
Sep 10 22:01:21.515191 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.515031    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwfrj\" (UniqueName: \"kubernetes.io/projected/064a91af-7e3d-4e71-b743-04d68d63174e-kube-api-access-zwfrj\") pod \"network-check-target-rrm25\" (UID: \"064a91af-7e3d-4e71-b743-04d68d63174e\") " pod="openshift-network-diagnostics/network-check-target-rrm25"
Sep 10 22:01:21.515191 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.515056    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b73372d8-bb63-4e98-82e3-05c929fad192-etc-sysctl-conf\") pod \"tuned-dth7z\" (UID: \"b73372d8-bb63-4e98-82e3-05c929fad192\") " pod="openshift-cluster-node-tuning-operator/tuned-dth7z"
Sep 10 22:01:21.515191 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:01:21.515060    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 22:01:21.515191 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.515075    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b73372d8-bb63-4e98-82e3-05c929fad192-run\") pod \"tuned-dth7z\" (UID: \"b73372d8-bb63-4e98-82e3-05c929fad192\") " pod="openshift-cluster-node-tuning-operator/tuned-dth7z"
Sep 10 22:01:21.515191 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.515099    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fe63ec98-ecdb-4555-8799-a1f3e9558da3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-x4xzs\" (UID: \"fe63ec98-ecdb-4555-8799-a1f3e9558da3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x4xzs"
Sep 10 22:01:21.515191 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:01:21.515128    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d6f601b-532d-4f1f-9580-977997448ea9-original-pull-secret podName:0d6f601b-532d-4f1f-9580-977997448ea9 nodeName:}" failed. No retries permitted until 2026-09-10 22:01:22.015102924 +0000 UTC m=+2.971170615 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0d6f601b-532d-4f1f-9580-977997448ea9-original-pull-secret") pod "global-pull-secret-syncer-n9r9x" (UID: "0d6f601b-532d-4f1f-9580-977997448ea9") : object "kube-system"/"original-pull-secret" not registered
Sep 10 22:01:21.515191 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.515161    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fe63ec98-ecdb-4555-8799-a1f3e9558da3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-x4xzs\" (UID: \"fe63ec98-ecdb-4555-8799-a1f3e9558da3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x4xzs"
Sep 10 22:01:21.515191 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.515160    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fe63ec98-ecdb-4555-8799-a1f3e9558da3-sys-fs\") pod \"aws-ebs-csi-driver-node-x4xzs\" (UID: \"fe63ec98-ecdb-4555-8799-a1f3e9558da3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x4xzs"
Sep 10 22:01:21.515191 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.515195    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chqgf\" (UniqueName: \"kubernetes.io/projected/fe63ec98-ecdb-4555-8799-a1f3e9558da3-kube-api-access-chqgf\") pod \"aws-ebs-csi-driver-node-x4xzs\" (UID: \"fe63ec98-ecdb-4555-8799-a1f3e9558da3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x4xzs"
Sep 10 22:01:21.515895 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.515209    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fe63ec98-ecdb-4555-8799-a1f3e9558da3-sys-fs\") pod \"aws-ebs-csi-driver-node-x4xzs\" (UID: \"fe63ec98-ecdb-4555-8799-a1f3e9558da3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x4xzs"
Sep 10 22:01:21.515895 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.515222    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8750bcd5-12c7-45a5-9157-84ed58d80d24-ovnkube-script-lib\") pod \"ovnkube-node-zbjsk\" (UID: \"8750bcd5-12c7-45a5-9157-84ed58d80d24\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbjsk"
Sep 10 22:01:21.515895 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.515244    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3cf12f53-0edf-43cc-913f-50b1669152b2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dt25t\" (UID: \"3cf12f53-0edf-43cc-913f-50b1669152b2\") " pod="openshift-multus/multus-additional-cni-plugins-dt25t"
Sep 10 22:01:21.515895 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.514708    2582 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 10 22:01:21.515895 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.514645    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fe63ec98-ecdb-4555-8799-a1f3e9558da3-socket-dir\") pod \"aws-ebs-csi-driver-node-x4xzs\" (UID: \"fe63ec98-ecdb-4555-8799-a1f3e9558da3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x4xzs"
Sep 10 22:01:21.515895 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.515337    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2k6gp\" (UniqueName: \"kubernetes.io/projected/7821d1b4-78c1-4500-84e4-1b1fc03ad836-kube-api-access-2k6gp\") pod \"iptables-alerter-p8f5t\" (UID: \"7821d1b4-78c1-4500-84e4-1b1fc03ad836\") " pod="openshift-network-operator/iptables-alerter-p8f5t"
Sep 10 22:01:21.515895 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.515408    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b73372d8-bb63-4e98-82e3-05c929fad192-etc-sysctl-conf\") pod \"tuned-dth7z\" (UID: \"b73372d8-bb63-4e98-82e3-05c929fad192\") " pod="openshift-cluster-node-tuning-operator/tuned-dth7z"
Sep 10 22:01:21.515895 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.515426    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b73372d8-bb63-4e98-82e3-05c929fad192-etc-systemd\") pod \"tuned-dth7z\" (UID: \"b73372d8-bb63-4e98-82e3-05c929fad192\") " pod="openshift-cluster-node-tuning-operator/tuned-dth7z"
Sep 10 22:01:21.515895 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.515489    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b73372d8-bb63-4e98-82e3-05c929fad192-etc-systemd\") pod \"tuned-dth7z\" (UID: \"b73372d8-bb63-4e98-82e3-05c929fad192\") " pod="openshift-cluster-node-tuning-operator/tuned-dth7z"
Sep 10 22:01:21.515895 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.515503    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b73372d8-bb63-4e98-82e3-05c929fad192-run\") pod \"tuned-dth7z\" (UID: \"b73372d8-bb63-4e98-82e3-05c929fad192\") " pod="openshift-cluster-node-tuning-operator/tuned-dth7z"
Sep 10 22:01:21.515895 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.515648    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b73372d8-bb63-4e98-82e3-05c929fad192-lib-modules\") pod \"tuned-dth7z\" (UID: \"b73372d8-bb63-4e98-82e3-05c929fad192\") " pod="openshift-cluster-node-tuning-operator/tuned-dth7z"
Sep 10 22:01:21.515895 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.515681    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tv6xk\" (UniqueName: \"kubernetes.io/projected/1731f853-3630-4b20-b78c-5243c08f20e4-kube-api-access-tv6xk\") pod \"multus-8rwdz\" (UID: \"1731f853-3630-4b20-b78c-5243c08f20e4\") " pod="openshift-multus/multus-8rwdz"
Sep 10 22:01:21.515895 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.515720    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8750bcd5-12c7-45a5-9157-84ed58d80d24-node-log\") pod \"ovnkube-node-zbjsk\" (UID: \"8750bcd5-12c7-45a5-9157-84ed58d80d24\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbjsk"
Sep 10 22:01:21.515895 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.515743    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3cf12f53-0edf-43cc-913f-50b1669152b2-cni-binary-copy\") pod \"multus-additional-cni-plugins-dt25t\" (UID: \"3cf12f53-0edf-43cc-913f-50b1669152b2\") " pod="openshift-multus/multus-additional-cni-plugins-dt25t"
Sep 10 22:01:21.515895 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.515773    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1731f853-3630-4b20-b78c-5243c08f20e4-host-var-lib-kubelet\") pod \"multus-8rwdz\" (UID: \"1731f853-3630-4b20-b78c-5243c08f20e4\") " pod="openshift-multus/multus-8rwdz"
Sep 10 22:01:21.515895 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.515776    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b73372d8-bb63-4e98-82e3-05c929fad192-lib-modules\") pod \"tuned-dth7z\" (UID: \"b73372d8-bb63-4e98-82e3-05c929fad192\") " pod="openshift-cluster-node-tuning-operator/tuned-dth7z"
Sep 10 22:01:21.515895 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.515804    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8750bcd5-12c7-45a5-9157-84ed58d80d24-run-openvswitch\") pod \"ovnkube-node-zbjsk\" (UID: \"8750bcd5-12c7-45a5-9157-84ed58d80d24\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbjsk"
Sep 10 22:01:21.516675 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.515833    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8750bcd5-12c7-45a5-9157-84ed58d80d24-run-ovn\") pod \"ovnkube-node-zbjsk\" (UID: \"8750bcd5-12c7-45a5-9157-84ed58d80d24\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbjsk"
Sep 10 22:01:21.516675 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.515869    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f0c8381d-52b6-4511-90be-59aebfef283a-agent-certs\") pod \"konnectivity-agent-cjdvg\" (UID: \"f0c8381d-52b6-4511-90be-59aebfef283a\") " pod="kube-system/konnectivity-agent-cjdvg"
Sep 10 22:01:21.516675 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.515903    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b73372d8-bb63-4e98-82e3-05c929fad192-sys\") pod \"tuned-dth7z\" (UID: \"b73372d8-bb63-4e98-82e3-05c929fad192\") " pod="openshift-cluster-node-tuning-operator/tuned-dth7z"
Sep 10 22:01:21.516675 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.515944    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6c7caf6-cae8-4a81-98c5-e4db1671eff5-metrics-certs\") pod \"network-metrics-daemon-xw7pd\" (UID: \"a6c7caf6-cae8-4a81-98c5-e4db1671eff5\") " pod="openshift-multus/network-metrics-daemon-xw7pd"
Sep 10 22:01:21.516675 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.515970    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1731f853-3630-4b20-b78c-5243c08f20e4-multus-socket-dir-parent\") pod \"multus-8rwdz\" (UID: \"1731f853-3630-4b20-b78c-5243c08f20e4\") " pod="openshift-multus/multus-8rwdz"
Sep 10 22:01:21.516675 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.516018    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1731f853-3630-4b20-b78c-5243c08f20e4-multus-daemon-config\") pod \"multus-8rwdz\" (UID: \"1731f853-3630-4b20-b78c-5243c08f20e4\") " pod="openshift-multus/multus-8rwdz"
Sep 10 22:01:21.516675 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.516041    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8750bcd5-12c7-45a5-9157-84ed58d80d24-host-kubelet\") pod \"ovnkube-node-zbjsk\" (UID: \"8750bcd5-12c7-45a5-9157-84ed58d80d24\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbjsk"
Sep 10 22:01:21.516675 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.516063    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8750bcd5-12c7-45a5-9157-84ed58d80d24-etc-openvswitch\") pod \"ovnkube-node-zbjsk\" (UID: \"8750bcd5-12c7-45a5-9157-84ed58d80d24\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbjsk"
Sep 10 22:01:21.516675 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:01:21.516146    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 22:01:21.516675 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.516178    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8750bcd5-12c7-45a5-9157-84ed58d80d24-ovn-node-metrics-cert\") pod \"ovnkube-node-zbjsk\" (UID: \"8750bcd5-12c7-45a5-9157-84ed58d80d24\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbjsk"
Sep 10 22:01:21.516675 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:01:21.516203    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6c7caf6-cae8-4a81-98c5-e4db1671eff5-metrics-certs podName:a6c7caf6-cae8-4a81-98c5-e4db1671eff5 nodeName:}" failed. No retries permitted until 2026-09-10 22:01:22.016186663 +0000 UTC m=+2.972254360 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a6c7caf6-cae8-4a81-98c5-e4db1671eff5-metrics-certs") pod "network-metrics-daemon-xw7pd" (UID: "a6c7caf6-cae8-4a81-98c5-e4db1671eff5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 22:01:21.516675 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.516246    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b73372d8-bb63-4e98-82e3-05c929fad192-sys\") pod \"tuned-dth7z\" (UID: \"b73372d8-bb63-4e98-82e3-05c929fad192\") " pod="openshift-cluster-node-tuning-operator/tuned-dth7z"
Sep 10 22:01:21.516675 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.516283    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzhtv\" (UniqueName: \"kubernetes.io/projected/f430e7fc-0f5e-48e2-9ea6-f97c277cdb06-kube-api-access-vzhtv\") pod \"node-ca-74bb6\" (UID: \"f430e7fc-0f5e-48e2-9ea6-f97c277cdb06\") " pod="openshift-image-registry/node-ca-74bb6"
Sep 10 22:01:21.516675 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.516319    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b73372d8-bb63-4e98-82e3-05c929fad192-etc-sysctl-d\") pod \"tuned-dth7z\" (UID: \"b73372d8-bb63-4e98-82e3-05c929fad192\") " pod="openshift-cluster-node-tuning-operator/tuned-dth7z"
Sep 10 22:01:21.516675 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.516439    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1731f853-3630-4b20-b78c-5243c08f20e4-multus-cni-dir\") pod \"multus-8rwdz\" (UID: \"1731f853-3630-4b20-b78c-5243c08f20e4\") " pod="openshift-multus/multus-8rwdz"
Sep 10 22:01:21.516675 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.516480    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrgtp\" (UniqueName: \"kubernetes.io/projected/8750bcd5-12c7-45a5-9157-84ed58d80d24-kube-api-access-nrgtp\") pod \"ovnkube-node-zbjsk\" (UID: \"8750bcd5-12c7-45a5-9157-84ed58d80d24\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbjsk"
Sep 10 22:01:21.516675 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.516527    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b73372d8-bb63-4e98-82e3-05c929fad192-var-lib-kubelet\") pod \"tuned-dth7z\" (UID: \"b73372d8-bb63-4e98-82e3-05c929fad192\") " pod="openshift-cluster-node-tuning-operator/tuned-dth7z"
Sep 10 22:01:21.517330 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.516513    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b73372d8-bb63-4e98-82e3-05c929fad192-etc-sysctl-d\") pod \"tuned-dth7z\" (UID: \"b73372d8-bb63-4e98-82e3-05c929fad192\") " pod="openshift-cluster-node-tuning-operator/tuned-dth7z"
Sep 10 22:01:21.517330 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.516588    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8750bcd5-12c7-45a5-9157-84ed58d80d24-host-slash\") pod \"ovnkube-node-zbjsk\" (UID: \"8750bcd5-12c7-45a5-9157-84ed58d80d24\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbjsk"
Sep 10 22:01:21.517330 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.516613    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8750bcd5-12c7-45a5-9157-84ed58d80d24-host-cni-netd\") pod \"ovnkube-node-zbjsk\" (UID: \"8750bcd5-12c7-45a5-9157-84ed58d80d24\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbjsk"
Sep 10 22:01:21.517330 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.516612    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b73372d8-bb63-4e98-82e3-05c929fad192-var-lib-kubelet\") pod \"tuned-dth7z\" (UID: \"b73372d8-bb63-4e98-82e3-05c929fad192\") " pod="openshift-cluster-node-tuning-operator/tuned-dth7z"
Sep 10 22:01:21.517330 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.516637    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3cf12f53-0edf-43cc-913f-50b1669152b2-system-cni-dir\") pod \"multus-additional-cni-plugins-dt25t\" (UID: \"3cf12f53-0edf-43cc-913f-50b1669152b2\") " pod="openshift-multus/multus-additional-cni-plugins-dt25t"
Sep 10 22:01:21.517330 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.516686    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b73372d8-bb63-4e98-82e3-05c929fad192-etc-tuned\") pod \"tuned-dth7z\" (UID: \"b73372d8-bb63-4e98-82e3-05c929fad192\") " pod="openshift-cluster-node-tuning-operator/tuned-dth7z"
Sep 10 22:01:21.517330 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.516759    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16caf96d-d15f-45e5-b003-1aec5786ea58-tmp-dir\") pod \"node-resolver-ncrzz\" (UID: \"16caf96d-d15f-45e5-b003-1aec5786ea58\") " pod="openshift-dns/node-resolver-ncrzz"
Sep 10 22:01:21.517330 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.516800    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8750bcd5-12c7-45a5-9157-84ed58d80d24-log-socket\") pod \"ovnkube-node-zbjsk\" (UID: \"8750bcd5-12c7-45a5-9157-84ed58d80d24\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbjsk"
Sep 10 22:01:21.517330 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.516827    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7821d1b4-78c1-4500-84e4-1b1fc03ad836-host-slash\") pod \"iptables-alerter-p8f5t\" (UID: \"7821d1b4-78c1-4500-84e4-1b1fc03ad836\") " pod="openshift-network-operator/iptables-alerter-p8f5t"
Sep 10 22:01:21.517330 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.516886    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vzmd\" (UniqueName: \"kubernetes.io/projected/b73372d8-bb63-4e98-82e3-05c929fad192-kube-api-access-9vzmd\") pod \"tuned-dth7z\" (UID: \"b73372d8-bb63-4e98-82e3-05c929fad192\") " pod="openshift-cluster-node-tuning-operator/tuned-dth7z"
Sep 10 22:01:21.517330 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.516914    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fe63ec98-ecdb-4555-8799-a1f3e9558da3-etc-selinux\") pod \"aws-ebs-csi-driver-node-x4xzs\" (UID: \"fe63ec98-ecdb-4555-8799-a1f3e9558da3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x4xzs"
Sep 10 22:01:21.517330 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.516937    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1731f853-3630-4b20-b78c-5243c08f20e4-cnibin\") pod \"multus-8rwdz\" (UID: \"1731f853-3630-4b20-b78c-5243c08f20e4\") " pod="openshift-multus/multus-8rwdz"
Sep 10 22:01:21.517330 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.516989    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1731f853-3630-4b20-b78c-5243c08f20e4-cni-binary-copy\") pod \"multus-8rwdz\" (UID: \"1731f853-3630-4b20-b78c-5243c08f20e4\") " pod="openshift-multus/multus-8rwdz"
Sep 10 22:01:21.517330 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.517013    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8750bcd5-12c7-45a5-9157-84ed58d80d24-run-systemd\") pod \"ovnkube-node-zbjsk\" (UID: \"8750bcd5-12c7-45a5-9157-84ed58d80d24\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbjsk"
Sep 10 22:01:21.517330 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.517037    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8750bcd5-12c7-45a5-9157-84ed58d80d24-host-cni-bin\") pod \"ovnkube-node-zbjsk\" (UID: \"8750bcd5-12c7-45a5-9157-84ed58d80d24\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbjsk"
Sep 10 22:01:21.517330 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.517055    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fe63ec98-ecdb-4555-8799-a1f3e9558da3-etc-selinux\") pod \"aws-ebs-csi-driver-node-x4xzs\" (UID: \"fe63ec98-ecdb-4555-8799-a1f3e9558da3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x4xzs"
Sep 10 22:01:21.517330 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.517059    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3cf12f53-0edf-43cc-913f-50b1669152b2-cnibin\") pod \"multus-additional-cni-plugins-dt25t\" (UID: \"3cf12f53-0edf-43cc-913f-50b1669152b2\") " pod="openshift-multus/multus-additional-cni-plugins-dt25t"
Sep 10 22:01:21.518008 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.517113    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7821d1b4-78c1-4500-84e4-1b1fc03ad836-host-slash\") pod \"iptables-alerter-p8f5t\" (UID: \"7821d1b4-78c1-4500-84e4-1b1fc03ad836\") " pod="openshift-network-operator/iptables-alerter-p8f5t"
Sep 10 22:01:21.518008 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.517332    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fe63ec98-ecdb-4555-8799-a1f3e9558da3-device-dir\") pod \"aws-ebs-csi-driver-node-x4xzs\" (UID: \"fe63ec98-ecdb-4555-8799-a1f3e9558da3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x4xzs"
Sep 10 22:01:21.518008 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.517369    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1731f853-3630-4b20-b78c-5243c08f20e4-multus-conf-dir\") pod \"multus-8rwdz\" (UID: \"1731f853-3630-4b20-b78c-5243c08f20e4\") " pod="openshift-multus/multus-8rwdz"
Sep 10 22:01:21.518008 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.517388    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16caf96d-d15f-45e5-b003-1aec5786ea58-tmp-dir\") pod \"node-resolver-ncrzz\" (UID: \"16caf96d-d15f-45e5-b003-1aec5786ea58\") " pod="openshift-dns/node-resolver-ncrzz"
Sep 10 22:01:21.518008 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.517429    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fe63ec98-ecdb-4555-8799-a1f3e9558da3-device-dir\") pod \"aws-ebs-csi-driver-node-x4xzs\" (UID: \"fe63ec98-ecdb-4555-8799-a1f3e9558da3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x4xzs"
Sep 10 22:01:21.518008 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.517440    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f430e7fc-0f5e-48e2-9ea6-f97c277cdb06-host\") pod \"node-ca-74bb6\" (UID: \"f430e7fc-0f5e-48e2-9ea6-f97c277cdb06\") " pod="openshift-image-registry/node-ca-74bb6"
Sep 10 22:01:21.518008 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.517474    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fe63ec98-ecdb-4555-8799-a1f3e9558da3-registration-dir\") pod \"aws-ebs-csi-driver-node-x4xzs\" (UID: \"fe63ec98-ecdb-4555-8799-a1f3e9558da3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x4xzs"
Sep 10 22:01:21.518008 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.517487    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f430e7fc-0f5e-48e2-9ea6-f97c277cdb06-host\") pod \"node-ca-74bb6\" (UID: \"f430e7fc-0f5e-48e2-9ea6-f97c277cdb06\") " pod="openshift-image-registry/node-ca-74bb6"
Sep 10 22:01:21.518008 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.517513    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7821d1b4-78c1-4500-84e4-1b1fc03ad836-iptables-alerter-script\") pod \"iptables-alerter-p8f5t\" (UID: \"7821d1b4-78c1-4500-84e4-1b1fc03ad836\") " pod="openshift-network-operator/iptables-alerter-p8f5t"
Sep 10 22:01:21.518008 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.517518    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fe63ec98-ecdb-4555-8799-a1f3e9558da3-registration-dir\") pod \"aws-ebs-csi-driver-node-x4xzs\" (UID: \"fe63ec98-ecdb-4555-8799-a1f3e9558da3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x4xzs"
Sep 10 22:01:21.518008 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.517549    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b73372d8-bb63-4e98-82e3-05c929fad192-etc-sysconfig\") pod \"tuned-dth7z\" (UID: \"b73372d8-bb63-4e98-82e3-05c929fad192\") " pod="openshift-cluster-node-tuning-operator/tuned-dth7z"
Sep 10 22:01:21.518008 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.517572    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/16caf96d-d15f-45e5-b003-1aec5786ea58-hosts-file\") pod \"node-resolver-ncrzz\" (UID: \"16caf96d-d15f-45e5-b003-1aec5786ea58\") " pod="openshift-dns/node-resolver-ncrzz"
Sep 10 22:01:21.518008 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.517596    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1731f853-3630-4b20-b78c-5243c08f20e4-host-run-netns\") pod \"multus-8rwdz\" (UID: \"1731f853-3630-4b20-b78c-5243c08f20e4\") " pod="openshift-multus/multus-8rwdz"
Sep 10 22:01:21.518008 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.517624    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1731f853-3630-4b20-b78c-5243c08f20e4-host-var-lib-cni-multus\") pod \"multus-8rwdz\" (UID: \"1731f853-3630-4b20-b78c-5243c08f20e4\") " pod="openshift-multus/multus-8rwdz"
Sep 10 22:01:21.518008 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.517667    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/16caf96d-d15f-45e5-b003-1aec5786ea58-hosts-file\") pod \"node-resolver-ncrzz\" (UID: \"16caf96d-d15f-45e5-b003-1aec5786ea58\") " pod="openshift-dns/node-resolver-ncrzz"
Sep 10 22:01:21.518008 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.517671    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8750bcd5-12c7-45a5-9157-84ed58d80d24-ovnkube-config\") pod \"ovnkube-node-zbjsk\" (UID: \"8750bcd5-12c7-45a5-9157-84ed58d80d24\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbjsk"
Sep 10 22:01:21.518008 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.517626    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b73372d8-bb63-4e98-82e3-05c929fad192-etc-sysconfig\") pod \"tuned-dth7z\" (UID: \"b73372d8-bb63-4e98-82e3-05c929fad192\") " pod="openshift-cluster-node-tuning-operator/tuned-dth7z"
Sep 10 22:01:21.518822 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.517706    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3cf12f53-0edf-43cc-913f-50b1669152b2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dt25t\" (UID: \"3cf12f53-0edf-43cc-913f-50b1669152b2\") " pod="openshift-multus/multus-additional-cni-plugins-dt25t"
Sep 10 22:01:21.518822 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.518035    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7821d1b4-78c1-4500-84e4-1b1fc03ad836-iptables-alerter-script\") pod \"iptables-alerter-p8f5t\" (UID: \"7821d1b4-78c1-4500-84e4-1b1fc03ad836\") " pod="openshift-network-operator/iptables-alerter-p8f5t"
Sep 10 22:01:21.519865 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.519828    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b73372d8-bb63-4e98-82e3-05c929fad192-tmp\") pod \"tuned-dth7z\" (UID: \"b73372d8-bb63-4e98-82e3-05c929fad192\") " pod="openshift-cluster-node-tuning-operator/tuned-dth7z"
Sep 10 22:01:21.520162 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.520134    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b73372d8-bb63-4e98-82e3-05c929fad192-etc-tuned\") pod \"tuned-dth7z\" (UID: \"b73372d8-bb63-4e98-82e3-05c929fad192\") " pod="openshift-cluster-node-tuning-operator/tuned-dth7z"
Sep 10 22:01:21.521679 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.521636    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7w6k\" (UniqueName: \"kubernetes.io/projected/16caf96d-d15f-45e5-b003-1aec5786ea58-kube-api-access-x7w6k\") pod \"node-resolver-ncrzz\" (UID: \"16caf96d-d15f-45e5-b003-1aec5786ea58\") " pod="openshift-dns/node-resolver-ncrzz"
Sep 10 22:01:21.527055 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:01:21.527036    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 22:01:21.527163 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:01:21.527062    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 22:01:21.527163 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:01:21.527076    2582 projected.go:196] Error preparing data for projected volume kube-api-access-zwfrj for pod openshift-network-diagnostics/network-check-target-rrm25: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 22:01:21.527163 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:01:21.527132    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/064a91af-7e3d-4e71-b743-04d68d63174e-kube-api-access-zwfrj podName:064a91af-7e3d-4e71-b743-04d68d63174e nodeName:}" failed. No retries permitted until 2026-09-10 22:01:22.027115154 +0000 UTC m=+2.983182839 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-zwfrj" (UniqueName: "kubernetes.io/projected/064a91af-7e3d-4e71-b743-04d68d63174e-kube-api-access-zwfrj") pod "network-check-target-rrm25" (UID: "064a91af-7e3d-4e71-b743-04d68d63174e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 22:01:21.529164 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.529138    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzhtv\" (UniqueName: \"kubernetes.io/projected/f430e7fc-0f5e-48e2-9ea6-f97c277cdb06-kube-api-access-vzhtv\") pod \"node-ca-74bb6\" (UID: \"f430e7fc-0f5e-48e2-9ea6-f97c277cdb06\") " pod="openshift-image-registry/node-ca-74bb6"
Sep 10 22:01:21.529775 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.529753    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-chqgf\" (UniqueName: \"kubernetes.io/projected/fe63ec98-ecdb-4555-8799-a1f3e9558da3-kube-api-access-chqgf\") pod \"aws-ebs-csi-driver-node-x4xzs\" (UID: \"fe63ec98-ecdb-4555-8799-a1f3e9558da3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x4xzs"
Sep 10 22:01:21.530181 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.530115    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vzmd\" (UniqueName: \"kubernetes.io/projected/b73372d8-bb63-4e98-82e3-05c929fad192-kube-api-access-9vzmd\") pod \"tuned-dth7z\" (UID: \"b73372d8-bb63-4e98-82e3-05c929fad192\") " pod="openshift-cluster-node-tuning-operator/tuned-dth7z"
Sep 10 22:01:21.530989 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.530824    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvt5v\" (UniqueName: \"kubernetes.io/projected/a6c7caf6-cae8-4a81-98c5-e4db1671eff5-kube-api-access-rvt5v\") pod \"network-metrics-daemon-xw7pd\" (UID: \"a6c7caf6-cae8-4a81-98c5-e4db1671eff5\") " pod="openshift-multus/network-metrics-daemon-xw7pd"
Sep 10 22:01:21.531317 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.531293    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2k6gp\" (UniqueName: \"kubernetes.io/projected/7821d1b4-78c1-4500-84e4-1b1fc03ad836-kube-api-access-2k6gp\") pod \"iptables-alerter-p8f5t\" (UID: \"7821d1b4-78c1-4500-84e4-1b1fc03ad836\") " pod="openshift-network-operator/iptables-alerter-p8f5t"
Sep 10 22:01:21.618284 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.618192    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1731f853-3630-4b20-b78c-5243c08f20e4-host-run-netns\") pod \"multus-8rwdz\" (UID: \"1731f853-3630-4b20-b78c-5243c08f20e4\") " pod="openshift-multus/multus-8rwdz"
Sep 10 22:01:21.618284 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.618233    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1731f853-3630-4b20-b78c-5243c08f20e4-host-var-lib-cni-multus\") pod \"multus-8rwdz\" (UID: \"1731f853-3630-4b20-b78c-5243c08f20e4\") " pod="openshift-multus/multus-8rwdz"
Sep 10 22:01:21.618284 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.618255    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8750bcd5-12c7-45a5-9157-84ed58d80d24-ovnkube-config\") pod \"ovnkube-node-zbjsk\" (UID: \"8750bcd5-12c7-45a5-9157-84ed58d80d24\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbjsk"
Sep 10 22:01:21.618284 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.618279    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3cf12f53-0edf-43cc-913f-50b1669152b2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dt25t\" (UID: \"3cf12f53-0edf-43cc-913f-50b1669152b2\") " pod="openshift-multus/multus-additional-cni-plugins-dt25t"
Sep 10 22:01:21.618608 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.618307    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqsbj\" (UniqueName: \"kubernetes.io/projected/3cf12f53-0edf-43cc-913f-50b1669152b2-kube-api-access-fqsbj\") pod \"multus-additional-cni-plugins-dt25t\" (UID: \"3cf12f53-0edf-43cc-913f-50b1669152b2\") " pod="openshift-multus/multus-additional-cni-plugins-dt25t"
Sep 10 22:01:21.618608 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.618314    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1731f853-3630-4b20-b78c-5243c08f20e4-host-run-netns\") pod \"multus-8rwdz\" (UID: \"1731f853-3630-4b20-b78c-5243c08f20e4\") " pod="openshift-multus/multus-8rwdz"
Sep 10 22:01:21.618608 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.618321    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1731f853-3630-4b20-b78c-5243c08f20e4-host-var-lib-cni-multus\") pod \"multus-8rwdz\" (UID: \"1731f853-3630-4b20-b78c-5243c08f20e4\") " pod="openshift-multus/multus-8rwdz"
Sep 10 22:01:21.618608 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.618331    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3cf12f53-0edf-43cc-913f-50b1669152b2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dt25t\" (UID: \"3cf12f53-0edf-43cc-913f-50b1669152b2\") " pod="openshift-multus/multus-additional-cni-plugins-dt25t"
Sep 10 22:01:21.618608 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.618400    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1731f853-3630-4b20-b78c-5243c08f20e4-host-var-lib-cni-bin\") pod \"multus-8rwdz\" (UID: \"1731f853-3630-4b20-b78c-5243c08f20e4\") " pod="openshift-multus/multus-8rwdz"
Sep 10 22:01:21.618608 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.618428    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8750bcd5-12c7-45a5-9157-84ed58d80d24-host-run-netns\") pod \"ovnkube-node-zbjsk\" (UID: \"8750bcd5-12c7-45a5-9157-84ed58d80d24\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbjsk"
Sep 10 22:01:21.618608 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.618453    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8750bcd5-12c7-45a5-9157-84ed58d80d24-env-overrides\") pod \"ovnkube-node-zbjsk\" (UID: \"8750bcd5-12c7-45a5-9157-84ed58d80d24\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbjsk"
Sep 10 22:01:21.618608 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.618478    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8750bcd5-12c7-45a5-9157-84ed58d80d24-systemd-units\") pod \"ovnkube-node-zbjsk\" (UID: \"8750bcd5-12c7-45a5-9157-84ed58d80d24\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbjsk"
Sep 10 22:01:21.618608 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.618499    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8750bcd5-12c7-45a5-9157-84ed58d80d24-var-lib-openvswitch\") pod \"ovnkube-node-zbjsk\" (UID: \"8750bcd5-12c7-45a5-9157-84ed58d80d24\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbjsk"
Sep 10 22:01:21.618608 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.618521    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8750bcd5-12c7-45a5-9157-84ed58d80d24-host-run-ovn-kubernetes\") pod \"ovnkube-node-zbjsk\" (UID: \"8750bcd5-12c7-45a5-9157-84ed58d80d24\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbjsk"
Sep 10 22:01:21.618608 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.618525    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3cf12f53-0edf-43cc-913f-50b1669152b2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dt25t\" (UID: \"3cf12f53-0edf-43cc-913f-50b1669152b2\") " pod="openshift-multus/multus-additional-cni-plugins-dt25t"
Sep 10 22:01:21.618608 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.618548    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8750bcd5-12c7-45a5-9157-84ed58d80d24-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zbjsk\" (UID: \"8750bcd5-12c7-45a5-9157-84ed58d80d24\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbjsk"
Sep 10 22:01:21.618608 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.618576    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f0c8381d-52b6-4511-90be-59aebfef283a-konnectivity-ca\") pod \"konnectivity-agent-cjdvg\" (UID: \"f0c8381d-52b6-4511-90be-59aebfef283a\") " pod="kube-system/konnectivity-agent-cjdvg"
Sep 10 22:01:21.618608 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.618599    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1731f853-3630-4b20-b78c-5243c08f20e4-host-run-k8s-cni-cncf-io\") pod \"multus-8rwdz\" (UID: \"1731f853-3630-4b20-b78c-5243c08f20e4\") " pod="openshift-multus/multus-8rwdz"
Sep 10 22:01:21.619227 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.618619    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1731f853-3630-4b20-b78c-5243c08f20e4-host-var-lib-cni-bin\") pod \"multus-8rwdz\" (UID: \"1731f853-3630-4b20-b78c-5243c08f20e4\") " pod="openshift-multus/multus-8rwdz"
Sep 10 22:01:21.619227 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.618624    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1731f853-3630-4b20-b78c-5243c08f20e4-host-run-multus-certs\") pod \"multus-8rwdz\" (UID: \"1731f853-3630-4b20-b78c-5243c08f20e4\") " pod="openshift-multus/multus-8rwdz"
Sep 10 22:01:21.619227 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.618664    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1731f853-3630-4b20-b78c-5243c08f20e4-host-run-multus-certs\") pod \"multus-8rwdz\" (UID: \"1731f853-3630-4b20-b78c-5243c08f20e4\") " pod="openshift-multus/multus-8rwdz"
Sep 10 22:01:21.619227 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.618684    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1731f853-3630-4b20-b78c-5243c08f20e4-system-cni-dir\") pod \"multus-8rwdz\" (UID: \"1731f853-3630-4b20-b78c-5243c08f20e4\") " pod="openshift-multus/multus-8rwdz"
Sep 10 22:01:21.619227 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.618701    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1731f853-3630-4b20-b78c-5243c08f20e4-hostroot\") pod \"multus-8rwdz\" (UID: \"1731f853-3630-4b20-b78c-5243c08f20e4\") " pod="openshift-multus/multus-8rwdz"
Sep 10 22:01:21.619227 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.618705    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8750bcd5-12c7-45a5-9157-84ed58d80d24-systemd-units\") pod \"ovnkube-node-zbjsk\" (UID: \"8750bcd5-12c7-45a5-9157-84ed58d80d24\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbjsk"
Sep 10 22:01:21.619227 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.618716    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1731f853-3630-4b20-b78c-5243c08f20e4-etc-kubernetes\") pod \"multus-8rwdz\" (UID: \"1731f853-3630-4b20-b78c-5243c08f20e4\") " pod="openshift-multus/multus-8rwdz"
Sep 10 22:01:21.619227 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.618754    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8750bcd5-12c7-45a5-9157-84ed58d80d24-ovnkube-script-lib\") pod \"ovnkube-node-zbjsk\" (UID: \"8750bcd5-12c7-45a5-9157-84ed58d80d24\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbjsk"
Sep 10 22:01:21.619227 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.618780    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3cf12f53-0edf-43cc-913f-50b1669152b2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dt25t\" (UID: \"3cf12f53-0edf-43cc-913f-50b1669152b2\") " pod="openshift-multus/multus-additional-cni-plugins-dt25t"
Sep 10 22:01:21.619227 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.618784    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8750bcd5-12c7-45a5-9157-84ed58d80d24-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zbjsk\" (UID: \"8750bcd5-12c7-45a5-9157-84ed58d80d24\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbjsk"
Sep 10 22:01:21.619227 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.618809    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tv6xk\" (UniqueName: \"kubernetes.io/projected/1731f853-3630-4b20-b78c-5243c08f20e4-kube-api-access-tv6xk\") pod \"multus-8rwdz\" (UID: \"1731f853-3630-4b20-b78c-5243c08f20e4\") " pod="openshift-multus/multus-8rwdz"
Sep 10 22:01:21.619227 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.618788    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8750bcd5-12c7-45a5-9157-84ed58d80d24-var-lib-openvswitch\") pod \"ovnkube-node-zbjsk\" (UID: \"8750bcd5-12c7-45a5-9157-84ed58d80d24\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbjsk"
Sep 10 22:01:21.619227 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.618834    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8750bcd5-12c7-45a5-9157-84ed58d80d24-host-run-netns\") pod \"ovnkube-node-zbjsk\" (UID: \"8750bcd5-12c7-45a5-9157-84ed58d80d24\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbjsk"
Sep 10 22:01:21.619227 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.618878    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8750bcd5-12c7-45a5-9157-84ed58d80d24-host-run-ovn-kubernetes\") pod \"ovnkube-node-zbjsk\" (UID: \"8750bcd5-12c7-45a5-9157-84ed58d80d24\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbjsk"
Sep 10 22:01:21.619227 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.618893    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8750bcd5-12c7-45a5-9157-84ed58d80d24-node-log\") pod \"ovnkube-node-zbjsk\" (UID: \"8750bcd5-12c7-45a5-9157-84ed58d80d24\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbjsk"
Sep 10 22:01:21.619227 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.618897    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1731f853-3630-4b20-b78c-5243c08f20e4-system-cni-dir\") pod \"multus-8rwdz\" (UID: \"1731f853-3630-4b20-b78c-5243c08f20e4\") " pod="openshift-multus/multus-8rwdz"
Sep 10 22:01:21.619227 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.618839    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8750bcd5-12c7-45a5-9157-84ed58d80d24-node-log\") pod \"ovnkube-node-zbjsk\" (UID: \"8750bcd5-12c7-45a5-9157-84ed58d80d24\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbjsk"
Sep 10 22:01:21.619227 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.618933    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1731f853-3630-4b20-b78c-5243c08f20e4-host-run-k8s-cni-cncf-io\") pod \"multus-8rwdz\" (UID: \"1731f853-3630-4b20-b78c-5243c08f20e4\") " pod="openshift-multus/multus-8rwdz"
Sep 10 22:01:21.620129 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.618938    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1731f853-3630-4b20-b78c-5243c08f20e4-etc-kubernetes\") pod \"multus-8rwdz\" (UID: \"1731f853-3630-4b20-b78c-5243c08f20e4\") " pod="openshift-multus/multus-8rwdz"
Sep 10 22:01:21.620129 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.618945    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3cf12f53-0edf-43cc-913f-50b1669152b2-cni-binary-copy\") pod \"multus-additional-cni-plugins-dt25t\" (UID: \"3cf12f53-0edf-43cc-913f-50b1669152b2\") " pod="openshift-multus/multus-additional-cni-plugins-dt25t"
Sep 10 22:01:21.620129 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.618934    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8750bcd5-12c7-45a5-9157-84ed58d80d24-env-overrides\") pod \"ovnkube-node-zbjsk\" (UID: \"8750bcd5-12c7-45a5-9157-84ed58d80d24\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbjsk"
Sep 10 22:01:21.620129 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.618938    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3cf12f53-0edf-43cc-913f-50b1669152b2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dt25t\" (UID: \"3cf12f53-0edf-43cc-913f-50b1669152b2\") " pod="openshift-multus/multus-additional-cni-plugins-dt25t"
Sep 10 22:01:21.620129 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.618985    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1731f853-3630-4b20-b78c-5243c08f20e4-hostroot\") pod \"multus-8rwdz\" (UID: \"1731f853-3630-4b20-b78c-5243c08f20e4\") " pod="openshift-multus/multus-8rwdz"
Sep 10 22:01:21.620129 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.618992    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8750bcd5-12c7-45a5-9157-84ed58d80d24-ovnkube-config\") pod \"ovnkube-node-zbjsk\" (UID: \"8750bcd5-12c7-45a5-9157-84ed58d80d24\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbjsk"
Sep 10 22:01:21.620129 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.619040    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1731f853-3630-4b20-b78c-5243c08f20e4-host-var-lib-kubelet\") pod \"multus-8rwdz\" (UID: \"1731f853-3630-4b20-b78c-5243c08f20e4\") " pod="openshift-multus/multus-8rwdz"
Sep 10 22:01:21.620129 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.619069    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8750bcd5-12c7-45a5-9157-84ed58d80d24-run-openvswitch\") pod \"ovnkube-node-zbjsk\" (UID: \"8750bcd5-12c7-45a5-9157-84ed58d80d24\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbjsk"
Sep 10 22:01:21.620129 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.619093    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8750bcd5-12c7-45a5-9157-84ed58d80d24-run-ovn\") pod \"ovnkube-node-zbjsk\" (UID: \"8750bcd5-12c7-45a5-9157-84ed58d80d24\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbjsk"
Sep 10 22:01:21.620129 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.619115    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f0c8381d-52b6-4511-90be-59aebfef283a-agent-certs\") pod \"konnectivity-agent-cjdvg\" (UID: \"f0c8381d-52b6-4511-90be-59aebfef283a\") " pod="kube-system/konnectivity-agent-cjdvg"
Sep 10 22:01:21.620129 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.619145    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8750bcd5-12c7-45a5-9157-84ed58d80d24-run-openvswitch\") pod \"ovnkube-node-zbjsk\" (UID: \"8750bcd5-12c7-45a5-9157-84ed58d80d24\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbjsk"
Sep 10 22:01:21.620129 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.619154    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1731f853-3630-4b20-b78c-5243c08f20e4-multus-socket-dir-parent\") pod \"multus-8rwdz\" (UID: \"1731f853-3630-4b20-b78c-5243c08f20e4\") " pod="openshift-multus/multus-8rwdz"
Sep 10 22:01:21.620129 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.619186    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1731f853-3630-4b20-b78c-5243c08f20e4-host-var-lib-kubelet\") pod \"multus-8rwdz\" (UID: \"1731f853-3630-4b20-b78c-5243c08f20e4\") " pod="openshift-multus/multus-8rwdz"
Sep 10 22:01:21.620129 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.619189    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1731f853-3630-4b20-b78c-5243c08f20e4-multus-daemon-config\") pod \"multus-8rwdz\" (UID: \"1731f853-3630-4b20-b78c-5243c08f20e4\") " pod="openshift-multus/multus-8rwdz"
Sep 10 22:01:21.620129 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.619221    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8750bcd5-12c7-45a5-9157-84ed58d80d24-host-kubelet\") pod \"ovnkube-node-zbjsk\" (UID: \"8750bcd5-12c7-45a5-9157-84ed58d80d24\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbjsk"
Sep 10 22:01:21.620129 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.619244    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8750bcd5-12c7-45a5-9157-84ed58d80d24-etc-openvswitch\") pod \"ovnkube-node-zbjsk\" (UID: \"8750bcd5-12c7-45a5-9157-84ed58d80d24\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbjsk"
Sep 10 22:01:21.620129 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.619252    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f0c8381d-52b6-4511-90be-59aebfef283a-konnectivity-ca\") pod \"konnectivity-agent-cjdvg\" (UID: \"f0c8381d-52b6-4511-90be-59aebfef283a\") " pod="kube-system/konnectivity-agent-cjdvg"
Sep 10 22:01:21.620129 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.619268    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8750bcd5-12c7-45a5-9157-84ed58d80d24-ovn-node-metrics-cert\") pod \"ovnkube-node-zbjsk\" (UID: \"8750bcd5-12c7-45a5-9157-84ed58d80d24\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbjsk"
Sep 10 22:01:21.620974 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.619300    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1731f853-3630-4b20-b78c-5243c08f20e4-multus-cni-dir\") pod \"multus-8rwdz\" (UID: \"1731f853-3630-4b20-b78c-5243c08f20e4\") " pod="openshift-multus/multus-8rwdz"
Sep 10 22:01:21.620974 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.619324    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrgtp\" (UniqueName: \"kubernetes.io/projected/8750bcd5-12c7-45a5-9157-84ed58d80d24-kube-api-access-nrgtp\") pod \"ovnkube-node-zbjsk\" (UID: \"8750bcd5-12c7-45a5-9157-84ed58d80d24\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbjsk"
Sep 10 22:01:21.620974 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.619351    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8750bcd5-12c7-45a5-9157-84ed58d80d24-host-slash\") pod \"ovnkube-node-zbjsk\" (UID: \"8750bcd5-12c7-45a5-9157-84ed58d80d24\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbjsk"
Sep 10 22:01:21.620974 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.619375    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8750bcd5-12c7-45a5-9157-84ed58d80d24-host-cni-netd\") pod \"ovnkube-node-zbjsk\" (UID: \"8750bcd5-12c7-45a5-9157-84ed58d80d24\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbjsk"
Sep 10 22:01:21.620974 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.619399    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3cf12f53-0edf-43cc-913f-50b1669152b2-system-cni-dir\") pod \"multus-additional-cni-plugins-dt25t\" (UID: \"3cf12f53-0edf-43cc-913f-50b1669152b2\") " pod="openshift-multus/multus-additional-cni-plugins-dt25t"
Sep 10 22:01:21.620974 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.619424    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8750bcd5-12c7-45a5-9157-84ed58d80d24-log-socket\") pod \"ovnkube-node-zbjsk\" (UID: \"8750bcd5-12c7-45a5-9157-84ed58d80d24\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbjsk"
Sep 10 22:01:21.620974 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.619451    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1731f853-3630-4b20-b78c-5243c08f20e4-cnibin\") pod \"multus-8rwdz\" (UID: \"1731f853-3630-4b20-b78c-5243c08f20e4\") " pod="openshift-multus/multus-8rwdz"
Sep 10 22:01:21.620974 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.619474    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1731f853-3630-4b20-b78c-5243c08f20e4-cni-binary-copy\") pod \"multus-8rwdz\" (UID: \"1731f853-3630-4b20-b78c-5243c08f20e4\") " pod="openshift-multus/multus-8rwdz"
Sep 10 22:01:21.620974 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.619479    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3cf12f53-0edf-43cc-913f-50b1669152b2-cni-binary-copy\") pod \"multus-additional-cni-plugins-dt25t\" (UID: \"3cf12f53-0edf-43cc-913f-50b1669152b2\") " pod="openshift-multus/multus-additional-cni-plugins-dt25t"
Sep 10 22:01:21.620974 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.619487    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8750bcd5-12c7-45a5-9157-84ed58d80d24-ovnkube-script-lib\") pod \"ovnkube-node-zbjsk\" (UID: \"8750bcd5-12c7-45a5-9157-84ed58d80d24\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbjsk"
Sep 10 22:01:21.620974 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.619500    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8750bcd5-12c7-45a5-9157-84ed58d80d24-run-systemd\") pod \"ovnkube-node-zbjsk\" (UID: \"8750bcd5-12c7-45a5-9157-84ed58d80d24\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbjsk"
Sep 10 22:01:21.620974 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.619511    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3cf12f53-0edf-43cc-913f-50b1669152b2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dt25t\" (UID: \"3cf12f53-0edf-43cc-913f-50b1669152b2\") " pod="openshift-multus/multus-additional-cni-plugins-dt25t"
Sep 10 22:01:21.620974 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.619540    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8750bcd5-12c7-45a5-9157-84ed58d80d24-run-ovn\") pod \"ovnkube-node-zbjsk\" (UID: \"8750bcd5-12c7-45a5-9157-84ed58d80d24\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbjsk"
Sep 10 22:01:21.620974 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.619547    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1731f853-3630-4b20-b78c-5243c08f20e4-multus-socket-dir-parent\") pod \"multus-8rwdz\" (UID: \"1731f853-3630-4b20-b78c-5243c08f20e4\") " pod="openshift-multus/multus-8rwdz"
Sep 10 22:01:21.620974 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.619551    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8750bcd5-12c7-45a5-9157-84ed58d80d24-host-cni-bin\") pod \"ovnkube-node-zbjsk\" (UID: \"8750bcd5-12c7-45a5-9157-84ed58d80d24\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbjsk"
Sep 10 22:01:21.620974 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.619582    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8750bcd5-12c7-45a5-9157-84ed58d80d24-etc-openvswitch\") pod \"ovnkube-node-zbjsk\" (UID: \"8750bcd5-12c7-45a5-9157-84ed58d80d24\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbjsk"
Sep 10 22:01:21.620974 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.619584    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3cf12f53-0edf-43cc-913f-50b1669152b2-cnibin\") pod \"multus-additional-cni-plugins-dt25t\" (UID: \"3cf12f53-0edf-43cc-913f-50b1669152b2\") " pod="openshift-multus/multus-additional-cni-plugins-dt25t"
Sep 10 22:01:21.621727 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.619594    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8750bcd5-12c7-45a5-9157-84ed58d80d24-host-cni-bin\") pod \"ovnkube-node-zbjsk\" (UID: \"8750bcd5-12c7-45a5-9157-84ed58d80d24\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbjsk"
Sep 10 22:01:21.621727 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.619624    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1731f853-3630-4b20-b78c-5243c08f20e4-multus-conf-dir\") pod \"multus-8rwdz\" (UID: \"1731f853-3630-4b20-b78c-5243c08f20e4\") " pod="openshift-multus/multus-8rwdz"
Sep 10 22:01:21.621727 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.619634    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3cf12f53-0edf-43cc-913f-50b1669152b2-cnibin\") pod \"multus-additional-cni-plugins-dt25t\" (UID: \"3cf12f53-0edf-43cc-913f-50b1669152b2\") " pod="openshift-multus/multus-additional-cni-plugins-dt25t"
Sep 10 22:01:21.621727 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.619643    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8750bcd5-12c7-45a5-9157-84ed58d80d24-log-socket\") pod \"ovnkube-node-zbjsk\" (UID: \"8750bcd5-12c7-45a5-9157-84ed58d80d24\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbjsk"
Sep 10 22:01:21.621727 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.619589    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8750bcd5-12c7-45a5-9157-84ed58d80d24-host-kubelet\") pod \"ovnkube-node-zbjsk\" (UID: \"8750bcd5-12c7-45a5-9157-84ed58d80d24\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbjsk"
Sep 10 22:01:21.621727 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.619683    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1731f853-3630-4b20-b78c-5243c08f20e4-multus-cni-dir\") pod \"multus-8rwdz\" (UID: \"1731f853-3630-4b20-b78c-5243c08f20e4\") " pod="openshift-multus/multus-8rwdz"
Sep 10 22:01:21.621727 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.619702    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1731f853-3630-4b20-b78c-5243c08f20e4-cnibin\") pod \"multus-8rwdz\" (UID: \"1731f853-3630-4b20-b78c-5243c08f20e4\") " pod="openshift-multus/multus-8rwdz"
Sep 10 22:01:21.621727 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.619635    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3cf12f53-0edf-43cc-913f-50b1669152b2-system-cni-dir\") pod \"multus-additional-cni-plugins-dt25t\" (UID: \"3cf12f53-0edf-43cc-913f-50b1669152b2\") " pod="openshift-multus/multus-additional-cni-plugins-dt25t"
Sep 10 22:01:21.621727 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.619707    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8750bcd5-12c7-45a5-9157-84ed58d80d24-run-systemd\") pod \"ovnkube-node-zbjsk\" (UID: \"8750bcd5-12c7-45a5-9157-84ed58d80d24\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbjsk"
Sep 10 22:01:21.621727 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.619724    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1731f853-3630-4b20-b78c-5243c08f20e4-multus-daemon-config\") pod \"multus-8rwdz\" (UID: \"1731f853-3630-4b20-b78c-5243c08f20e4\") " pod="openshift-multus/multus-8rwdz"
Sep 10 22:01:21.621727 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.619735    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8750bcd5-12c7-45a5-9157-84ed58d80d24-host-slash\") pod \"ovnkube-node-zbjsk\" (UID: \"8750bcd5-12c7-45a5-9157-84ed58d80d24\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbjsk"
Sep 10 22:01:21.621727 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.619700    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1731f853-3630-4b20-b78c-5243c08f20e4-multus-conf-dir\") pod \"multus-8rwdz\" (UID: \"1731f853-3630-4b20-b78c-5243c08f20e4\") " pod="openshift-multus/multus-8rwdz"
Sep 10 22:01:21.621727 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.619766    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8750bcd5-12c7-45a5-9157-84ed58d80d24-host-cni-netd\") pod \"ovnkube-node-zbjsk\" (UID: \"8750bcd5-12c7-45a5-9157-84ed58d80d24\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbjsk"
Sep 10 22:01:21.621727 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.620171    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1731f853-3630-4b20-b78c-5243c08f20e4-cni-binary-copy\") pod \"multus-8rwdz\" (UID: \"1731f853-3630-4b20-b78c-5243c08f20e4\") " pod="openshift-multus/multus-8rwdz"
Sep 10 22:01:21.622223 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.621927    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f0c8381d-52b6-4511-90be-59aebfef283a-agent-certs\") pod \"konnectivity-agent-cjdvg\" (UID: \"f0c8381d-52b6-4511-90be-59aebfef283a\") " pod="kube-system/konnectivity-agent-cjdvg"
Sep 10 22:01:21.622365 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.622341    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8750bcd5-12c7-45a5-9157-84ed58d80d24-ovn-node-metrics-cert\") pod \"ovnkube-node-zbjsk\" (UID: \"8750bcd5-12c7-45a5-9157-84ed58d80d24\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbjsk"
Sep 10 22:01:21.625889 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.625869    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tv6xk\" (UniqueName: \"kubernetes.io/projected/1731f853-3630-4b20-b78c-5243c08f20e4-kube-api-access-tv6xk\") pod \"multus-8rwdz\" (UID: \"1731f853-3630-4b20-b78c-5243c08f20e4\") " pod="openshift-multus/multus-8rwdz"
Sep 10 22:01:21.626412 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.626390    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqsbj\" (UniqueName: \"kubernetes.io/projected/3cf12f53-0edf-43cc-913f-50b1669152b2-kube-api-access-fqsbj\") pod \"multus-additional-cni-plugins-dt25t\" (UID: \"3cf12f53-0edf-43cc-913f-50b1669152b2\") " pod="openshift-multus/multus-additional-cni-plugins-dt25t"
Sep 10 22:01:21.627208 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.627191    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrgtp\" (UniqueName: \"kubernetes.io/projected/8750bcd5-12c7-45a5-9157-84ed58d80d24-kube-api-access-nrgtp\") pod \"ovnkube-node-zbjsk\" (UID: \"8750bcd5-12c7-45a5-9157-84ed58d80d24\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbjsk"
Sep 10 22:01:21.671276 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.671242    2582 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 22:01:21.693976 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.693932    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-74bb6"
Sep 10 22:01:21.703326 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.703300    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-p8f5t"
Sep 10 22:01:21.713988 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.713964    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-dth7z"
Sep 10 22:01:21.721746 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.721716    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-ncrzz"
Sep 10 22:01:21.729392 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.729354    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x4xzs"
Sep 10 22:01:21.737044 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.737018    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-8rwdz"
Sep 10 22:01:21.744860 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.744831    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-zbjsk"
Sep 10 22:01:21.753567 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.753548    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-dt25t"
Sep 10 22:01:21.760250 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:21.760223    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-cjdvg"
Sep 10 22:01:22.022503 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:22.022418    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0d6f601b-532d-4f1f-9580-977997448ea9-original-pull-secret\") pod \"global-pull-secret-syncer-n9r9x\" (UID: \"0d6f601b-532d-4f1f-9580-977997448ea9\") " pod="kube-system/global-pull-secret-syncer-n9r9x"
Sep 10 22:01:22.022503 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:22.022490    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6c7caf6-cae8-4a81-98c5-e4db1671eff5-metrics-certs\") pod \"network-metrics-daemon-xw7pd\" (UID: \"a6c7caf6-cae8-4a81-98c5-e4db1671eff5\") " pod="openshift-multus/network-metrics-daemon-xw7pd"
Sep 10 22:01:22.022731 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:01:22.022597    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 22:01:22.022731 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:01:22.022680    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d6f601b-532d-4f1f-9580-977997448ea9-original-pull-secret podName:0d6f601b-532d-4f1f-9580-977997448ea9 nodeName:}" failed. No retries permitted until 2026-09-10 22:01:23.022660262 +0000 UTC m=+3.978727969 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0d6f601b-532d-4f1f-9580-977997448ea9-original-pull-secret") pod "global-pull-secret-syncer-n9r9x" (UID: "0d6f601b-532d-4f1f-9580-977997448ea9") : object "kube-system"/"original-pull-secret" not registered
Sep 10 22:01:22.022731 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:01:22.022598    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 22:01:22.022893 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:01:22.022761    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6c7caf6-cae8-4a81-98c5-e4db1671eff5-metrics-certs podName:a6c7caf6-cae8-4a81-98c5-e4db1671eff5 nodeName:}" failed. No retries permitted until 2026-09-10 22:01:23.02274372 +0000 UTC m=+3.978811420 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a6c7caf6-cae8-4a81-98c5-e4db1671eff5-metrics-certs") pod "network-metrics-daemon-xw7pd" (UID: "a6c7caf6-cae8-4a81-98c5-e4db1671eff5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 22:01:22.123602 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:22.123566    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwfrj\" (UniqueName: \"kubernetes.io/projected/064a91af-7e3d-4e71-b743-04d68d63174e-kube-api-access-zwfrj\") pod \"network-check-target-rrm25\" (UID: \"064a91af-7e3d-4e71-b743-04d68d63174e\") " pod="openshift-network-diagnostics/network-check-target-rrm25"
Sep 10 22:01:22.123776 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:01:22.123740    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 22:01:22.123776 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:01:22.123765    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 22:01:22.123911 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:01:22.123779    2582 projected.go:196] Error preparing data for projected volume kube-api-access-zwfrj for pod openshift-network-diagnostics/network-check-target-rrm25: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 22:01:22.123911 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:01:22.123843    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/064a91af-7e3d-4e71-b743-04d68d63174e-kube-api-access-zwfrj podName:064a91af-7e3d-4e71-b743-04d68d63174e nodeName:}" failed. No retries permitted until 2026-09-10 22:01:23.123825837 +0000 UTC m=+4.079893543 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-zwfrj" (UniqueName: "kubernetes.io/projected/064a91af-7e3d-4e71-b743-04d68d63174e-kube-api-access-zwfrj") pod "network-check-target-rrm25" (UID: "064a91af-7e3d-4e71-b743-04d68d63174e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 22:01:22.444245 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:22.444153    2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-09 21:56:20 +0000 UTC" deadline="2028-06-23 11:10:05.173661734 +0000 UTC"
Sep 10 22:01:22.444245 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:22.444203    2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15637h8m42.729461135s"
Sep 10 22:01:22.492525 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:22.492504    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rrm25"
Sep 10 22:01:22.492667 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:01:22.492603    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rrm25" podUID="064a91af-7e3d-4e71-b743-04d68d63174e"
Sep 10 22:01:22.515181 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:22.515150    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1731f853_3630_4b20_b78c_5243c08f20e4.slice/crio-0efaac0800f4da2c88f32eb1cc65b1959fbdec127f0d05229a48c5fe84506924 WatchSource:0}: Error finding container 0efaac0800f4da2c88f32eb1cc65b1959fbdec127f0d05229a48c5fe84506924: Status 404 returned error can't find the container with id 0efaac0800f4da2c88f32eb1cc65b1959fbdec127f0d05229a48c5fe84506924
Sep 10 22:01:22.516636 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:22.516615    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16caf96d_d15f_45e5_b003_1aec5786ea58.slice/crio-c1d976b72b82d414a496ae44791141f6e34d2e533a5ae3342962a01d53d9a049 WatchSource:0}: Error finding container c1d976b72b82d414a496ae44791141f6e34d2e533a5ae3342962a01d53d9a049: Status 404 returned error can't find the container with id c1d976b72b82d414a496ae44791141f6e34d2e533a5ae3342962a01d53d9a049
Sep 10 22:01:22.518019 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:22.517995    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0c8381d_52b6_4511_90be_59aebfef283a.slice/crio-acd49f2254ef245a3504f8b7ab1e57e84f06189a48ffbcbcf7acf02531acceb0 WatchSource:0}: Error finding container acd49f2254ef245a3504f8b7ab1e57e84f06189a48ffbcbcf7acf02531acceb0: Status 404 returned error can't find the container with id acd49f2254ef245a3504f8b7ab1e57e84f06189a48ffbcbcf7acf02531acceb0
Sep 10 22:01:22.521560 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:22.521541    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe63ec98_ecdb_4555_8799_a1f3e9558da3.slice/crio-b13c09d82a3b442235338a4760b1a17752794c7a1333bde49ee44a36c7c1cc6a WatchSource:0}: Error finding container b13c09d82a3b442235338a4760b1a17752794c7a1333bde49ee44a36c7c1cc6a: Status 404 returned error can't find the container with id b13c09d82a3b442235338a4760b1a17752794c7a1333bde49ee44a36c7c1cc6a
Sep 10 22:01:22.523101 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:22.523068    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb73372d8_bb63_4e98_82e3_05c929fad192.slice/crio-bd179a7e7a57b61c2441f0520b2bd15e73731ef0c4589102fdfc77439f88c0b0 WatchSource:0}: Error finding container bd179a7e7a57b61c2441f0520b2bd15e73731ef0c4589102fdfc77439f88c0b0: Status 404 returned error can't find the container with id bd179a7e7a57b61c2441f0520b2bd15e73731ef0c4589102fdfc77439f88c0b0
Sep 10 22:01:22.524176 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:22.523794    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8750bcd5_12c7_45a5_9157_84ed58d80d24.slice/crio-d302c2bb151f5b51b2b5afdcf9045a54c2bbea4d2e53782852a816e1ac3d4bcf WatchSource:0}: Error finding container d302c2bb151f5b51b2b5afdcf9045a54c2bbea4d2e53782852a816e1ac3d4bcf: Status 404 returned error can't find the container with id d302c2bb151f5b51b2b5afdcf9045a54c2bbea4d2e53782852a816e1ac3d4bcf
Sep 10 22:01:22.525468 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:22.525329    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cf12f53_0edf_43cc_913f_50b1669152b2.slice/crio-56e06a6a8f1a1461cd716cb47355f4c04dac616f0c969ebd8c6b41ef5e1a62e0 WatchSource:0}: Error finding container 56e06a6a8f1a1461cd716cb47355f4c04dac616f0c969ebd8c6b41ef5e1a62e0: Status 404 returned error can't find the container with id 56e06a6a8f1a1461cd716cb47355f4c04dac616f0c969ebd8c6b41ef5e1a62e0
Sep 10 22:01:22.525736 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:22.525691    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf430e7fc_0f5e_48e2_9ea6_f97c277cdb06.slice/crio-773dea586819766eed8b315b779967e297fdaac08932caeab063e3cf14b7073f WatchSource:0}: Error finding container 773dea586819766eed8b315b779967e297fdaac08932caeab063e3cf14b7073f: Status 404 returned error can't find the container with id 773dea586819766eed8b315b779967e297fdaac08932caeab063e3cf14b7073f
Sep 10 22:01:22.527127 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:22.527042    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7821d1b4_78c1_4500_84e4_1b1fc03ad836.slice/crio-0c5b0468880d0fc365cca29c58c9ab8c5b7bd29480bac7cb2029d30d7f197b9d WatchSource:0}: Error finding container 0c5b0468880d0fc365cca29c58c9ab8c5b7bd29480bac7cb2029d30d7f197b9d: Status 404 returned error can't find the container with id 0c5b0468880d0fc365cca29c58c9ab8c5b7bd29480bac7cb2029d30d7f197b9d
Sep 10 22:01:23.029622 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:23.029539    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0d6f601b-532d-4f1f-9580-977997448ea9-original-pull-secret\") pod \"global-pull-secret-syncer-n9r9x\" (UID: \"0d6f601b-532d-4f1f-9580-977997448ea9\") " pod="kube-system/global-pull-secret-syncer-n9r9x"
Sep 10 22:01:23.029622 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:23.029617    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6c7caf6-cae8-4a81-98c5-e4db1671eff5-metrics-certs\") pod \"network-metrics-daemon-xw7pd\" (UID: \"a6c7caf6-cae8-4a81-98c5-e4db1671eff5\") " pod="openshift-multus/network-metrics-daemon-xw7pd"
Sep 10 22:01:23.029830 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:01:23.029713    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 22:01:23.029830 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:01:23.029776    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6c7caf6-cae8-4a81-98c5-e4db1671eff5-metrics-certs podName:a6c7caf6-cae8-4a81-98c5-e4db1671eff5 nodeName:}" failed. No retries permitted until 2026-09-10 22:01:25.029756846 +0000 UTC m=+5.985824529 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a6c7caf6-cae8-4a81-98c5-e4db1671eff5-metrics-certs") pod "network-metrics-daemon-xw7pd" (UID: "a6c7caf6-cae8-4a81-98c5-e4db1671eff5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 22:01:23.029830 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:01:23.029713    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 22:01:23.030046 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:01:23.029838    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d6f601b-532d-4f1f-9580-977997448ea9-original-pull-secret podName:0d6f601b-532d-4f1f-9580-977997448ea9 nodeName:}" failed. No retries permitted until 2026-09-10 22:01:25.029825315 +0000 UTC m=+5.985893002 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0d6f601b-532d-4f1f-9580-977997448ea9-original-pull-secret") pod "global-pull-secret-syncer-n9r9x" (UID: "0d6f601b-532d-4f1f-9580-977997448ea9") : object "kube-system"/"original-pull-secret" not registered
Sep 10 22:01:23.130892 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:23.130690    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwfrj\" (UniqueName: \"kubernetes.io/projected/064a91af-7e3d-4e71-b743-04d68d63174e-kube-api-access-zwfrj\") pod \"network-check-target-rrm25\" (UID: \"064a91af-7e3d-4e71-b743-04d68d63174e\") " pod="openshift-network-diagnostics/network-check-target-rrm25"
Sep 10 22:01:23.130892 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:01:23.130892    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 22:01:23.131112 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:01:23.130915    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 22:01:23.131112 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:01:23.130927    2582 projected.go:196] Error preparing data for projected volume kube-api-access-zwfrj for pod openshift-network-diagnostics/network-check-target-rrm25: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 22:01:23.131112 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:01:23.130987    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/064a91af-7e3d-4e71-b743-04d68d63174e-kube-api-access-zwfrj podName:064a91af-7e3d-4e71-b743-04d68d63174e nodeName:}" failed. No retries permitted until 2026-09-10 22:01:25.130968696 +0000 UTC m=+6.087036384 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-zwfrj" (UniqueName: "kubernetes.io/projected/064a91af-7e3d-4e71-b743-04d68d63174e-kube-api-access-zwfrj") pod "network-check-target-rrm25" (UID: "064a91af-7e3d-4e71-b743-04d68d63174e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 22:01:23.503525 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:23.503450    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xw7pd"
Sep 10 22:01:23.504355 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:01:23.503583    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xw7pd" podUID="a6c7caf6-cae8-4a81-98c5-e4db1671eff5"
Sep 10 22:01:23.504355 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:23.503971    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n9r9x"
Sep 10 22:01:23.504355 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:01:23.504049    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n9r9x" podUID="0d6f601b-532d-4f1f-9580-977997448ea9"
Sep 10 22:01:23.520183 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:23.520151    2582 generic.go:356] "Generic (PLEG): container finished" podID="6ea49b038141fb7080f5d505e03c9c21" containerID="5d3d4d6e8218aa361e6a3c8230f6073576d313997f60592c99a5f14fef9015d4" exitCode=0
Sep 10 22:01:23.520329 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:23.520238    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-144.ec2.internal" event={"ID":"6ea49b038141fb7080f5d505e03c9c21","Type":"ContainerDied","Data":"5d3d4d6e8218aa361e6a3c8230f6073576d313997f60592c99a5f14fef9015d4"}
Sep 10 22:01:23.531951 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:23.531921    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-144.ec2.internal" event={"ID":"f31b7187b87e2b509b6e74131f844880","Type":"ContainerStarted","Data":"14bf71871d95f5472a53040e2379fefcffa4f16dcfa2920121e849dced282774"}
Sep 10 22:01:23.541118 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:23.541090    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dt25t" event={"ID":"3cf12f53-0edf-43cc-913f-50b1669152b2","Type":"ContainerStarted","Data":"56e06a6a8f1a1461cd716cb47355f4c04dac616f0c969ebd8c6b41ef5e1a62e0"}
Sep 10 22:01:23.549588 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:23.549539    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-144.ec2.internal" podStartSLOduration=3.549521429 podStartE2EDuration="3.549521429s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 22:01:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:01:23.548916686 +0000 UTC m=+4.504984394" watchObservedRunningTime="2026-09-10 22:01:23.549521429 +0000 UTC m=+4.505589136"
Sep 10 22:01:23.553024 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:23.552971    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-dth7z" event={"ID":"b73372d8-bb63-4e98-82e3-05c929fad192","Type":"ContainerStarted","Data":"bd179a7e7a57b61c2441f0520b2bd15e73731ef0c4589102fdfc77439f88c0b0"}
Sep 10 22:01:23.570290 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:23.570071    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x4xzs" event={"ID":"fe63ec98-ecdb-4555-8799-a1f3e9558da3","Type":"ContainerStarted","Data":"b13c09d82a3b442235338a4760b1a17752794c7a1333bde49ee44a36c7c1cc6a"}
Sep 10 22:01:23.573977 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:23.573954    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-ncrzz" event={"ID":"16caf96d-d15f-45e5-b003-1aec5786ea58","Type":"ContainerStarted","Data":"c1d976b72b82d414a496ae44791141f6e34d2e533a5ae3342962a01d53d9a049"}
Sep 10 22:01:23.589580 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:23.589550    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-8rwdz" event={"ID":"1731f853-3630-4b20-b78c-5243c08f20e4","Type":"ContainerStarted","Data":"0efaac0800f4da2c88f32eb1cc65b1959fbdec127f0d05229a48c5fe84506924"}
Sep 10 22:01:23.605398 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:23.605369    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-p8f5t" event={"ID":"7821d1b4-78c1-4500-84e4-1b1fc03ad836","Type":"ContainerStarted","Data":"0c5b0468880d0fc365cca29c58c9ab8c5b7bd29480bac7cb2029d30d7f197b9d"}
Sep 10 22:01:23.617049 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:23.616982    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-74bb6" event={"ID":"f430e7fc-0f5e-48e2-9ea6-f97c277cdb06","Type":"ContainerStarted","Data":"773dea586819766eed8b315b779967e297fdaac08932caeab063e3cf14b7073f"}
Sep 10 22:01:23.630573 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:23.630542    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zbjsk" event={"ID":"8750bcd5-12c7-45a5-9157-84ed58d80d24","Type":"ContainerStarted","Data":"d302c2bb151f5b51b2b5afdcf9045a54c2bbea4d2e53782852a816e1ac3d4bcf"}
Sep 10 22:01:23.637214 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:23.637187    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-cjdvg" event={"ID":"f0c8381d-52b6-4511-90be-59aebfef283a","Type":"ContainerStarted","Data":"acd49f2254ef245a3504f8b7ab1e57e84f06189a48ffbcbcf7acf02531acceb0"}
Sep 10 22:01:24.492662 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:24.492569    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rrm25"
Sep 10 22:01:24.492831 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:01:24.492711    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rrm25" podUID="064a91af-7e3d-4e71-b743-04d68d63174e"
Sep 10 22:01:24.655868 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:24.655184    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-144.ec2.internal" event={"ID":"6ea49b038141fb7080f5d505e03c9c21","Type":"ContainerStarted","Data":"1b7f77ba8e450e5d3a5a1979ceee8c26f92735b05a6e9011ef4d8a94db256638"}
Sep 10 22:01:24.669482 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:24.669413    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-144.ec2.internal" podStartSLOduration=4.669389679 podStartE2EDuration="4.669389679s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 22:01:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:01:24.668376404 +0000 UTC m=+5.624444111" watchObservedRunningTime="2026-09-10 22:01:24.669389679 +0000 UTC m=+5.625457386"
Sep 10 22:01:25.047496 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:25.047458    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6c7caf6-cae8-4a81-98c5-e4db1671eff5-metrics-certs\") pod \"network-metrics-daemon-xw7pd\" (UID: \"a6c7caf6-cae8-4a81-98c5-e4db1671eff5\") " pod="openshift-multus/network-metrics-daemon-xw7pd"
Sep 10 22:01:25.047674 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:25.047545    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0d6f601b-532d-4f1f-9580-977997448ea9-original-pull-secret\") pod \"global-pull-secret-syncer-n9r9x\" (UID: \"0d6f601b-532d-4f1f-9580-977997448ea9\") " pod="kube-system/global-pull-secret-syncer-n9r9x"
Sep 10 22:01:25.047768 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:01:25.047703    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 22:01:25.047823 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:01:25.047769    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d6f601b-532d-4f1f-9580-977997448ea9-original-pull-secret podName:0d6f601b-532d-4f1f-9580-977997448ea9 nodeName:}" failed. No retries permitted until 2026-09-10 22:01:29.047748877 +0000 UTC m=+10.003816567 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0d6f601b-532d-4f1f-9580-977997448ea9-original-pull-secret") pod "global-pull-secret-syncer-n9r9x" (UID: "0d6f601b-532d-4f1f-9580-977997448ea9") : object "kube-system"/"original-pull-secret" not registered
Sep 10 22:01:25.048226 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:01:25.048200    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 22:01:25.048324 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:01:25.048263    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6c7caf6-cae8-4a81-98c5-e4db1671eff5-metrics-certs podName:a6c7caf6-cae8-4a81-98c5-e4db1671eff5 nodeName:}" failed. No retries permitted until 2026-09-10 22:01:29.048244777 +0000 UTC m=+10.004312483 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a6c7caf6-cae8-4a81-98c5-e4db1671eff5-metrics-certs") pod "network-metrics-daemon-xw7pd" (UID: "a6c7caf6-cae8-4a81-98c5-e4db1671eff5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 22:01:25.147960 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:25.147919    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwfrj\" (UniqueName: \"kubernetes.io/projected/064a91af-7e3d-4e71-b743-04d68d63174e-kube-api-access-zwfrj\") pod \"network-check-target-rrm25\" (UID: \"064a91af-7e3d-4e71-b743-04d68d63174e\") " pod="openshift-network-diagnostics/network-check-target-rrm25"
Sep 10 22:01:25.148169 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:01:25.148132    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 22:01:25.148169 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:01:25.148156    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 22:01:25.148169 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:01:25.148169    2582 projected.go:196] Error preparing data for projected volume kube-api-access-zwfrj for pod openshift-network-diagnostics/network-check-target-rrm25: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 22:01:25.148341 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:01:25.148227    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/064a91af-7e3d-4e71-b743-04d68d63174e-kube-api-access-zwfrj podName:064a91af-7e3d-4e71-b743-04d68d63174e nodeName:}" failed. No retries permitted until 2026-09-10 22:01:29.148207744 +0000 UTC m=+10.104275434 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-zwfrj" (UniqueName: "kubernetes.io/projected/064a91af-7e3d-4e71-b743-04d68d63174e-kube-api-access-zwfrj") pod "network-check-target-rrm25" (UID: "064a91af-7e3d-4e71-b743-04d68d63174e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 22:01:25.492373 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:25.492293    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xw7pd"
Sep 10 22:01:25.492523 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:01:25.492440    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xw7pd" podUID="a6c7caf6-cae8-4a81-98c5-e4db1671eff5"
Sep 10 22:01:25.492869 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:25.492829    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n9r9x"
Sep 10 22:01:25.493003 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:01:25.492960    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n9r9x" podUID="0d6f601b-532d-4f1f-9580-977997448ea9"
Sep 10 22:01:26.492670 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:26.492283    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rrm25"
Sep 10 22:01:26.492670 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:01:26.492417    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rrm25" podUID="064a91af-7e3d-4e71-b743-04d68d63174e"
Sep 10 22:01:27.496559 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:27.496487    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xw7pd"
Sep 10 22:01:27.497044 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:01:27.496622    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xw7pd" podUID="a6c7caf6-cae8-4a81-98c5-e4db1671eff5"
Sep 10 22:01:27.497044 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:27.496739    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n9r9x"
Sep 10 22:01:27.497044 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:01:27.496925    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n9r9x" podUID="0d6f601b-532d-4f1f-9580-977997448ea9"
Sep 10 22:01:28.492685 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:28.492570    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rrm25"
Sep 10 22:01:28.492886 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:01:28.492680    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rrm25" podUID="064a91af-7e3d-4e71-b743-04d68d63174e"
Sep 10 22:01:29.083382 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:29.083088    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0d6f601b-532d-4f1f-9580-977997448ea9-original-pull-secret\") pod \"global-pull-secret-syncer-n9r9x\" (UID: \"0d6f601b-532d-4f1f-9580-977997448ea9\") " pod="kube-system/global-pull-secret-syncer-n9r9x"
Sep 10 22:01:29.083382 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:29.083168    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6c7caf6-cae8-4a81-98c5-e4db1671eff5-metrics-certs\") pod \"network-metrics-daemon-xw7pd\" (UID: \"a6c7caf6-cae8-4a81-98c5-e4db1671eff5\") " pod="openshift-multus/network-metrics-daemon-xw7pd"
Sep 10 22:01:29.083382 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:01:29.083216    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 22:01:29.083382 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:01:29.083296    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d6f601b-532d-4f1f-9580-977997448ea9-original-pull-secret podName:0d6f601b-532d-4f1f-9580-977997448ea9 nodeName:}" failed. No retries permitted until 2026-09-10 22:01:37.083273687 +0000 UTC m=+18.039341386 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0d6f601b-532d-4f1f-9580-977997448ea9-original-pull-secret") pod "global-pull-secret-syncer-n9r9x" (UID: "0d6f601b-532d-4f1f-9580-977997448ea9") : object "kube-system"/"original-pull-secret" not registered
Sep 10 22:01:29.083382 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:01:29.083298    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 22:01:29.083382 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:01:29.083351    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6c7caf6-cae8-4a81-98c5-e4db1671eff5-metrics-certs podName:a6c7caf6-cae8-4a81-98c5-e4db1671eff5 nodeName:}" failed. No retries permitted until 2026-09-10 22:01:37.083332555 +0000 UTC m=+18.039400242 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a6c7caf6-cae8-4a81-98c5-e4db1671eff5-metrics-certs") pod "network-metrics-daemon-xw7pd" (UID: "a6c7caf6-cae8-4a81-98c5-e4db1671eff5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 22:01:29.184120 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:29.184081    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwfrj\" (UniqueName: \"kubernetes.io/projected/064a91af-7e3d-4e71-b743-04d68d63174e-kube-api-access-zwfrj\") pod \"network-check-target-rrm25\" (UID: \"064a91af-7e3d-4e71-b743-04d68d63174e\") " pod="openshift-network-diagnostics/network-check-target-rrm25"
Sep 10 22:01:29.184289 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:01:29.184261    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 22:01:29.184289 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:01:29.184283    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 22:01:29.184402 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:01:29.184297    2582 projected.go:196] Error preparing data for projected volume kube-api-access-zwfrj for pod openshift-network-diagnostics/network-check-target-rrm25: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 22:01:29.184402 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:01:29.184353    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/064a91af-7e3d-4e71-b743-04d68d63174e-kube-api-access-zwfrj podName:064a91af-7e3d-4e71-b743-04d68d63174e nodeName:}" failed. No retries permitted until 2026-09-10 22:01:37.184334287 +0000 UTC m=+18.140401985 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-zwfrj" (UniqueName: "kubernetes.io/projected/064a91af-7e3d-4e71-b743-04d68d63174e-kube-api-access-zwfrj") pod "network-check-target-rrm25" (UID: "064a91af-7e3d-4e71-b743-04d68d63174e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 22:01:29.495760 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:29.495693    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xw7pd"
Sep 10 22:01:29.495930 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:01:29.495825    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xw7pd" podUID="a6c7caf6-cae8-4a81-98c5-e4db1671eff5"
Sep 10 22:01:29.495930 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:29.495881    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n9r9x"
Sep 10 22:01:29.496053 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:01:29.495997    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n9r9x" podUID="0d6f601b-532d-4f1f-9580-977997448ea9"
Sep 10 22:01:30.492271 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:30.492234    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rrm25"
Sep 10 22:01:30.492729 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:01:30.492368    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rrm25" podUID="064a91af-7e3d-4e71-b743-04d68d63174e"
Sep 10 22:01:31.492905 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:31.492568    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n9r9x"
Sep 10 22:01:31.492905 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:01:31.492714    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n9r9x" podUID="0d6f601b-532d-4f1f-9580-977997448ea9"
Sep 10 22:01:31.493360 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:31.493073    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xw7pd"
Sep 10 22:01:31.493360 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:01:31.493182    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xw7pd" podUID="a6c7caf6-cae8-4a81-98c5-e4db1671eff5"
Sep 10 22:01:32.493029 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:32.492993    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rrm25"
Sep 10 22:01:32.493472 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:01:32.493124    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rrm25" podUID="064a91af-7e3d-4e71-b743-04d68d63174e"
Sep 10 22:01:33.492311 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:33.492272    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n9r9x"
Sep 10 22:01:33.492496 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:01:33.492415    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n9r9x" podUID="0d6f601b-532d-4f1f-9580-977997448ea9"
Sep 10 22:01:33.492638 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:33.492278    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xw7pd"
Sep 10 22:01:33.492792 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:01:33.492759    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xw7pd" podUID="a6c7caf6-cae8-4a81-98c5-e4db1671eff5"
Sep 10 22:01:34.492937 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:34.492904    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rrm25"
Sep 10 22:01:34.493372 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:01:34.493032    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rrm25" podUID="064a91af-7e3d-4e71-b743-04d68d63174e"
Sep 10 22:01:35.495783 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:35.495755    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xw7pd"
Sep 10 22:01:35.496256 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:01:35.495902    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xw7pd" podUID="a6c7caf6-cae8-4a81-98c5-e4db1671eff5"
Sep 10 22:01:35.496256 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:35.495951    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n9r9x"
Sep 10 22:01:35.496256 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:01:35.496056    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n9r9x" podUID="0d6f601b-532d-4f1f-9580-977997448ea9"
Sep 10 22:01:36.492791 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:36.492755    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rrm25"
Sep 10 22:01:36.492979 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:01:36.492930    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rrm25" podUID="064a91af-7e3d-4e71-b743-04d68d63174e"
Sep 10 22:01:37.142303 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:37.142253    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0d6f601b-532d-4f1f-9580-977997448ea9-original-pull-secret\") pod \"global-pull-secret-syncer-n9r9x\" (UID: \"0d6f601b-532d-4f1f-9580-977997448ea9\") " pod="kube-system/global-pull-secret-syncer-n9r9x"
Sep 10 22:01:37.142775 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:37.142337    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6c7caf6-cae8-4a81-98c5-e4db1671eff5-metrics-certs\") pod \"network-metrics-daemon-xw7pd\" (UID: \"a6c7caf6-cae8-4a81-98c5-e4db1671eff5\") " pod="openshift-multus/network-metrics-daemon-xw7pd"
Sep 10 22:01:37.142775 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:01:37.142431    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 22:01:37.142775 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:01:37.142451    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 22:01:37.142775 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:01:37.142518    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d6f601b-532d-4f1f-9580-977997448ea9-original-pull-secret podName:0d6f601b-532d-4f1f-9580-977997448ea9 nodeName:}" failed. No retries permitted until 2026-09-10 22:01:53.142495863 +0000 UTC m=+34.098563567 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0d6f601b-532d-4f1f-9580-977997448ea9-original-pull-secret") pod "global-pull-secret-syncer-n9r9x" (UID: "0d6f601b-532d-4f1f-9580-977997448ea9") : object "kube-system"/"original-pull-secret" not registered
Sep 10 22:01:37.142775 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:01:37.142542    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6c7caf6-cae8-4a81-98c5-e4db1671eff5-metrics-certs podName:a6c7caf6-cae8-4a81-98c5-e4db1671eff5 nodeName:}" failed. No retries permitted until 2026-09-10 22:01:53.142531468 +0000 UTC m=+34.098599164 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a6c7caf6-cae8-4a81-98c5-e4db1671eff5-metrics-certs") pod "network-metrics-daemon-xw7pd" (UID: "a6c7caf6-cae8-4a81-98c5-e4db1671eff5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 22:01:37.242979 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:37.242942    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwfrj\" (UniqueName: \"kubernetes.io/projected/064a91af-7e3d-4e71-b743-04d68d63174e-kube-api-access-zwfrj\") pod \"network-check-target-rrm25\" (UID: \"064a91af-7e3d-4e71-b743-04d68d63174e\") " pod="openshift-network-diagnostics/network-check-target-rrm25"
Sep 10 22:01:37.243163 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:01:37.243117    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 22:01:37.243163 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:01:37.243137    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 22:01:37.243163 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:01:37.243146    2582 projected.go:196] Error preparing data for projected volume kube-api-access-zwfrj for pod openshift-network-diagnostics/network-check-target-rrm25: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 22:01:37.243304 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:01:37.243201    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/064a91af-7e3d-4e71-b743-04d68d63174e-kube-api-access-zwfrj podName:064a91af-7e3d-4e71-b743-04d68d63174e nodeName:}" failed. No retries permitted until 2026-09-10 22:01:53.243186209 +0000 UTC m=+34.199253916 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-zwfrj" (UniqueName: "kubernetes.io/projected/064a91af-7e3d-4e71-b743-04d68d63174e-kube-api-access-zwfrj") pod "network-check-target-rrm25" (UID: "064a91af-7e3d-4e71-b743-04d68d63174e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 22:01:37.495169 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:37.495094    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xw7pd"
Sep 10 22:01:37.495320 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:01:37.495210    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xw7pd" podUID="a6c7caf6-cae8-4a81-98c5-e4db1671eff5"
Sep 10 22:01:37.495320 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:37.495269    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n9r9x"
Sep 10 22:01:37.495423 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:01:37.495395    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n9r9x" podUID="0d6f601b-532d-4f1f-9580-977997448ea9"
Sep 10 22:01:38.493063 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:38.493023    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rrm25"
Sep 10 22:01:38.493486 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:01:38.493140    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rrm25" podUID="064a91af-7e3d-4e71-b743-04d68d63174e"
Sep 10 22:01:39.492888 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:39.492800    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n9r9x"
Sep 10 22:01:39.493088 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:39.492915    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xw7pd"
Sep 10 22:01:39.493088 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:01:39.492938    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n9r9x" podUID="0d6f601b-532d-4f1f-9580-977997448ea9"
Sep 10 22:01:39.493088 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:01:39.493027    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xw7pd" podUID="a6c7caf6-cae8-4a81-98c5-e4db1671eff5"
Sep 10 22:01:40.492872 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:40.492593    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rrm25"
Sep 10 22:01:40.493047 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:01:40.492873    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rrm25" podUID="064a91af-7e3d-4e71-b743-04d68d63174e"
Sep 10 22:01:40.703450 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:40.703413    2582 generic.go:356] "Generic (PLEG): container finished" podID="3cf12f53-0edf-43cc-913f-50b1669152b2" containerID="f746ad73801a9ad80dd7eabccef7707094fe493e43b054958bfc0397d935f866" exitCode=0
Sep 10 22:01:40.704178 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:40.703504    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dt25t" event={"ID":"3cf12f53-0edf-43cc-913f-50b1669152b2","Type":"ContainerDied","Data":"f746ad73801a9ad80dd7eabccef7707094fe493e43b054958bfc0397d935f866"}
Sep 10 22:01:40.705155 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:40.705125    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-dth7z" event={"ID":"b73372d8-bb63-4e98-82e3-05c929fad192","Type":"ContainerStarted","Data":"b55e1648b45286afec6a4350625b62d5865b5a953e56a2e6180d084e2d7bf2f3"}
Sep 10 22:01:40.706805 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:40.706782    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x4xzs" event={"ID":"fe63ec98-ecdb-4555-8799-a1f3e9558da3","Type":"ContainerStarted","Data":"ff9d94e4ac14667a2fa7645d071f01f5e4e7638c8fcab00155038b03c837318f"}
Sep 10 22:01:40.708391 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:40.708364    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-ncrzz" event={"ID":"16caf96d-d15f-45e5-b003-1aec5786ea58","Type":"ContainerStarted","Data":"7088674abed49512d40c4baf51dc7d634d65359fa70f988f2ba0c39dec30e7a0"}
Sep 10 22:01:40.710072 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:40.710041    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-8rwdz" event={"ID":"1731f853-3630-4b20-b78c-5243c08f20e4","Type":"ContainerStarted","Data":"7cb91a78a98465a924820a96fc08236dacea5b02c02337df2d6beb2ada628322"}
Sep 10 22:01:40.711574 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:40.711553    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-74bb6" event={"ID":"f430e7fc-0f5e-48e2-9ea6-f97c277cdb06","Type":"ContainerStarted","Data":"81d0b5dcc507cf0566fe27c22567bce4fde96ade37e95e9df0a94fac23064311"}
Sep 10 22:01:40.714455 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:40.714435    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zbjsk" event={"ID":"8750bcd5-12c7-45a5-9157-84ed58d80d24","Type":"ContainerStarted","Data":"198a8a55db3c730e7de0391e46d44b2675497bcf1c388367713321eab90f30fa"}
Sep 10 22:01:40.714616 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:40.714490    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zbjsk" event={"ID":"8750bcd5-12c7-45a5-9157-84ed58d80d24","Type":"ContainerStarted","Data":"3e5ed15e5a5a3aaf6907a8d7094352a9f24f1111fd0f1910b78d6a1369a02efb"}
Sep 10 22:01:40.714616 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:40.714538    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zbjsk" event={"ID":"8750bcd5-12c7-45a5-9157-84ed58d80d24","Type":"ContainerStarted","Data":"8810086976c5bc9d9100604a63a25eda9d89df6263b5411af6a66a648d06ccf6"}
Sep 10 22:01:40.714616 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:40.714572    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zbjsk" event={"ID":"8750bcd5-12c7-45a5-9157-84ed58d80d24","Type":"ContainerStarted","Data":"eab89088f0b2ffff2a572e02368a66013db6287cdd3d6d8544037e06b7cdf1c9"}
Sep 10 22:01:40.714616 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:40.714606    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zbjsk" event={"ID":"8750bcd5-12c7-45a5-9157-84ed58d80d24","Type":"ContainerStarted","Data":"5f3ecadf04fea8b80c501d6de08b594ce97e736e2d406f2e122fc25e7a780837"}
Sep 10 22:01:40.716145 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:40.716125    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-cjdvg" event={"ID":"f0c8381d-52b6-4511-90be-59aebfef283a","Type":"ContainerStarted","Data":"72b375f0461aefa5d6e661cdd2f64d18d93934cd4d0e5d78f0f03bfef03cf5a6"}
Sep 10 22:01:40.748493 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:40.748449    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-74bb6" podStartSLOduration=4.178949427 podStartE2EDuration="21.748436343s" totalImagesPullingTime="17.569486916s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 22:01:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:01:40.748246224 +0000 UTC m=+21.704313932" watchObservedRunningTime="2026-09-10 22:01:40.748436343 +0000 UTC m=+21.704504048"
Sep 10 22:01:40.769033 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:40.768982    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-dth7z" podStartSLOduration=4.195569596 podStartE2EDuration="21.768967032s" totalImagesPullingTime="17.573397436s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 22:01:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:01:40.768506768 +0000 UTC m=+21.724574464" watchObservedRunningTime="2026-09-10 22:01:40.768967032 +0000 UTC m=+21.725034738"
Sep 10 22:01:40.821442 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:40.821389    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-8rwdz" podStartSLOduration=4.178323871 podStartE2EDuration="21.821375326s" totalImagesPullingTime="17.643051455s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 22:01:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:01:40.820972224 +0000 UTC m=+21.777039931" watchObservedRunningTime="2026-09-10 22:01:40.821375326 +0000 UTC m=+21.777443032"
Sep 10 22:01:40.821655 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:40.821626    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-ncrzz" podStartSLOduration=4.247077449 podStartE2EDuration="21.821616331s" totalImagesPullingTime="17.574538882s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 22:01:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:01:40.797677346 +0000 UTC m=+21.753745051" watchObservedRunningTime="2026-09-10 22:01:40.821616331 +0000 UTC m=+21.777684038"
Sep 10 22:01:41.268889 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:41.268739    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-cjdvg"
Sep 10 22:01:41.269724 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:41.269701    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-cjdvg"
Sep 10 22:01:41.285441 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:41.285382    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-cjdvg" podStartSLOduration=8.493161247 podStartE2EDuration="22.285361274s" totalImagesPullingTime="13.792200027s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 22:01:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:01:40.840560879 +0000 UTC m=+21.796628584" watchObservedRunningTime="2026-09-10 22:01:41.285361274 +0000 UTC m=+22.241429008"
Sep 10 22:01:41.492424 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:41.492395    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n9r9x"
Sep 10 22:01:41.492552 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:01:41.492528    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n9r9x" podUID="0d6f601b-532d-4f1f-9580-977997448ea9"
Sep 10 22:01:41.492601 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:41.492570    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xw7pd"
Sep 10 22:01:41.492682 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:01:41.492659    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xw7pd" podUID="a6c7caf6-cae8-4a81-98c5-e4db1671eff5"
Sep 10 22:01:41.722135 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:41.722105    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-p8f5t" event={"ID":"7821d1b4-78c1-4500-84e4-1b1fc03ad836","Type":"ContainerStarted","Data":"9a9948f596901292a81d60101497d912eb9c6c64c5f7e9db57604ff6c18784fc"}
Sep 10 22:01:41.726201 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:41.726164    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zbjsk" event={"ID":"8750bcd5-12c7-45a5-9157-84ed58d80d24","Type":"ContainerStarted","Data":"0fb128a271fe7d52f330fae521d69d0cf10ffe7b0e409a140aa48105ad974c45"}
Sep 10 22:01:41.726933 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:41.726913    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-cjdvg"
Sep 10 22:01:41.727027 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:41.726975    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-cjdvg"
Sep 10 22:01:41.749185 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:41.749161    2582 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 10 22:01:41.776279 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:41.776205    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-p8f5t" podStartSLOduration=5.208136613 podStartE2EDuration="22.776189362s" totalImagesPullingTime="17.568052749s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 22:01:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:01:41.74621134 +0000 UTC m=+22.702279047" watchObservedRunningTime="2026-09-10 22:01:41.776189362 +0000 UTC m=+22.732257130"
Sep 10 22:01:42.492749 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:42.492713    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rrm25"
Sep 10 22:01:42.492932 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:42.492703    2582 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-10T22:01:41.749182015Z","UUID":"bd19af91-e22e-477a-9b39-c80775852931","Handler":null,"Name":"","Endpoint":""}
Sep 10 22:01:42.492932 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:01:42.492871    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rrm25" podUID="064a91af-7e3d-4e71-b743-04d68d63174e"
Sep 10 22:01:42.494210 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:42.494191    2582 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 10 22:01:42.494344 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:42.494225    2582 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 10 22:01:42.732264 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:42.732227    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x4xzs" event={"ID":"fe63ec98-ecdb-4555-8799-a1f3e9558da3","Type":"ContainerStarted","Data":"b243fec2d7005a21eb0bd970275120ed55d577d89edc7782999b527421f2ff5c"}
Sep 10 22:01:43.492188 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:43.492096    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n9r9x"
Sep 10 22:01:43.492334 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:01:43.492209    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n9r9x" podUID="0d6f601b-532d-4f1f-9580-977997448ea9"
Sep 10 22:01:43.492334 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:43.492243    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xw7pd"
Sep 10 22:01:43.492334 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:01:43.492297    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xw7pd" podUID="a6c7caf6-cae8-4a81-98c5-e4db1671eff5"
Sep 10 22:01:43.738137 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:43.738103    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zbjsk" event={"ID":"8750bcd5-12c7-45a5-9157-84ed58d80d24","Type":"ContainerStarted","Data":"61feb73f44adb9390062a9303dda6fc398542be20bd2af0310a93d9cf644a71d"}
Sep 10 22:01:43.740716 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:43.740680    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x4xzs" event={"ID":"fe63ec98-ecdb-4555-8799-a1f3e9558da3","Type":"ContainerStarted","Data":"c093c1c50f8c5b376c5592784163d67ca0998e13706dd4c60dafa274411a9a5a"}
Sep 10 22:01:43.758807 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:43.758718    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x4xzs" podStartSLOduration=4.48531727 podStartE2EDuration="24.758703662s" totalImagesPullingTime="20.273386392s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 22:01:19 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:01:43.758270922 +0000 UTC m=+24.714338629" watchObservedRunningTime="2026-09-10 22:01:43.758703662 +0000 UTC m=+24.714771372"
Sep 10 22:01:44.492640 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:44.492609    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rrm25"
Sep 10 22:01:44.492868 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:01:44.492737    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rrm25" podUID="064a91af-7e3d-4e71-b743-04d68d63174e"
Sep 10 22:01:45.493065 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:45.492913    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n9r9x"
Sep 10 22:01:45.493594 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:45.492913    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xw7pd"
Sep 10 22:01:45.493594 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:01:45.493176    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n9r9x" podUID="0d6f601b-532d-4f1f-9580-977997448ea9"
Sep 10 22:01:45.493594 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:01:45.493305    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xw7pd" podUID="a6c7caf6-cae8-4a81-98c5-e4db1671eff5"
Sep 10 22:01:45.749999 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:45.749966    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zbjsk" event={"ID":"8750bcd5-12c7-45a5-9157-84ed58d80d24","Type":"ContainerStarted","Data":"62c92b1f1d78cee7304690ca88ea86ebf7f2291d69c6321ec3ca9b5354c6d7ab"}
Sep 10 22:01:45.750222 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:45.750201    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-zbjsk"
Sep 10 22:01:45.750413 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:45.750274    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-zbjsk"
Sep 10 22:01:45.752302 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:45.752277    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dt25t" event={"ID":"3cf12f53-0edf-43cc-913f-50b1669152b2","Type":"ContainerStarted","Data":"83227c909893d2f978b80e5ab635c2d3a1f9b69c47a5513671c6b91c0fa55293"}
Sep 10 22:01:45.765287 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:45.765266    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zbjsk"
Sep 10 22:01:45.777425 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:45.777381    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-zbjsk" podStartSLOduration=9.156365516 podStartE2EDuration="26.77736424s" totalImagesPullingTime="17.620998724s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 22:01:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:01:45.77680314 +0000 UTC m=+26.732870845" watchObservedRunningTime="2026-09-10 22:01:45.77736424 +0000 UTC m=+26.733431967"
Sep 10 22:01:46.492628 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:46.492595    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rrm25"
Sep 10 22:01:46.492830 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:01:46.492706    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rrm25" podUID="064a91af-7e3d-4e71-b743-04d68d63174e"
Sep 10 22:01:46.756688 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:46.756605    2582 generic.go:356] "Generic (PLEG): container finished" podID="3cf12f53-0edf-43cc-913f-50b1669152b2" containerID="83227c909893d2f978b80e5ab635c2d3a1f9b69c47a5513671c6b91c0fa55293" exitCode=0
Sep 10 22:01:46.757261 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:46.756695    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dt25t" event={"ID":"3cf12f53-0edf-43cc-913f-50b1669152b2","Type":"ContainerDied","Data":"83227c909893d2f978b80e5ab635c2d3a1f9b69c47a5513671c6b91c0fa55293"}
Sep 10 22:01:46.757261 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:46.757126    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-zbjsk"
Sep 10 22:01:46.772881 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:46.772840    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zbjsk"
Sep 10 22:01:47.492225 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:47.491997    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xw7pd"
Sep 10 22:01:47.492412 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:47.492002    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n9r9x"
Sep 10 22:01:47.492412 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:01:47.492334    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xw7pd" podUID="a6c7caf6-cae8-4a81-98c5-e4db1671eff5"
Sep 10 22:01:47.492412 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:01:47.492382    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n9r9x" podUID="0d6f601b-532d-4f1f-9580-977997448ea9"
Sep 10 22:01:47.496093 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:47.496065    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-rrm25"]
Sep 10 22:01:47.496244 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:47.496157    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rrm25"
Sep 10 22:01:47.496306 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:01:47.496245    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rrm25" podUID="064a91af-7e3d-4e71-b743-04d68d63174e"
Sep 10 22:01:47.496825 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:47.496801    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-n9r9x"]
Sep 10 22:01:47.498700 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:47.498673    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-xw7pd"]
Sep 10 22:01:47.758659 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:47.758585    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n9r9x"
Sep 10 22:01:47.759047 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:47.758590    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xw7pd"
Sep 10 22:01:47.759047 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:01:47.758687    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n9r9x" podUID="0d6f601b-532d-4f1f-9580-977997448ea9"
Sep 10 22:01:47.759047 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:01:47.758758    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xw7pd" podUID="a6c7caf6-cae8-4a81-98c5-e4db1671eff5"
Sep 10 22:01:48.763046 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:48.762972    2582 generic.go:356] "Generic (PLEG): container finished" podID="3cf12f53-0edf-43cc-913f-50b1669152b2" containerID="28380792c6a72e3e81821f23c985935ec1196d6e306e21f7c6ce248564528941" exitCode=0
Sep 10 22:01:48.763413 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:48.763059    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dt25t" event={"ID":"3cf12f53-0edf-43cc-913f-50b1669152b2","Type":"ContainerDied","Data":"28380792c6a72e3e81821f23c985935ec1196d6e306e21f7c6ce248564528941"}
Sep 10 22:01:49.493476 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:49.493442    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rrm25"
Sep 10 22:01:49.493615 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:49.493590    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xw7pd"
Sep 10 22:01:49.493660 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:01:49.493601    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rrm25" podUID="064a91af-7e3d-4e71-b743-04d68d63174e"
Sep 10 22:01:49.493711 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:01:49.493695    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xw7pd" podUID="a6c7caf6-cae8-4a81-98c5-e4db1671eff5"
Sep 10 22:01:49.493821 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:49.493721    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n9r9x"
Sep 10 22:01:49.493821 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:01:49.493779    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n9r9x" podUID="0d6f601b-532d-4f1f-9580-977997448ea9"
Sep 10 22:01:50.769374 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:50.769112    2582 generic.go:356] "Generic (PLEG): container finished" podID="3cf12f53-0edf-43cc-913f-50b1669152b2" containerID="ccd7597ae67bfeda5fc254ea8c63e78bf63a79f899e36901f1da3d5cefbb821a" exitCode=0
Sep 10 22:01:50.769374 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:50.769192    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dt25t" event={"ID":"3cf12f53-0edf-43cc-913f-50b1669152b2","Type":"ContainerDied","Data":"ccd7597ae67bfeda5fc254ea8c63e78bf63a79f899e36901f1da3d5cefbb821a"}
Sep 10 22:01:51.492877 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:51.492787    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n9r9x"
Sep 10 22:01:51.492877 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:51.492804    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rrm25"
Sep 10 22:01:51.492877 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:51.492790    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xw7pd"
Sep 10 22:01:51.493107 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:01:51.492920    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rrm25" podUID="064a91af-7e3d-4e71-b743-04d68d63174e"
Sep 10 22:01:51.493107 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:01:51.493008    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xw7pd" podUID="a6c7caf6-cae8-4a81-98c5-e4db1671eff5"
Sep 10 22:01:51.493190 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:01:51.493109    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n9r9x" podUID="0d6f601b-532d-4f1f-9580-977997448ea9"
Sep 10 22:01:52.882734 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:52.882654    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-144.ec2.internal" event="NodeReady"
Sep 10 22:01:52.883245 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:52.882805    2582 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 10 22:01:52.928367 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:52.928333    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-xck4c"]
Sep 10 22:01:52.932617 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:52.932589    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-5p6sf"]
Sep 10 22:01:52.932778 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:52.932729    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-xck4c"
Sep 10 22:01:52.934311 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:52.934288    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-5p6sf"
Sep 10 22:01:52.934961 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:52.934935    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 10 22:01:52.935083 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:52.934937    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 10 22:01:52.935145 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:52.935095    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-qt6wb\""
Sep 10 22:01:52.936499 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:52.936302    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 10 22:01:52.936499 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:52.936310    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-fw62z\""
Sep 10 22:01:52.936499 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:52.936301    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 10 22:01:52.936499 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:52.936396    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 10 22:01:52.938676 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:52.938656    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-5p6sf"]
Sep 10 22:01:52.942769 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:52.942389    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-xck4c"]
Sep 10 22:01:53.060120 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:53.060092    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d4e1da6-4cc3-41a7-a8be-1fdb06fd6759-tmp-dir\") pod \"dns-default-xck4c\" (UID: \"9d4e1da6-4cc3-41a7-a8be-1fdb06fd6759\") " pod="openshift-dns/dns-default-xck4c"
Sep 10 22:01:53.060273 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:53.060129    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jv5nx\" (UniqueName: \"kubernetes.io/projected/9d4e1da6-4cc3-41a7-a8be-1fdb06fd6759-kube-api-access-jv5nx\") pod \"dns-default-xck4c\" (UID: \"9d4e1da6-4cc3-41a7-a8be-1fdb06fd6759\") " pod="openshift-dns/dns-default-xck4c"
Sep 10 22:01:53.060273 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:53.060148    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79cg8\" (UniqueName: \"kubernetes.io/projected/b8b665fe-9846-4157-8922-433c2e13f6fa-kube-api-access-79cg8\") pod \"ingress-canary-5p6sf\" (UID: \"b8b665fe-9846-4157-8922-433c2e13f6fa\") " pod="openshift-ingress-canary/ingress-canary-5p6sf"
Sep 10 22:01:53.060273 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:53.060170    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9d4e1da6-4cc3-41a7-a8be-1fdb06fd6759-metrics-tls\") pod \"dns-default-xck4c\" (UID: \"9d4e1da6-4cc3-41a7-a8be-1fdb06fd6759\") " pod="openshift-dns/dns-default-xck4c"
Sep 10 22:01:53.060273 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:53.060252    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9d4e1da6-4cc3-41a7-a8be-1fdb06fd6759-config-volume\") pod \"dns-default-xck4c\" (UID: \"9d4e1da6-4cc3-41a7-a8be-1fdb06fd6759\") " pod="openshift-dns/dns-default-xck4c"
Sep 10 22:01:53.060467 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:53.060361    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b8b665fe-9846-4157-8922-433c2e13f6fa-cert\") pod \"ingress-canary-5p6sf\" (UID: \"b8b665fe-9846-4157-8922-433c2e13f6fa\") " pod="openshift-ingress-canary/ingress-canary-5p6sf"
Sep 10 22:01:53.161474 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:53.161392    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jv5nx\" (UniqueName: \"kubernetes.io/projected/9d4e1da6-4cc3-41a7-a8be-1fdb06fd6759-kube-api-access-jv5nx\") pod \"dns-default-xck4c\" (UID: \"9d4e1da6-4cc3-41a7-a8be-1fdb06fd6759\") " pod="openshift-dns/dns-default-xck4c"
Sep 10 22:01:53.161474 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:53.161430    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79cg8\" (UniqueName: \"kubernetes.io/projected/b8b665fe-9846-4157-8922-433c2e13f6fa-kube-api-access-79cg8\") pod \"ingress-canary-5p6sf\" (UID: \"b8b665fe-9846-4157-8922-433c2e13f6fa\") " pod="openshift-ingress-canary/ingress-canary-5p6sf"
Sep 10 22:01:53.161474 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:53.161455    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9d4e1da6-4cc3-41a7-a8be-1fdb06fd6759-metrics-tls\") pod \"dns-default-xck4c\" (UID: \"9d4e1da6-4cc3-41a7-a8be-1fdb06fd6759\") " pod="openshift-dns/dns-default-xck4c"
Sep 10 22:01:53.161744 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:01:53.161563    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 22:01:53.161744 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:01:53.161633    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d4e1da6-4cc3-41a7-a8be-1fdb06fd6759-metrics-tls podName:9d4e1da6-4cc3-41a7-a8be-1fdb06fd6759 nodeName:}" failed. No retries permitted until 2026-09-10 22:01:53.66161234 +0000 UTC m=+34.617680037 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9d4e1da6-4cc3-41a7-a8be-1fdb06fd6759-metrics-tls") pod "dns-default-xck4c" (UID: "9d4e1da6-4cc3-41a7-a8be-1fdb06fd6759") : secret "dns-default-metrics-tls" not found
Sep 10 22:01:53.161744 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:53.161651    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9d4e1da6-4cc3-41a7-a8be-1fdb06fd6759-config-volume\") pod \"dns-default-xck4c\" (UID: \"9d4e1da6-4cc3-41a7-a8be-1fdb06fd6759\") " pod="openshift-dns/dns-default-xck4c"
Sep 10 22:01:53.161744 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:53.161684    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0d6f601b-532d-4f1f-9580-977997448ea9-original-pull-secret\") pod \"global-pull-secret-syncer-n9r9x\" (UID: \"0d6f601b-532d-4f1f-9580-977997448ea9\") " pod="kube-system/global-pull-secret-syncer-n9r9x"
Sep 10 22:01:53.161744 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:53.161739    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6c7caf6-cae8-4a81-98c5-e4db1671eff5-metrics-certs\") pod \"network-metrics-daemon-xw7pd\" (UID: \"a6c7caf6-cae8-4a81-98c5-e4db1671eff5\") " pod="openshift-multus/network-metrics-daemon-xw7pd"
Sep 10 22:01:53.162089 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:53.161768    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b8b665fe-9846-4157-8922-433c2e13f6fa-cert\") pod \"ingress-canary-5p6sf\" (UID: \"b8b665fe-9846-4157-8922-433c2e13f6fa\") " pod="openshift-ingress-canary/ingress-canary-5p6sf"
Sep 10 22:01:53.162089 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:53.161813    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d4e1da6-4cc3-41a7-a8be-1fdb06fd6759-tmp-dir\") pod \"dns-default-xck4c\" (UID: \"9d4e1da6-4cc3-41a7-a8be-1fdb06fd6759\") " pod="openshift-dns/dns-default-xck4c"
Sep 10 22:01:53.162089 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:01:53.161835    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 22:01:53.162089 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:01:53.161899    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 22:01:53.162089 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:01:53.161924    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d6f601b-532d-4f1f-9580-977997448ea9-original-pull-secret podName:0d6f601b-532d-4f1f-9580-977997448ea9 nodeName:}" failed. No retries permitted until 2026-09-10 22:02:25.161906597 +0000 UTC m=+66.117974282 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0d6f601b-532d-4f1f-9580-977997448ea9-original-pull-secret") pod "global-pull-secret-syncer-n9r9x" (UID: "0d6f601b-532d-4f1f-9580-977997448ea9") : object "kube-system"/"original-pull-secret" not registered
Sep 10 22:01:53.162089 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:01:53.161952    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6c7caf6-cae8-4a81-98c5-e4db1671eff5-metrics-certs podName:a6c7caf6-cae8-4a81-98c5-e4db1671eff5 nodeName:}" failed. No retries permitted until 2026-09-10 22:02:25.161939317 +0000 UTC m=+66.118007001 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a6c7caf6-cae8-4a81-98c5-e4db1671eff5-metrics-certs") pod "network-metrics-daemon-xw7pd" (UID: "a6c7caf6-cae8-4a81-98c5-e4db1671eff5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 22:01:53.162346 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:53.162269    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d4e1da6-4cc3-41a7-a8be-1fdb06fd6759-tmp-dir\") pod \"dns-default-xck4c\" (UID: \"9d4e1da6-4cc3-41a7-a8be-1fdb06fd6759\") " pod="openshift-dns/dns-default-xck4c"
Sep 10 22:01:53.162346 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:53.162309    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9d4e1da6-4cc3-41a7-a8be-1fdb06fd6759-config-volume\") pod \"dns-default-xck4c\" (UID: \"9d4e1da6-4cc3-41a7-a8be-1fdb06fd6759\") " pod="openshift-dns/dns-default-xck4c"
Sep 10 22:01:53.166296 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:53.166275    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b8b665fe-9846-4157-8922-433c2e13f6fa-cert\") pod \"ingress-canary-5p6sf\" (UID: \"b8b665fe-9846-4157-8922-433c2e13f6fa\") " pod="openshift-ingress-canary/ingress-canary-5p6sf"
Sep 10 22:01:53.170368 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:53.170337    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-79cg8\" (UniqueName: \"kubernetes.io/projected/b8b665fe-9846-4157-8922-433c2e13f6fa-kube-api-access-79cg8\") pod \"ingress-canary-5p6sf\" (UID: \"b8b665fe-9846-4157-8922-433c2e13f6fa\") " pod="openshift-ingress-canary/ingress-canary-5p6sf"
Sep 10 22:01:53.170368 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:53.170356    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jv5nx\" (UniqueName: \"kubernetes.io/projected/9d4e1da6-4cc3-41a7-a8be-1fdb06fd6759-kube-api-access-jv5nx\") pod \"dns-default-xck4c\" (UID: \"9d4e1da6-4cc3-41a7-a8be-1fdb06fd6759\") " pod="openshift-dns/dns-default-xck4c"
Sep 10 22:01:53.255032 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:53.254996    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-5p6sf"
Sep 10 22:01:53.262768 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:53.262744    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwfrj\" (UniqueName: \"kubernetes.io/projected/064a91af-7e3d-4e71-b743-04d68d63174e-kube-api-access-zwfrj\") pod \"network-check-target-rrm25\" (UID: \"064a91af-7e3d-4e71-b743-04d68d63174e\") " pod="openshift-network-diagnostics/network-check-target-rrm25"
Sep 10 22:01:53.262912 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:01:53.262892    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 22:01:53.262912 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:01:53.262915    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 22:01:53.263075 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:01:53.262931    2582 projected.go:196] Error preparing data for projected volume kube-api-access-zwfrj for pod openshift-network-diagnostics/network-check-target-rrm25: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 22:01:53.263075 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:01:53.262990    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/064a91af-7e3d-4e71-b743-04d68d63174e-kube-api-access-zwfrj podName:064a91af-7e3d-4e71-b743-04d68d63174e nodeName:}" failed. No retries permitted until 2026-09-10 22:02:25.262970651 +0000 UTC m=+66.219038351 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-zwfrj" (UniqueName: "kubernetes.io/projected/064a91af-7e3d-4e71-b743-04d68d63174e-kube-api-access-zwfrj") pod "network-check-target-rrm25" (UID: "064a91af-7e3d-4e71-b743-04d68d63174e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 22:01:53.492611 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:53.492529    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xw7pd"
Sep 10 22:01:53.492611 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:53.492529    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n9r9x"
Sep 10 22:01:53.492823 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:53.492533    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rrm25"
Sep 10 22:01:53.496273 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:53.496113    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 10 22:01:53.496273 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:53.496142    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-vfb6l\""
Sep 10 22:01:53.496273 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:53.496173    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 10 22:01:53.496515 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:53.496364    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-5csvl\""
Sep 10 22:01:53.496515 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:53.496461    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 10 22:01:53.496618 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:53.496555    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 10 22:01:53.536299 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:53.536274    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-5p6sf"]
Sep 10 22:01:53.540666 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:01:53.540640    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8b665fe_9846_4157_8922_433c2e13f6fa.slice/crio-49aac6bc27dfe04bc456c06428c979cbd6ba785c890ee5d29ccd7511d7f3dcc4 WatchSource:0}: Error finding container 49aac6bc27dfe04bc456c06428c979cbd6ba785c890ee5d29ccd7511d7f3dcc4: Status 404 returned error can't find the container with id 49aac6bc27dfe04bc456c06428c979cbd6ba785c890ee5d29ccd7511d7f3dcc4
Sep 10 22:01:53.666567 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:53.666531    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9d4e1da6-4cc3-41a7-a8be-1fdb06fd6759-metrics-tls\") pod \"dns-default-xck4c\" (UID: \"9d4e1da6-4cc3-41a7-a8be-1fdb06fd6759\") " pod="openshift-dns/dns-default-xck4c"
Sep 10 22:01:53.666758 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:01:53.666691    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 22:01:53.666822 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:01:53.666762    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d4e1da6-4cc3-41a7-a8be-1fdb06fd6759-metrics-tls podName:9d4e1da6-4cc3-41a7-a8be-1fdb06fd6759 nodeName:}" failed. No retries permitted until 2026-09-10 22:01:54.666740555 +0000 UTC m=+35.622808238 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9d4e1da6-4cc3-41a7-a8be-1fdb06fd6759-metrics-tls") pod "dns-default-xck4c" (UID: "9d4e1da6-4cc3-41a7-a8be-1fdb06fd6759") : secret "dns-default-metrics-tls" not found
Sep 10 22:01:53.778422 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:53.778336    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-5p6sf" event={"ID":"b8b665fe-9846-4157-8922-433c2e13f6fa","Type":"ContainerStarted","Data":"49aac6bc27dfe04bc456c06428c979cbd6ba785c890ee5d29ccd7511d7f3dcc4"}
Sep 10 22:01:54.673249 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:54.673211    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9d4e1da6-4cc3-41a7-a8be-1fdb06fd6759-metrics-tls\") pod \"dns-default-xck4c\" (UID: \"9d4e1da6-4cc3-41a7-a8be-1fdb06fd6759\") " pod="openshift-dns/dns-default-xck4c"
Sep 10 22:01:54.673974 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:01:54.673356    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 22:01:54.673974 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:01:54.673426    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d4e1da6-4cc3-41a7-a8be-1fdb06fd6759-metrics-tls podName:9d4e1da6-4cc3-41a7-a8be-1fdb06fd6759 nodeName:}" failed. No retries permitted until 2026-09-10 22:01:56.673405346 +0000 UTC m=+37.629473033 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9d4e1da6-4cc3-41a7-a8be-1fdb06fd6759-metrics-tls") pod "dns-default-xck4c" (UID: "9d4e1da6-4cc3-41a7-a8be-1fdb06fd6759") : secret "dns-default-metrics-tls" not found
Sep 10 22:01:55.787206 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:55.787167    2582 generic.go:356] "Generic (PLEG): container finished" podID="3cf12f53-0edf-43cc-913f-50b1669152b2" containerID="29738b32dde7209fe7e86240e5ecc426af212de24003895ec8883af641c79e29" exitCode=0
Sep 10 22:01:55.787620 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:55.787247    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dt25t" event={"ID":"3cf12f53-0edf-43cc-913f-50b1669152b2","Type":"ContainerDied","Data":"29738b32dde7209fe7e86240e5ecc426af212de24003895ec8883af641c79e29"}
Sep 10 22:01:56.690029 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:56.689994    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9d4e1da6-4cc3-41a7-a8be-1fdb06fd6759-metrics-tls\") pod \"dns-default-xck4c\" (UID: \"9d4e1da6-4cc3-41a7-a8be-1fdb06fd6759\") " pod="openshift-dns/dns-default-xck4c"
Sep 10 22:01:56.690199 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:01:56.690159    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 22:01:56.690250 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:01:56.690229    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d4e1da6-4cc3-41a7-a8be-1fdb06fd6759-metrics-tls podName:9d4e1da6-4cc3-41a7-a8be-1fdb06fd6759 nodeName:}" failed. No retries permitted until 2026-09-10 22:02:00.690213621 +0000 UTC m=+41.646281305 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9d4e1da6-4cc3-41a7-a8be-1fdb06fd6759-metrics-tls") pod "dns-default-xck4c" (UID: "9d4e1da6-4cc3-41a7-a8be-1fdb06fd6759") : secret "dns-default-metrics-tls" not found
Sep 10 22:01:56.793943 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:56.793903    2582 generic.go:356] "Generic (PLEG): container finished" podID="3cf12f53-0edf-43cc-913f-50b1669152b2" containerID="1469e3af8457fe13c449f417262c8e064214c712ca84dd921f2890136a87878f" exitCode=0
Sep 10 22:01:56.794412 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:56.793999    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dt25t" event={"ID":"3cf12f53-0edf-43cc-913f-50b1669152b2","Type":"ContainerDied","Data":"1469e3af8457fe13c449f417262c8e064214c712ca84dd921f2890136a87878f"}
Sep 10 22:01:57.799607 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:57.799568    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dt25t" event={"ID":"3cf12f53-0edf-43cc-913f-50b1669152b2","Type":"ContainerStarted","Data":"bc4f120847f40be9d110be4c06c30553a6f1c408474c706d952bea0065479cf7"}
Sep 10 22:01:57.801099 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:57.801076    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-5p6sf" event={"ID":"b8b665fe-9846-4157-8922-433c2e13f6fa","Type":"ContainerStarted","Data":"750f55ad5bb8d48515c1da12d9bbba56194dc84e72d8a216480c75768e6855d9"}
Sep 10 22:01:57.820823 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:57.820773    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-dt25t" podStartSLOduration=9.383204422 podStartE2EDuration="38.820758392s" totalImagesPullingTime="29.43755397s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 22:01:19 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:01:57.820382383 +0000 UTC m=+38.776450090" watchObservedRunningTime="2026-09-10 22:01:57.820758392 +0000 UTC m=+38.776826098"
Sep 10 22:01:57.835309 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:01:57.835267    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-5p6sf" podStartSLOduration=2.084689908 podStartE2EDuration="5.835255064s" totalImagesPullingTime="3.750565156s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 22:01:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:01:57.834797981 +0000 UTC m=+38.790865699" watchObservedRunningTime="2026-09-10 22:01:57.835255064 +0000 UTC m=+38.791322770"
Sep 10 22:02:00.715758 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:00.715578    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9d4e1da6-4cc3-41a7-a8be-1fdb06fd6759-metrics-tls\") pod \"dns-default-xck4c\" (UID: \"9d4e1da6-4cc3-41a7-a8be-1fdb06fd6759\") " pod="openshift-dns/dns-default-xck4c"
Sep 10 22:02:00.716146 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:02:00.715723    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 22:02:00.716146 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:02:00.715883    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d4e1da6-4cc3-41a7-a8be-1fdb06fd6759-metrics-tls podName:9d4e1da6-4cc3-41a7-a8be-1fdb06fd6759 nodeName:}" failed. No retries permitted until 2026-09-10 22:02:08.715867501 +0000 UTC m=+49.671935198 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9d4e1da6-4cc3-41a7-a8be-1fdb06fd6759-metrics-tls") pod "dns-default-xck4c" (UID: "9d4e1da6-4cc3-41a7-a8be-1fdb06fd6759") : secret "dns-default-metrics-tls" not found
Sep 10 22:02:08.773268 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:08.773221    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9d4e1da6-4cc3-41a7-a8be-1fdb06fd6759-metrics-tls\") pod \"dns-default-xck4c\" (UID: \"9d4e1da6-4cc3-41a7-a8be-1fdb06fd6759\") " pod="openshift-dns/dns-default-xck4c"
Sep 10 22:02:08.773731 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:02:08.773388    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 22:02:08.773731 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:02:08.773473    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d4e1da6-4cc3-41a7-a8be-1fdb06fd6759-metrics-tls podName:9d4e1da6-4cc3-41a7-a8be-1fdb06fd6759 nodeName:}" failed. No retries permitted until 2026-09-10 22:02:24.773455778 +0000 UTC m=+65.729523462 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9d4e1da6-4cc3-41a7-a8be-1fdb06fd6759-metrics-tls") pod "dns-default-xck4c" (UID: "9d4e1da6-4cc3-41a7-a8be-1fdb06fd6759") : secret "dns-default-metrics-tls" not found
Sep 10 22:02:18.773763 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:18.773728    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zbjsk"
Sep 10 22:02:24.785212 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:24.785166    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9d4e1da6-4cc3-41a7-a8be-1fdb06fd6759-metrics-tls\") pod \"dns-default-xck4c\" (UID: \"9d4e1da6-4cc3-41a7-a8be-1fdb06fd6759\") " pod="openshift-dns/dns-default-xck4c"
Sep 10 22:02:24.785786 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:02:24.785345    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 22:02:24.785786 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:02:24.785433    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d4e1da6-4cc3-41a7-a8be-1fdb06fd6759-metrics-tls podName:9d4e1da6-4cc3-41a7-a8be-1fdb06fd6759 nodeName:}" failed. No retries permitted until 2026-09-10 22:02:56.785409674 +0000 UTC m=+97.741477375 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9d4e1da6-4cc3-41a7-a8be-1fdb06fd6759-metrics-tls") pod "dns-default-xck4c" (UID: "9d4e1da6-4cc3-41a7-a8be-1fdb06fd6759") : secret "dns-default-metrics-tls" not found
Sep 10 22:02:25.189068 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:25.189034    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0d6f601b-532d-4f1f-9580-977997448ea9-original-pull-secret\") pod \"global-pull-secret-syncer-n9r9x\" (UID: \"0d6f601b-532d-4f1f-9580-977997448ea9\") " pod="kube-system/global-pull-secret-syncer-n9r9x"
Sep 10 22:02:25.189220 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:25.189101    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6c7caf6-cae8-4a81-98c5-e4db1671eff5-metrics-certs\") pod \"network-metrics-daemon-xw7pd\" (UID: \"a6c7caf6-cae8-4a81-98c5-e4db1671eff5\") " pod="openshift-multus/network-metrics-daemon-xw7pd"
Sep 10 22:02:25.191422 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:25.191407    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 10 22:02:25.191467 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:25.191445    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 10 22:02:25.200098 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:02:25.200080    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 10 22:02:25.200160 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:02:25.200151    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6c7caf6-cae8-4a81-98c5-e4db1671eff5-metrics-certs podName:a6c7caf6-cae8-4a81-98c5-e4db1671eff5 nodeName:}" failed. No retries permitted until 2026-09-10 22:03:29.20011735 +0000 UTC m=+130.156185034 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a6c7caf6-cae8-4a81-98c5-e4db1671eff5-metrics-certs") pod "network-metrics-daemon-xw7pd" (UID: "a6c7caf6-cae8-4a81-98c5-e4db1671eff5") : secret "metrics-daemon-secret" not found
Sep 10 22:02:25.204661 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:25.204639    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0d6f601b-532d-4f1f-9580-977997448ea9-original-pull-secret\") pod \"global-pull-secret-syncer-n9r9x\" (UID: \"0d6f601b-532d-4f1f-9580-977997448ea9\") " pod="kube-system/global-pull-secret-syncer-n9r9x"
Sep 10 22:02:25.290133 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:25.290096    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwfrj\" (UniqueName: \"kubernetes.io/projected/064a91af-7e3d-4e71-b743-04d68d63174e-kube-api-access-zwfrj\") pod \"network-check-target-rrm25\" (UID: \"064a91af-7e3d-4e71-b743-04d68d63174e\") " pod="openshift-network-diagnostics/network-check-target-rrm25"
Sep 10 22:02:25.292327 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:25.292309    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 10 22:02:25.302944 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:25.302928    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 10 22:02:25.313784 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:25.313764    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwfrj\" (UniqueName: \"kubernetes.io/projected/064a91af-7e3d-4e71-b743-04d68d63174e-kube-api-access-zwfrj\") pod \"network-check-target-rrm25\" (UID: \"064a91af-7e3d-4e71-b743-04d68d63174e\") " pod="openshift-network-diagnostics/network-check-target-rrm25"
Sep 10 22:02:25.317552 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:25.317534    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n9r9x"
Sep 10 22:02:25.583405 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:25.583327    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-n9r9x"]
Sep 10 22:02:25.586745 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:02:25.586718    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d6f601b_532d_4f1f_9580_977997448ea9.slice/crio-0805e993e22a4e1179ab5a51b2f603e95c3025ce6e692cee45f8ee0d2d085015 WatchSource:0}: Error finding container 0805e993e22a4e1179ab5a51b2f603e95c3025ce6e692cee45f8ee0d2d085015: Status 404 returned error can't find the container with id 0805e993e22a4e1179ab5a51b2f603e95c3025ce6e692cee45f8ee0d2d085015
Sep 10 22:02:25.613618 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:25.613592    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-5csvl\""
Sep 10 22:02:25.622043 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:25.622028    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rrm25"
Sep 10 22:02:25.867512 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:25.867430    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-n9r9x" event={"ID":"0d6f601b-532d-4f1f-9580-977997448ea9","Type":"ContainerStarted","Data":"0805e993e22a4e1179ab5a51b2f603e95c3025ce6e692cee45f8ee0d2d085015"}
Sep 10 22:02:25.886845 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:25.886817    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-rrm25"]
Sep 10 22:02:25.889392 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:02:25.889371    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod064a91af_7e3d_4e71_b743_04d68d63174e.slice/crio-476e1f690cb34e373073870078bb7e5337e3f82a3070e411a68d3a5cd58c5bc6 WatchSource:0}: Error finding container 476e1f690cb34e373073870078bb7e5337e3f82a3070e411a68d3a5cd58c5bc6: Status 404 returned error can't find the container with id 476e1f690cb34e373073870078bb7e5337e3f82a3070e411a68d3a5cd58c5bc6
Sep 10 22:02:26.872023 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:26.871985    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-rrm25" event={"ID":"064a91af-7e3d-4e71-b743-04d68d63174e","Type":"ContainerStarted","Data":"476e1f690cb34e373073870078bb7e5337e3f82a3070e411a68d3a5cd58c5bc6"}
Sep 10 22:02:28.448999 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:28.448956    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-xl6k8"]
Sep 10 22:02:28.452164 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:28.452132    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-xl6k8"
Sep 10 22:02:28.454485 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:28.454443    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Sep 10 22:02:28.454631 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:28.454571    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-w4xzx\""
Sep 10 22:02:28.454631 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:28.454587    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Sep 10 22:02:28.454990 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:28.454967    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Sep 10 22:02:28.462364 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:28.462339    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-xl6k8"]
Sep 10 22:02:28.465927 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:28.465882    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-49png"]
Sep 10 22:02:28.468095 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:28.468073    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-49png"
Sep 10 22:02:28.470358 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:28.470186    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-ctz6d\""
Sep 10 22:02:28.479805 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:28.479779    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-49png"]
Sep 10 22:02:28.511901 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:28.511867    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmkp9\" (UniqueName: \"kubernetes.io/projected/14d6dc12-a6d1-4fce-999a-102fcd5100fe-kube-api-access-hmkp9\") pod \"network-check-source-6f8b768455-49png\" (UID: \"14d6dc12-a6d1-4fce-999a-102fcd5100fe\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-49png"
Sep 10 22:02:28.512082 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:28.511937    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5kdw\" (UniqueName: \"kubernetes.io/projected/a121acc4-9c7f-4dca-8d25-29cce37120a7-kube-api-access-k5kdw\") pod \"cluster-samples-operator-86499c89c4-xl6k8\" (UID: \"a121acc4-9c7f-4dca-8d25-29cce37120a7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-xl6k8"
Sep 10 22:02:28.512082 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:28.512003    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a121acc4-9c7f-4dca-8d25-29cce37120a7-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-xl6k8\" (UID: \"a121acc4-9c7f-4dca-8d25-29cce37120a7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-xl6k8"
Sep 10 22:02:28.555665 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:28.555605    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5844b856db-bhv4x"]
Sep 10 22:02:28.557699 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:28.557675    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-bhv4x"
Sep 10 22:02:28.561815 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:28.561795    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Sep 10 22:02:28.562115 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:28.562095    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 10 22:02:28.562203 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:28.562186    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Sep 10 22:02:28.562259 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:28.562187    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 10 22:02:28.562601 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:28.562585    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-5rz5s\""
Sep 10 22:02:28.570887 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:28.570869    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Sep 10 22:02:28.583227 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:28.583200    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-bhv4x"]
Sep 10 22:02:28.613264 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:28.613235    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3aaaf117-e959-49e2-93f0-342a657439c5-service-ca-bundle\") pod \"insights-operator-5844b856db-bhv4x\" (UID: \"3aaaf117-e959-49e2-93f0-342a657439c5\") " pod="openshift-insights/insights-operator-5844b856db-bhv4x"
Sep 10 22:02:28.613426 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:28.613272    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z247d\" (UniqueName: \"kubernetes.io/projected/3aaaf117-e959-49e2-93f0-342a657439c5-kube-api-access-z247d\") pod \"insights-operator-5844b856db-bhv4x\" (UID: \"3aaaf117-e959-49e2-93f0-342a657439c5\") " pod="openshift-insights/insights-operator-5844b856db-bhv4x"
Sep 10 22:02:28.613426 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:28.613321    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3aaaf117-e959-49e2-93f0-342a657439c5-trusted-ca-bundle\") pod \"insights-operator-5844b856db-bhv4x\" (UID: \"3aaaf117-e959-49e2-93f0-342a657439c5\") " pod="openshift-insights/insights-operator-5844b856db-bhv4x"
Sep 10 22:02:28.613426 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:28.613344    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3aaaf117-e959-49e2-93f0-342a657439c5-serving-cert\") pod \"insights-operator-5844b856db-bhv4x\" (UID: \"3aaaf117-e959-49e2-93f0-342a657439c5\") " pod="openshift-insights/insights-operator-5844b856db-bhv4x"
Sep 10 22:02:28.613426 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:28.613384    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmkp9\" (UniqueName: \"kubernetes.io/projected/14d6dc12-a6d1-4fce-999a-102fcd5100fe-kube-api-access-hmkp9\") pod \"network-check-source-6f8b768455-49png\" (UID: \"14d6dc12-a6d1-4fce-999a-102fcd5100fe\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-49png"
Sep 10 22:02:28.613426 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:28.613402    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3aaaf117-e959-49e2-93f0-342a657439c5-tmp\") pod \"insights-operator-5844b856db-bhv4x\" (UID: \"3aaaf117-e959-49e2-93f0-342a657439c5\") " pod="openshift-insights/insights-operator-5844b856db-bhv4x"
Sep 10 22:02:28.613742 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:28.613464    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5kdw\" (UniqueName: \"kubernetes.io/projected/a121acc4-9c7f-4dca-8d25-29cce37120a7-kube-api-access-k5kdw\") pod \"cluster-samples-operator-86499c89c4-xl6k8\" (UID: \"a121acc4-9c7f-4dca-8d25-29cce37120a7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-xl6k8"
Sep 10 22:02:28.613742 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:28.613490    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/3aaaf117-e959-49e2-93f0-342a657439c5-snapshots\") pod \"insights-operator-5844b856db-bhv4x\" (UID: \"3aaaf117-e959-49e2-93f0-342a657439c5\") " pod="openshift-insights/insights-operator-5844b856db-bhv4x"
Sep 10 22:02:28.613742 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:28.613558    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a121acc4-9c7f-4dca-8d25-29cce37120a7-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-xl6k8\" (UID: \"a121acc4-9c7f-4dca-8d25-29cce37120a7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-xl6k8"
Sep 10 22:02:28.613742 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:02:28.613639    2582 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 10 22:02:28.613742 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:02:28.613700    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a121acc4-9c7f-4dca-8d25-29cce37120a7-samples-operator-tls podName:a121acc4-9c7f-4dca-8d25-29cce37120a7 nodeName:}" failed. No retries permitted until 2026-09-10 22:02:29.113676685 +0000 UTC m=+70.069744369 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/a121acc4-9c7f-4dca-8d25-29cce37120a7-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-xl6k8" (UID: "a121acc4-9c7f-4dca-8d25-29cce37120a7") : secret "samples-operator-tls" not found
Sep 10 22:02:28.626238 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:28.626210    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5kdw\" (UniqueName: \"kubernetes.io/projected/a121acc4-9c7f-4dca-8d25-29cce37120a7-kube-api-access-k5kdw\") pod \"cluster-samples-operator-86499c89c4-xl6k8\" (UID: \"a121acc4-9c7f-4dca-8d25-29cce37120a7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-xl6k8"
Sep 10 22:02:28.626238 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:28.626220    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmkp9\" (UniqueName: \"kubernetes.io/projected/14d6dc12-a6d1-4fce-999a-102fcd5100fe-kube-api-access-hmkp9\") pod \"network-check-source-6f8b768455-49png\" (UID: \"14d6dc12-a6d1-4fce-999a-102fcd5100fe\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-49png"
Sep 10 22:02:28.650431 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:28.650397    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-w6vlb"]
Sep 10 22:02:28.653160 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:28.652524    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-w6vlb"
Sep 10 22:02:28.653996 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:28.653968    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-66fc76d68b-5wp97"]
Sep 10 22:02:28.655103 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:28.654619    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Sep 10 22:02:28.655103 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:28.654667    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Sep 10 22:02:28.655103 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:28.654950    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Sep 10 22:02:28.655103 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:28.655026    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Sep 10 22:02:28.655532 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:28.655506    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-crlwg\""
Sep 10 22:02:28.656094 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:28.656035    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-66fc76d68b-5wp97"
Sep 10 22:02:28.657839 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:28.657812    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 10 22:02:28.657839 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:28.657830    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Sep 10 22:02:28.658046 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:28.657994    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Sep 10 22:02:28.658737 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:28.658583    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Sep 10 22:02:28.658737 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:28.658602    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-sdn8b\""
Sep 10 22:02:28.658737 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:28.658621    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 10 22:02:28.658997 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:28.658791    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Sep 10 22:02:28.661630 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:28.661609    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Sep 10 22:02:28.665871 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:28.665831    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-w6vlb"]
Sep 10 22:02:28.669314 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:28.669294    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-66fc76d68b-5wp97"]
Sep 10 22:02:28.714255 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:28.714172    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3aaaf117-e959-49e2-93f0-342a657439c5-service-ca-bundle\") pod \"insights-operator-5844b856db-bhv4x\" (UID: \"3aaaf117-e959-49e2-93f0-342a657439c5\") " pod="openshift-insights/insights-operator-5844b856db-bhv4x"
Sep 10 22:02:28.714255 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:28.714205    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z247d\" (UniqueName: \"kubernetes.io/projected/3aaaf117-e959-49e2-93f0-342a657439c5-kube-api-access-z247d\") pod \"insights-operator-5844b856db-bhv4x\" (UID: \"3aaaf117-e959-49e2-93f0-342a657439c5\") " pod="openshift-insights/insights-operator-5844b856db-bhv4x"
Sep 10 22:02:28.714255 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:28.714230    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8517c9a7-382a-40fd-bc52-e6da90ebc584-config\") pod \"console-operator-696884fb9c-w6vlb\" (UID: \"8517c9a7-382a-40fd-bc52-e6da90ebc584\") " pod="openshift-console-operator/console-operator-696884fb9c-w6vlb"
Sep 10 22:02:28.714485 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:28.714262    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3aaaf117-e959-49e2-93f0-342a657439c5-trusted-ca-bundle\") pod \"insights-operator-5844b856db-bhv4x\" (UID: \"3aaaf117-e959-49e2-93f0-342a657439c5\") " pod="openshift-insights/insights-operator-5844b856db-bhv4x"
Sep 10 22:02:28.714485 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:28.714279    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3aaaf117-e959-49e2-93f0-342a657439c5-serving-cert\") pod \"insights-operator-5844b856db-bhv4x\" (UID: \"3aaaf117-e959-49e2-93f0-342a657439c5\") " pod="openshift-insights/insights-operator-5844b856db-bhv4x"
Sep 10 22:02:28.714485 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:28.714301    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zf9l6\" (UniqueName: \"kubernetes.io/projected/8517c9a7-382a-40fd-bc52-e6da90ebc584-kube-api-access-zf9l6\") pod \"console-operator-696884fb9c-w6vlb\" (UID: \"8517c9a7-382a-40fd-bc52-e6da90ebc584\") " pod="openshift-console-operator/console-operator-696884fb9c-w6vlb"
Sep 10 22:02:28.714485 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:28.714322    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/97c76d14-c853-48fa-bb5c-b7c553b4f899-stats-auth\") pod \"router-default-66fc76d68b-5wp97\" (UID: \"97c76d14-c853-48fa-bb5c-b7c553b4f899\") " pod="openshift-ingress/router-default-66fc76d68b-5wp97"
Sep 10 22:02:28.714485 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:28.714351    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/97c76d14-c853-48fa-bb5c-b7c553b4f899-default-certificate\") pod \"router-default-66fc76d68b-5wp97\" (UID: \"97c76d14-c853-48fa-bb5c-b7c553b4f899\") " pod="openshift-ingress/router-default-66fc76d68b-5wp97"
Sep 10 22:02:28.714485 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:28.714376    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqvh8\" (UniqueName: \"kubernetes.io/projected/97c76d14-c853-48fa-bb5c-b7c553b4f899-kube-api-access-mqvh8\") pod \"router-default-66fc76d68b-5wp97\" (UID: \"97c76d14-c853-48fa-bb5c-b7c553b4f899\") " pod="openshift-ingress/router-default-66fc76d68b-5wp97"
Sep 10 22:02:28.714485 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:28.714412    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3aaaf117-e959-49e2-93f0-342a657439c5-tmp\") pod \"insights-operator-5844b856db-bhv4x\" (UID: \"3aaaf117-e959-49e2-93f0-342a657439c5\") " pod="openshift-insights/insights-operator-5844b856db-bhv4x"
Sep 10 22:02:28.714485 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:28.714436    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/97c76d14-c853-48fa-bb5c-b7c553b4f899-service-ca-bundle\") pod \"router-default-66fc76d68b-5wp97\" (UID: \"97c76d14-c853-48fa-bb5c-b7c553b4f899\") " pod="openshift-ingress/router-default-66fc76d68b-5wp97"
Sep 10 22:02:28.714485 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:28.714461    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8517c9a7-382a-40fd-bc52-e6da90ebc584-serving-cert\") pod \"console-operator-696884fb9c-w6vlb\" (UID: \"8517c9a7-382a-40fd-bc52-e6da90ebc584\") " pod="openshift-console-operator/console-operator-696884fb9c-w6vlb"
Sep 10 22:02:28.714881 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:28.714492    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8517c9a7-382a-40fd-bc52-e6da90ebc584-trusted-ca\") pod \"console-operator-696884fb9c-w6vlb\" (UID: \"8517c9a7-382a-40fd-bc52-e6da90ebc584\") " pod="openshift-console-operator/console-operator-696884fb9c-w6vlb"
Sep 10 22:02:28.714881 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:28.714545    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/3aaaf117-e959-49e2-93f0-342a657439c5-snapshots\") pod \"insights-operator-5844b856db-bhv4x\" (UID: \"3aaaf117-e959-49e2-93f0-342a657439c5\") " pod="openshift-insights/insights-operator-5844b856db-bhv4x"
Sep 10 22:02:28.714881 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:28.714570    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/97c76d14-c853-48fa-bb5c-b7c553b4f899-metrics-certs\") pod \"router-default-66fc76d68b-5wp97\" (UID: \"97c76d14-c853-48fa-bb5c-b7c553b4f899\") " pod="openshift-ingress/router-default-66fc76d68b-5wp97"
Sep 10 22:02:28.714881 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:28.714786    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3aaaf117-e959-49e2-93f0-342a657439c5-tmp\") pod \"insights-operator-5844b856db-bhv4x\" (UID: \"3aaaf117-e959-49e2-93f0-342a657439c5\") " pod="openshift-insights/insights-operator-5844b856db-bhv4x"
Sep 10 22:02:28.715081 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:28.714958    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3aaaf117-e959-49e2-93f0-342a657439c5-service-ca-bundle\") pod \"insights-operator-5844b856db-bhv4x\" (UID: \"3aaaf117-e959-49e2-93f0-342a657439c5\") " pod="openshift-insights/insights-operator-5844b856db-bhv4x"
Sep 10 22:02:28.715238 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:28.715193    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3aaaf117-e959-49e2-93f0-342a657439c5-trusted-ca-bundle\") pod \"insights-operator-5844b856db-bhv4x\" (UID: \"3aaaf117-e959-49e2-93f0-342a657439c5\") " pod="openshift-insights/insights-operator-5844b856db-bhv4x"
Sep 10 22:02:28.715369 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:28.715250    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/3aaaf117-e959-49e2-93f0-342a657439c5-snapshots\") pod \"insights-operator-5844b856db-bhv4x\" (UID: \"3aaaf117-e959-49e2-93f0-342a657439c5\") " pod="openshift-insights/insights-operator-5844b856db-bhv4x"
Sep 10 22:02:28.717002 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:28.716982    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3aaaf117-e959-49e2-93f0-342a657439c5-serving-cert\") pod \"insights-operator-5844b856db-bhv4x\" (UID: \"3aaaf117-e959-49e2-93f0-342a657439c5\") " pod="openshift-insights/insights-operator-5844b856db-bhv4x"
Sep 10 22:02:28.722100 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:28.722075    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z247d\" (UniqueName: \"kubernetes.io/projected/3aaaf117-e959-49e2-93f0-342a657439c5-kube-api-access-z247d\") pod \"insights-operator-5844b856db-bhv4x\" (UID: \"3aaaf117-e959-49e2-93f0-342a657439c5\") " pod="openshift-insights/insights-operator-5844b856db-bhv4x"
Sep 10 22:02:28.778897 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:28.778833    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-49png"
Sep 10 22:02:28.815194 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:28.815161    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zf9l6\" (UniqueName: \"kubernetes.io/projected/8517c9a7-382a-40fd-bc52-e6da90ebc584-kube-api-access-zf9l6\") pod \"console-operator-696884fb9c-w6vlb\" (UID: \"8517c9a7-382a-40fd-bc52-e6da90ebc584\") " pod="openshift-console-operator/console-operator-696884fb9c-w6vlb"
Sep 10 22:02:28.815335 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:28.815198    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/97c76d14-c853-48fa-bb5c-b7c553b4f899-stats-auth\") pod \"router-default-66fc76d68b-5wp97\" (UID: \"97c76d14-c853-48fa-bb5c-b7c553b4f899\") " pod="openshift-ingress/router-default-66fc76d68b-5wp97"
Sep 10 22:02:28.815335 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:28.815220    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/97c76d14-c853-48fa-bb5c-b7c553b4f899-default-certificate\") pod \"router-default-66fc76d68b-5wp97\" (UID: \"97c76d14-c853-48fa-bb5c-b7c553b4f899\") " pod="openshift-ingress/router-default-66fc76d68b-5wp97"
Sep 10 22:02:28.815500 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:28.815470    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqvh8\" (UniqueName: \"kubernetes.io/projected/97c76d14-c853-48fa-bb5c-b7c553b4f899-kube-api-access-mqvh8\") pod \"router-default-66fc76d68b-5wp97\" (UID: \"97c76d14-c853-48fa-bb5c-b7c553b4f899\") " pod="openshift-ingress/router-default-66fc76d68b-5wp97"
Sep 10 22:02:28.815607 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:28.815536    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/97c76d14-c853-48fa-bb5c-b7c553b4f899-service-ca-bundle\") pod \"router-default-66fc76d68b-5wp97\" (UID: \"97c76d14-c853-48fa-bb5c-b7c553b4f899\") " pod="openshift-ingress/router-default-66fc76d68b-5wp97"
Sep 10 22:02:28.815607 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:28.815581    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8517c9a7-382a-40fd-bc52-e6da90ebc584-serving-cert\") pod \"console-operator-696884fb9c-w6vlb\" (UID: \"8517c9a7-382a-40fd-bc52-e6da90ebc584\") " pod="openshift-console-operator/console-operator-696884fb9c-w6vlb"
Sep 10 22:02:28.815722 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:28.815624    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8517c9a7-382a-40fd-bc52-e6da90ebc584-trusted-ca\") pod \"console-operator-696884fb9c-w6vlb\" (UID: \"8517c9a7-382a-40fd-bc52-e6da90ebc584\") " pod="openshift-console-operator/console-operator-696884fb9c-w6vlb"
Sep 10 22:02:28.815722 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:28.815702    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/97c76d14-c853-48fa-bb5c-b7c553b4f899-metrics-certs\") pod \"router-default-66fc76d68b-5wp97\" (UID: \"97c76d14-c853-48fa-bb5c-b7c553b4f899\") " pod="openshift-ingress/router-default-66fc76d68b-5wp97"
Sep 10 22:02:28.815814 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:28.815780    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8517c9a7-382a-40fd-bc52-e6da90ebc584-config\") pod \"console-operator-696884fb9c-w6vlb\" (UID: \"8517c9a7-382a-40fd-bc52-e6da90ebc584\") " pod="openshift-console-operator/console-operator-696884fb9c-w6vlb"
Sep 10 22:02:28.815986 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:02:28.815966    2582 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 10 22:02:28.816060 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:02:28.816038    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/97c76d14-c853-48fa-bb5c-b7c553b4f899-metrics-certs podName:97c76d14-c853-48fa-bb5c-b7c553b4f899 nodeName:}" failed. No retries permitted until 2026-09-10 22:02:29.316017078 +0000 UTC m=+70.272084780 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/97c76d14-c853-48fa-bb5c-b7c553b4f899-metrics-certs") pod "router-default-66fc76d68b-5wp97" (UID: "97c76d14-c853-48fa-bb5c-b7c553b4f899") : secret "router-metrics-certs-default" not found
Sep 10 22:02:28.816124 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:02:28.816068    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/97c76d14-c853-48fa-bb5c-b7c553b4f899-service-ca-bundle podName:97c76d14-c853-48fa-bb5c-b7c553b4f899 nodeName:}" failed. No retries permitted until 2026-09-10 22:02:29.316055504 +0000 UTC m=+70.272123195 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/97c76d14-c853-48fa-bb5c-b7c553b4f899-service-ca-bundle") pod "router-default-66fc76d68b-5wp97" (UID: "97c76d14-c853-48fa-bb5c-b7c553b4f899") : configmap references non-existent config key: service-ca.crt
Sep 10 22:02:28.816559 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:28.816536    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8517c9a7-382a-40fd-bc52-e6da90ebc584-config\") pod \"console-operator-696884fb9c-w6vlb\" (UID: \"8517c9a7-382a-40fd-bc52-e6da90ebc584\") " pod="openshift-console-operator/console-operator-696884fb9c-w6vlb"
Sep 10 22:02:28.816640 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:28.816616    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8517c9a7-382a-40fd-bc52-e6da90ebc584-trusted-ca\") pod \"console-operator-696884fb9c-w6vlb\" (UID: \"8517c9a7-382a-40fd-bc52-e6da90ebc584\") " pod="openshift-console-operator/console-operator-696884fb9c-w6vlb"
Sep 10 22:02:28.818149 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:28.818120    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/97c76d14-c853-48fa-bb5c-b7c553b4f899-default-certificate\") pod \"router-default-66fc76d68b-5wp97\" (UID: \"97c76d14-c853-48fa-bb5c-b7c553b4f899\") " pod="openshift-ingress/router-default-66fc76d68b-5wp97"
Sep 10 22:02:28.818446 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:28.818424    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/97c76d14-c853-48fa-bb5c-b7c553b4f899-stats-auth\") pod \"router-default-66fc76d68b-5wp97\" (UID: \"97c76d14-c853-48fa-bb5c-b7c553b4f899\") " pod="openshift-ingress/router-default-66fc76d68b-5wp97"
Sep 10 22:02:28.818498 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:28.818485    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8517c9a7-382a-40fd-bc52-e6da90ebc584-serving-cert\") pod \"console-operator-696884fb9c-w6vlb\" (UID: \"8517c9a7-382a-40fd-bc52-e6da90ebc584\") " pod="openshift-console-operator/console-operator-696884fb9c-w6vlb"
Sep 10 22:02:28.826130 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:28.826107    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqvh8\" (UniqueName: \"kubernetes.io/projected/97c76d14-c853-48fa-bb5c-b7c553b4f899-kube-api-access-mqvh8\") pod \"router-default-66fc76d68b-5wp97\" (UID: \"97c76d14-c853-48fa-bb5c-b7c553b4f899\") " pod="openshift-ingress/router-default-66fc76d68b-5wp97"
Sep 10 22:02:28.826351 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:28.826331    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zf9l6\" (UniqueName: \"kubernetes.io/projected/8517c9a7-382a-40fd-bc52-e6da90ebc584-kube-api-access-zf9l6\") pod \"console-operator-696884fb9c-w6vlb\" (UID: \"8517c9a7-382a-40fd-bc52-e6da90ebc584\") " pod="openshift-console-operator/console-operator-696884fb9c-w6vlb"
Sep 10 22:02:28.868374 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:28.868336    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-bhv4x"
Sep 10 22:02:28.966587 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:28.966506    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-w6vlb"
Sep 10 22:02:29.118634 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:29.118596    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a121acc4-9c7f-4dca-8d25-29cce37120a7-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-xl6k8\" (UID: \"a121acc4-9c7f-4dca-8d25-29cce37120a7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-xl6k8"
Sep 10 22:02:29.118822 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:02:29.118749    2582 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 10 22:02:29.118895 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:02:29.118824    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a121acc4-9c7f-4dca-8d25-29cce37120a7-samples-operator-tls podName:a121acc4-9c7f-4dca-8d25-29cce37120a7 nodeName:}" failed. No retries permitted until 2026-09-10 22:02:30.118806085 +0000 UTC m=+71.074873787 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/a121acc4-9c7f-4dca-8d25-29cce37120a7-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-xl6k8" (UID: "a121acc4-9c7f-4dca-8d25-29cce37120a7") : secret "samples-operator-tls" not found
Sep 10 22:02:29.320643 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:29.320544    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/97c76d14-c853-48fa-bb5c-b7c553b4f899-service-ca-bundle\") pod \"router-default-66fc76d68b-5wp97\" (UID: \"97c76d14-c853-48fa-bb5c-b7c553b4f899\") " pod="openshift-ingress/router-default-66fc76d68b-5wp97"
Sep 10 22:02:29.320643 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:29.320605    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/97c76d14-c853-48fa-bb5c-b7c553b4f899-metrics-certs\") pod \"router-default-66fc76d68b-5wp97\" (UID: \"97c76d14-c853-48fa-bb5c-b7c553b4f899\") " pod="openshift-ingress/router-default-66fc76d68b-5wp97"
Sep 10 22:02:29.320887 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:02:29.320744    2582 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 10 22:02:29.320887 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:02:29.320748    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/97c76d14-c853-48fa-bb5c-b7c553b4f899-service-ca-bundle podName:97c76d14-c853-48fa-bb5c-b7c553b4f899 nodeName:}" failed. No retries permitted until 2026-09-10 22:02:30.320724216 +0000 UTC m=+71.276791900 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/97c76d14-c853-48fa-bb5c-b7c553b4f899-service-ca-bundle") pod "router-default-66fc76d68b-5wp97" (UID: "97c76d14-c853-48fa-bb5c-b7c553b4f899") : configmap references non-existent config key: service-ca.crt
Sep 10 22:02:29.320887 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:02:29.320799    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/97c76d14-c853-48fa-bb5c-b7c553b4f899-metrics-certs podName:97c76d14-c853-48fa-bb5c-b7c553b4f899 nodeName:}" failed. No retries permitted until 2026-09-10 22:02:30.320783371 +0000 UTC m=+71.276851054 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/97c76d14-c853-48fa-bb5c-b7c553b4f899-metrics-certs") pod "router-default-66fc76d68b-5wp97" (UID: "97c76d14-c853-48fa-bb5c-b7c553b4f899") : secret "router-metrics-certs-default" not found
Sep 10 22:02:30.127794 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:30.127749    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a121acc4-9c7f-4dca-8d25-29cce37120a7-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-xl6k8\" (UID: \"a121acc4-9c7f-4dca-8d25-29cce37120a7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-xl6k8"
Sep 10 22:02:30.128300 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:02:30.127972    2582 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 10 22:02:30.128300 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:02:30.128060    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a121acc4-9c7f-4dca-8d25-29cce37120a7-samples-operator-tls podName:a121acc4-9c7f-4dca-8d25-29cce37120a7 nodeName:}" failed. No retries permitted until 2026-09-10 22:02:32.128038632 +0000 UTC m=+73.084106317 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/a121acc4-9c7f-4dca-8d25-29cce37120a7-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-xl6k8" (UID: "a121acc4-9c7f-4dca-8d25-29cce37120a7") : secret "samples-operator-tls" not found
Sep 10 22:02:30.329690 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:30.329651    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/97c76d14-c853-48fa-bb5c-b7c553b4f899-service-ca-bundle\") pod \"router-default-66fc76d68b-5wp97\" (UID: \"97c76d14-c853-48fa-bb5c-b7c553b4f899\") " pod="openshift-ingress/router-default-66fc76d68b-5wp97"
Sep 10 22:02:30.329955 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:30.329714    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/97c76d14-c853-48fa-bb5c-b7c553b4f899-metrics-certs\") pod \"router-default-66fc76d68b-5wp97\" (UID: \"97c76d14-c853-48fa-bb5c-b7c553b4f899\") " pod="openshift-ingress/router-default-66fc76d68b-5wp97"
Sep 10 22:02:30.329955 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:02:30.329839    2582 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 10 22:02:30.329955 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:02:30.329875    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/97c76d14-c853-48fa-bb5c-b7c553b4f899-service-ca-bundle podName:97c76d14-c853-48fa-bb5c-b7c553b4f899 nodeName:}" failed. No retries permitted until 2026-09-10 22:02:32.329841611 +0000 UTC m=+73.285909295 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/97c76d14-c853-48fa-bb5c-b7c553b4f899-service-ca-bundle") pod "router-default-66fc76d68b-5wp97" (UID: "97c76d14-c853-48fa-bb5c-b7c553b4f899") : configmap references non-existent config key: service-ca.crt
Sep 10 22:02:30.329955 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:02:30.329901    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/97c76d14-c853-48fa-bb5c-b7c553b4f899-metrics-certs podName:97c76d14-c853-48fa-bb5c-b7c553b4f899 nodeName:}" failed. No retries permitted until 2026-09-10 22:02:32.32989375 +0000 UTC m=+73.285961433 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/97c76d14-c853-48fa-bb5c-b7c553b4f899-metrics-certs") pod "router-default-66fc76d68b-5wp97" (UID: "97c76d14-c853-48fa-bb5c-b7c553b4f899") : secret "router-metrics-certs-default" not found
Sep 10 22:02:30.761566 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:30.761490    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-bhv4x"]
Sep 10 22:02:30.764407 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:30.764340    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-w6vlb"]
Sep 10 22:02:30.765253 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:02:30.765216    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3aaaf117_e959_49e2_93f0_342a657439c5.slice/crio-2cf302f92d4492ed803d7d44217f5512293760a541fd8870c59494c5768a2e70 WatchSource:0}: Error finding container 2cf302f92d4492ed803d7d44217f5512293760a541fd8870c59494c5768a2e70: Status 404 returned error can't find the container with id 2cf302f92d4492ed803d7d44217f5512293760a541fd8870c59494c5768a2e70
Sep 10 22:02:30.765442 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:30.765381    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-49png"]
Sep 10 22:02:30.770072 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:02:30.770041    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14d6dc12_a6d1_4fce_999a_102fcd5100fe.slice/crio-95f2d76a3ae7bbfb6ba9c510674eec6b190640c095bfa5d976e75a4f86511e94 WatchSource:0}: Error finding container 95f2d76a3ae7bbfb6ba9c510674eec6b190640c095bfa5d976e75a4f86511e94: Status 404 returned error can't find the container with id 95f2d76a3ae7bbfb6ba9c510674eec6b190640c095bfa5d976e75a4f86511e94
Sep 10 22:02:30.770366 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:02:30.770335    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8517c9a7_382a_40fd_bc52_e6da90ebc584.slice/crio-49c59fa0e0c0a8f3d05bd6ca1afbcab579ffc15646e572881c5fa1c460b8d5fd WatchSource:0}: Error finding container 49c59fa0e0c0a8f3d05bd6ca1afbcab579ffc15646e572881c5fa1c460b8d5fd: Status 404 returned error can't find the container with id 49c59fa0e0c0a8f3d05bd6ca1afbcab579ffc15646e572881c5fa1c460b8d5fd
Sep 10 22:02:30.885193 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:30.885158    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-w6vlb" event={"ID":"8517c9a7-382a-40fd-bc52-e6da90ebc584","Type":"ContainerStarted","Data":"49c59fa0e0c0a8f3d05bd6ca1afbcab579ffc15646e572881c5fa1c460b8d5fd"}
Sep 10 22:02:30.886809 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:30.886786    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-rrm25" event={"ID":"064a91af-7e3d-4e71-b743-04d68d63174e","Type":"ContainerStarted","Data":"11c73bf07817472403167b390baeebda4308eb52d20343e3b82be44dce526671"}
Sep 10 22:02:30.886948 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:30.886883    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-rrm25"
Sep 10 22:02:30.888438 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:30.888415    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-n9r9x" event={"ID":"0d6f601b-532d-4f1f-9580-977997448ea9","Type":"ContainerStarted","Data":"d42bb01863d84f41590822192f3a97141422cb16b58ee0eccaeb84cd900444a3"}
Sep 10 22:02:30.889967 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:30.889940    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-49png" event={"ID":"14d6dc12-a6d1-4fce-999a-102fcd5100fe","Type":"ContainerStarted","Data":"95f2d76a3ae7bbfb6ba9c510674eec6b190640c095bfa5d976e75a4f86511e94"}
Sep 10 22:02:30.891024 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:30.891006    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-bhv4x" event={"ID":"3aaaf117-e959-49e2-93f0-342a657439c5","Type":"ContainerStarted","Data":"2cf302f92d4492ed803d7d44217f5512293760a541fd8870c59494c5768a2e70"}
Sep 10 22:02:30.906439 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:30.906372    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-rrm25" podStartSLOduration=67.31303732 podStartE2EDuration="1m11.906357354s" totalImagesPullingTime="4.593320034s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 22:01:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:02:30.906208399 +0000 UTC m=+71.862276106" watchObservedRunningTime="2026-09-10 22:02:30.906357354 +0000 UTC m=+71.862425062"
Sep 10 22:02:30.921352 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:30.921311    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-n9r9x" podStartSLOduration=67.008043561 podStartE2EDuration="1m11.921296153s" totalImagesPullingTime="4.913252592s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 22:01:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:02:30.920475077 +0000 UTC m=+71.876542782" watchObservedRunningTime="2026-09-10 22:02:30.921296153 +0000 UTC m=+71.877363858"
Sep 10 22:02:31.895941 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:31.895900    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-49png" event={"ID":"14d6dc12-a6d1-4fce-999a-102fcd5100fe","Type":"ContainerStarted","Data":"e9ebc51cabc78c09d097189a1320ac4f33c414ee809c71ed2909f6f3fe9ab993"}
Sep 10 22:02:31.915441 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:31.915384    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f8b768455-49png" podStartSLOduration=3.915364721 podStartE2EDuration="3.915364721s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 22:02:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:02:31.914195548 +0000 UTC m=+72.870263255" watchObservedRunningTime="2026-09-10 22:02:31.915364721 +0000 UTC m=+72.871432428"
Sep 10 22:02:32.145911 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:32.145863    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a121acc4-9c7f-4dca-8d25-29cce37120a7-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-xl6k8\" (UID: \"a121acc4-9c7f-4dca-8d25-29cce37120a7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-xl6k8"
Sep 10 22:02:32.146134 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:02:32.146051    2582 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 10 22:02:32.146134 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:02:32.146132    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a121acc4-9c7f-4dca-8d25-29cce37120a7-samples-operator-tls podName:a121acc4-9c7f-4dca-8d25-29cce37120a7 nodeName:}" failed. No retries permitted until 2026-09-10 22:02:36.146111051 +0000 UTC m=+77.102178738 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/a121acc4-9c7f-4dca-8d25-29cce37120a7-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-xl6k8" (UID: "a121acc4-9c7f-4dca-8d25-29cce37120a7") : secret "samples-operator-tls" not found
Sep 10 22:02:32.310614 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:32.309819    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-hcz8r"]
Sep 10 22:02:32.314014 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:32.312938    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-hcz8r"
Sep 10 22:02:32.315844 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:32.315765    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-682x9\""
Sep 10 22:02:32.316399 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:32.316377    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Sep 10 22:02:32.316514 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:32.316483    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Sep 10 22:02:32.325188 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:32.325165    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-hcz8r"]
Sep 10 22:02:32.348062 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:32.348029    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/97c76d14-c853-48fa-bb5c-b7c553b4f899-service-ca-bundle\") pod \"router-default-66fc76d68b-5wp97\" (UID: \"97c76d14-c853-48fa-bb5c-b7c553b4f899\") " pod="openshift-ingress/router-default-66fc76d68b-5wp97"
Sep 10 22:02:32.348228 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:32.348081    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6bx7\" (UniqueName: \"kubernetes.io/projected/759ec1b7-bee2-429e-aac1-2256e6e8dea4-kube-api-access-x6bx7\") pod \"migrator-8b4d697c5-hcz8r\" (UID: \"759ec1b7-bee2-429e-aac1-2256e6e8dea4\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-hcz8r"
Sep 10 22:02:32.348228 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:32.348148    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/97c76d14-c853-48fa-bb5c-b7c553b4f899-metrics-certs\") pod \"router-default-66fc76d68b-5wp97\" (UID: \"97c76d14-c853-48fa-bb5c-b7c553b4f899\") " pod="openshift-ingress/router-default-66fc76d68b-5wp97"
Sep 10 22:02:32.348330 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:02:32.348225    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/97c76d14-c853-48fa-bb5c-b7c553b4f899-service-ca-bundle podName:97c76d14-c853-48fa-bb5c-b7c553b4f899 nodeName:}" failed. No retries permitted until 2026-09-10 22:02:36.348201931 +0000 UTC m=+77.304269634 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/97c76d14-c853-48fa-bb5c-b7c553b4f899-service-ca-bundle") pod "router-default-66fc76d68b-5wp97" (UID: "97c76d14-c853-48fa-bb5c-b7c553b4f899") : configmap references non-existent config key: service-ca.crt
Sep 10 22:02:32.348330 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:02:32.348263    2582 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 10 22:02:32.348330 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:02:32.348312    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/97c76d14-c853-48fa-bb5c-b7c553b4f899-metrics-certs podName:97c76d14-c853-48fa-bb5c-b7c553b4f899 nodeName:}" failed. No retries permitted until 2026-09-10 22:02:36.348296687 +0000 UTC m=+77.304364393 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/97c76d14-c853-48fa-bb5c-b7c553b4f899-metrics-certs") pod "router-default-66fc76d68b-5wp97" (UID: "97c76d14-c853-48fa-bb5c-b7c553b4f899") : secret "router-metrics-certs-default" not found
Sep 10 22:02:32.449719 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:32.449677    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6bx7\" (UniqueName: \"kubernetes.io/projected/759ec1b7-bee2-429e-aac1-2256e6e8dea4-kube-api-access-x6bx7\") pod \"migrator-8b4d697c5-hcz8r\" (UID: \"759ec1b7-bee2-429e-aac1-2256e6e8dea4\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-hcz8r"
Sep 10 22:02:32.463302 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:32.463268    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6bx7\" (UniqueName: \"kubernetes.io/projected/759ec1b7-bee2-429e-aac1-2256e6e8dea4-kube-api-access-x6bx7\") pod \"migrator-8b4d697c5-hcz8r\" (UID: \"759ec1b7-bee2-429e-aac1-2256e6e8dea4\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-hcz8r"
Sep 10 22:02:32.625412 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:32.625370    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-hcz8r"
Sep 10 22:02:33.393918 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:33.393837    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-hcz8r"]
Sep 10 22:02:33.396958 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:02:33.396930    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod759ec1b7_bee2_429e_aac1_2256e6e8dea4.slice/crio-6b930b3c5049c643b0c8851194cda0129d81976510e5253a34d70eb25be1d153 WatchSource:0}: Error finding container 6b930b3c5049c643b0c8851194cda0129d81976510e5253a34d70eb25be1d153: Status 404 returned error can't find the container with id 6b930b3c5049c643b0c8851194cda0129d81976510e5253a34d70eb25be1d153
Sep 10 22:02:33.905199 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:33.905100    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-bhv4x" event={"ID":"3aaaf117-e959-49e2-93f0-342a657439c5","Type":"ContainerStarted","Data":"df9f7d8ebdb7af48f463bbdceb6841a29c4b02b07ba5d5e3618255ebbffa08a8"}
Sep 10 22:02:33.906996 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:33.906976    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-w6vlb_8517c9a7-382a-40fd-bc52-e6da90ebc584/console-operator/0.log"
Sep 10 22:02:33.907150 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:33.907013    2582 generic.go:356] "Generic (PLEG): container finished" podID="8517c9a7-382a-40fd-bc52-e6da90ebc584" containerID="e4641124cfd3fd47bc8bea4f1dce93b4ef69a2ce8192b3974493cbd5586c9c7b" exitCode=255
Sep 10 22:02:33.907150 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:33.907080    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-w6vlb" event={"ID":"8517c9a7-382a-40fd-bc52-e6da90ebc584","Type":"ContainerDied","Data":"e4641124cfd3fd47bc8bea4f1dce93b4ef69a2ce8192b3974493cbd5586c9c7b"}
Sep 10 22:02:33.907331 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:33.907308    2582 scope.go:117] "RemoveContainer" containerID="e4641124cfd3fd47bc8bea4f1dce93b4ef69a2ce8192b3974493cbd5586c9c7b"
Sep 10 22:02:33.908707 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:33.908688    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-hcz8r" event={"ID":"759ec1b7-bee2-429e-aac1-2256e6e8dea4","Type":"ContainerStarted","Data":"6b930b3c5049c643b0c8851194cda0129d81976510e5253a34d70eb25be1d153"}
Sep 10 22:02:33.923609 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:33.923560    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5844b856db-bhv4x" podStartSLOduration=3.561406324 podStartE2EDuration="5.923541878s" totalImagesPullingTime="2.362135554s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 22:02:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:02:33.922133669 +0000 UTC m=+74.878201376" watchObservedRunningTime="2026-09-10 22:02:33.923541878 +0000 UTC m=+74.879609585"
Sep 10 22:02:34.913657 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:34.913625    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-w6vlb_8517c9a7-382a-40fd-bc52-e6da90ebc584/console-operator/1.log"
Sep 10 22:02:34.914117 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:34.914088    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-w6vlb_8517c9a7-382a-40fd-bc52-e6da90ebc584/console-operator/0.log"
Sep 10 22:02:34.914188 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:34.914130    2582 generic.go:356] "Generic (PLEG): container finished" podID="8517c9a7-382a-40fd-bc52-e6da90ebc584" containerID="3013de141d8b302036e3dbf993be8ebac9be10d7a50f9c4fd99feb063d168166" exitCode=255
Sep 10 22:02:34.914253 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:34.914214    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-w6vlb" event={"ID":"8517c9a7-382a-40fd-bc52-e6da90ebc584","Type":"ContainerDied","Data":"3013de141d8b302036e3dbf993be8ebac9be10d7a50f9c4fd99feb063d168166"}
Sep 10 22:02:34.914313 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:34.914297    2582 scope.go:117] "RemoveContainer" containerID="e4641124cfd3fd47bc8bea4f1dce93b4ef69a2ce8192b3974493cbd5586c9c7b"
Sep 10 22:02:34.914458 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:34.914441    2582 scope.go:117] "RemoveContainer" containerID="3013de141d8b302036e3dbf993be8ebac9be10d7a50f9c4fd99feb063d168166"
Sep 10 22:02:34.914654 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:02:34.914633    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-w6vlb_openshift-console-operator(8517c9a7-382a-40fd-bc52-e6da90ebc584)\"" pod="openshift-console-operator/console-operator-696884fb9c-w6vlb" podUID="8517c9a7-382a-40fd-bc52-e6da90ebc584"
Sep 10 22:02:34.916072 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:34.916044    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-hcz8r" event={"ID":"759ec1b7-bee2-429e-aac1-2256e6e8dea4","Type":"ContainerStarted","Data":"9a0bf85e43a99037f0c29c5ec22490afa1af6e72210f00c0fd8785663552394f"}
Sep 10 22:02:34.916181 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:34.916093    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-hcz8r" event={"ID":"759ec1b7-bee2-429e-aac1-2256e6e8dea4","Type":"ContainerStarted","Data":"e137297207da02741aa34e4312f01b709dbab5505319a271f9436e33a548c638"}
Sep 10 22:02:34.950116 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:34.950058    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-hcz8r" podStartSLOduration=1.89208927 podStartE2EDuration="2.950035759s" totalImagesPullingTime="1.057946489s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 22:02:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:02:34.948741253 +0000 UTC m=+75.904808958" watchObservedRunningTime="2026-09-10 22:02:34.950035759 +0000 UTC m=+75.906103467"
Sep 10 22:02:35.038504 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:35.038469    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-hrpd5"]
Sep 10 22:02:35.041134 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:35.041118    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-hrpd5"
Sep 10 22:02:35.042891 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:35.042863    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 10 22:02:35.043096 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:35.043072    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-fvnxt\""
Sep 10 22:02:35.043180 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:35.043094    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 10 22:02:35.059030 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:35.058996    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-hrpd5"]
Sep 10 22:02:35.071322 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:35.071298    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/29d71552-94a7-4491-aced-1141665a8b0b-data-volume\") pod \"insights-runtime-extractor-hrpd5\" (UID: \"29d71552-94a7-4491-aced-1141665a8b0b\") " pod="openshift-insights/insights-runtime-extractor-hrpd5"
Sep 10 22:02:35.071423 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:35.071348    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/29d71552-94a7-4491-aced-1141665a8b0b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hrpd5\" (UID: \"29d71552-94a7-4491-aced-1141665a8b0b\") " pod="openshift-insights/insights-runtime-extractor-hrpd5"
Sep 10 22:02:35.071423 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:35.071383    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/29d71552-94a7-4491-aced-1141665a8b0b-crio-socket\") pod \"insights-runtime-extractor-hrpd5\" (UID: \"29d71552-94a7-4491-aced-1141665a8b0b\") " pod="openshift-insights/insights-runtime-extractor-hrpd5"
Sep 10 22:02:35.071510 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:35.071491    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wltfm\" (UniqueName: \"kubernetes.io/projected/29d71552-94a7-4491-aced-1141665a8b0b-kube-api-access-wltfm\") pod \"insights-runtime-extractor-hrpd5\" (UID: \"29d71552-94a7-4491-aced-1141665a8b0b\") " pod="openshift-insights/insights-runtime-extractor-hrpd5"
Sep 10 22:02:35.071685 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:35.071522    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/29d71552-94a7-4491-aced-1141665a8b0b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hrpd5\" (UID: \"29d71552-94a7-4491-aced-1141665a8b0b\") " pod="openshift-insights/insights-runtime-extractor-hrpd5"
Sep 10 22:02:35.171955 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:35.171829    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wltfm\" (UniqueName: \"kubernetes.io/projected/29d71552-94a7-4491-aced-1141665a8b0b-kube-api-access-wltfm\") pod \"insights-runtime-extractor-hrpd5\" (UID: \"29d71552-94a7-4491-aced-1141665a8b0b\") " pod="openshift-insights/insights-runtime-extractor-hrpd5"
Sep 10 22:02:35.171955 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:35.171898    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/29d71552-94a7-4491-aced-1141665a8b0b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hrpd5\" (UID: \"29d71552-94a7-4491-aced-1141665a8b0b\") " pod="openshift-insights/insights-runtime-extractor-hrpd5"
Sep 10 22:02:35.171955 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:35.171945    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/29d71552-94a7-4491-aced-1141665a8b0b-data-volume\") pod \"insights-runtime-extractor-hrpd5\" (UID: \"29d71552-94a7-4491-aced-1141665a8b0b\") " pod="openshift-insights/insights-runtime-extractor-hrpd5"
Sep 10 22:02:35.172201 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:02:35.172064    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 22:02:35.172201 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:02:35.172114    2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 10 22:02:35.172201 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:02:35.172147    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29d71552-94a7-4491-aced-1141665a8b0b-insights-runtime-extractor-tls podName:29d71552-94a7-4491-aced-1141665a8b0b nodeName:}" failed. No retries permitted until 2026-09-10 22:02:35.672123997 +0000 UTC m=+76.628191703 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/29d71552-94a7-4491-aced-1141665a8b0b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hrpd5" (UID: "29d71552-94a7-4491-aced-1141665a8b0b") : secret "insights-runtime-extractor-tls" not found
Sep 10 22:02:35.172201 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:02:35.172173    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/29d71552-94a7-4491-aced-1141665a8b0b-kube-rbac-proxy-cm podName:29d71552-94a7-4491-aced-1141665a8b0b nodeName:}" failed. No retries permitted until 2026-09-10 22:02:35.672158807 +0000 UTC m=+76.628226492 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/29d71552-94a7-4491-aced-1141665a8b0b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hrpd5" (UID: "29d71552-94a7-4491-aced-1141665a8b0b") : configmap "kube-rbac-proxy" not found
Sep 10 22:02:35.172201 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:35.172065    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/29d71552-94a7-4491-aced-1141665a8b0b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hrpd5\" (UID: \"29d71552-94a7-4491-aced-1141665a8b0b\") " pod="openshift-insights/insights-runtime-extractor-hrpd5"
Sep 10 22:02:35.172388 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:35.172252    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/29d71552-94a7-4491-aced-1141665a8b0b-crio-socket\") pod \"insights-runtime-extractor-hrpd5\" (UID: \"29d71552-94a7-4491-aced-1141665a8b0b\") " pod="openshift-insights/insights-runtime-extractor-hrpd5"
Sep 10 22:02:35.172388 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:35.172305    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/29d71552-94a7-4491-aced-1141665a8b0b-data-volume\") pod \"insights-runtime-extractor-hrpd5\" (UID: \"29d71552-94a7-4491-aced-1141665a8b0b\") " pod="openshift-insights/insights-runtime-extractor-hrpd5"
Sep 10 22:02:35.172388 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:35.172331    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/29d71552-94a7-4491-aced-1141665a8b0b-crio-socket\") pod \"insights-runtime-extractor-hrpd5\" (UID: \"29d71552-94a7-4491-aced-1141665a8b0b\") " pod="openshift-insights/insights-runtime-extractor-hrpd5"
Sep 10 22:02:35.188729 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:35.188702    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wltfm\" (UniqueName: \"kubernetes.io/projected/29d71552-94a7-4491-aced-1141665a8b0b-kube-api-access-wltfm\") pod \"insights-runtime-extractor-hrpd5\" (UID: \"29d71552-94a7-4491-aced-1141665a8b0b\") " pod="openshift-insights/insights-runtime-extractor-hrpd5"
Sep 10 22:02:35.679257 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:35.679215    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/29d71552-94a7-4491-aced-1141665a8b0b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hrpd5\" (UID: \"29d71552-94a7-4491-aced-1141665a8b0b\") " pod="openshift-insights/insights-runtime-extractor-hrpd5"
Sep 10 22:02:35.679458 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:35.679302    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/29d71552-94a7-4491-aced-1141665a8b0b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hrpd5\" (UID: \"29d71552-94a7-4491-aced-1141665a8b0b\") " pod="openshift-insights/insights-runtime-extractor-hrpd5"
Sep 10 22:02:35.679458 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:02:35.679362    2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 10 22:02:35.679458 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:02:35.679409    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 22:02:35.679458 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:02:35.679444    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/29d71552-94a7-4491-aced-1141665a8b0b-kube-rbac-proxy-cm podName:29d71552-94a7-4491-aced-1141665a8b0b nodeName:}" failed. No retries permitted until 2026-09-10 22:02:36.679423546 +0000 UTC m=+77.635491250 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/29d71552-94a7-4491-aced-1141665a8b0b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hrpd5" (UID: "29d71552-94a7-4491-aced-1141665a8b0b") : configmap "kube-rbac-proxy" not found
Sep 10 22:02:35.679731 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:02:35.679464    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29d71552-94a7-4491-aced-1141665a8b0b-insights-runtime-extractor-tls podName:29d71552-94a7-4491-aced-1141665a8b0b nodeName:}" failed. No retries permitted until 2026-09-10 22:02:36.679454603 +0000 UTC m=+77.635522291 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/29d71552-94a7-4491-aced-1141665a8b0b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hrpd5" (UID: "29d71552-94a7-4491-aced-1141665a8b0b") : secret "insights-runtime-extractor-tls" not found
Sep 10 22:02:35.921320 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:35.921291    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-w6vlb_8517c9a7-382a-40fd-bc52-e6da90ebc584/console-operator/1.log"
Sep 10 22:02:35.921777 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:35.921763    2582 scope.go:117] "RemoveContainer" containerID="3013de141d8b302036e3dbf993be8ebac9be10d7a50f9c4fd99feb063d168166"
Sep 10 22:02:35.921935 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:02:35.921917    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-w6vlb_openshift-console-operator(8517c9a7-382a-40fd-bc52-e6da90ebc584)\"" pod="openshift-console-operator/console-operator-696884fb9c-w6vlb" podUID="8517c9a7-382a-40fd-bc52-e6da90ebc584"
Sep 10 22:02:36.184517 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:36.184479    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a121acc4-9c7f-4dca-8d25-29cce37120a7-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-xl6k8\" (UID: \"a121acc4-9c7f-4dca-8d25-29cce37120a7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-xl6k8"
Sep 10 22:02:36.184681 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:02:36.184643    2582 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 10 22:02:36.184721 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:02:36.184714    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a121acc4-9c7f-4dca-8d25-29cce37120a7-samples-operator-tls podName:a121acc4-9c7f-4dca-8d25-29cce37120a7 nodeName:}" failed. No retries permitted until 2026-09-10 22:02:44.184697304 +0000 UTC m=+85.140764987 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/a121acc4-9c7f-4dca-8d25-29cce37120a7-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-xl6k8" (UID: "a121acc4-9c7f-4dca-8d25-29cce37120a7") : secret "samples-operator-tls" not found
Sep 10 22:02:36.386255 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:36.386218    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/97c76d14-c853-48fa-bb5c-b7c553b4f899-metrics-certs\") pod \"router-default-66fc76d68b-5wp97\" (UID: \"97c76d14-c853-48fa-bb5c-b7c553b4f899\") " pod="openshift-ingress/router-default-66fc76d68b-5wp97"
Sep 10 22:02:36.386409 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:02:36.386369    2582 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 10 22:02:36.386409 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:36.386399    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/97c76d14-c853-48fa-bb5c-b7c553b4f899-service-ca-bundle\") pod \"router-default-66fc76d68b-5wp97\" (UID: \"97c76d14-c853-48fa-bb5c-b7c553b4f899\") " pod="openshift-ingress/router-default-66fc76d68b-5wp97"
Sep 10 22:02:36.386483 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:02:36.386431    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/97c76d14-c853-48fa-bb5c-b7c553b4f899-metrics-certs podName:97c76d14-c853-48fa-bb5c-b7c553b4f899 nodeName:}" failed. No retries permitted until 2026-09-10 22:02:44.386414612 +0000 UTC m=+85.342482296 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/97c76d14-c853-48fa-bb5c-b7c553b4f899-metrics-certs") pod "router-default-66fc76d68b-5wp97" (UID: "97c76d14-c853-48fa-bb5c-b7c553b4f899") : secret "router-metrics-certs-default" not found
Sep 10 22:02:36.386524 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:02:36.386509    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/97c76d14-c853-48fa-bb5c-b7c553b4f899-service-ca-bundle podName:97c76d14-c853-48fa-bb5c-b7c553b4f899 nodeName:}" failed. No retries permitted until 2026-09-10 22:02:44.386497579 +0000 UTC m=+85.342565263 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/97c76d14-c853-48fa-bb5c-b7c553b4f899-service-ca-bundle") pod "router-default-66fc76d68b-5wp97" (UID: "97c76d14-c853-48fa-bb5c-b7c553b4f899") : configmap references non-existent config key: service-ca.crt
Sep 10 22:02:36.689338 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:36.689311    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/29d71552-94a7-4491-aced-1141665a8b0b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hrpd5\" (UID: \"29d71552-94a7-4491-aced-1141665a8b0b\") " pod="openshift-insights/insights-runtime-extractor-hrpd5"
Sep 10 22:02:36.689480 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:36.689383    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/29d71552-94a7-4491-aced-1141665a8b0b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hrpd5\" (UID: \"29d71552-94a7-4491-aced-1141665a8b0b\") " pod="openshift-insights/insights-runtime-extractor-hrpd5"
Sep 10 22:02:36.689480 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:02:36.689447    2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 10 22:02:36.689480 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:02:36.689464    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 22:02:36.689582 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:02:36.689513    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/29d71552-94a7-4491-aced-1141665a8b0b-kube-rbac-proxy-cm podName:29d71552-94a7-4491-aced-1141665a8b0b nodeName:}" failed. No retries permitted until 2026-09-10 22:02:38.689498389 +0000 UTC m=+79.645566073 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/29d71552-94a7-4491-aced-1141665a8b0b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hrpd5" (UID: "29d71552-94a7-4491-aced-1141665a8b0b") : configmap "kube-rbac-proxy" not found
Sep 10 22:02:36.689582 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:02:36.689526    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29d71552-94a7-4491-aced-1141665a8b0b-insights-runtime-extractor-tls podName:29d71552-94a7-4491-aced-1141665a8b0b nodeName:}" failed. No retries permitted until 2026-09-10 22:02:38.689520156 +0000 UTC m=+79.645587841 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/29d71552-94a7-4491-aced-1141665a8b0b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hrpd5" (UID: "29d71552-94a7-4491-aced-1141665a8b0b") : secret "insights-runtime-extractor-tls" not found
Sep 10 22:02:38.001254 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:38.001227    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-ncrzz_16caf96d-d15f-45e5-b003-1aec5786ea58/dns-node-resolver/0.log"
Sep 10 22:02:38.705167 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:38.705126    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/29d71552-94a7-4491-aced-1141665a8b0b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hrpd5\" (UID: \"29d71552-94a7-4491-aced-1141665a8b0b\") " pod="openshift-insights/insights-runtime-extractor-hrpd5"
Sep 10 22:02:38.705326 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:02:38.705258    2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 10 22:02:38.705326 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:38.705281    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/29d71552-94a7-4491-aced-1141665a8b0b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hrpd5\" (UID: \"29d71552-94a7-4491-aced-1141665a8b0b\") " pod="openshift-insights/insights-runtime-extractor-hrpd5"
Sep 10 22:02:38.705436 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:02:38.705334    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/29d71552-94a7-4491-aced-1141665a8b0b-kube-rbac-proxy-cm podName:29d71552-94a7-4491-aced-1141665a8b0b nodeName:}" failed. No retries permitted until 2026-09-10 22:02:42.705318342 +0000 UTC m=+83.661386025 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/29d71552-94a7-4491-aced-1141665a8b0b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hrpd5" (UID: "29d71552-94a7-4491-aced-1141665a8b0b") : configmap "kube-rbac-proxy" not found
Sep 10 22:02:38.705436 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:02:38.705369    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 22:02:38.705436 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:02:38.705413    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29d71552-94a7-4491-aced-1141665a8b0b-insights-runtime-extractor-tls podName:29d71552-94a7-4491-aced-1141665a8b0b nodeName:}" failed. No retries permitted until 2026-09-10 22:02:42.70540158 +0000 UTC m=+83.661469264 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/29d71552-94a7-4491-aced-1141665a8b0b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hrpd5" (UID: "29d71552-94a7-4491-aced-1141665a8b0b") : secret "insights-runtime-extractor-tls" not found
Sep 10 22:02:38.800064 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:38.800038    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-74bb6_f430e7fc-0f5e-48e2-9ea6-f97c277cdb06/node-ca/0.log"
Sep 10 22:02:38.967719 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:38.967619    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-w6vlb"
Sep 10 22:02:38.967719 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:38.967686    2582 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-696884fb9c-w6vlb"
Sep 10 22:02:38.968036 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:38.968023    2582 scope.go:117] "RemoveContainer" containerID="3013de141d8b302036e3dbf993be8ebac9be10d7a50f9c4fd99feb063d168166"
Sep 10 22:02:38.968182 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:02:38.968165    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-w6vlb_openshift-console-operator(8517c9a7-382a-40fd-bc52-e6da90ebc584)\"" pod="openshift-console-operator/console-operator-696884fb9c-w6vlb" podUID="8517c9a7-382a-40fd-bc52-e6da90ebc584"
Sep 10 22:02:39.602203 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:39.602177    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-5p6sf_b8b665fe-9846-4157-8922-433c2e13f6fa/serve-healthcheck-canary/0.log"
Sep 10 22:02:40.003279 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:40.003246    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-hcz8r_759ec1b7-bee2-429e-aac1-2256e6e8dea4/migrator/0.log"
Sep 10 22:02:40.201033 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:40.201001    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-hcz8r_759ec1b7-bee2-429e-aac1-2256e6e8dea4/graceful-termination/0.log"
Sep 10 22:02:42.736694 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:42.736575    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/29d71552-94a7-4491-aced-1141665a8b0b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hrpd5\" (UID: \"29d71552-94a7-4491-aced-1141665a8b0b\") " pod="openshift-insights/insights-runtime-extractor-hrpd5"
Sep 10 22:02:42.736694 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:42.736640    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/29d71552-94a7-4491-aced-1141665a8b0b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hrpd5\" (UID: \"29d71552-94a7-4491-aced-1141665a8b0b\") " pod="openshift-insights/insights-runtime-extractor-hrpd5"
Sep 10 22:02:42.737096 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:02:42.736721    2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 10 22:02:42.737096 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:02:42.736745    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 22:02:42.737096 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:02:42.736785    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/29d71552-94a7-4491-aced-1141665a8b0b-kube-rbac-proxy-cm podName:29d71552-94a7-4491-aced-1141665a8b0b nodeName:}" failed. No retries permitted until 2026-09-10 22:02:50.736767628 +0000 UTC m=+91.692835326 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/29d71552-94a7-4491-aced-1141665a8b0b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hrpd5" (UID: "29d71552-94a7-4491-aced-1141665a8b0b") : configmap "kube-rbac-proxy" not found
Sep 10 22:02:42.737096 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:02:42.736803    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29d71552-94a7-4491-aced-1141665a8b0b-insights-runtime-extractor-tls podName:29d71552-94a7-4491-aced-1141665a8b0b nodeName:}" failed. No retries permitted until 2026-09-10 22:02:50.736793854 +0000 UTC m=+91.692861538 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/29d71552-94a7-4491-aced-1141665a8b0b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hrpd5" (UID: "29d71552-94a7-4491-aced-1141665a8b0b") : secret "insights-runtime-extractor-tls" not found
Sep 10 22:02:44.249324 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:44.249291    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a121acc4-9c7f-4dca-8d25-29cce37120a7-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-xl6k8\" (UID: \"a121acc4-9c7f-4dca-8d25-29cce37120a7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-xl6k8"
Sep 10 22:02:44.251766 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:44.251737    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a121acc4-9c7f-4dca-8d25-29cce37120a7-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-xl6k8\" (UID: \"a121acc4-9c7f-4dca-8d25-29cce37120a7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-xl6k8"
Sep 10 22:02:44.365455 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:44.365414    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-xl6k8"
Sep 10 22:02:44.451341 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:44.451310    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/97c76d14-c853-48fa-bb5c-b7c553b4f899-metrics-certs\") pod \"router-default-66fc76d68b-5wp97\" (UID: \"97c76d14-c853-48fa-bb5c-b7c553b4f899\") " pod="openshift-ingress/router-default-66fc76d68b-5wp97"
Sep 10 22:02:44.451522 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:44.451502    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/97c76d14-c853-48fa-bb5c-b7c553b4f899-service-ca-bundle\") pod \"router-default-66fc76d68b-5wp97\" (UID: \"97c76d14-c853-48fa-bb5c-b7c553b4f899\") " pod="openshift-ingress/router-default-66fc76d68b-5wp97"
Sep 10 22:02:44.452076 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:44.452052    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/97c76d14-c853-48fa-bb5c-b7c553b4f899-service-ca-bundle\") pod \"router-default-66fc76d68b-5wp97\" (UID: \"97c76d14-c853-48fa-bb5c-b7c553b4f899\") " pod="openshift-ingress/router-default-66fc76d68b-5wp97"
Sep 10 22:02:44.453695 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:44.453663    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/97c76d14-c853-48fa-bb5c-b7c553b4f899-metrics-certs\") pod \"router-default-66fc76d68b-5wp97\" (UID: \"97c76d14-c853-48fa-bb5c-b7c553b4f899\") " pod="openshift-ingress/router-default-66fc76d68b-5wp97"
Sep 10 22:02:44.573690 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:44.573613    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-66fc76d68b-5wp97"
Sep 10 22:02:44.633425 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:44.633398    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-xl6k8"]
Sep 10 22:02:44.841937 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:44.841811    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-66fc76d68b-5wp97"]
Sep 10 22:02:44.846109 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:02:44.846081    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97c76d14_c853_48fa_bb5c_b7c553b4f899.slice/crio-02023204c518bf56d37b6003a7229045f27d7f7d3d5a4699e6abb853e58f3694 WatchSource:0}: Error finding container 02023204c518bf56d37b6003a7229045f27d7f7d3d5a4699e6abb853e58f3694: Status 404 returned error can't find the container with id 02023204c518bf56d37b6003a7229045f27d7f7d3d5a4699e6abb853e58f3694
Sep 10 22:02:44.950891 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:44.950834    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-xl6k8" event={"ID":"a121acc4-9c7f-4dca-8d25-29cce37120a7","Type":"ContainerStarted","Data":"3c4dbf5d99dce8d3d0a38039aa3274b6b19fd0fbd83ac1ee280578c7121c7335"}
Sep 10 22:02:44.952258 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:44.952227    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-66fc76d68b-5wp97" event={"ID":"97c76d14-c853-48fa-bb5c-b7c553b4f899","Type":"ContainerStarted","Data":"818d288012dde419edf69136fc3a6950e2d88105bf64f617ac79a85926053a27"}
Sep 10 22:02:44.952378 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:44.952273    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-66fc76d68b-5wp97" event={"ID":"97c76d14-c853-48fa-bb5c-b7c553b4f899","Type":"ContainerStarted","Data":"02023204c518bf56d37b6003a7229045f27d7f7d3d5a4699e6abb853e58f3694"}
Sep 10 22:02:44.972729 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:44.972668    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-66fc76d68b-5wp97" podStartSLOduration=16.972642272 podStartE2EDuration="16.972642272s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 22:02:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:02:44.971228578 +0000 UTC m=+85.927296305" watchObservedRunningTime="2026-09-10 22:02:44.972642272 +0000 UTC m=+85.928709979"
Sep 10 22:02:45.574064 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:45.574023    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-66fc76d68b-5wp97"
Sep 10 22:02:45.578007 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:45.577979    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-66fc76d68b-5wp97"
Sep 10 22:02:45.956735 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:45.956690    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-66fc76d68b-5wp97"
Sep 10 22:02:45.958176 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:45.958151    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-66fc76d68b-5wp97"
Sep 10 22:02:46.961277 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:46.961238    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-xl6k8" event={"ID":"a121acc4-9c7f-4dca-8d25-29cce37120a7","Type":"ContainerStarted","Data":"2bb66701ade75f328e11fd7a1920498a5e4e1610087621c5059d723a48b63ae4"}
Sep 10 22:02:46.961761 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:46.961301    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-xl6k8" event={"ID":"a121acc4-9c7f-4dca-8d25-29cce37120a7","Type":"ContainerStarted","Data":"a6344a6a830ddacdcf73c29153484ac6067cfe70715e9c9c24a675d2a467d7c7"}
Sep 10 22:02:46.981139 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:46.981090    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-xl6k8" podStartSLOduration=17.293376599 podStartE2EDuration="18.981075029s" totalImagesPullingTime="1.68769843s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 22:02:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:02:46.979115425 +0000 UTC m=+87.935183131" watchObservedRunningTime="2026-09-10 22:02:46.981075029 +0000 UTC m=+87.937142768"
Sep 10 22:02:50.492776 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:50.492745    2582 scope.go:117] "RemoveContainer" containerID="3013de141d8b302036e3dbf993be8ebac9be10d7a50f9c4fd99feb063d168166"
Sep 10 22:02:50.804401 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:50.804328    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/29d71552-94a7-4491-aced-1141665a8b0b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hrpd5\" (UID: \"29d71552-94a7-4491-aced-1141665a8b0b\") " pod="openshift-insights/insights-runtime-extractor-hrpd5"
Sep 10 22:02:50.804401 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:50.804378    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/29d71552-94a7-4491-aced-1141665a8b0b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hrpd5\" (UID: \"29d71552-94a7-4491-aced-1141665a8b0b\") " pod="openshift-insights/insights-runtime-extractor-hrpd5"
Sep 10 22:02:50.804823 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:50.804805    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/29d71552-94a7-4491-aced-1141665a8b0b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hrpd5\" (UID: \"29d71552-94a7-4491-aced-1141665a8b0b\") " pod="openshift-insights/insights-runtime-extractor-hrpd5"
Sep 10 22:02:50.806571 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:50.806540    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/29d71552-94a7-4491-aced-1141665a8b0b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hrpd5\" (UID: \"29d71552-94a7-4491-aced-1141665a8b0b\") " pod="openshift-insights/insights-runtime-extractor-hrpd5"
Sep 10 22:02:50.949247 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:50.949210    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-hrpd5"
Sep 10 22:02:50.975630 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:50.975604    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-w6vlb_8517c9a7-382a-40fd-bc52-e6da90ebc584/console-operator/2.log"
Sep 10 22:02:50.976078 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:50.976062    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-w6vlb_8517c9a7-382a-40fd-bc52-e6da90ebc584/console-operator/1.log"
Sep 10 22:02:50.976157 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:50.976094    2582 generic.go:356] "Generic (PLEG): container finished" podID="8517c9a7-382a-40fd-bc52-e6da90ebc584" containerID="ac01c923477f1b6a4a73956c4a99ee7441a7a6a1d5497baf8d6de2012d41c934" exitCode=255
Sep 10 22:02:50.976157 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:50.976132    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-w6vlb" event={"ID":"8517c9a7-382a-40fd-bc52-e6da90ebc584","Type":"ContainerDied","Data":"ac01c923477f1b6a4a73956c4a99ee7441a7a6a1d5497baf8d6de2012d41c934"}
Sep 10 22:02:50.976216 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:50.976175    2582 scope.go:117] "RemoveContainer" containerID="3013de141d8b302036e3dbf993be8ebac9be10d7a50f9c4fd99feb063d168166"
Sep 10 22:02:50.976556 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:50.976537    2582 scope.go:117] "RemoveContainer" containerID="ac01c923477f1b6a4a73956c4a99ee7441a7a6a1d5497baf8d6de2012d41c934"
Sep 10 22:02:50.976751 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:02:50.976728    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-696884fb9c-w6vlb_openshift-console-operator(8517c9a7-382a-40fd-bc52-e6da90ebc584)\"" pod="openshift-console-operator/console-operator-696884fb9c-w6vlb" podUID="8517c9a7-382a-40fd-bc52-e6da90ebc584"
Sep 10 22:02:51.220810 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:51.220774    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-hrpd5"]
Sep 10 22:02:51.225524 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:02:51.225497    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29d71552_94a7_4491_aced_1141665a8b0b.slice/crio-a7fc52275dd2dae6a76830c79d9d0496f9a9b3419dfa18cacc1852cd23a05d5f WatchSource:0}: Error finding container a7fc52275dd2dae6a76830c79d9d0496f9a9b3419dfa18cacc1852cd23a05d5f: Status 404 returned error can't find the container with id a7fc52275dd2dae6a76830c79d9d0496f9a9b3419dfa18cacc1852cd23a05d5f
Sep 10 22:02:51.981080 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:51.981048    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-w6vlb_8517c9a7-382a-40fd-bc52-e6da90ebc584/console-operator/2.log"
Sep 10 22:02:51.982968 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:51.982946    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hrpd5" event={"ID":"29d71552-94a7-4491-aced-1141665a8b0b","Type":"ContainerStarted","Data":"35c626bd359d1de07230f2dc7ff9b5d802791358b3866b75d25560dc146fc18b"}
Sep 10 22:02:51.983065 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:51.982997    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hrpd5" event={"ID":"29d71552-94a7-4491-aced-1141665a8b0b","Type":"ContainerStarted","Data":"f3e718374846b27f2adcf4da1084df04e611d1e478ac526d9a0996585ea22c00"}
Sep 10 22:02:51.983065 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:51.983018    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hrpd5" event={"ID":"29d71552-94a7-4491-aced-1141665a8b0b","Type":"ContainerStarted","Data":"a7fc52275dd2dae6a76830c79d9d0496f9a9b3419dfa18cacc1852cd23a05d5f"}
Sep 10 22:02:53.992006 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:53.991959    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hrpd5" event={"ID":"29d71552-94a7-4491-aced-1141665a8b0b","Type":"ContainerStarted","Data":"76d9418b57f3227d3f6e05857c8fa6ed4e219f6da667324b832e25a699ed452b"}
Sep 10 22:02:54.011199 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:54.011159    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-hrpd5" podStartSLOduration=17.034633221 podStartE2EDuration="19.01114679s" totalImagesPullingTime="1.976513569s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 22:02:35 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:02:54.010377174 +0000 UTC m=+94.966444891" watchObservedRunningTime="2026-09-10 22:02:54.01114679 +0000 UTC m=+94.967214496"
Sep 10 22:02:56.850202 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:56.850167    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9d4e1da6-4cc3-41a7-a8be-1fdb06fd6759-metrics-tls\") pod \"dns-default-xck4c\" (UID: \"9d4e1da6-4cc3-41a7-a8be-1fdb06fd6759\") " pod="openshift-dns/dns-default-xck4c"
Sep 10 22:02:56.852449 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:56.852427    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9d4e1da6-4cc3-41a7-a8be-1fdb06fd6759-metrics-tls\") pod \"dns-default-xck4c\" (UID: \"9d4e1da6-4cc3-41a7-a8be-1fdb06fd6759\") " pod="openshift-dns/dns-default-xck4c"
Sep 10 22:02:57.150651 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:57.150621    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d4cdbdbb7-mll4z"]
Sep 10 22:02:57.151761 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:57.151733    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-qt6wb\""
Sep 10 22:02:57.153010 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:57.152986    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d4cdbdbb7-mll4z"
Sep 10 22:02:57.154696 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:57.154673    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 10 22:02:57.155253 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:57.155231    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 10 22:02:57.155444 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:57.155425    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 10 22:02:57.155444 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:57.155444    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 10 22:02:57.155444 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:57.155451    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 10 22:02:57.155672 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:57.155511    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 10 22:02:57.155771 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:57.155756    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 10 22:02:57.157882 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:57.157860    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-xck4c"
Sep 10 22:02:57.173231 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:57.173208    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d4cdbdbb7-mll4z"]
Sep 10 22:02:57.254375 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:57.254343    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/8dbb883b-69b3-4f38-846d-7106ad885618-ca\") pod \"cluster-proxy-proxy-agent-7d4cdbdbb7-mll4z\" (UID: \"8dbb883b-69b3-4f38-846d-7106ad885618\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d4cdbdbb7-mll4z"
Sep 10 22:02:57.254515 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:57.254381    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8dbb883b-69b3-4f38-846d-7106ad885618-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7d4cdbdbb7-mll4z\" (UID: \"8dbb883b-69b3-4f38-846d-7106ad885618\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d4cdbdbb7-mll4z"
Sep 10 22:02:57.254515 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:57.254405    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/8dbb883b-69b3-4f38-846d-7106ad885618-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7d4cdbdbb7-mll4z\" (UID: \"8dbb883b-69b3-4f38-846d-7106ad885618\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d4cdbdbb7-mll4z"
Sep 10 22:02:57.254515 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:57.254486    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bsh9q\" (UniqueName: \"kubernetes.io/projected/8dbb883b-69b3-4f38-846d-7106ad885618-kube-api-access-bsh9q\") pod \"cluster-proxy-proxy-agent-7d4cdbdbb7-mll4z\" (UID: \"8dbb883b-69b3-4f38-846d-7106ad885618\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d4cdbdbb7-mll4z"
Sep 10 22:02:57.254614 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:57.254516    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/8dbb883b-69b3-4f38-846d-7106ad885618-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7d4cdbdbb7-mll4z\" (UID: \"8dbb883b-69b3-4f38-846d-7106ad885618\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d4cdbdbb7-mll4z"
Sep 10 22:02:57.254614 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:57.254539    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/8dbb883b-69b3-4f38-846d-7106ad885618-hub\") pod \"cluster-proxy-proxy-agent-7d4cdbdbb7-mll4z\" (UID: \"8dbb883b-69b3-4f38-846d-7106ad885618\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d4cdbdbb7-mll4z"
Sep 10 22:02:57.354961 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:57.354925    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/8dbb883b-69b3-4f38-846d-7106ad885618-hub\") pod \"cluster-proxy-proxy-agent-7d4cdbdbb7-mll4z\" (UID: \"8dbb883b-69b3-4f38-846d-7106ad885618\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d4cdbdbb7-mll4z"
Sep 10 22:02:57.355127 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:57.355003    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/8dbb883b-69b3-4f38-846d-7106ad885618-ca\") pod \"cluster-proxy-proxy-agent-7d4cdbdbb7-mll4z\" (UID: \"8dbb883b-69b3-4f38-846d-7106ad885618\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d4cdbdbb7-mll4z"
Sep 10 22:02:57.355127 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:57.355035    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8dbb883b-69b3-4f38-846d-7106ad885618-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7d4cdbdbb7-mll4z\" (UID: \"8dbb883b-69b3-4f38-846d-7106ad885618\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d4cdbdbb7-mll4z"
Sep 10 22:02:57.355127 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:57.355066    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/8dbb883b-69b3-4f38-846d-7106ad885618-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7d4cdbdbb7-mll4z\" (UID: \"8dbb883b-69b3-4f38-846d-7106ad885618\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d4cdbdbb7-mll4z"
Sep 10 22:02:57.355287 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:57.355202    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bsh9q\" (UniqueName: \"kubernetes.io/projected/8dbb883b-69b3-4f38-846d-7106ad885618-kube-api-access-bsh9q\") pod \"cluster-proxy-proxy-agent-7d4cdbdbb7-mll4z\" (UID: \"8dbb883b-69b3-4f38-846d-7106ad885618\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d4cdbdbb7-mll4z"
Sep 10 22:02:57.355287 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:57.355252    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/8dbb883b-69b3-4f38-846d-7106ad885618-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7d4cdbdbb7-mll4z\" (UID: \"8dbb883b-69b3-4f38-846d-7106ad885618\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d4cdbdbb7-mll4z"
Sep 10 22:02:57.356001 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:57.355972    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/8dbb883b-69b3-4f38-846d-7106ad885618-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7d4cdbdbb7-mll4z\" (UID: \"8dbb883b-69b3-4f38-846d-7106ad885618\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d4cdbdbb7-mll4z"
Sep 10 22:02:57.357515 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:57.357492    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/8dbb883b-69b3-4f38-846d-7106ad885618-ca\") pod \"cluster-proxy-proxy-agent-7d4cdbdbb7-mll4z\" (UID: \"8dbb883b-69b3-4f38-846d-7106ad885618\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d4cdbdbb7-mll4z"
Sep 10 22:02:57.357700 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:57.357680    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8dbb883b-69b3-4f38-846d-7106ad885618-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7d4cdbdbb7-mll4z\" (UID: \"8dbb883b-69b3-4f38-846d-7106ad885618\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d4cdbdbb7-mll4z"
Sep 10 22:02:57.357748 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:57.357712    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/8dbb883b-69b3-4f38-846d-7106ad885618-hub\") pod \"cluster-proxy-proxy-agent-7d4cdbdbb7-mll4z\" (UID: \"8dbb883b-69b3-4f38-846d-7106ad885618\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d4cdbdbb7-mll4z"
Sep 10 22:02:57.357748 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:57.357719    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/8dbb883b-69b3-4f38-846d-7106ad885618-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7d4cdbdbb7-mll4z\" (UID: \"8dbb883b-69b3-4f38-846d-7106ad885618\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d4cdbdbb7-mll4z"
Sep 10 22:02:57.367583 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:57.367560    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bsh9q\" (UniqueName: \"kubernetes.io/projected/8dbb883b-69b3-4f38-846d-7106ad885618-kube-api-access-bsh9q\") pod \"cluster-proxy-proxy-agent-7d4cdbdbb7-mll4z\" (UID: \"8dbb883b-69b3-4f38-846d-7106ad885618\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d4cdbdbb7-mll4z"
Sep 10 22:02:57.431031 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:57.431007    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-xck4c"]
Sep 10 22:02:57.434424 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:02:57.434401    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d4e1da6_4cc3_41a7_a8be_1fdb06fd6759.slice/crio-a6d50945043d13863e021abc00456327cf5b3ae92955cf2cb8fda13853421c5d WatchSource:0}: Error finding container a6d50945043d13863e021abc00456327cf5b3ae92955cf2cb8fda13853421c5d: Status 404 returned error can't find the container with id a6d50945043d13863e021abc00456327cf5b3ae92955cf2cb8fda13853421c5d
Sep 10 22:02:57.470280 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:57.470258    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d4cdbdbb7-mll4z"
Sep 10 22:02:57.740140 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:57.740064    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d4cdbdbb7-mll4z"]
Sep 10 22:02:57.742943 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:02:57.742907    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8dbb883b_69b3_4f38_846d_7106ad885618.slice/crio-405ac2c3c5a2f30ae33167cc970eb8d95866345e14682cb5ebf2e367baa74629 WatchSource:0}: Error finding container 405ac2c3c5a2f30ae33167cc970eb8d95866345e14682cb5ebf2e367baa74629: Status 404 returned error can't find the container with id 405ac2c3c5a2f30ae33167cc970eb8d95866345e14682cb5ebf2e367baa74629
Sep 10 22:02:58.005487 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:58.005398    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xck4c" event={"ID":"9d4e1da6-4cc3-41a7-a8be-1fdb06fd6759","Type":"ContainerStarted","Data":"a6d50945043d13863e021abc00456327cf5b3ae92955cf2cb8fda13853421c5d"}
Sep 10 22:02:58.006570 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:58.006545    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d4cdbdbb7-mll4z" event={"ID":"8dbb883b-69b3-4f38-846d-7106ad885618","Type":"ContainerStarted","Data":"405ac2c3c5a2f30ae33167cc970eb8d95866345e14682cb5ebf2e367baa74629"}
Sep 10 22:02:58.967315 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:58.967277    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-w6vlb"
Sep 10 22:02:58.967512 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:58.967350    2582 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-696884fb9c-w6vlb"
Sep 10 22:02:58.967763 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:02:58.967737    2582 scope.go:117] "RemoveContainer" containerID="ac01c923477f1b6a4a73956c4a99ee7441a7a6a1d5497baf8d6de2012d41c934"
Sep 10 22:02:58.968059 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:02:58.967961    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-696884fb9c-w6vlb_openshift-console-operator(8517c9a7-382a-40fd-bc52-e6da90ebc584)\"" pod="openshift-console-operator/console-operator-696884fb9c-w6vlb" podUID="8517c9a7-382a-40fd-bc52-e6da90ebc584"
Sep 10 22:03:01.898752 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:03:01.898719    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-rrm25"
Sep 10 22:03:02.025410 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:03:02.025372    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d4cdbdbb7-mll4z" event={"ID":"8dbb883b-69b3-4f38-846d-7106ad885618","Type":"ContainerStarted","Data":"9279d66ccb75d38f760f4c6b180b6cb136ea07558eb079160b42816b228425c3"}
Sep 10 22:03:02.025589 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:03:02.025422    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d4cdbdbb7-mll4z" event={"ID":"8dbb883b-69b3-4f38-846d-7106ad885618","Type":"ContainerStarted","Data":"49d2c5a88288df8b5e8c2c6261a4822f1cef88ba173f574dc176da3f806fe114"}
Sep 10 22:03:02.025589 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:03:02.025441    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d4cdbdbb7-mll4z" event={"ID":"8dbb883b-69b3-4f38-846d-7106ad885618","Type":"ContainerStarted","Data":"6fe76b5ba2d90741df29ff7b50f90c7a687e30ae1d5ca90a1d402e7e877415ca"}
Sep 10 22:03:02.027108 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:03:02.027079    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xck4c" event={"ID":"9d4e1da6-4cc3-41a7-a8be-1fdb06fd6759","Type":"ContainerStarted","Data":"1f03bad08a003c67b68e3af8f69bac54d47c49739c415dbca4c44bdc4b794749"}
Sep 10 22:03:02.027247 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:03:02.027121    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xck4c" event={"ID":"9d4e1da6-4cc3-41a7-a8be-1fdb06fd6759","Type":"ContainerStarted","Data":"852941ee45d21ff4b67e73b5ae1a091924968093d3f432d4c43d5f0d748d36ee"}
Sep 10 22:03:02.027247 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:03:02.027226    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-xck4c"
Sep 10 22:03:02.044807 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:03:02.044765    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d4cdbdbb7-mll4z" podStartSLOduration=1.5598993920000002 podStartE2EDuration="5.044752326s" totalImagesPullingTime="3.484852934s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 22:02:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:03:02.044322847 +0000 UTC m=+103.000390564" watchObservedRunningTime="2026-09-10 22:03:02.044752326 +0000 UTC m=+103.000820031"
Sep 10 22:03:02.061308 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:03:02.061269    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-xck4c" podStartSLOduration=66.307959799 podStartE2EDuration="1m10.061259575s" totalImagesPullingTime="3.753299776s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 22:01:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:03:02.060004509 +0000 UTC m=+103.016072215" watchObservedRunningTime="2026-09-10 22:03:02.061259575 +0000 UTC m=+103.017327319"
Sep 10 22:03:10.493228 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:03:10.493192    2582 scope.go:117] "RemoveContainer" containerID="ac01c923477f1b6a4a73956c4a99ee7441a7a6a1d5497baf8d6de2012d41c934"
Sep 10 22:03:10.493685 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:03:10.493361    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-696884fb9c-w6vlb_openshift-console-operator(8517c9a7-382a-40fd-bc52-e6da90ebc584)\"" pod="openshift-console-operator/console-operator-696884fb9c-w6vlb" podUID="8517c9a7-382a-40fd-bc52-e6da90ebc584"
Sep 10 22:03:12.033258 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:03:12.033222    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-xck4c"
Sep 10 22:03:12.389320 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:03:12.389228    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-kcl7j"]
Sep 10 22:03:12.391333 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:03:12.391317    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-kcl7j"
Sep 10 22:03:12.397929 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:03:12.397912    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 10 22:03:12.398778 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:03:12.398761    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 10 22:03:12.398778 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:03:12.398770    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 10 22:03:12.399488 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:03:12.399471    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 10 22:03:12.399630 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:03:12.399614    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 10 22:03:12.399983 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:03:12.399967    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 10 22:03:12.400738 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:03:12.400716    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-7v8kf\""
Sep 10 22:03:12.569932 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:03:12.569892    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a43f9f15-f001-4c47-8676-2cdd638c206a-root\") pod \"node-exporter-kcl7j\" (UID: \"a43f9f15-f001-4c47-8676-2cdd638c206a\") " pod="openshift-monitoring/node-exporter-kcl7j"
Sep 10 22:03:12.570136 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:03:12.569933    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a43f9f15-f001-4c47-8676-2cdd638c206a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-kcl7j\" (UID: \"a43f9f15-f001-4c47-8676-2cdd638c206a\") " pod="openshift-monitoring/node-exporter-kcl7j"
Sep 10 22:03:12.570136 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:03:12.569971    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a43f9f15-f001-4c47-8676-2cdd638c206a-sys\") pod \"node-exporter-kcl7j\" (UID: \"a43f9f15-f001-4c47-8676-2cdd638c206a\") " pod="openshift-monitoring/node-exporter-kcl7j"
Sep 10 22:03:12.570136 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:03:12.570047    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a43f9f15-f001-4c47-8676-2cdd638c206a-node-exporter-accelerators-collector-config\") pod \"node-exporter-kcl7j\" (UID: \"a43f9f15-f001-4c47-8676-2cdd638c206a\") " pod="openshift-monitoring/node-exporter-kcl7j"
Sep 10 22:03:12.570136 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:03:12.570123    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a43f9f15-f001-4c47-8676-2cdd638c206a-node-exporter-textfile\") pod \"node-exporter-kcl7j\" (UID: \"a43f9f15-f001-4c47-8676-2cdd638c206a\") " pod="openshift-monitoring/node-exporter-kcl7j"
Sep 10 22:03:12.570297 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:03:12.570171    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a43f9f15-f001-4c47-8676-2cdd638c206a-metrics-client-ca\") pod \"node-exporter-kcl7j\" (UID: \"a43f9f15-f001-4c47-8676-2cdd638c206a\") " pod="openshift-monitoring/node-exporter-kcl7j"
Sep 10 22:03:12.570297 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:03:12.570236    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a43f9f15-f001-4c47-8676-2cdd638c206a-node-exporter-tls\") pod \"node-exporter-kcl7j\" (UID: \"a43f9f15-f001-4c47-8676-2cdd638c206a\") " pod="openshift-monitoring/node-exporter-kcl7j"
Sep 10 22:03:12.570297 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:03:12.570253    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddj62\" (UniqueName: \"kubernetes.io/projected/a43f9f15-f001-4c47-8676-2cdd638c206a-kube-api-access-ddj62\") pod \"node-exporter-kcl7j\" (UID: \"a43f9f15-f001-4c47-8676-2cdd638c206a\") " pod="openshift-monitoring/node-exporter-kcl7j"
Sep 10 22:03:12.570297 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:03:12.570269    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a43f9f15-f001-4c47-8676-2cdd638c206a-node-exporter-wtmp\") pod \"node-exporter-kcl7j\" (UID: \"a43f9f15-f001-4c47-8676-2cdd638c206a\") " pod="openshift-monitoring/node-exporter-kcl7j"
Sep 10 22:03:12.671323 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:03:12.671283    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a43f9f15-f001-4c47-8676-2cdd638c206a-metrics-client-ca\") pod \"node-exporter-kcl7j\" (UID: \"a43f9f15-f001-4c47-8676-2cdd638c206a\") " pod="openshift-monitoring/node-exporter-kcl7j"
Sep 10 22:03:12.671528 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:03:12.671352    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a43f9f15-f001-4c47-8676-2cdd638c206a-node-exporter-tls\") pod \"node-exporter-kcl7j\" (UID: \"a43f9f15-f001-4c47-8676-2cdd638c206a\") " pod="openshift-monitoring/node-exporter-kcl7j"
Sep 10 22:03:12.671528 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:03:12.671370    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddj62\" (UniqueName: \"kubernetes.io/projected/a43f9f15-f001-4c47-8676-2cdd638c206a-kube-api-access-ddj62\") pod \"node-exporter-kcl7j\" (UID: \"a43f9f15-f001-4c47-8676-2cdd638c206a\") " pod="openshift-monitoring/node-exporter-kcl7j"
Sep 10 22:03:12.671528 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:03:12.671386    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a43f9f15-f001-4c47-8676-2cdd638c206a-node-exporter-wtmp\") pod \"node-exporter-kcl7j\" (UID: \"a43f9f15-f001-4c47-8676-2cdd638c206a\") " pod="openshift-monitoring/node-exporter-kcl7j"
Sep 10 22:03:12.671528 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:03:12.671411    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a43f9f15-f001-4c47-8676-2cdd638c206a-root\") pod \"node-exporter-kcl7j\" (UID: \"a43f9f15-f001-4c47-8676-2cdd638c206a\") " pod="openshift-monitoring/node-exporter-kcl7j"
Sep 10 22:03:12.671528 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:03:12.671432    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a43f9f15-f001-4c47-8676-2cdd638c206a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-kcl7j\" (UID: \"a43f9f15-f001-4c47-8676-2cdd638c206a\") " pod="openshift-monitoring/node-exporter-kcl7j"
Sep 10 22:03:12.671528 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:03:12.671464    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a43f9f15-f001-4c47-8676-2cdd638c206a-sys\") pod \"node-exporter-kcl7j\" (UID: \"a43f9f15-f001-4c47-8676-2cdd638c206a\") " pod="openshift-monitoring/node-exporter-kcl7j"
Sep 10 22:03:12.671528 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:03:12.671494    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a43f9f15-f001-4c47-8676-2cdd638c206a-node-exporter-accelerators-collector-config\") pod \"node-exporter-kcl7j\" (UID: \"a43f9f15-f001-4c47-8676-2cdd638c206a\") " pod="openshift-monitoring/node-exporter-kcl7j"
Sep 10 22:03:12.671528 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:03:12.671505    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a43f9f15-f001-4c47-8676-2cdd638c206a-root\") pod \"node-exporter-kcl7j\" (UID: \"a43f9f15-f001-4c47-8676-2cdd638c206a\") " pod="openshift-monitoring/node-exporter-kcl7j"
Sep 10 22:03:12.671528 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:03:12.671521    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a43f9f15-f001-4c47-8676-2cdd638c206a-node-exporter-textfile\") pod \"node-exporter-kcl7j\" (UID: \"a43f9f15-f001-4c47-8676-2cdd638c206a\") " pod="openshift-monitoring/node-exporter-kcl7j"
Sep 10 22:03:12.672024 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:03:12.671597    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a43f9f15-f001-4c47-8676-2cdd638c206a-sys\") pod \"node-exporter-kcl7j\" (UID: \"a43f9f15-f001-4c47-8676-2cdd638c206a\") " pod="openshift-monitoring/node-exporter-kcl7j"
Sep 10 22:03:12.672024 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:03:12.671638    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a43f9f15-f001-4c47-8676-2cdd638c206a-node-exporter-wtmp\") pod \"node-exporter-kcl7j\" (UID: \"a43f9f15-f001-4c47-8676-2cdd638c206a\") " pod="openshift-monitoring/node-exporter-kcl7j"
Sep 10 22:03:12.672024 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:03:12.671910    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a43f9f15-f001-4c47-8676-2cdd638c206a-node-exporter-textfile\") pod \"node-exporter-kcl7j\" (UID: \"a43f9f15-f001-4c47-8676-2cdd638c206a\") " pod="openshift-monitoring/node-exporter-kcl7j"
Sep 10 22:03:12.672185 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:03:12.672142    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a43f9f15-f001-4c47-8676-2cdd638c206a-node-exporter-accelerators-collector-config\") pod \"node-exporter-kcl7j\" (UID: \"a43f9f15-f001-4c47-8676-2cdd638c206a\") " pod="openshift-monitoring/node-exporter-kcl7j"
Sep 10 22:03:12.672595 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:03:12.672572    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a43f9f15-f001-4c47-8676-2cdd638c206a-metrics-client-ca\") pod \"node-exporter-kcl7j\" (UID: \"a43f9f15-f001-4c47-8676-2cdd638c206a\") " pod="openshift-monitoring/node-exporter-kcl7j"
Sep 10 22:03:12.673887 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:03:12.673869    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a43f9f15-f001-4c47-8676-2cdd638c206a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-kcl7j\" (UID: \"a43f9f15-f001-4c47-8676-2cdd638c206a\") " pod="openshift-monitoring/node-exporter-kcl7j"
Sep 10 22:03:12.673938 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:03:12.673925    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a43f9f15-f001-4c47-8676-2cdd638c206a-node-exporter-tls\") pod \"node-exporter-kcl7j\" (UID: \"a43f9f15-f001-4c47-8676-2cdd638c206a\") " pod="openshift-monitoring/node-exporter-kcl7j"
Sep 10 22:03:12.681417 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:03:12.681388    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddj62\" (UniqueName: \"kubernetes.io/projected/a43f9f15-f001-4c47-8676-2cdd638c206a-kube-api-access-ddj62\") pod \"node-exporter-kcl7j\" (UID: \"a43f9f15-f001-4c47-8676-2cdd638c206a\") " pod="openshift-monitoring/node-exporter-kcl7j"
Sep 10 22:03:12.700426 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:03:12.700406    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-kcl7j"
Sep 10 22:03:12.709681 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:03:12.709658    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda43f9f15_f001_4c47_8676_2cdd638c206a.slice/crio-f72297276ff657443d8b42b2c7fafd3ccb50de6f1f4b07d2e9cef08dab7f92e4 WatchSource:0}: Error finding container f72297276ff657443d8b42b2c7fafd3ccb50de6f1f4b07d2e9cef08dab7f92e4: Status 404 returned error can't find the container with id f72297276ff657443d8b42b2c7fafd3ccb50de6f1f4b07d2e9cef08dab7f92e4
Sep 10 22:03:13.066083 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:03:13.065998    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kcl7j" event={"ID":"a43f9f15-f001-4c47-8676-2cdd638c206a","Type":"ContainerStarted","Data":"f72297276ff657443d8b42b2c7fafd3ccb50de6f1f4b07d2e9cef08dab7f92e4"}
Sep 10 22:03:14.071805 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:03:14.071721    2582 generic.go:356] "Generic (PLEG): container finished" podID="a43f9f15-f001-4c47-8676-2cdd638c206a" containerID="ca8bf4f31ec4343cb794d085302ca3145b5a06981156f64baf2059511d68fc96" exitCode=0
Sep 10 22:03:14.072170 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:03:14.071808    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kcl7j" event={"ID":"a43f9f15-f001-4c47-8676-2cdd638c206a","Type":"ContainerDied","Data":"ca8bf4f31ec4343cb794d085302ca3145b5a06981156f64baf2059511d68fc96"}
Sep 10 22:03:15.077235 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:03:15.077201    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kcl7j" event={"ID":"a43f9f15-f001-4c47-8676-2cdd638c206a","Type":"ContainerStarted","Data":"0f16030e22bdcaac837746f0e09e3a2a499b23bab9ca7bd53d6d2ab63fedde4c"}
Sep 10 22:03:15.077605 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:03:15.077282    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kcl7j" event={"ID":"a43f9f15-f001-4c47-8676-2cdd638c206a","Type":"ContainerStarted","Data":"e5250239323c19746ba5b813e37e5c3ec62829d0dc5aa730a2717d7e1b3027e7"}
Sep 10 22:03:15.100307 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:03:15.100262    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-kcl7j" podStartSLOduration=2.080847034 podStartE2EDuration="3.100247195s" totalImagesPullingTime="955.363081ms" totalInitContainerRuntime="64.03708ms" isStatefulPod=true podCreationTimestamp="2026-09-10 22:03:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:03:15.098604934 +0000 UTC m=+116.054672637" watchObservedRunningTime="2026-09-10 22:03:15.100247195 +0000 UTC m=+116.056314901"
Sep 10 22:03:24.493030 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:03:24.492991    2582 scope.go:117] "RemoveContainer" containerID="ac01c923477f1b6a4a73956c4a99ee7441a7a6a1d5497baf8d6de2012d41c934"
Sep 10 22:03:25.112397 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:03:25.112369    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-w6vlb_8517c9a7-382a-40fd-bc52-e6da90ebc584/console-operator/2.log"
Sep 10 22:03:25.112566 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:03:25.112454    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-w6vlb" event={"ID":"8517c9a7-382a-40fd-bc52-e6da90ebc584","Type":"ContainerStarted","Data":"7f35ae2f561ca5189b84605dc848b1b5a5d2c99c4a415e88458b5fd1a0834119"}
Sep 10 22:03:25.112790 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:03:25.112763    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-w6vlb"
Sep 10 22:03:25.130798 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:03:25.130754    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-696884fb9c-w6vlb" podStartSLOduration=54.768593429 podStartE2EDuration="57.130722712s" totalImagesPullingTime="2.362129283s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 22:02:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:03:25.130533077 +0000 UTC m=+126.086600797" watchObservedRunningTime="2026-09-10 22:03:25.130722712 +0000 UTC m=+126.086790415"
Sep 10 22:03:25.381431 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:03:25.381347    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-696884fb9c-w6vlb"
Sep 10 22:03:29.209130 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:03:29.209096    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6c7caf6-cae8-4a81-98c5-e4db1671eff5-metrics-certs\") pod \"network-metrics-daemon-xw7pd\" (UID: \"a6c7caf6-cae8-4a81-98c5-e4db1671eff5\") " pod="openshift-multus/network-metrics-daemon-xw7pd"
Sep 10 22:03:29.211415 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:03:29.211393    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6c7caf6-cae8-4a81-98c5-e4db1671eff5-metrics-certs\") pod \"network-metrics-daemon-xw7pd\" (UID: \"a6c7caf6-cae8-4a81-98c5-e4db1671eff5\") " pod="openshift-multus/network-metrics-daemon-xw7pd"
Sep 10 22:03:29.506162 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:03:29.506087    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-vfb6l\""
Sep 10 22:03:29.514273 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:03:29.514249    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xw7pd"
Sep 10 22:03:29.781684 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:03:29.781611    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-xw7pd"]
Sep 10 22:03:29.784356 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:03:29.784329    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6c7caf6_cae8_4a81_98c5_e4db1671eff5.slice/crio-b4ee706feb9dad6ab4fc57e07efb80b8babf47206e996d2e6e989b4497c9f116 WatchSource:0}: Error finding container b4ee706feb9dad6ab4fc57e07efb80b8babf47206e996d2e6e989b4497c9f116: Status 404 returned error can't find the container with id b4ee706feb9dad6ab4fc57e07efb80b8babf47206e996d2e6e989b4497c9f116
Sep 10 22:03:30.129111 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:03:30.129028    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xw7pd" event={"ID":"a6c7caf6-cae8-4a81-98c5-e4db1671eff5","Type":"ContainerStarted","Data":"b4ee706feb9dad6ab4fc57e07efb80b8babf47206e996d2e6e989b4497c9f116"}
Sep 10 22:03:31.135124 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:03:31.135024    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xw7pd" event={"ID":"a6c7caf6-cae8-4a81-98c5-e4db1671eff5","Type":"ContainerStarted","Data":"45ccbb69deebc037671e050dc19eac386f07655b12a4af7d613f52f2ec35e330"}
Sep 10 22:03:31.135484 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:03:31.135149    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xw7pd" event={"ID":"a6c7caf6-cae8-4a81-98c5-e4db1671eff5","Type":"ContainerStarted","Data":"d68bf8cb6334d7c63974eee378f1dde91ab7c2c48541de081f4f59f00525fc0a"}
Sep 10 22:03:31.150825 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:03:31.150773    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-xw7pd" podStartSLOduration=131.106320376 podStartE2EDuration="2m12.150757445s" totalImagesPullingTime="1.044437069s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 22:01:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:03:31.149409041 +0000 UTC m=+132.105476758" watchObservedRunningTime="2026-09-10 22:03:31.150757445 +0000 UTC m=+132.106825178"
Sep 10 22:03:54.217799 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:03:54.217762    2582 generic.go:356] "Generic (PLEG): container finished" podID="3aaaf117-e959-49e2-93f0-342a657439c5" containerID="df9f7d8ebdb7af48f463bbdceb6841a29c4b02b07ba5d5e3618255ebbffa08a8" exitCode=0
Sep 10 22:03:54.218273 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:03:54.217794    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-bhv4x" event={"ID":"3aaaf117-e959-49e2-93f0-342a657439c5","Type":"ContainerDied","Data":"df9f7d8ebdb7af48f463bbdceb6841a29c4b02b07ba5d5e3618255ebbffa08a8"}
Sep 10 22:03:54.218273 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:03:54.218102    2582 scope.go:117] "RemoveContainer" containerID="df9f7d8ebdb7af48f463bbdceb6841a29c4b02b07ba5d5e3618255ebbffa08a8"
Sep 10 22:03:55.223054 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:03:55.223018    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-bhv4x" event={"ID":"3aaaf117-e959-49e2-93f0-342a657439c5","Type":"ContainerStarted","Data":"f3292fa4cc01df88b223fa74f947fbc37907a14525d59cf0a4666cda58abbace"}
Sep 10 22:06:19.405930 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:06:19.405886    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-w6vlb_8517c9a7-382a-40fd-bc52-e6da90ebc584/console-operator/2.log"
Sep 10 22:06:19.410604 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:06:19.410580    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-w6vlb_8517c9a7-382a-40fd-bc52-e6da90ebc584/console-operator/2.log"
Sep 10 22:06:19.423966 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:06:19.423932    2582 kubelet.go:1648] "Image garbage collection succeeded"
Sep 10 22:07:28.602206 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:07:28.602117    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-m9dhp"]
Sep 10 22:07:28.605131 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:07:28.605108    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-m9dhp"
Sep 10 22:07:28.607613 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:07:28.607588    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 10 22:07:28.607728 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:07:28.607588    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 10 22:07:28.607728 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:07:28.607590    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-n55jb\""
Sep 10 22:07:28.612940 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:07:28.612913    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-m9dhp"]
Sep 10 22:07:28.712442 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:07:28.712396    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/45a6a874-366b-44c6-8e2d-59b370cbb643-tmp\") pod \"openshift-lws-operator-fd8ccff4c-m9dhp\" (UID: \"45a6a874-366b-44c6-8e2d-59b370cbb643\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-m9dhp"
Sep 10 22:07:28.712627 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:07:28.712471    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97vb4\" (UniqueName: \"kubernetes.io/projected/45a6a874-366b-44c6-8e2d-59b370cbb643-kube-api-access-97vb4\") pod \"openshift-lws-operator-fd8ccff4c-m9dhp\" (UID: \"45a6a874-366b-44c6-8e2d-59b370cbb643\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-m9dhp"
Sep 10 22:07:28.813839 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:07:28.813797    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97vb4\" (UniqueName: \"kubernetes.io/projected/45a6a874-366b-44c6-8e2d-59b370cbb643-kube-api-access-97vb4\") pod \"openshift-lws-operator-fd8ccff4c-m9dhp\" (UID: \"45a6a874-366b-44c6-8e2d-59b370cbb643\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-m9dhp"
Sep 10 22:07:28.814025 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:07:28.813880    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/45a6a874-366b-44c6-8e2d-59b370cbb643-tmp\") pod \"openshift-lws-operator-fd8ccff4c-m9dhp\" (UID: \"45a6a874-366b-44c6-8e2d-59b370cbb643\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-m9dhp"
Sep 10 22:07:28.814256 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:07:28.814237    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/45a6a874-366b-44c6-8e2d-59b370cbb643-tmp\") pod \"openshift-lws-operator-fd8ccff4c-m9dhp\" (UID: \"45a6a874-366b-44c6-8e2d-59b370cbb643\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-m9dhp"
Sep 10 22:07:28.821794 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:07:28.821769    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-97vb4\" (UniqueName: \"kubernetes.io/projected/45a6a874-366b-44c6-8e2d-59b370cbb643-kube-api-access-97vb4\") pod \"openshift-lws-operator-fd8ccff4c-m9dhp\" (UID: \"45a6a874-366b-44c6-8e2d-59b370cbb643\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-m9dhp"
Sep 10 22:07:28.913913 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:07:28.913875    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-m9dhp"
Sep 10 22:07:29.183630 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:07:29.183550    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-m9dhp"]
Sep 10 22:07:29.187286 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:07:29.187256    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod45a6a874_366b_44c6_8e2d_59b370cbb643.slice/crio-ae1abe29c30ca6a26ce5ccf647e94a27720fe5c95f73b777a9be362a9731617a WatchSource:0}: Error finding container ae1abe29c30ca6a26ce5ccf647e94a27720fe5c95f73b777a9be362a9731617a: Status 404 returned error can't find the container with id ae1abe29c30ca6a26ce5ccf647e94a27720fe5c95f73b777a9be362a9731617a
Sep 10 22:07:29.189216 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:07:29.189200    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 22:07:29.957478 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:07:29.957432    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-m9dhp" event={"ID":"45a6a874-366b-44c6-8e2d-59b370cbb643","Type":"ContainerStarted","Data":"ae1abe29c30ca6a26ce5ccf647e94a27720fe5c95f73b777a9be362a9731617a"}
Sep 10 22:07:31.967570 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:07:31.967527    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-m9dhp" event={"ID":"45a6a874-366b-44c6-8e2d-59b370cbb643","Type":"ContainerStarted","Data":"8763eb896cbeb5f21f8ee424e667185d4b1b7f6aa9c5e585135e35a2f6af5f6d"}
Sep 10 22:07:31.983603 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:07:31.983544    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-m9dhp" podStartSLOduration=1.709131219 podStartE2EDuration="3.983527506s" totalImagesPullingTime="2.274396287s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 22:07:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:07:31.98267088 +0000 UTC m=+372.938738588" watchObservedRunningTime="2026-09-10 22:07:31.983527506 +0000 UTC m=+372.939595213"
Sep 10 22:07:48.488468 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:07:48.488437    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-76d44b6d5b-vjglf"]
Sep 10 22:07:48.493310 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:07:48.493293    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-76d44b6d5b-vjglf"
Sep 10 22:07:48.498668 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:07:48.498644    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 10 22:07:48.498668 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:07:48.498644    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-5r2tg\""
Sep 10 22:07:48.498880 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:07:48.498693    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 10 22:07:48.498880 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:07:48.498746    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 10 22:07:48.499009 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:07:48.498993    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 10 22:07:48.512406 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:07:48.512385    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-76d44b6d5b-vjglf"]
Sep 10 22:07:48.559121 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:07:48.559095    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lt6b\" (UniqueName: \"kubernetes.io/projected/d16e4185-bc86-4442-81a4-9ca18f1b49d2-kube-api-access-6lt6b\") pod \"opendatahub-operator-controller-manager-76d44b6d5b-vjglf\" (UID: \"d16e4185-bc86-4442-81a4-9ca18f1b49d2\") " pod="opendatahub/opendatahub-operator-controller-manager-76d44b6d5b-vjglf"
Sep 10 22:07:48.559233 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:07:48.559133    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d16e4185-bc86-4442-81a4-9ca18f1b49d2-manifests\") pod \"opendatahub-operator-controller-manager-76d44b6d5b-vjglf\" (UID: \"d16e4185-bc86-4442-81a4-9ca18f1b49d2\") " pod="opendatahub/opendatahub-operator-controller-manager-76d44b6d5b-vjglf"
Sep 10 22:07:48.559233 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:07:48.559164    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d16e4185-bc86-4442-81a4-9ca18f1b49d2-apiservice-cert\") pod \"opendatahub-operator-controller-manager-76d44b6d5b-vjglf\" (UID: \"d16e4185-bc86-4442-81a4-9ca18f1b49d2\") " pod="opendatahub/opendatahub-operator-controller-manager-76d44b6d5b-vjglf"
Sep 10 22:07:48.559233 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:07:48.559223    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d16e4185-bc86-4442-81a4-9ca18f1b49d2-webhook-cert\") pod \"opendatahub-operator-controller-manager-76d44b6d5b-vjglf\" (UID: \"d16e4185-bc86-4442-81a4-9ca18f1b49d2\") " pod="opendatahub/opendatahub-operator-controller-manager-76d44b6d5b-vjglf"
Sep 10 22:07:48.559348 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:07:48.559249    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d16e4185-bc86-4442-81a4-9ca18f1b49d2-tmp\") pod \"opendatahub-operator-controller-manager-76d44b6d5b-vjglf\" (UID: \"d16e4185-bc86-4442-81a4-9ca18f1b49d2\") " pod="opendatahub/opendatahub-operator-controller-manager-76d44b6d5b-vjglf"
Sep 10 22:07:48.659991 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:07:48.659966    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d16e4185-bc86-4442-81a4-9ca18f1b49d2-webhook-cert\") pod \"opendatahub-operator-controller-manager-76d44b6d5b-vjglf\" (UID: \"d16e4185-bc86-4442-81a4-9ca18f1b49d2\") " pod="opendatahub/opendatahub-operator-controller-manager-76d44b6d5b-vjglf"
Sep 10 22:07:48.660106 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:07:48.659996    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d16e4185-bc86-4442-81a4-9ca18f1b49d2-tmp\") pod \"opendatahub-operator-controller-manager-76d44b6d5b-vjglf\" (UID: \"d16e4185-bc86-4442-81a4-9ca18f1b49d2\") " pod="opendatahub/opendatahub-operator-controller-manager-76d44b6d5b-vjglf"
Sep 10 22:07:48.660106 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:07:48.660019    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lt6b\" (UniqueName: \"kubernetes.io/projected/d16e4185-bc86-4442-81a4-9ca18f1b49d2-kube-api-access-6lt6b\") pod \"opendatahub-operator-controller-manager-76d44b6d5b-vjglf\" (UID: \"d16e4185-bc86-4442-81a4-9ca18f1b49d2\") " pod="opendatahub/opendatahub-operator-controller-manager-76d44b6d5b-vjglf"
Sep 10 22:07:48.660106 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:07:48.660046    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d16e4185-bc86-4442-81a4-9ca18f1b49d2-manifests\") pod \"opendatahub-operator-controller-manager-76d44b6d5b-vjglf\" (UID: \"d16e4185-bc86-4442-81a4-9ca18f1b49d2\") " pod="opendatahub/opendatahub-operator-controller-manager-76d44b6d5b-vjglf"
Sep 10 22:07:48.660106 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:07:48.660074    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d16e4185-bc86-4442-81a4-9ca18f1b49d2-apiservice-cert\") pod \"opendatahub-operator-controller-manager-76d44b6d5b-vjglf\" (UID: \"d16e4185-bc86-4442-81a4-9ca18f1b49d2\") " pod="opendatahub/opendatahub-operator-controller-manager-76d44b6d5b-vjglf"
Sep 10 22:07:48.660453 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:07:48.660427    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d16e4185-bc86-4442-81a4-9ca18f1b49d2-manifests\") pod \"opendatahub-operator-controller-manager-76d44b6d5b-vjglf\" (UID: \"d16e4185-bc86-4442-81a4-9ca18f1b49d2\") " pod="opendatahub/opendatahub-operator-controller-manager-76d44b6d5b-vjglf"
Sep 10 22:07:48.662182 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:07:48.662159    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d16e4185-bc86-4442-81a4-9ca18f1b49d2-tmp\") pod \"opendatahub-operator-controller-manager-76d44b6d5b-vjglf\" (UID: \"d16e4185-bc86-4442-81a4-9ca18f1b49d2\") " pod="opendatahub/opendatahub-operator-controller-manager-76d44b6d5b-vjglf"
Sep 10 22:07:48.662500 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:07:48.662483    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d16e4185-bc86-4442-81a4-9ca18f1b49d2-apiservice-cert\") pod \"opendatahub-operator-controller-manager-76d44b6d5b-vjglf\" (UID: \"d16e4185-bc86-4442-81a4-9ca18f1b49d2\") " pod="opendatahub/opendatahub-operator-controller-manager-76d44b6d5b-vjglf"
Sep 10 22:07:48.662544 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:07:48.662514    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d16e4185-bc86-4442-81a4-9ca18f1b49d2-webhook-cert\") pod \"opendatahub-operator-controller-manager-76d44b6d5b-vjglf\" (UID: \"d16e4185-bc86-4442-81a4-9ca18f1b49d2\") " pod="opendatahub/opendatahub-operator-controller-manager-76d44b6d5b-vjglf"
Sep 10 22:07:48.668714 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:07:48.668694    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lt6b\" (UniqueName: \"kubernetes.io/projected/d16e4185-bc86-4442-81a4-9ca18f1b49d2-kube-api-access-6lt6b\") pod \"opendatahub-operator-controller-manager-76d44b6d5b-vjglf\" (UID: \"d16e4185-bc86-4442-81a4-9ca18f1b49d2\") " pod="opendatahub/opendatahub-operator-controller-manager-76d44b6d5b-vjglf"
Sep 10 22:07:48.803335 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:07:48.803272    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-76d44b6d5b-vjglf"
Sep 10 22:07:49.103187 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:07:49.103104    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-76d44b6d5b-vjglf"]
Sep 10 22:07:49.107332 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:07:49.107304    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd16e4185_bc86_4442_81a4_9ca18f1b49d2.slice/crio-d93ad91602693d33786c24434bab04f49a194da549f096ee7488d2c38b700920 WatchSource:0}: Error finding container d93ad91602693d33786c24434bab04f49a194da549f096ee7488d2c38b700920: Status 404 returned error can't find the container with id d93ad91602693d33786c24434bab04f49a194da549f096ee7488d2c38b700920
Sep 10 22:07:50.034376 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:07:50.034327    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-76d44b6d5b-vjglf" event={"ID":"d16e4185-bc86-4442-81a4-9ca18f1b49d2","Type":"ContainerStarted","Data":"d93ad91602693d33786c24434bab04f49a194da549f096ee7488d2c38b700920"}
Sep 10 22:07:53.048792 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:07:53.048754    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-76d44b6d5b-vjglf" event={"ID":"d16e4185-bc86-4442-81a4-9ca18f1b49d2","Type":"ContainerStarted","Data":"f1f384fc20530f3ec2380f389d7c3f21e4f668f47e1bfd646b99d7199262f7c5"}
Sep 10 22:07:54.053731 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:07:54.053695    2582 generic.go:356] "Generic (PLEG): container finished" podID="d16e4185-bc86-4442-81a4-9ca18f1b49d2" containerID="f1f384fc20530f3ec2380f389d7c3f21e4f668f47e1bfd646b99d7199262f7c5" exitCode=0
Sep 10 22:07:54.054143 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:07:54.053761    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-76d44b6d5b-vjglf" event={"ID":"d16e4185-bc86-4442-81a4-9ca18f1b49d2","Type":"ContainerDied","Data":"f1f384fc20530f3ec2380f389d7c3f21e4f668f47e1bfd646b99d7199262f7c5"}
Sep 10 22:07:55.059384 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:07:55.059348    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-76d44b6d5b-vjglf" event={"ID":"d16e4185-bc86-4442-81a4-9ca18f1b49d2","Type":"ContainerStarted","Data":"7a0d033f667ed0fbbf4bf4f3a41629c365bf9c3d0555685ba79a7fa3b7454da0"}
Sep 10 22:07:55.059776 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:07:55.059477    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-76d44b6d5b-vjglf"
Sep 10 22:07:55.082566 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:07:55.082517    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-76d44b6d5b-vjglf" podStartSLOduration=2.7672029719999998 podStartE2EDuration="7.082502251s" totalImagesPullingTime="3.287144271s" totalInitContainerRuntime="1.028155008s" isStatefulPod=false podCreationTimestamp="2026-09-10 22:07:48 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:07:55.080566597 +0000 UTC m=+396.036634302" watchObservedRunningTime="2026-09-10 22:07:55.082502251 +0000 UTC m=+396.038569957"
Sep 10 22:08:06.065372 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:08:06.065337    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-76d44b6d5b-vjglf"
Sep 10 22:08:08.456772 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:08:08.456736    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-69ccfddbf-qbrxk"]
Sep 10 22:08:08.465313 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:08:08.465283    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-69ccfddbf-qbrxk"
Sep 10 22:08:08.467355 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:08:08.467331    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 10 22:08:08.467760 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:08:08.467740    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-z59kw\""
Sep 10 22:08:08.469473 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:08:08.469453    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 10 22:08:08.482454 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:08:08.482429    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-69ccfddbf-qbrxk"]
Sep 10 22:08:08.507604 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:08:08.507571    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qnlf\" (UniqueName: \"kubernetes.io/projected/eff019c7-6e59-467a-a49e-8b761744ac47-kube-api-access-7qnlf\") pod \"kube-auth-proxy-69ccfddbf-qbrxk\" (UID: \"eff019c7-6e59-467a-a49e-8b761744ac47\") " pod="openshift-ingress/kube-auth-proxy-69ccfddbf-qbrxk"
Sep 10 22:08:08.507788 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:08:08.507619    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eff019c7-6e59-467a-a49e-8b761744ac47-tls-certs\") pod \"kube-auth-proxy-69ccfddbf-qbrxk\" (UID: \"eff019c7-6e59-467a-a49e-8b761744ac47\") " pod="openshift-ingress/kube-auth-proxy-69ccfddbf-qbrxk"
Sep 10 22:08:08.507788 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:08:08.507655    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eff019c7-6e59-467a-a49e-8b761744ac47-tmp\") pod \"kube-auth-proxy-69ccfddbf-qbrxk\" (UID: \"eff019c7-6e59-467a-a49e-8b761744ac47\") " pod="openshift-ingress/kube-auth-proxy-69ccfddbf-qbrxk"
Sep 10 22:08:08.608654 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:08:08.608617    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qnlf\" (UniqueName: \"kubernetes.io/projected/eff019c7-6e59-467a-a49e-8b761744ac47-kube-api-access-7qnlf\") pod \"kube-auth-proxy-69ccfddbf-qbrxk\" (UID: \"eff019c7-6e59-467a-a49e-8b761744ac47\") " pod="openshift-ingress/kube-auth-proxy-69ccfddbf-qbrxk"
Sep 10 22:08:08.608874 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:08:08.608670    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eff019c7-6e59-467a-a49e-8b761744ac47-tls-certs\") pod \"kube-auth-proxy-69ccfddbf-qbrxk\" (UID: \"eff019c7-6e59-467a-a49e-8b761744ac47\") " pod="openshift-ingress/kube-auth-proxy-69ccfddbf-qbrxk"
Sep 10 22:08:08.608874 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:08:08.608720    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eff019c7-6e59-467a-a49e-8b761744ac47-tmp\") pod \"kube-auth-proxy-69ccfddbf-qbrxk\" (UID: \"eff019c7-6e59-467a-a49e-8b761744ac47\") " pod="openshift-ingress/kube-auth-proxy-69ccfddbf-qbrxk"
Sep 10 22:08:08.608968 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:08:08.608869    2582 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found
Sep 10 22:08:08.608968 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:08:08.608933    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eff019c7-6e59-467a-a49e-8b761744ac47-tls-certs podName:eff019c7-6e59-467a-a49e-8b761744ac47 nodeName:}" failed. No retries permitted until 2026-09-10 22:08:09.108916764 +0000 UTC m=+410.064984448 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/eff019c7-6e59-467a-a49e-8b761744ac47-tls-certs") pod "kube-auth-proxy-69ccfddbf-qbrxk" (UID: "eff019c7-6e59-467a-a49e-8b761744ac47") : secret "kube-auth-proxy-tls" not found
Sep 10 22:08:08.611033 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:08:08.611014    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eff019c7-6e59-467a-a49e-8b761744ac47-tmp\") pod \"kube-auth-proxy-69ccfddbf-qbrxk\" (UID: \"eff019c7-6e59-467a-a49e-8b761744ac47\") " pod="openshift-ingress/kube-auth-proxy-69ccfddbf-qbrxk"
Sep 10 22:08:08.628001 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:08:08.627976    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qnlf\" (UniqueName: \"kubernetes.io/projected/eff019c7-6e59-467a-a49e-8b761744ac47-kube-api-access-7qnlf\") pod \"kube-auth-proxy-69ccfddbf-qbrxk\" (UID: \"eff019c7-6e59-467a-a49e-8b761744ac47\") " pod="openshift-ingress/kube-auth-proxy-69ccfddbf-qbrxk"
Sep 10 22:08:09.113427 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:08:09.113382    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eff019c7-6e59-467a-a49e-8b761744ac47-tls-certs\") pod \"kube-auth-proxy-69ccfddbf-qbrxk\" (UID: \"eff019c7-6e59-467a-a49e-8b761744ac47\") " pod="openshift-ingress/kube-auth-proxy-69ccfddbf-qbrxk"
Sep 10 22:08:09.115924 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:08:09.115890    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eff019c7-6e59-467a-a49e-8b761744ac47-tls-certs\") pod \"kube-auth-proxy-69ccfddbf-qbrxk\" (UID: \"eff019c7-6e59-467a-a49e-8b761744ac47\") " pod="openshift-ingress/kube-auth-proxy-69ccfddbf-qbrxk"
Sep 10 22:08:09.375106 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:08:09.375017    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-69ccfddbf-qbrxk"
Sep 10 22:08:09.656061 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:08:09.656026    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-69ccfddbf-qbrxk"]
Sep 10 22:08:09.659785 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:08:09.659754    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeff019c7_6e59_467a_a49e_8b761744ac47.slice/crio-468bdaa00f7e9014ca422512faa1474426099dcc52b2272d72db0493409b167a WatchSource:0}: Error finding container 468bdaa00f7e9014ca422512faa1474426099dcc52b2272d72db0493409b167a: Status 404 returned error can't find the container with id 468bdaa00f7e9014ca422512faa1474426099dcc52b2272d72db0493409b167a
Sep 10 22:08:10.113254 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:08:10.113161    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-69ccfddbf-qbrxk" event={"ID":"eff019c7-6e59-467a-a49e-8b761744ac47","Type":"ContainerStarted","Data":"468bdaa00f7e9014ca422512faa1474426099dcc52b2272d72db0493409b167a"}
Sep 10 22:08:13.127911 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:08:13.127870    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-69ccfddbf-qbrxk" event={"ID":"eff019c7-6e59-467a-a49e-8b761744ac47","Type":"ContainerStarted","Data":"1eff5d1a94bc2fe591d917704910a2bb9135e0a270a973a9a48a5b1e311d96d4"}
Sep 10 22:08:13.145484 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:08:13.145431    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-69ccfddbf-qbrxk" podStartSLOduration=2.44067684 podStartE2EDuration="5.145415309s" totalImagesPullingTime="2.704738469s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 22:08:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:08:13.1437896 +0000 UTC m=+414.099857305" watchObservedRunningTime="2026-09-10 22:08:13.145415309 +0000 UTC m=+414.101483016"
Sep 10 22:08:15.636024 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:08:15.635989    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-q8r9d"]
Sep 10 22:08:15.639113 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:08:15.639097    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-q8r9d"
Sep 10 22:08:15.641715 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:08:15.641695    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 10 22:08:15.641814 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:08:15.641732    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-cbjf2\""
Sep 10 22:08:15.658801 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:08:15.658773    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-q8r9d"]
Sep 10 22:08:15.769280 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:08:15.769251    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2892c71-f59d-454f-81dc-168d4259d13f-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-q8r9d\" (UID: \"d2892c71-f59d-454f-81dc-168d4259d13f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-q8r9d"
Sep 10 22:08:15.769280 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:08:15.769284    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2892c71-f59d-454f-81dc-168d4259d13f-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-q8r9d\" (UID: \"d2892c71-f59d-454f-81dc-168d4259d13f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-q8r9d"
Sep 10 22:08:15.769479 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:08:15.769307    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2892c71-f59d-454f-81dc-168d4259d13f-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-q8r9d\" (UID: \"d2892c71-f59d-454f-81dc-168d4259d13f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-q8r9d"
Sep 10 22:08:15.769479 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:08:15.769355    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9g87\" (UniqueName: \"kubernetes.io/projected/d2892c71-f59d-454f-81dc-168d4259d13f-kube-api-access-q9g87\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-q8r9d\" (UID: \"d2892c71-f59d-454f-81dc-168d4259d13f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-q8r9d"
Sep 10 22:08:15.769479 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:08:15.769409    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2892c71-f59d-454f-81dc-168d4259d13f-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-q8r9d\" (UID: \"d2892c71-f59d-454f-81dc-168d4259d13f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-q8r9d"
Sep 10 22:08:15.769479 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:08:15.769424    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2892c71-f59d-454f-81dc-168d4259d13f-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-q8r9d\" (UID: \"d2892c71-f59d-454f-81dc-168d4259d13f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-q8r9d"
Sep 10 22:08:15.769479 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:08:15.769438    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2892c71-f59d-454f-81dc-168d4259d13f-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-q8r9d\" (UID: \"d2892c71-f59d-454f-81dc-168d4259d13f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-q8r9d"
Sep 10 22:08:15.769627 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:08:15.769491    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2892c71-f59d-454f-81dc-168d4259d13f-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-q8r9d\" (UID: \"d2892c71-f59d-454f-81dc-168d4259d13f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-q8r9d"
Sep 10 22:08:15.769627 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:08:15.769521    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2892c71-f59d-454f-81dc-168d4259d13f-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-q8r9d\" (UID: \"d2892c71-f59d-454f-81dc-168d4259d13f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-q8r9d"
Sep 10 22:08:15.870827 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:08:15.870792    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2892c71-f59d-454f-81dc-168d4259d13f-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-q8r9d\" (UID: \"d2892c71-f59d-454f-81dc-168d4259d13f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-q8r9d"
Sep 10 22:08:15.870827 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:08:15.870827    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2892c71-f59d-454f-81dc-168d4259d13f-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-q8r9d\" (UID: \"d2892c71-f59d-454f-81dc-168d4259d13f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-q8r9d"
Sep 10 22:08:15.871052 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:08:15.870868    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2892c71-f59d-454f-81dc-168d4259d13f-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-q8r9d\" (UID: \"d2892c71-f59d-454f-81dc-168d4259d13f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-q8r9d"
Sep 10 22:08:15.871052 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:08:15.870906    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2892c71-f59d-454f-81dc-168d4259d13f-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-q8r9d\" (UID: \"d2892c71-f59d-454f-81dc-168d4259d13f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-q8r9d"
Sep 10 22:08:15.871052 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:08:15.870940    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2892c71-f59d-454f-81dc-168d4259d13f-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-q8r9d\" (UID: \"d2892c71-f59d-454f-81dc-168d4259d13f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-q8r9d"
Sep 10 22:08:15.871052 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:08:15.871011    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2892c71-f59d-454f-81dc-168d4259d13f-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-q8r9d\" (UID: \"d2892c71-f59d-454f-81dc-168d4259d13f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-q8r9d"
Sep 10 22:08:15.871052 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:08:15.871040    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2892c71-f59d-454f-81dc-168d4259d13f-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-q8r9d\" (UID: \"d2892c71-f59d-454f-81dc-168d4259d13f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-q8r9d"
Sep 10 22:08:15.871310 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:08:15.871077    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2892c71-f59d-454f-81dc-168d4259d13f-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-q8r9d\" (UID: \"d2892c71-f59d-454f-81dc-168d4259d13f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-q8r9d"
Sep 10 22:08:15.871310 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:08:15.871105    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9g87\" (UniqueName: \"kubernetes.io/projected/d2892c71-f59d-454f-81dc-168d4259d13f-kube-api-access-q9g87\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-q8r9d\" (UID: \"d2892c71-f59d-454f-81dc-168d4259d13f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-q8r9d"
Sep 10 22:08:15.871310 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:08:15.871295    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2892c71-f59d-454f-81dc-168d4259d13f-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-q8r9d\" (UID: \"d2892c71-f59d-454f-81dc-168d4259d13f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-q8r9d"
Sep 10 22:08:15.871466 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:08:15.871356    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2892c71-f59d-454f-81dc-168d4259d13f-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-q8r9d\" (UID: \"d2892c71-f59d-454f-81dc-168d4259d13f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-q8r9d"
Sep 10 22:08:15.871466 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:08:15.871447    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2892c71-f59d-454f-81dc-168d4259d13f-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-q8r9d\" (UID: \"d2892c71-f59d-454f-81dc-168d4259d13f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-q8r9d"
Sep 10 22:08:15.871568 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:08:15.871525    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2892c71-f59d-454f-81dc-168d4259d13f-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-q8r9d\" (UID: \"d2892c71-f59d-454f-81dc-168d4259d13f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-q8r9d"
Sep 10 22:08:15.871868 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:08:15.871832    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2892c71-f59d-454f-81dc-168d4259d13f-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-q8r9d\" (UID: \"d2892c71-f59d-454f-81dc-168d4259d13f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-q8r9d"
Sep 10 22:08:15.873287 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:08:15.873265    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2892c71-f59d-454f-81dc-168d4259d13f-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-q8r9d\" (UID: \"d2892c71-f59d-454f-81dc-168d4259d13f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-q8r9d"
Sep 10 22:08:15.873527 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:08:15.873500    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2892c71-f59d-454f-81dc-168d4259d13f-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-q8r9d\" (UID: \"d2892c71-f59d-454f-81dc-168d4259d13f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-q8r9d"
Sep 10 22:08:15.882078 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:08:15.882053    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2892c71-f59d-454f-81dc-168d4259d13f-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-q8r9d\" (UID: \"d2892c71-f59d-454f-81dc-168d4259d13f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-q8r9d"
Sep 10 22:08:15.883364 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:08:15.883343    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9g87\" (UniqueName: \"kubernetes.io/projected/d2892c71-f59d-454f-81dc-168d4259d13f-kube-api-access-q9g87\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-q8r9d\" (UID: \"d2892c71-f59d-454f-81dc-168d4259d13f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-q8r9d"
Sep 10 22:08:15.948345 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:08:15.948321    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-q8r9d"
Sep 10 22:08:16.232030 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:08:16.231979    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-q8r9d"]
Sep 10 22:08:16.234228 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:08:16.234199    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2892c71_f59d_454f_81dc_168d4259d13f.slice/crio-23f9201e3d4db87f4e056d2eb3802e7356301a83f302e19bec8c483a740c28c1 WatchSource:0}: Error finding container 23f9201e3d4db87f4e056d2eb3802e7356301a83f302e19bec8c483a740c28c1: Status 404 returned error can't find the container with id 23f9201e3d4db87f4e056d2eb3802e7356301a83f302e19bec8c483a740c28c1
Sep 10 22:08:17.145933 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:08:17.145896    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-q8r9d" event={"ID":"d2892c71-f59d-454f-81dc-168d4259d13f","Type":"ContainerStarted","Data":"23f9201e3d4db87f4e056d2eb3802e7356301a83f302e19bec8c483a740c28c1"}
Sep 10 22:08:19.110523 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:08:19.110489    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-8fs8w"]
Sep 10 22:08:19.120461 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:08:19.120436    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-8fs8w"
Sep 10 22:08:19.122739 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:08:19.122710    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Sep 10 22:08:19.122739 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:08:19.122732    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-7chhd\""
Sep 10 22:08:19.123556 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:08:19.123533    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-8fs8w"]
Sep 10 22:08:19.193588 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:08:19.193538    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 22:08:19.193765 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:08:19.193621    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 22:08:19.193765 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:08:19.193647    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 22:08:19.201690 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:08:19.201666    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8z46\" (UniqueName: \"kubernetes.io/projected/fadc83c6-d3c0-42fd-bf1c-b9137805a88a-kube-api-access-v8z46\") pod \"odh-model-controller-7998c67dc4-8fs8w\" (UID: \"fadc83c6-d3c0-42fd-bf1c-b9137805a88a\") " pod="opendatahub/odh-model-controller-7998c67dc4-8fs8w"
Sep 10 22:08:19.201756 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:08:19.201697    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fadc83c6-d3c0-42fd-bf1c-b9137805a88a-cert\") pod \"odh-model-controller-7998c67dc4-8fs8w\" (UID: \"fadc83c6-d3c0-42fd-bf1c-b9137805a88a\") " pod="opendatahub/odh-model-controller-7998c67dc4-8fs8w"
Sep 10 22:08:19.303007 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:08:19.302969    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8z46\" (UniqueName: \"kubernetes.io/projected/fadc83c6-d3c0-42fd-bf1c-b9137805a88a-kube-api-access-v8z46\") pod \"odh-model-controller-7998c67dc4-8fs8w\" (UID: \"fadc83c6-d3c0-42fd-bf1c-b9137805a88a\") " pod="opendatahub/odh-model-controller-7998c67dc4-8fs8w"
Sep 10 22:08:19.303195 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:08:19.303015    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fadc83c6-d3c0-42fd-bf1c-b9137805a88a-cert\") pod \"odh-model-controller-7998c67dc4-8fs8w\" (UID: \"fadc83c6-d3c0-42fd-bf1c-b9137805a88a\") " pod="opendatahub/odh-model-controller-7998c67dc4-8fs8w"
Sep 10 22:08:19.303195 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:08:19.303138    2582 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Sep 10 22:08:19.303303 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:08:19.303200    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fadc83c6-d3c0-42fd-bf1c-b9137805a88a-cert podName:fadc83c6-d3c0-42fd-bf1c-b9137805a88a nodeName:}" failed. No retries permitted until 2026-09-10 22:08:19.80318008 +0000 UTC m=+420.759247764 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/fadc83c6-d3c0-42fd-bf1c-b9137805a88a-cert") pod "odh-model-controller-7998c67dc4-8fs8w" (UID: "fadc83c6-d3c0-42fd-bf1c-b9137805a88a") : secret "odh-model-controller-webhook-cert" not found
Sep 10 22:08:19.311412 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:08:19.311388    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8z46\" (UniqueName: \"kubernetes.io/projected/fadc83c6-d3c0-42fd-bf1c-b9137805a88a-kube-api-access-v8z46\") pod \"odh-model-controller-7998c67dc4-8fs8w\" (UID: \"fadc83c6-d3c0-42fd-bf1c-b9137805a88a\") " pod="opendatahub/odh-model-controller-7998c67dc4-8fs8w"
Sep 10 22:08:19.728788 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:08:19.728759    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-r2b97"]
Sep 10 22:08:19.733250 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:08:19.733234    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-r2b97"
Sep 10 22:08:19.735988 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:08:19.735969    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Sep 10 22:08:19.736099 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:08:19.735973    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-2ftj8\""
Sep 10 22:08:19.750181 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:08:19.750154    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-r2b97"]
Sep 10 22:08:19.807779 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:08:19.807743    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fadc83c6-d3c0-42fd-bf1c-b9137805a88a-cert\") pod \"odh-model-controller-7998c67dc4-8fs8w\" (UID: \"fadc83c6-d3c0-42fd-bf1c-b9137805a88a\") " pod="opendatahub/odh-model-controller-7998c67dc4-8fs8w"
Sep 10 22:08:19.807989 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:08:19.807786    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvq8t\" (UniqueName: \"kubernetes.io/projected/6760a7a5-e21c-4606-89a3-f78554990043-kube-api-access-wvq8t\") pod \"model-serving-api-54d89bd5fc-r2b97\" (UID: \"6760a7a5-e21c-4606-89a3-f78554990043\") " pod="opendatahub/model-serving-api-54d89bd5fc-r2b97"
Sep 10 22:08:19.807989 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:08:19.807842    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6760a7a5-e21c-4606-89a3-f78554990043-tls-certs\") pod \"model-serving-api-54d89bd5fc-r2b97\" (UID: \"6760a7a5-e21c-4606-89a3-f78554990043\") " pod="opendatahub/model-serving-api-54d89bd5fc-r2b97"
Sep 10 22:08:19.807989 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:08:19.807933    2582 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Sep 10 22:08:19.808113 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:08:19.807999    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fadc83c6-d3c0-42fd-bf1c-b9137805a88a-cert podName:fadc83c6-d3c0-42fd-bf1c-b9137805a88a nodeName:}" failed. No retries permitted until 2026-09-10 22:08:20.807981615 +0000 UTC m=+421.764049322 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/fadc83c6-d3c0-42fd-bf1c-b9137805a88a-cert") pod "odh-model-controller-7998c67dc4-8fs8w" (UID: "fadc83c6-d3c0-42fd-bf1c-b9137805a88a") : secret "odh-model-controller-webhook-cert" not found
Sep 10 22:08:19.908362 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:08:19.908326    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6760a7a5-e21c-4606-89a3-f78554990043-tls-certs\") pod \"model-serving-api-54d89bd5fc-r2b97\" (UID: \"6760a7a5-e21c-4606-89a3-f78554990043\") " pod="opendatahub/model-serving-api-54d89bd5fc-r2b97"
Sep 10 22:08:19.908539 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:08:19.908472    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvq8t\" (UniqueName: \"kubernetes.io/projected/6760a7a5-e21c-4606-89a3-f78554990043-kube-api-access-wvq8t\") pod \"model-serving-api-54d89bd5fc-r2b97\" (UID: \"6760a7a5-e21c-4606-89a3-f78554990043\") " pod="opendatahub/model-serving-api-54d89bd5fc-r2b97"
Sep 10 22:08:19.908539 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:08:19.908481    2582 secret.go:189] Couldn't get secret opendatahub/model-serving-api-tls: secret "model-serving-api-tls" not found
Sep 10 22:08:19.908610 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:08:19.908544    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6760a7a5-e21c-4606-89a3-f78554990043-tls-certs podName:6760a7a5-e21c-4606-89a3-f78554990043 nodeName:}" failed. No retries permitted until 2026-09-10 22:08:20.40852548 +0000 UTC m=+421.364593167 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/6760a7a5-e21c-4606-89a3-f78554990043-tls-certs") pod "model-serving-api-54d89bd5fc-r2b97" (UID: "6760a7a5-e21c-4606-89a3-f78554990043") : secret "model-serving-api-tls" not found
Sep 10 22:08:19.916527 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:08:19.916495    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvq8t\" (UniqueName: \"kubernetes.io/projected/6760a7a5-e21c-4606-89a3-f78554990043-kube-api-access-wvq8t\") pod \"model-serving-api-54d89bd5fc-r2b97\" (UID: \"6760a7a5-e21c-4606-89a3-f78554990043\") " pod="opendatahub/model-serving-api-54d89bd5fc-r2b97"
Sep 10 22:08:20.162013 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:08:20.161977    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-q8r9d" event={"ID":"d2892c71-f59d-454f-81dc-168d4259d13f","Type":"ContainerStarted","Data":"ce66b69d71fb56fa8c72acdfaef72e5fa63400d8ea308d6f1daa601915416bd2"}
Sep 10 22:08:20.181712 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:08:20.181575    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-q8r9d" podStartSLOduration=2.224468227 podStartE2EDuration="5.181555046s" totalImagesPullingTime="2.957086819s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 22:08:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:08:20.181377031 +0000 UTC m=+421.137444739" watchObservedRunningTime="2026-09-10 22:08:20.181555046 +0000 UTC m=+421.137622732"
Sep 10 22:08:20.414336 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:08:20.414245    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6760a7a5-e21c-4606-89a3-f78554990043-tls-certs\") pod \"model-serving-api-54d89bd5fc-r2b97\" (UID: \"6760a7a5-e21c-4606-89a3-f78554990043\") " pod="opendatahub/model-serving-api-54d89bd5fc-r2b97"
Sep 10 22:08:20.416669 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:08:20.416640    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6760a7a5-e21c-4606-89a3-f78554990043-tls-certs\") pod \"model-serving-api-54d89bd5fc-r2b97\" (UID: \"6760a7a5-e21c-4606-89a3-f78554990043\") " pod="opendatahub/model-serving-api-54d89bd5fc-r2b97"
Sep 10 22:08:20.643078 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:08:20.643027    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-r2b97"
Sep 10 22:08:20.818586 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:08:20.818527    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fadc83c6-d3c0-42fd-bf1c-b9137805a88a-cert\") pod \"odh-model-controller-7998c67dc4-8fs8w\" (UID: \"fadc83c6-d3c0-42fd-bf1c-b9137805a88a\") " pod="opendatahub/odh-model-controller-7998c67dc4-8fs8w"
Sep 10 22:08:20.820996 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:08:20.820957    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fadc83c6-d3c0-42fd-bf1c-b9137805a88a-cert\") pod \"odh-model-controller-7998c67dc4-8fs8w\" (UID: \"fadc83c6-d3c0-42fd-bf1c-b9137805a88a\") " pod="opendatahub/odh-model-controller-7998c67dc4-8fs8w"
Sep 10 22:08:20.913398 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:08:20.913370    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-r2b97"]
Sep 10 22:08:20.915683 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:08:20.915658    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6760a7a5_e21c_4606_89a3_f78554990043.slice/crio-99cafbf24e2e2896bc5b8322ee725199b38aaf576c9568cd75dd5e1669d45e0a WatchSource:0}: Error finding container 99cafbf24e2e2896bc5b8322ee725199b38aaf576c9568cd75dd5e1669d45e0a: Status 404 returned error can't find the container with id 99cafbf24e2e2896bc5b8322ee725199b38aaf576c9568cd75dd5e1669d45e0a
Sep 10 22:08:20.934296 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:08:20.934266    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-7chhd\""
Sep 10 22:08:20.943385 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:08:20.943312    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-8fs8w"
Sep 10 22:08:20.949505 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:08:20.949486    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-q8r9d"
Sep 10 22:08:20.954486 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:08:20.954462    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-q8r9d"
Sep 10 22:08:21.167685 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:08:21.167644    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-r2b97" event={"ID":"6760a7a5-e21c-4606-89a3-f78554990043","Type":"ContainerStarted","Data":"99cafbf24e2e2896bc5b8322ee725199b38aaf576c9568cd75dd5e1669d45e0a"}
Sep 10 22:08:21.168201 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:08:21.167896    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-q8r9d"
Sep 10 22:08:21.168803 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:08:21.168786    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-q8r9d"
Sep 10 22:08:21.216391 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:08:21.216366    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-8fs8w"]
Sep 10 22:08:21.218775 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:08:21.218735    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfadc83c6_d3c0_42fd_bf1c_b9137805a88a.slice/crio-0c962e0b1594ebe675ddbd485a12007ca630e9aabda8f0f42fed1a8593afd7da WatchSource:0}: Error finding container 0c962e0b1594ebe675ddbd485a12007ca630e9aabda8f0f42fed1a8593afd7da: Status 404 returned error can't find the container with id 0c962e0b1594ebe675ddbd485a12007ca630e9aabda8f0f42fed1a8593afd7da
Sep 10 22:08:22.174271 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:08:22.174232    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-8fs8w" event={"ID":"fadc83c6-d3c0-42fd-bf1c-b9137805a88a","Type":"ContainerStarted","Data":"0c962e0b1594ebe675ddbd485a12007ca630e9aabda8f0f42fed1a8593afd7da"}
Sep 10 22:08:23.180678 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:08:23.180637    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-r2b97" event={"ID":"6760a7a5-e21c-4606-89a3-f78554990043","Type":"ContainerStarted","Data":"2f2231966eccc82f6976f590c06a8378a383ef59f952821844c97d75fd463785"}
Sep 10 22:08:23.197065 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:08:23.197015    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-r2b97" podStartSLOduration=2.540826079 podStartE2EDuration="4.196996778s" totalImagesPullingTime="1.656170699s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 22:08:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:08:23.195754762 +0000 UTC m=+424.151822491" watchObservedRunningTime="2026-09-10 22:08:23.196996778 +0000 UTC m=+424.153064481"
Sep 10 22:08:24.187130 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:08:24.187083    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-8fs8w" event={"ID":"fadc83c6-d3c0-42fd-bf1c-b9137805a88a","Type":"ContainerStarted","Data":"9e7f20ba98c2dc264ec4964393e1565fa18d5881fb11d7c64707b53c2b8da1a7"}
Sep 10 22:08:24.187786 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:08:24.187214    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-8fs8w"
Sep 10 22:08:24.187786 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:08:24.187266    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-r2b97"
Sep 10 22:08:24.204430 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:08:24.204370    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-8fs8w" podStartSLOduration=2.5392249700000002 podStartE2EDuration="5.204351852s" totalImagesPullingTime="2.665126882s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 22:08:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:08:24.203533791 +0000 UTC m=+425.159601499" watchObservedRunningTime="2026-09-10 22:08:24.204351852 +0000 UTC m=+425.160419559"
Sep 10 22:08:25.224320 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:08:25.224286    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-56f58b8775-bsmcq"]
Sep 10 22:08:25.229996 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:08:25.229973    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-56f58b8775-bsmcq"
Sep 10 22:08:25.232188 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:08:25.232167    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 10 22:08:25.232828 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:08:25.232805    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-4whms\""
Sep 10 22:08:25.232943 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:08:25.232814    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 10 22:08:25.232943 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:08:25.232865    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 10 22:08:25.236037 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:08:25.236016    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-56f58b8775-bsmcq"]
Sep 10 22:08:25.355195 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:08:25.355157    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66c55\" (UniqueName: \"kubernetes.io/projected/a2719a25-d4d6-419f-8310-56426b57f105-kube-api-access-66c55\") pod \"lws-controller-manager-56f58b8775-bsmcq\" (UID: \"a2719a25-d4d6-419f-8310-56426b57f105\") " pod="openshift-lws-operator/lws-controller-manager-56f58b8775-bsmcq"
Sep 10 22:08:25.355195 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:08:25.355202    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a2719a25-d4d6-419f-8310-56426b57f105-metrics-cert\") pod \"lws-controller-manager-56f58b8775-bsmcq\" (UID: \"a2719a25-d4d6-419f-8310-56426b57f105\") " pod="openshift-lws-operator/lws-controller-manager-56f58b8775-bsmcq"
Sep 10 22:08:25.355438 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:08:25.355229    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a2719a25-d4d6-419f-8310-56426b57f105-cert\") pod \"lws-controller-manager-56f58b8775-bsmcq\" (UID: \"a2719a25-d4d6-419f-8310-56426b57f105\") " pod="openshift-lws-operator/lws-controller-manager-56f58b8775-bsmcq"
Sep 10 22:08:25.355438 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:08:25.355312    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/a2719a25-d4d6-419f-8310-56426b57f105-manager-config\") pod \"lws-controller-manager-56f58b8775-bsmcq\" (UID: \"a2719a25-d4d6-419f-8310-56426b57f105\") " pod="openshift-lws-operator/lws-controller-manager-56f58b8775-bsmcq"
Sep 10 22:08:25.456399 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:08:25.456350    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66c55\" (UniqueName: \"kubernetes.io/projected/a2719a25-d4d6-419f-8310-56426b57f105-kube-api-access-66c55\") pod \"lws-controller-manager-56f58b8775-bsmcq\" (UID: \"a2719a25-d4d6-419f-8310-56426b57f105\") " pod="openshift-lws-operator/lws-controller-manager-56f58b8775-bsmcq"
Sep 10 22:08:25.456399 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:08:25.456402    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a2719a25-d4d6-419f-8310-56426b57f105-metrics-cert\") pod \"lws-controller-manager-56f58b8775-bsmcq\" (UID: \"a2719a25-d4d6-419f-8310-56426b57f105\") " pod="openshift-lws-operator/lws-controller-manager-56f58b8775-bsmcq"
Sep 10 22:08:25.456666 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:08:25.456425    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a2719a25-d4d6-419f-8310-56426b57f105-cert\") pod \"lws-controller-manager-56f58b8775-bsmcq\" (UID: \"a2719a25-d4d6-419f-8310-56426b57f105\") " pod="openshift-lws-operator/lws-controller-manager-56f58b8775-bsmcq"
Sep 10 22:08:25.456666 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:08:25.456541    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/a2719a25-d4d6-419f-8310-56426b57f105-manager-config\") pod \"lws-controller-manager-56f58b8775-bsmcq\" (UID: \"a2719a25-d4d6-419f-8310-56426b57f105\") " pod="openshift-lws-operator/lws-controller-manager-56f58b8775-bsmcq"
Sep 10 22:08:25.457203 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:08:25.457179    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/a2719a25-d4d6-419f-8310-56426b57f105-manager-config\") pod \"lws-controller-manager-56f58b8775-bsmcq\" (UID: \"a2719a25-d4d6-419f-8310-56426b57f105\") " pod="openshift-lws-operator/lws-controller-manager-56f58b8775-bsmcq"
Sep 10 22:08:25.459035 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:08:25.459016    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a2719a25-d4d6-419f-8310-56426b57f105-cert\") pod \"lws-controller-manager-56f58b8775-bsmcq\" (UID: \"a2719a25-d4d6-419f-8310-56426b57f105\") " pod="openshift-lws-operator/lws-controller-manager-56f58b8775-bsmcq"
Sep 10 22:08:25.459249 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:08:25.459077    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a2719a25-d4d6-419f-8310-56426b57f105-metrics-cert\") pod \"lws-controller-manager-56f58b8775-bsmcq\" (UID: \"a2719a25-d4d6-419f-8310-56426b57f105\") " pod="openshift-lws-operator/lws-controller-manager-56f58b8775-bsmcq"
Sep 10 22:08:25.475105 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:08:25.475048    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-66c55\" (UniqueName: \"kubernetes.io/projected/a2719a25-d4d6-419f-8310-56426b57f105-kube-api-access-66c55\") pod \"lws-controller-manager-56f58b8775-bsmcq\" (UID: \"a2719a25-d4d6-419f-8310-56426b57f105\") " pod="openshift-lws-operator/lws-controller-manager-56f58b8775-bsmcq"
Sep 10 22:08:25.539736 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:08:25.539706    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-56f58b8775-bsmcq"
Sep 10 22:08:25.825150 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:08:25.825122    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-56f58b8775-bsmcq"]
Sep 10 22:08:25.827779 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:08:25.827752    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2719a25_d4d6_419f_8310_56426b57f105.slice/crio-81930039b2cdd7355c8800ae2eb4015e994a47905b4cda82e86e7d65b3e779a4 WatchSource:0}: Error finding container 81930039b2cdd7355c8800ae2eb4015e994a47905b4cda82e86e7d65b3e779a4: Status 404 returned error can't find the container with id 81930039b2cdd7355c8800ae2eb4015e994a47905b4cda82e86e7d65b3e779a4
Sep 10 22:08:26.196701 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:08:26.196654    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-56f58b8775-bsmcq" event={"ID":"a2719a25-d4d6-419f-8310-56426b57f105","Type":"ContainerStarted","Data":"81930039b2cdd7355c8800ae2eb4015e994a47905b4cda82e86e7d65b3e779a4"}
Sep 10 22:08:28.206656 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:08:28.206621    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-56f58b8775-bsmcq" event={"ID":"a2719a25-d4d6-419f-8310-56426b57f105","Type":"ContainerStarted","Data":"d71a72360006b36e7e41e146be8e70c183f1e9dfa6622492bebc027ef1bfaf1d"}
Sep 10 22:08:28.207075 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:08:28.206691    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-56f58b8775-bsmcq"
Sep 10 22:08:28.241136 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:08:28.241035    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-56f58b8775-bsmcq" podStartSLOduration=1.786591439 podStartE2EDuration="3.241018714s" totalImagesPullingTime="1.454427275s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 22:08:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:08:28.240193731 +0000 UTC m=+429.196261435" watchObservedRunningTime="2026-09-10 22:08:28.241018714 +0000 UTC m=+429.197086420"
Sep 10 22:08:35.193208 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:08:35.193178    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-8fs8w"
Sep 10 22:08:35.195232 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:08:35.195215    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-r2b97"
Sep 10 22:08:39.214920 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:08:39.214887    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-56f58b8775-bsmcq"
Sep 10 22:10:39.246646 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:10:39.246575    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-pgc8r"]
Sep 10 22:10:39.249823 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:10:39.249803    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-pgc8r"
Sep 10 22:10:39.251753 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:10:39.251730    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 10 22:10:39.251845 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:10:39.251767    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 10 22:10:39.252183 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:10:39.252166    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-sbbd5\""
Sep 10 22:10:39.255871 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:10:39.255838    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-pgc8r"]
Sep 10 22:10:39.317593 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:10:39.317569    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57jhr\" (UniqueName: \"kubernetes.io/projected/92d8ea29-adef-4ebf-a33d-dc203c662fb2-kube-api-access-57jhr\") pod \"authorino-7bcfd8f9cd-pgc8r\" (UID: \"92d8ea29-adef-4ebf-a33d-dc203c662fb2\") " pod="kuadrant-system/authorino-7bcfd8f9cd-pgc8r"
Sep 10 22:10:39.418663 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:10:39.418637    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57jhr\" (UniqueName: \"kubernetes.io/projected/92d8ea29-adef-4ebf-a33d-dc203c662fb2-kube-api-access-57jhr\") pod \"authorino-7bcfd8f9cd-pgc8r\" (UID: \"92d8ea29-adef-4ebf-a33d-dc203c662fb2\") " pod="kuadrant-system/authorino-7bcfd8f9cd-pgc8r"
Sep 10 22:10:39.427111 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:10:39.427085    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-57jhr\" (UniqueName: \"kubernetes.io/projected/92d8ea29-adef-4ebf-a33d-dc203c662fb2-kube-api-access-57jhr\") pod \"authorino-7bcfd8f9cd-pgc8r\" (UID: \"92d8ea29-adef-4ebf-a33d-dc203c662fb2\") " pod="kuadrant-system/authorino-7bcfd8f9cd-pgc8r"
Sep 10 22:10:39.448642 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:10:39.448620    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59c974bd48-jj9fw"]
Sep 10 22:10:39.451685 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:10:39.451671    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-jj9fw"
Sep 10 22:10:39.456411 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:10:39.456389    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-jj9fw"]
Sep 10 22:10:39.519259 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:10:39.519193    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lp77k\" (UniqueName: \"kubernetes.io/projected/e43ade8d-035c-438d-a448-5a698f5023eb-kube-api-access-lp77k\") pod \"authorino-59c974bd48-jj9fw\" (UID: \"e43ade8d-035c-438d-a448-5a698f5023eb\") " pod="kuadrant-system/authorino-59c974bd48-jj9fw"
Sep 10 22:10:39.558368 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:10:39.558343    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-pgc8r"
Sep 10 22:10:39.621362 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:10:39.619977    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lp77k\" (UniqueName: \"kubernetes.io/projected/e43ade8d-035c-438d-a448-5a698f5023eb-kube-api-access-lp77k\") pod \"authorino-59c974bd48-jj9fw\" (UID: \"e43ade8d-035c-438d-a448-5a698f5023eb\") " pod="kuadrant-system/authorino-59c974bd48-jj9fw"
Sep 10 22:10:39.629518 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:10:39.629492    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lp77k\" (UniqueName: \"kubernetes.io/projected/e43ade8d-035c-438d-a448-5a698f5023eb-kube-api-access-lp77k\") pod \"authorino-59c974bd48-jj9fw\" (UID: \"e43ade8d-035c-438d-a448-5a698f5023eb\") " pod="kuadrant-system/authorino-59c974bd48-jj9fw"
Sep 10 22:10:39.761937 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:10:39.761900    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-jj9fw"
Sep 10 22:10:39.832752 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:10:39.832566    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-pgc8r"]
Sep 10 22:10:39.837231 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:10:39.837189    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod92d8ea29_adef_4ebf_a33d_dc203c662fb2.slice/crio-9c13b4231d56459bffb486a815fc140e9926a1843f4608b88f6408c6ffc3c325 WatchSource:0}: Error finding container 9c13b4231d56459bffb486a815fc140e9926a1843f4608b88f6408c6ffc3c325: Status 404 returned error can't find the container with id 9c13b4231d56459bffb486a815fc140e9926a1843f4608b88f6408c6ffc3c325
Sep 10 22:10:40.033043 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:10:40.033017    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-jj9fw"]
Sep 10 22:10:40.035080 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:10:40.035050    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode43ade8d_035c_438d_a448_5a698f5023eb.slice/crio-bf49a5382455f69c0cfbb9f5076a96d6a378a68c62685b1da7626ec3a94b95eb WatchSource:0}: Error finding container bf49a5382455f69c0cfbb9f5076a96d6a378a68c62685b1da7626ec3a94b95eb: Status 404 returned error can't find the container with id bf49a5382455f69c0cfbb9f5076a96d6a378a68c62685b1da7626ec3a94b95eb
Sep 10 22:10:40.757935 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:10:40.757895    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-jj9fw" event={"ID":"e43ade8d-035c-438d-a448-5a698f5023eb","Type":"ContainerStarted","Data":"bf49a5382455f69c0cfbb9f5076a96d6a378a68c62685b1da7626ec3a94b95eb"}
Sep 10 22:10:40.759375 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:10:40.759345    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-pgc8r" event={"ID":"92d8ea29-adef-4ebf-a33d-dc203c662fb2","Type":"ContainerStarted","Data":"9c13b4231d56459bffb486a815fc140e9926a1843f4608b88f6408c6ffc3c325"}
Sep 10 22:10:43.775538 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:10:43.775505    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-jj9fw" event={"ID":"e43ade8d-035c-438d-a448-5a698f5023eb","Type":"ContainerStarted","Data":"aa282b11766a45e7e3cd4f986eff32e00ff0a71bbeba13a662849f2f522cffb6"}
Sep 10 22:10:43.777007 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:10:43.776980    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-pgc8r" event={"ID":"92d8ea29-adef-4ebf-a33d-dc203c662fb2","Type":"ContainerStarted","Data":"ecaf60c8f03301c54457d9c4dd37667ea41273be25eca869380e176cfaa390ee"}
Sep 10 22:10:43.789507 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:10:43.789465    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59c974bd48-jj9fw" podStartSLOduration=2.104391677 podStartE2EDuration="4.789450159s" totalImagesPullingTime="2.685058482s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 22:10:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:10:43.788578989 +0000 UTC m=+564.744646694" watchObservedRunningTime="2026-09-10 22:10:43.789450159 +0000 UTC m=+564.745517865"
Sep 10 22:10:43.802595 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:10:43.802553    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7bcfd8f9cd-pgc8r" podStartSLOduration=1.929853272 podStartE2EDuration="4.802533887s" totalImagesPullingTime="2.872680615s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 22:10:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:10:43.801736541 +0000 UTC m=+564.757804251" watchObservedRunningTime="2026-09-10 22:10:43.802533887 +0000 UTC m=+564.758601593"
Sep 10 22:10:43.828374 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:10:43.828347    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-pgc8r"]
Sep 10 22:10:45.785173 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:10:45.785134    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7bcfd8f9cd-pgc8r" podUID="92d8ea29-adef-4ebf-a33d-dc203c662fb2" containerName="authorino" containerID="cri-o://ecaf60c8f03301c54457d9c4dd37667ea41273be25eca869380e176cfaa390ee" gracePeriod=30
Sep 10 22:10:46.022619 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:10:46.022596    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-pgc8r"
Sep 10 22:10:46.175608 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:10:46.175584    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-57jhr\" (UniqueName: \"kubernetes.io/projected/92d8ea29-adef-4ebf-a33d-dc203c662fb2-kube-api-access-57jhr\") pod \"92d8ea29-adef-4ebf-a33d-dc203c662fb2\" (UID: \"92d8ea29-adef-4ebf-a33d-dc203c662fb2\") "
Sep 10 22:10:46.177717 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:10:46.177695    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92d8ea29-adef-4ebf-a33d-dc203c662fb2-kube-api-access-57jhr" (OuterVolumeSpecName: "kube-api-access-57jhr") pod "92d8ea29-adef-4ebf-a33d-dc203c662fb2" (UID: "92d8ea29-adef-4ebf-a33d-dc203c662fb2"). InnerVolumeSpecName "kube-api-access-57jhr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 22:10:46.276298 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:10:46.276270    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-57jhr\" (UniqueName: \"kubernetes.io/projected/92d8ea29-adef-4ebf-a33d-dc203c662fb2-kube-api-access-57jhr\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:10:46.789519 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:10:46.789486    2582 generic.go:356] "Generic (PLEG): container finished" podID="92d8ea29-adef-4ebf-a33d-dc203c662fb2" containerID="ecaf60c8f03301c54457d9c4dd37667ea41273be25eca869380e176cfaa390ee" exitCode=0
Sep 10 22:10:46.789897 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:10:46.789524    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-pgc8r"
Sep 10 22:10:46.789897 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:10:46.789564    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-pgc8r" event={"ID":"92d8ea29-adef-4ebf-a33d-dc203c662fb2","Type":"ContainerDied","Data":"ecaf60c8f03301c54457d9c4dd37667ea41273be25eca869380e176cfaa390ee"}
Sep 10 22:10:46.789897 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:10:46.789605    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-pgc8r" event={"ID":"92d8ea29-adef-4ebf-a33d-dc203c662fb2","Type":"ContainerDied","Data":"9c13b4231d56459bffb486a815fc140e9926a1843f4608b88f6408c6ffc3c325"}
Sep 10 22:10:46.789897 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:10:46.789623    2582 scope.go:117] "RemoveContainer" containerID="ecaf60c8f03301c54457d9c4dd37667ea41273be25eca869380e176cfaa390ee"
Sep 10 22:10:46.798342 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:10:46.798322    2582 scope.go:117] "RemoveContainer" containerID="ecaf60c8f03301c54457d9c4dd37667ea41273be25eca869380e176cfaa390ee"
Sep 10 22:10:46.798614 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:10:46.798594    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ecaf60c8f03301c54457d9c4dd37667ea41273be25eca869380e176cfaa390ee\": container with ID starting with ecaf60c8f03301c54457d9c4dd37667ea41273be25eca869380e176cfaa390ee not found: ID does not exist" containerID="ecaf60c8f03301c54457d9c4dd37667ea41273be25eca869380e176cfaa390ee"
Sep 10 22:10:46.798658 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:10:46.798624    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ecaf60c8f03301c54457d9c4dd37667ea41273be25eca869380e176cfaa390ee"} err="failed to get container status \"ecaf60c8f03301c54457d9c4dd37667ea41273be25eca869380e176cfaa390ee\": rpc error: code = NotFound desc = could not find container \"ecaf60c8f03301c54457d9c4dd37667ea41273be25eca869380e176cfaa390ee\": container with ID starting with ecaf60c8f03301c54457d9c4dd37667ea41273be25eca869380e176cfaa390ee not found: ID does not exist"
Sep 10 22:10:46.809351 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:10:46.809326    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-pgc8r"]
Sep 10 22:10:46.811796 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:10:46.811775    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-pgc8r"]
Sep 10 22:10:47.503378 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:10:47.503340    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="92d8ea29-adef-4ebf-a33d-dc203c662fb2" path="/var/lib/kubelet/pods/92d8ea29-adef-4ebf-a33d-dc203c662fb2/volumes"
Sep 10 22:11:05.097399 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:11:05.097361    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-557f876f69-6bxsm"]
Sep 10 22:11:05.097803 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:11:05.097668    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92d8ea29-adef-4ebf-a33d-dc203c662fb2" containerName="authorino"
Sep 10 22:11:05.097803 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:11:05.097678    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="92d8ea29-adef-4ebf-a33d-dc203c662fb2" containerName="authorino"
Sep 10 22:11:05.097803 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:11:05.097730    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="92d8ea29-adef-4ebf-a33d-dc203c662fb2" containerName="authorino"
Sep 10 22:11:05.101838 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:11:05.101820    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-6bxsm"
Sep 10 22:11:05.109032 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:11:05.109000    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-557f876f69-6bxsm"]
Sep 10 22:11:05.129602 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:11:05.129565    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tst4\" (UniqueName: \"kubernetes.io/projected/c424dd7c-ac15-4c55-ab18-cff0d06c4c7f-kube-api-access-4tst4\") pod \"authorino-557f876f69-6bxsm\" (UID: \"c424dd7c-ac15-4c55-ab18-cff0d06c4c7f\") " pod="kuadrant-system/authorino-557f876f69-6bxsm"
Sep 10 22:11:05.170982 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:11:05.170941    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-6bxsm"]
Sep 10 22:11:05.171205 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:11:05.171185    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-4tst4], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-557f876f69-6bxsm" podUID="c424dd7c-ac15-4c55-ab18-cff0d06c4c7f"
Sep 10 22:11:05.203667 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:11:05.203637    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-bdskz"]
Sep 10 22:11:05.206614 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:11:05.206598    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-bdskz"
Sep 10 22:11:05.208760 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:11:05.208742    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 10 22:11:05.213770 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:11:05.213747    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-bdskz"]
Sep 10 22:11:05.230660 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:11:05.230629    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/43c4db39-0a76-407a-8873-e04465bfd0f6-tls-cert\") pod \"authorino-6d4dfc649f-bdskz\" (UID: \"43c4db39-0a76-407a-8873-e04465bfd0f6\") " pod="kuadrant-system/authorino-6d4dfc649f-bdskz"
Sep 10 22:11:05.230816 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:11:05.230670    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzw2x\" (UniqueName: \"kubernetes.io/projected/43c4db39-0a76-407a-8873-e04465bfd0f6-kube-api-access-tzw2x\") pod \"authorino-6d4dfc649f-bdskz\" (UID: \"43c4db39-0a76-407a-8873-e04465bfd0f6\") " pod="kuadrant-system/authorino-6d4dfc649f-bdskz"
Sep 10 22:11:05.230816 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:11:05.230739    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tst4\" (UniqueName: \"kubernetes.io/projected/c424dd7c-ac15-4c55-ab18-cff0d06c4c7f-kube-api-access-4tst4\") pod \"authorino-557f876f69-6bxsm\" (UID: \"c424dd7c-ac15-4c55-ab18-cff0d06c4c7f\") " pod="kuadrant-system/authorino-557f876f69-6bxsm"
Sep 10 22:11:05.238176 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:11:05.238158    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tst4\" (UniqueName: \"kubernetes.io/projected/c424dd7c-ac15-4c55-ab18-cff0d06c4c7f-kube-api-access-4tst4\") pod \"authorino-557f876f69-6bxsm\" (UID: \"c424dd7c-ac15-4c55-ab18-cff0d06c4c7f\") " pod="kuadrant-system/authorino-557f876f69-6bxsm"
Sep 10 22:11:05.331871 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:11:05.331817    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/43c4db39-0a76-407a-8873-e04465bfd0f6-tls-cert\") pod \"authorino-6d4dfc649f-bdskz\" (UID: \"43c4db39-0a76-407a-8873-e04465bfd0f6\") " pod="kuadrant-system/authorino-6d4dfc649f-bdskz"
Sep 10 22:11:05.332092 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:11:05.331913    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzw2x\" (UniqueName: \"kubernetes.io/projected/43c4db39-0a76-407a-8873-e04465bfd0f6-kube-api-access-tzw2x\") pod \"authorino-6d4dfc649f-bdskz\" (UID: \"43c4db39-0a76-407a-8873-e04465bfd0f6\") " pod="kuadrant-system/authorino-6d4dfc649f-bdskz"
Sep 10 22:11:05.333562 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:11:05.333539    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-bdskz"]
Sep 10 22:11:05.333773 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:11:05.333754    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-tzw2x tls-cert], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6d4dfc649f-bdskz" podUID="43c4db39-0a76-407a-8873-e04465bfd0f6"
Sep 10 22:11:05.334493 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:11:05.334473    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/43c4db39-0a76-407a-8873-e04465bfd0f6-tls-cert\") pod \"authorino-6d4dfc649f-bdskz\" (UID: \"43c4db39-0a76-407a-8873-e04465bfd0f6\") " pod="kuadrant-system/authorino-6d4dfc649f-bdskz"
Sep 10 22:11:05.346960 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:11:05.346928    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzw2x\" (UniqueName: \"kubernetes.io/projected/43c4db39-0a76-407a-8873-e04465bfd0f6-kube-api-access-tzw2x\") pod \"authorino-6d4dfc649f-bdskz\" (UID: \"43c4db39-0a76-407a-8873-e04465bfd0f6\") " pod="kuadrant-system/authorino-6d4dfc649f-bdskz"
Sep 10 22:11:05.354632 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:11:05.354559    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6864568d57-lm54k"]
Sep 10 22:11:05.357751 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:11:05.357727    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6864568d57-lm54k"
Sep 10 22:11:05.367001 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:11:05.366972    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6864568d57-lm54k"]
Sep 10 22:11:05.433358 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:11:05.433320    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2gz2\" (UniqueName: \"kubernetes.io/projected/b054a710-bc16-4450-8850-841808d286d8-kube-api-access-p2gz2\") pod \"authorino-6864568d57-lm54k\" (UID: \"b054a710-bc16-4450-8850-841808d286d8\") " pod="kuadrant-system/authorino-6864568d57-lm54k"
Sep 10 22:11:05.433529 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:11:05.433377    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b054a710-bc16-4450-8850-841808d286d8-tls-cert\") pod \"authorino-6864568d57-lm54k\" (UID: \"b054a710-bc16-4450-8850-841808d286d8\") " pod="kuadrant-system/authorino-6864568d57-lm54k"
Sep 10 22:11:05.534324 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:11:05.534270    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2gz2\" (UniqueName: \"kubernetes.io/projected/b054a710-bc16-4450-8850-841808d286d8-kube-api-access-p2gz2\") pod \"authorino-6864568d57-lm54k\" (UID: \"b054a710-bc16-4450-8850-841808d286d8\") " pod="kuadrant-system/authorino-6864568d57-lm54k"
Sep 10 22:11:05.534496 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:11:05.534353    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b054a710-bc16-4450-8850-841808d286d8-tls-cert\") pod \"authorino-6864568d57-lm54k\" (UID: \"b054a710-bc16-4450-8850-841808d286d8\") " pod="kuadrant-system/authorino-6864568d57-lm54k"
Sep 10 22:11:05.536805 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:11:05.536781    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b054a710-bc16-4450-8850-841808d286d8-tls-cert\") pod \"authorino-6864568d57-lm54k\" (UID: \"b054a710-bc16-4450-8850-841808d286d8\") " pod="kuadrant-system/authorino-6864568d57-lm54k"
Sep 10 22:11:05.542705 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:11:05.542675    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2gz2\" (UniqueName: \"kubernetes.io/projected/b054a710-bc16-4450-8850-841808d286d8-kube-api-access-p2gz2\") pod \"authorino-6864568d57-lm54k\" (UID: \"b054a710-bc16-4450-8850-841808d286d8\") " pod="kuadrant-system/authorino-6864568d57-lm54k"
Sep 10 22:11:05.668983 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:11:05.668951    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6864568d57-lm54k"
Sep 10 22:11:05.868352 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:11:05.868318    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-bdskz"
Sep 10 22:11:05.868539 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:11:05.868318    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-6bxsm"
Sep 10 22:11:05.875578 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:11:05.875555    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-bdskz"
Sep 10 22:11:05.878134 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:11:05.878115    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-6bxsm"
Sep 10 22:11:05.936748 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:11:05.936624    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6864568d57-lm54k"]
Sep 10 22:11:05.938158 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:11:05.938139    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/43c4db39-0a76-407a-8873-e04465bfd0f6-tls-cert\") pod \"43c4db39-0a76-407a-8873-e04465bfd0f6\" (UID: \"43c4db39-0a76-407a-8873-e04465bfd0f6\") "
Sep 10 22:11:05.938204 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:11:05.938177    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tzw2x\" (UniqueName: \"kubernetes.io/projected/43c4db39-0a76-407a-8873-e04465bfd0f6-kube-api-access-tzw2x\") pod \"43c4db39-0a76-407a-8873-e04465bfd0f6\" (UID: \"43c4db39-0a76-407a-8873-e04465bfd0f6\") "
Sep 10 22:11:05.938258 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:11:05.938203    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4tst4\" (UniqueName: \"kubernetes.io/projected/c424dd7c-ac15-4c55-ab18-cff0d06c4c7f-kube-api-access-4tst4\") pod \"c424dd7c-ac15-4c55-ab18-cff0d06c4c7f\" (UID: \"c424dd7c-ac15-4c55-ab18-cff0d06c4c7f\") "
Sep 10 22:11:05.939586 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:11:05.939552    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb054a710_bc16_4450_8850_841808d286d8.slice/crio-3288c10c97ebb0040ec08c27e016463047eaecbcf57ebbb050446271324b06d6 WatchSource:0}: Error finding container 3288c10c97ebb0040ec08c27e016463047eaecbcf57ebbb050446271324b06d6: Status 404 returned error can't find the container with id 3288c10c97ebb0040ec08c27e016463047eaecbcf57ebbb050446271324b06d6
Sep 10 22:11:05.940522 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:11:05.940494    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c424dd7c-ac15-4c55-ab18-cff0d06c4c7f-kube-api-access-4tst4" (OuterVolumeSpecName: "kube-api-access-4tst4") pod "c424dd7c-ac15-4c55-ab18-cff0d06c4c7f" (UID: "c424dd7c-ac15-4c55-ab18-cff0d06c4c7f"). InnerVolumeSpecName "kube-api-access-4tst4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 22:11:05.940612 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:11:05.940535    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43c4db39-0a76-407a-8873-e04465bfd0f6-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "43c4db39-0a76-407a-8873-e04465bfd0f6" (UID: "43c4db39-0a76-407a-8873-e04465bfd0f6"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 22:11:05.940612 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:11:05.940557    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43c4db39-0a76-407a-8873-e04465bfd0f6-kube-api-access-tzw2x" (OuterVolumeSpecName: "kube-api-access-tzw2x") pod "43c4db39-0a76-407a-8873-e04465bfd0f6" (UID: "43c4db39-0a76-407a-8873-e04465bfd0f6"). InnerVolumeSpecName "kube-api-access-tzw2x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 22:11:06.039346 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:11:06.039306    2582 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/43c4db39-0a76-407a-8873-e04465bfd0f6-tls-cert\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:11:06.039346 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:11:06.039340    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzw2x\" (UniqueName: \"kubernetes.io/projected/43c4db39-0a76-407a-8873-e04465bfd0f6-kube-api-access-tzw2x\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:11:06.039346 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:11:06.039354    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4tst4\" (UniqueName: \"kubernetes.io/projected/c424dd7c-ac15-4c55-ab18-cff0d06c4c7f-kube-api-access-4tst4\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:11:06.873798 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:11:06.873704    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6864568d57-lm54k" event={"ID":"b054a710-bc16-4450-8850-841808d286d8","Type":"ContainerStarted","Data":"415b69f132406d4c6cadb331011050e022139560aa1aa27e44cbf8200be4d2fc"}
Sep 10 22:11:06.873798 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:11:06.873755    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6864568d57-lm54k" event={"ID":"b054a710-bc16-4450-8850-841808d286d8","Type":"ContainerStarted","Data":"3288c10c97ebb0040ec08c27e016463047eaecbcf57ebbb050446271324b06d6"}
Sep 10 22:11:06.874224 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:11:06.873809    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-bdskz"
Sep 10 22:11:06.874224 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:11:06.874011    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-6bxsm"
Sep 10 22:11:06.888165 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:11:06.888123    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-6864568d57-lm54k" podStartSLOduration=1.318256347 podStartE2EDuration="1.888108915s" totalImagesPullingTime="569.852568ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 22:11:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:11:06.887393807 +0000 UTC m=+587.843461513" watchObservedRunningTime="2026-09-10 22:11:06.888108915 +0000 UTC m=+587.844176621"
Sep 10 22:11:06.908533 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:11:06.908492    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-6bxsm"]
Sep 10 22:11:06.915473 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:11:06.915440    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-jj9fw"]
Sep 10 22:11:06.915679 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:11:06.915651    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59c974bd48-jj9fw" podUID="e43ade8d-035c-438d-a448-5a698f5023eb" containerName="authorino" containerID="cri-o://aa282b11766a45e7e3cd4f986eff32e00ff0a71bbeba13a662849f2f522cffb6" gracePeriod=30
Sep 10 22:11:06.917937 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:11:06.917910    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-557f876f69-6bxsm"]
Sep 10 22:11:06.940487 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:11:06.940451    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-bdskz"]
Sep 10 22:11:06.944573 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:11:06.944538    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-bdskz"]
Sep 10 22:11:07.160416 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:11:07.160392    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-jj9fw"
Sep 10 22:11:07.250181 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:11:07.250151    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lp77k\" (UniqueName: \"kubernetes.io/projected/e43ade8d-035c-438d-a448-5a698f5023eb-kube-api-access-lp77k\") pod \"e43ade8d-035c-438d-a448-5a698f5023eb\" (UID: \"e43ade8d-035c-438d-a448-5a698f5023eb\") "
Sep 10 22:11:07.252233 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:11:07.252202    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e43ade8d-035c-438d-a448-5a698f5023eb-kube-api-access-lp77k" (OuterVolumeSpecName: "kube-api-access-lp77k") pod "e43ade8d-035c-438d-a448-5a698f5023eb" (UID: "e43ade8d-035c-438d-a448-5a698f5023eb"). InnerVolumeSpecName "kube-api-access-lp77k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 22:11:07.350781 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:11:07.350738    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lp77k\" (UniqueName: \"kubernetes.io/projected/e43ade8d-035c-438d-a448-5a698f5023eb-kube-api-access-lp77k\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:11:07.498478 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:11:07.498444    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="43c4db39-0a76-407a-8873-e04465bfd0f6" path="/var/lib/kubelet/pods/43c4db39-0a76-407a-8873-e04465bfd0f6/volumes"
Sep 10 22:11:07.498674 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:11:07.498663    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c424dd7c-ac15-4c55-ab18-cff0d06c4c7f" path="/var/lib/kubelet/pods/c424dd7c-ac15-4c55-ab18-cff0d06c4c7f/volumes"
Sep 10 22:11:07.880119 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:11:07.880020    2582 generic.go:356] "Generic (PLEG): container finished" podID="e43ade8d-035c-438d-a448-5a698f5023eb" containerID="aa282b11766a45e7e3cd4f986eff32e00ff0a71bbeba13a662849f2f522cffb6" exitCode=0
Sep 10 22:11:07.880119 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:11:07.880059    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-jj9fw"
Sep 10 22:11:07.880119 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:11:07.880102    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-jj9fw" event={"ID":"e43ade8d-035c-438d-a448-5a698f5023eb","Type":"ContainerDied","Data":"aa282b11766a45e7e3cd4f986eff32e00ff0a71bbeba13a662849f2f522cffb6"}
Sep 10 22:11:07.880642 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:11:07.880142    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-jj9fw" event={"ID":"e43ade8d-035c-438d-a448-5a698f5023eb","Type":"ContainerDied","Data":"bf49a5382455f69c0cfbb9f5076a96d6a378a68c62685b1da7626ec3a94b95eb"}
Sep 10 22:11:07.880642 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:11:07.880160    2582 scope.go:117] "RemoveContainer" containerID="aa282b11766a45e7e3cd4f986eff32e00ff0a71bbeba13a662849f2f522cffb6"
Sep 10 22:11:07.890263 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:11:07.890243    2582 scope.go:117] "RemoveContainer" containerID="aa282b11766a45e7e3cd4f986eff32e00ff0a71bbeba13a662849f2f522cffb6"
Sep 10 22:11:07.890552 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:11:07.890535    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa282b11766a45e7e3cd4f986eff32e00ff0a71bbeba13a662849f2f522cffb6\": container with ID starting with aa282b11766a45e7e3cd4f986eff32e00ff0a71bbeba13a662849f2f522cffb6 not found: ID does not exist" containerID="aa282b11766a45e7e3cd4f986eff32e00ff0a71bbeba13a662849f2f522cffb6"
Sep 10 22:11:07.890600 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:11:07.890562    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa282b11766a45e7e3cd4f986eff32e00ff0a71bbeba13a662849f2f522cffb6"} err="failed to get container status \"aa282b11766a45e7e3cd4f986eff32e00ff0a71bbeba13a662849f2f522cffb6\": rpc error: code = NotFound desc = could not find container \"aa282b11766a45e7e3cd4f986eff32e00ff0a71bbeba13a662849f2f522cffb6\": container with ID starting with aa282b11766a45e7e3cd4f986eff32e00ff0a71bbeba13a662849f2f522cffb6 not found: ID does not exist"
Sep 10 22:11:07.893989 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:11:07.893966    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-jj9fw"]
Sep 10 22:11:07.899678 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:11:07.899548    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59c974bd48-jj9fw"]
Sep 10 22:11:09.497798 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:11:09.497763    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e43ade8d-035c-438d-a448-5a698f5023eb" path="/var/lib/kubelet/pods/e43ade8d-035c-438d-a448-5a698f5023eb/volumes"
Sep 10 22:11:19.438702 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:11:19.438672    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-w6vlb_8517c9a7-382a-40fd-bc52-e6da90ebc584/console-operator/2.log"
Sep 10 22:11:19.440825 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:11:19.440801    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-w6vlb_8517c9a7-382a-40fd-bc52-e6da90ebc584/console-operator/2.log"
Sep 10 22:12:18.696425 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:12:18.696389    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-4c59l"]
Sep 10 22:12:18.696888 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:12:18.696719    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e43ade8d-035c-438d-a448-5a698f5023eb" containerName="authorino"
Sep 10 22:12:18.696888 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:12:18.696731    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="e43ade8d-035c-438d-a448-5a698f5023eb" containerName="authorino"
Sep 10 22:12:18.696888 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:12:18.696783    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="e43ade8d-035c-438d-a448-5a698f5023eb" containerName="authorino"
Sep 10 22:12:18.699367 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:12:18.699352    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4c59l"
Sep 10 22:12:18.705388 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:12:18.705363    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 10 22:12:18.705535 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:12:18.705462    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 10 22:12:18.705633 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:12:18.705575    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-2s65g\""
Sep 10 22:12:18.705752 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:12:18.705734    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Sep 10 22:12:18.709140 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:12:18.709116    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-4c59l"]
Sep 10 22:12:18.839252 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:12:18.839223    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/68299c8c-a2fe-4c4e-ad50-9d2bf78d13f5-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-4c59l\" (UID: \"68299c8c-a2fe-4c4e-ad50-9d2bf78d13f5\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4c59l"
Sep 10 22:12:18.839427 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:12:18.839260    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2g6ll\" (UniqueName: \"kubernetes.io/projected/68299c8c-a2fe-4c4e-ad50-9d2bf78d13f5-kube-api-access-2g6ll\") pod \"e2e-distinct-simulated-kserve-59975d69f9-4c59l\" (UID: \"68299c8c-a2fe-4c4e-ad50-9d2bf78d13f5\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4c59l"
Sep 10 22:12:18.839427 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:12:18.839290    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68299c8c-a2fe-4c4e-ad50-9d2bf78d13f5-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-4c59l\" (UID: \"68299c8c-a2fe-4c4e-ad50-9d2bf78d13f5\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4c59l"
Sep 10 22:12:18.839427 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:12:18.839368    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68299c8c-a2fe-4c4e-ad50-9d2bf78d13f5-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-4c59l\" (UID: \"68299c8c-a2fe-4c4e-ad50-9d2bf78d13f5\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4c59l"
Sep 10 22:12:18.839427 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:12:18.839408    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/68299c8c-a2fe-4c4e-ad50-9d2bf78d13f5-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-4c59l\" (UID: \"68299c8c-a2fe-4c4e-ad50-9d2bf78d13f5\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4c59l"
Sep 10 22:12:18.839559 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:12:18.839448    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68299c8c-a2fe-4c4e-ad50-9d2bf78d13f5-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-4c59l\" (UID: \"68299c8c-a2fe-4c4e-ad50-9d2bf78d13f5\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4c59l"
Sep 10 22:12:18.940749 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:12:18.940711    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/68299c8c-a2fe-4c4e-ad50-9d2bf78d13f5-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-4c59l\" (UID: \"68299c8c-a2fe-4c4e-ad50-9d2bf78d13f5\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4c59l"
Sep 10 22:12:18.940933 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:12:18.940753    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68299c8c-a2fe-4c4e-ad50-9d2bf78d13f5-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-4c59l\" (UID: \"68299c8c-a2fe-4c4e-ad50-9d2bf78d13f5\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4c59l"
Sep 10 22:12:18.940933 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:12:18.940794    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/68299c8c-a2fe-4c4e-ad50-9d2bf78d13f5-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-4c59l\" (UID: \"68299c8c-a2fe-4c4e-ad50-9d2bf78d13f5\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4c59l"
Sep 10 22:12:18.940933 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:12:18.940819    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2g6ll\" (UniqueName: \"kubernetes.io/projected/68299c8c-a2fe-4c4e-ad50-9d2bf78d13f5-kube-api-access-2g6ll\") pod \"e2e-distinct-simulated-kserve-59975d69f9-4c59l\" (UID: \"68299c8c-a2fe-4c4e-ad50-9d2bf78d13f5\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4c59l"
Sep 10 22:12:18.940933 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:12:18.940872    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68299c8c-a2fe-4c4e-ad50-9d2bf78d13f5-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-4c59l\" (UID: \"68299c8c-a2fe-4c4e-ad50-9d2bf78d13f5\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4c59l"
Sep 10 22:12:18.940933 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:12:18.940901    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68299c8c-a2fe-4c4e-ad50-9d2bf78d13f5-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-4c59l\" (UID: \"68299c8c-a2fe-4c4e-ad50-9d2bf78d13f5\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4c59l"
Sep 10 22:12:18.941197 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:12:18.941180    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/68299c8c-a2fe-4c4e-ad50-9d2bf78d13f5-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-4c59l\" (UID: \"68299c8c-a2fe-4c4e-ad50-9d2bf78d13f5\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4c59l"
Sep 10 22:12:18.941254 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:12:18.941190    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68299c8c-a2fe-4c4e-ad50-9d2bf78d13f5-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-4c59l\" (UID: \"68299c8c-a2fe-4c4e-ad50-9d2bf78d13f5\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4c59l"
Sep 10 22:12:18.941254 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:12:18.941243    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68299c8c-a2fe-4c4e-ad50-9d2bf78d13f5-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-4c59l\" (UID: \"68299c8c-a2fe-4c4e-ad50-9d2bf78d13f5\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4c59l"
Sep 10 22:12:18.943183 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:12:18.943159    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/68299c8c-a2fe-4c4e-ad50-9d2bf78d13f5-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-4c59l\" (UID: \"68299c8c-a2fe-4c4e-ad50-9d2bf78d13f5\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4c59l"
Sep 10 22:12:18.943461 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:12:18.943442    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68299c8c-a2fe-4c4e-ad50-9d2bf78d13f5-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-4c59l\" (UID: \"68299c8c-a2fe-4c4e-ad50-9d2bf78d13f5\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4c59l"
Sep 10 22:12:18.948784 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:12:18.948737    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2g6ll\" (UniqueName: \"kubernetes.io/projected/68299c8c-a2fe-4c4e-ad50-9d2bf78d13f5-kube-api-access-2g6ll\") pod \"e2e-distinct-simulated-kserve-59975d69f9-4c59l\" (UID: \"68299c8c-a2fe-4c4e-ad50-9d2bf78d13f5\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4c59l"
Sep 10 22:12:19.015566 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:12:19.015537    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4c59l"
Sep 10 22:12:19.285483 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:12:19.285458    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-4c59l"]
Sep 10 22:12:19.287605 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:12:19.287580    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68299c8c_a2fe_4c4e_ad50_9d2bf78d13f5.slice/crio-5245e893f3894597ab779d3dbb20b68a96c94737c12c520ef9c1a76d6642e645 WatchSource:0}: Error finding container 5245e893f3894597ab779d3dbb20b68a96c94737c12c520ef9c1a76d6642e645: Status 404 returned error can't find the container with id 5245e893f3894597ab779d3dbb20b68a96c94737c12c520ef9c1a76d6642e645
Sep 10 22:12:20.178697 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:12:20.178664    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4c59l" event={"ID":"68299c8c-a2fe-4c4e-ad50-9d2bf78d13f5","Type":"ContainerStarted","Data":"5245e893f3894597ab779d3dbb20b68a96c94737c12c520ef9c1a76d6642e645"}
Sep 10 22:12:21.185116 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:12:21.185080    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4c59l" event={"ID":"68299c8c-a2fe-4c4e-ad50-9d2bf78d13f5","Type":"ContainerStarted","Data":"13a0340b26ce65619dc21279cf416fa467771ab195c6e6c7e9d57e4be7a37759"}
Sep 10 22:12:21.213271 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:12:21.213207    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4c59l" podStartSLOduration=2.147782375 podStartE2EDuration="3.213184235s" totalImagesPullingTime="1.06540186s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 22:12:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:12:21.212080496 +0000 UTC m=+662.168148202" watchObservedRunningTime="2026-09-10 22:12:21.213184235 +0000 UTC m=+662.169251942"
Sep 10 22:12:29.016048 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:12:29.016006    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4c59l"
Sep 10 22:12:29.016578 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:12:29.016069    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4c59l"
Sep 10 22:12:29.028729 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:12:29.028696    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4c59l"
Sep 10 22:12:29.235090 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:12:29.235054    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4c59l"
Sep 10 22:13:02.589153 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:13:02.589119    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-wb4r5"]
Sep 10 22:13:02.593254 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:13:02.593232    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-wb4r5"
Sep 10 22:13:02.595717 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:13:02.595693    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Sep 10 22:13:02.604893 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:13:02.604870    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-wb4r5"]
Sep 10 22:13:02.679748 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:13:02.679709    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18e39a69-1b48-40ca-9438-6210181c9f8c-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-wb4r5\" (UID: \"18e39a69-1b48-40ca-9438-6210181c9f8c\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-wb4r5"
Sep 10 22:13:02.679941 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:13:02.679768    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18e39a69-1b48-40ca-9438-6210181c9f8c-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-wb4r5\" (UID: \"18e39a69-1b48-40ca-9438-6210181c9f8c\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-wb4r5"
Sep 10 22:13:02.679941 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:13:02.679810    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18e39a69-1b48-40ca-9438-6210181c9f8c-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-wb4r5\" (UID: \"18e39a69-1b48-40ca-9438-6210181c9f8c\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-wb4r5"
Sep 10 22:13:02.679941 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:13:02.679831    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18e39a69-1b48-40ca-9438-6210181c9f8c-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-wb4r5\" (UID: \"18e39a69-1b48-40ca-9438-6210181c9f8c\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-wb4r5"
Sep 10 22:13:02.680055 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:13:02.679943    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ssdn9\" (UniqueName: \"kubernetes.io/projected/18e39a69-1b48-40ca-9438-6210181c9f8c-kube-api-access-ssdn9\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-wb4r5\" (UID: \"18e39a69-1b48-40ca-9438-6210181c9f8c\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-wb4r5"
Sep 10 22:13:02.680055 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:13:02.679987    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18e39a69-1b48-40ca-9438-6210181c9f8c-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-wb4r5\" (UID: \"18e39a69-1b48-40ca-9438-6210181c9f8c\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-wb4r5"
Sep 10 22:13:02.780795 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:13:02.780758    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18e39a69-1b48-40ca-9438-6210181c9f8c-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-wb4r5\" (UID: \"18e39a69-1b48-40ca-9438-6210181c9f8c\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-wb4r5"
Sep 10 22:13:02.780795 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:13:02.780799    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18e39a69-1b48-40ca-9438-6210181c9f8c-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-wb4r5\" (UID: \"18e39a69-1b48-40ca-9438-6210181c9f8c\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-wb4r5"
Sep 10 22:13:02.781033 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:13:02.780841    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ssdn9\" (UniqueName: \"kubernetes.io/projected/18e39a69-1b48-40ca-9438-6210181c9f8c-kube-api-access-ssdn9\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-wb4r5\" (UID: \"18e39a69-1b48-40ca-9438-6210181c9f8c\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-wb4r5"
Sep 10 22:13:02.781033 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:13:02.780894    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18e39a69-1b48-40ca-9438-6210181c9f8c-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-wb4r5\" (UID: \"18e39a69-1b48-40ca-9438-6210181c9f8c\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-wb4r5"
Sep 10 22:13:02.781033 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:13:02.780936    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18e39a69-1b48-40ca-9438-6210181c9f8c-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-wb4r5\" (UID: \"18e39a69-1b48-40ca-9438-6210181c9f8c\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-wb4r5"
Sep 10 22:13:02.781033 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:13:02.780988    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18e39a69-1b48-40ca-9438-6210181c9f8c-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-wb4r5\" (UID: \"18e39a69-1b48-40ca-9438-6210181c9f8c\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-wb4r5"
Sep 10 22:13:02.781335 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:13:02.781309    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18e39a69-1b48-40ca-9438-6210181c9f8c-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-wb4r5\" (UID: \"18e39a69-1b48-40ca-9438-6210181c9f8c\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-wb4r5"
Sep 10 22:13:02.781410 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:13:02.781335    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18e39a69-1b48-40ca-9438-6210181c9f8c-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-wb4r5\" (UID: \"18e39a69-1b48-40ca-9438-6210181c9f8c\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-wb4r5"
Sep 10 22:13:02.781410 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:13:02.781394    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18e39a69-1b48-40ca-9438-6210181c9f8c-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-wb4r5\" (UID: \"18e39a69-1b48-40ca-9438-6210181c9f8c\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-wb4r5"
Sep 10 22:13:02.783117 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:13:02.783095    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18e39a69-1b48-40ca-9438-6210181c9f8c-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-wb4r5\" (UID: \"18e39a69-1b48-40ca-9438-6210181c9f8c\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-wb4r5"
Sep 10 22:13:02.783258 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:13:02.783242    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18e39a69-1b48-40ca-9438-6210181c9f8c-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-wb4r5\" (UID: \"18e39a69-1b48-40ca-9438-6210181c9f8c\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-wb4r5"
Sep 10 22:13:02.789717 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:13:02.789686    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ssdn9\" (UniqueName: \"kubernetes.io/projected/18e39a69-1b48-40ca-9438-6210181c9f8c-kube-api-access-ssdn9\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-wb4r5\" (UID: \"18e39a69-1b48-40ca-9438-6210181c9f8c\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-wb4r5"
Sep 10 22:13:02.902662 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:13:02.902576    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-wb4r5"
Sep 10 22:13:03.173100 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:13:03.173066    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-wb4r5"]
Sep 10 22:13:03.177217 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:13:03.177188    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod18e39a69_1b48_40ca_9438_6210181c9f8c.slice/crio-36fb38ed35473eed33021da7defb695539032a004f484e229aa1a7a4bc7a2c55 WatchSource:0}: Error finding container 36fb38ed35473eed33021da7defb695539032a004f484e229aa1a7a4bc7a2c55: Status 404 returned error can't find the container with id 36fb38ed35473eed33021da7defb695539032a004f484e229aa1a7a4bc7a2c55
Sep 10 22:13:03.178485 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:13:03.178468    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 22:13:03.374068 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:13:03.374032    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-wb4r5" event={"ID":"18e39a69-1b48-40ca-9438-6210181c9f8c","Type":"ContainerStarted","Data":"36fb38ed35473eed33021da7defb695539032a004f484e229aa1a7a4bc7a2c55"}
Sep 10 22:13:04.380024 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:13:04.379988    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-wb4r5" event={"ID":"18e39a69-1b48-40ca-9438-6210181c9f8c","Type":"ContainerStarted","Data":"f0e2ceb5967f07e4f5cd7bb811468f9a128e8485d4a09ab3973f2aae74c0e127"}
Sep 10 22:13:04.402207 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:13:04.402156    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-wb4r5" podStartSLOduration=2.103038206 podStartE2EDuration="2.402141668s" totalImagesPullingTime="299.103462ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 22:13:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:13:04.401143237 +0000 UTC m=+705.357210945" watchObservedRunningTime="2026-09-10 22:13:04.402141668 +0000 UTC m=+705.358209373"
Sep 10 22:13:12.902995 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:13:12.902916    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-wb4r5"
Sep 10 22:13:12.902995 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:13:12.902971    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-wb4r5"
Sep 10 22:13:12.915518 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:13:12.915492    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-wb4r5"
Sep 10 22:13:13.431558 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:13:13.431525    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-wb4r5"
Sep 10 22:14:11.151719 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:11.151676    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6b96dfc79b-g9q5h"]
Sep 10 22:14:11.155047 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:11.155025    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6b96dfc79b-g9q5h"
Sep 10 22:14:11.163070 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:11.163046    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6b96dfc79b-g9q5h"]
Sep 10 22:14:11.237424 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:11.237387    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgmbq\" (UniqueName: \"kubernetes.io/projected/6bae3cbf-19db-4a96-bb9d-870154b86bc3-kube-api-access-tgmbq\") pod \"authorino-6b96dfc79b-g9q5h\" (UID: \"6bae3cbf-19db-4a96-bb9d-870154b86bc3\") " pod="kuadrant-system/authorino-6b96dfc79b-g9q5h"
Sep 10 22:14:11.237597 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:11.237439    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6bae3cbf-19db-4a96-bb9d-870154b86bc3-tls-cert\") pod \"authorino-6b96dfc79b-g9q5h\" (UID: \"6bae3cbf-19db-4a96-bb9d-870154b86bc3\") " pod="kuadrant-system/authorino-6b96dfc79b-g9q5h"
Sep 10 22:14:11.337961 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:11.337929    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgmbq\" (UniqueName: \"kubernetes.io/projected/6bae3cbf-19db-4a96-bb9d-870154b86bc3-kube-api-access-tgmbq\") pod \"authorino-6b96dfc79b-g9q5h\" (UID: \"6bae3cbf-19db-4a96-bb9d-870154b86bc3\") " pod="kuadrant-system/authorino-6b96dfc79b-g9q5h"
Sep 10 22:14:11.338106 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:11.337974    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6bae3cbf-19db-4a96-bb9d-870154b86bc3-tls-cert\") pod \"authorino-6b96dfc79b-g9q5h\" (UID: \"6bae3cbf-19db-4a96-bb9d-870154b86bc3\") " pod="kuadrant-system/authorino-6b96dfc79b-g9q5h"
Sep 10 22:14:11.340461 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:11.340429    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6bae3cbf-19db-4a96-bb9d-870154b86bc3-tls-cert\") pod \"authorino-6b96dfc79b-g9q5h\" (UID: \"6bae3cbf-19db-4a96-bb9d-870154b86bc3\") " pod="kuadrant-system/authorino-6b96dfc79b-g9q5h"
Sep 10 22:14:11.346144 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:11.346118    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgmbq\" (UniqueName: \"kubernetes.io/projected/6bae3cbf-19db-4a96-bb9d-870154b86bc3-kube-api-access-tgmbq\") pod \"authorino-6b96dfc79b-g9q5h\" (UID: \"6bae3cbf-19db-4a96-bb9d-870154b86bc3\") " pod="kuadrant-system/authorino-6b96dfc79b-g9q5h"
Sep 10 22:14:11.463805 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:11.463779    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6b96dfc79b-g9q5h"
Sep 10 22:14:11.740358 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:11.740331    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6b96dfc79b-g9q5h"]
Sep 10 22:14:11.742933 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:14:11.742903    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6bae3cbf_19db_4a96_bb9d_870154b86bc3.slice/crio-dbef29fa374a169953ccc6a6677e878761da1f77d5611b2ac7e796ab54496247 WatchSource:0}: Error finding container dbef29fa374a169953ccc6a6677e878761da1f77d5611b2ac7e796ab54496247: Status 404 returned error can't find the container with id dbef29fa374a169953ccc6a6677e878761da1f77d5611b2ac7e796ab54496247
Sep 10 22:14:12.673027 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:12.672993    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6b96dfc79b-g9q5h" event={"ID":"6bae3cbf-19db-4a96-bb9d-870154b86bc3","Type":"ContainerStarted","Data":"dbef29fa374a169953ccc6a6677e878761da1f77d5611b2ac7e796ab54496247"}
Sep 10 22:14:21.713128 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:21.713080    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6b96dfc79b-g9q5h" event={"ID":"6bae3cbf-19db-4a96-bb9d-870154b86bc3","Type":"ContainerStarted","Data":"06b7d0988da5d1e51388bc70ba4a546a2d34d5b7aa3c069471bb810d153ec182"}
Sep 10 22:14:21.738958 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:21.738836    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-6b96dfc79b-g9q5h" podStartSLOduration=0.996550167 podStartE2EDuration="10.738819351s" totalImagesPullingTime="9.742269184s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 22:14:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:14:21.736771714 +0000 UTC m=+782.692839420" watchObservedRunningTime="2026-09-10 22:14:21.738819351 +0000 UTC m=+782.694887056"
Sep 10 22:14:21.777521 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:21.777469    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6864568d57-lm54k"]
Sep 10 22:14:21.777746 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:21.777716    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-6864568d57-lm54k" podUID="b054a710-bc16-4450-8850-841808d286d8" containerName="authorino" containerID="cri-o://415b69f132406d4c6cadb331011050e022139560aa1aa27e44cbf8200be4d2fc" gracePeriod=30
Sep 10 22:14:22.035334 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:22.035308    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6864568d57-lm54k"
Sep 10 22:14:22.138116 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:22.138085    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b054a710-bc16-4450-8850-841808d286d8-tls-cert\") pod \"b054a710-bc16-4450-8850-841808d286d8\" (UID: \"b054a710-bc16-4450-8850-841808d286d8\") "
Sep 10 22:14:22.138353 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:22.138139    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p2gz2\" (UniqueName: \"kubernetes.io/projected/b054a710-bc16-4450-8850-841808d286d8-kube-api-access-p2gz2\") pod \"b054a710-bc16-4450-8850-841808d286d8\" (UID: \"b054a710-bc16-4450-8850-841808d286d8\") "
Sep 10 22:14:22.140278 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:22.140240    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b054a710-bc16-4450-8850-841808d286d8-kube-api-access-p2gz2" (OuterVolumeSpecName: "kube-api-access-p2gz2") pod "b054a710-bc16-4450-8850-841808d286d8" (UID: "b054a710-bc16-4450-8850-841808d286d8"). InnerVolumeSpecName "kube-api-access-p2gz2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 22:14:22.148248 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:22.148211    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b054a710-bc16-4450-8850-841808d286d8-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "b054a710-bc16-4450-8850-841808d286d8" (UID: "b054a710-bc16-4450-8850-841808d286d8"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 22:14:22.239724 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:22.239689    2582 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b054a710-bc16-4450-8850-841808d286d8-tls-cert\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:14:22.239724 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:22.239716    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2gz2\" (UniqueName: \"kubernetes.io/projected/b054a710-bc16-4450-8850-841808d286d8-kube-api-access-p2gz2\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:14:22.719143 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:22.719106    2582 generic.go:356] "Generic (PLEG): container finished" podID="b054a710-bc16-4450-8850-841808d286d8" containerID="415b69f132406d4c6cadb331011050e022139560aa1aa27e44cbf8200be4d2fc" exitCode=0
Sep 10 22:14:22.719543 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:22.719151    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6864568d57-lm54k"
Sep 10 22:14:22.719543 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:22.719204    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6864568d57-lm54k" event={"ID":"b054a710-bc16-4450-8850-841808d286d8","Type":"ContainerDied","Data":"415b69f132406d4c6cadb331011050e022139560aa1aa27e44cbf8200be4d2fc"}
Sep 10 22:14:22.719543 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:22.719255    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6864568d57-lm54k" event={"ID":"b054a710-bc16-4450-8850-841808d286d8","Type":"ContainerDied","Data":"3288c10c97ebb0040ec08c27e016463047eaecbcf57ebbb050446271324b06d6"}
Sep 10 22:14:22.719543 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:22.719282    2582 scope.go:117] "RemoveContainer" containerID="415b69f132406d4c6cadb331011050e022139560aa1aa27e44cbf8200be4d2fc"
Sep 10 22:14:22.729236 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:22.729213    2582 scope.go:117] "RemoveContainer" containerID="415b69f132406d4c6cadb331011050e022139560aa1aa27e44cbf8200be4d2fc"
Sep 10 22:14:22.729564 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:14:22.729544    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"415b69f132406d4c6cadb331011050e022139560aa1aa27e44cbf8200be4d2fc\": container with ID starting with 415b69f132406d4c6cadb331011050e022139560aa1aa27e44cbf8200be4d2fc not found: ID does not exist" containerID="415b69f132406d4c6cadb331011050e022139560aa1aa27e44cbf8200be4d2fc"
Sep 10 22:14:22.729631 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:22.729579    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"415b69f132406d4c6cadb331011050e022139560aa1aa27e44cbf8200be4d2fc"} err="failed to get container status \"415b69f132406d4c6cadb331011050e022139560aa1aa27e44cbf8200be4d2fc\": rpc error: code = NotFound desc = could not find container \"415b69f132406d4c6cadb331011050e022139560aa1aa27e44cbf8200be4d2fc\": container with ID starting with 415b69f132406d4c6cadb331011050e022139560aa1aa27e44cbf8200be4d2fc not found: ID does not exist"
Sep 10 22:14:22.739906 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:22.739878    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6864568d57-lm54k"]
Sep 10 22:14:22.745023 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:22.744990    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6864568d57-lm54k"]
Sep 10 22:14:23.498293 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:23.498258    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b054a710-bc16-4450-8850-841808d286d8" path="/var/lib/kubelet/pods/b054a710-bc16-4450-8850-841808d286d8/volumes"
Sep 10 22:14:40.769280 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:40.769244    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7520-e62155"]
Sep 10 22:14:40.769669 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:40.769600    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b054a710-bc16-4450-8850-841808d286d8" containerName="authorino"
Sep 10 22:14:40.769669 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:40.769612    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="b054a710-bc16-4450-8850-841808d286d8" containerName="authorino"
Sep 10 22:14:40.769669 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:40.769667    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="b054a710-bc16-4450-8850-841808d286d8" containerName="authorino"
Sep 10 22:14:40.773874 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:40.773845    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7520-e62155"
Sep 10 22:14:40.775894 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:40.775843    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-znq4v\""
Sep 10 22:14:40.776644 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:40.776623    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7520-e62155"]
Sep 10 22:14:40.902255 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:40.902222    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22vvh\" (UniqueName: \"kubernetes.io/projected/3d95e13c-20cc-482d-a221-022236762244-kube-api-access-22vvh\") pod \"test-curl-7520-e62155\" (UID: \"3d95e13c-20cc-482d-a221-022236762244\") " pod="opendatahub/test-curl-7520-e62155"
Sep 10 22:14:41.003248 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:41.003216    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22vvh\" (UniqueName: \"kubernetes.io/projected/3d95e13c-20cc-482d-a221-022236762244-kube-api-access-22vvh\") pod \"test-curl-7520-e62155\" (UID: \"3d95e13c-20cc-482d-a221-022236762244\") " pod="opendatahub/test-curl-7520-e62155"
Sep 10 22:14:41.011266 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:41.011247    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-22vvh\" (UniqueName: \"kubernetes.io/projected/3d95e13c-20cc-482d-a221-022236762244-kube-api-access-22vvh\") pod \"test-curl-7520-e62155\" (UID: \"3d95e13c-20cc-482d-a221-022236762244\") " pod="opendatahub/test-curl-7520-e62155"
Sep 10 22:14:41.084788 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:41.084729    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7520-e62155"
Sep 10 22:14:41.354785 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:41.354706    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7520-e62155"]
Sep 10 22:14:41.357525 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:14:41.357491    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3d95e13c_20cc_482d_a221_022236762244.slice/crio-244fef8a8cc3d870d28c6276cfe25f75e2401d9e78e31e962adfcd2696365e64 WatchSource:0}: Error finding container 244fef8a8cc3d870d28c6276cfe25f75e2401d9e78e31e962adfcd2696365e64: Status 404 returned error can't find the container with id 244fef8a8cc3d870d28c6276cfe25f75e2401d9e78e31e962adfcd2696365e64
Sep 10 22:14:41.801594 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:41.801563    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7520-e62155" event={"ID":"3d95e13c-20cc-482d-a221-022236762244","Type":"ContainerStarted","Data":"244fef8a8cc3d870d28c6276cfe25f75e2401d9e78e31e962adfcd2696365e64"}
Sep 10 22:14:42.808942 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:42.808194    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7520-e62155" event={"ID":"3d95e13c-20cc-482d-a221-022236762244","Type":"ContainerStarted","Data":"5defbfc6bf843f09fcd31b017babc2f0147eb07fa6ca79856d156a836bac18e9"}
Sep 10 22:14:42.826449 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:42.826392    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7520-e62155" podStartSLOduration=1.9188715570000001 podStartE2EDuration="2.826376343s" totalImagesPullingTime="907.504786ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 22:14:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:14:42.824722813 +0000 UTC m=+803.780790520" watchObservedRunningTime="2026-09-10 22:14:42.826376343 +0000 UTC m=+803.782444049"
Sep 10 22:14:43.575572 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:43.575540    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7520-e62155"]
Sep 10 22:14:43.579926 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:43.579900    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7520-e62155"]
Sep 10 22:14:43.707334 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:43.707303    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7520-9206c4"]
Sep 10 22:14:43.707624 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:43.707612    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d95e13c-20cc-482d-a221-022236762244" containerName="test-curl-7520-e62155"
Sep 10 22:14:43.707674 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:43.707626    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d95e13c-20cc-482d-a221-022236762244" containerName="test-curl-7520-e62155"
Sep 10 22:14:43.707705 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:43.707680    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d95e13c-20cc-482d-a221-022236762244" containerName="test-curl-7520-e62155"
Sep 10 22:14:43.710845 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:43.710828    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7520-9206c4"
Sep 10 22:14:43.718567 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:43.718533    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7520-9206c4"]
Sep 10 22:14:43.812842 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:14:43.812814    2582 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7520-e62155" container="test-curl-7520-e62155"
Sep 10 22:14:43.830360 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:43.830330    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dr8bn\" (UniqueName: \"kubernetes.io/projected/769badbb-f1b4-4d68-bb8d-8d757a1e7b2b-kube-api-access-dr8bn\") pod \"test-curl-7520-9206c4\" (UID: \"769badbb-f1b4-4d68-bb8d-8d757a1e7b2b\") " pod="opendatahub/test-curl-7520-9206c4"
Sep 10 22:14:43.931227 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:43.931190    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dr8bn\" (UniqueName: \"kubernetes.io/projected/769badbb-f1b4-4d68-bb8d-8d757a1e7b2b-kube-api-access-dr8bn\") pod \"test-curl-7520-9206c4\" (UID: \"769badbb-f1b4-4d68-bb8d-8d757a1e7b2b\") " pod="opendatahub/test-curl-7520-9206c4"
Sep 10 22:14:43.941025 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:43.940996    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dr8bn\" (UniqueName: \"kubernetes.io/projected/769badbb-f1b4-4d68-bb8d-8d757a1e7b2b-kube-api-access-dr8bn\") pod \"test-curl-7520-9206c4\" (UID: \"769badbb-f1b4-4d68-bb8d-8d757a1e7b2b\") " pod="opendatahub/test-curl-7520-9206c4"
Sep 10 22:14:44.021380 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:44.021310    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7520-9206c4"
Sep 10 22:14:44.295952 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:44.295877    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7520-9206c4"]
Sep 10 22:14:44.300048 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:14:44.300017    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod769badbb_f1b4_4d68_bb8d_8d757a1e7b2b.slice/crio-c62905dbb690f929ef6471292afa0783c89bf301297f7a21a614f1078e351311 WatchSource:0}: Error finding container c62905dbb690f929ef6471292afa0783c89bf301297f7a21a614f1078e351311: Status 404 returned error can't find the container with id c62905dbb690f929ef6471292afa0783c89bf301297f7a21a614f1078e351311
Sep 10 22:14:44.818794 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:44.818754    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7520-9206c4" event={"ID":"769badbb-f1b4-4d68-bb8d-8d757a1e7b2b","Type":"ContainerStarted","Data":"be4bb514e6ce0665adc7b0684f4436127d659ea151d7e6a9b410b1f56acb81ca"}
Sep 10 22:14:44.818794 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:44.818802    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7520-9206c4" event={"ID":"769badbb-f1b4-4d68-bb8d-8d757a1e7b2b","Type":"ContainerStarted","Data":"c62905dbb690f929ef6471292afa0783c89bf301297f7a21a614f1078e351311"}
Sep 10 22:14:44.819263 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:44.818914    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7520-e62155" podUID="3d95e13c-20cc-482d-a221-022236762244" containerName="test-curl-7520-e62155" containerID="cri-o://5defbfc6bf843f09fcd31b017babc2f0147eb07fa6ca79856d156a836bac18e9" gracePeriod=2
Sep 10 22:14:44.833417 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:44.833373    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7520-9206c4" podStartSLOduration=1.833356975 podStartE2EDuration="1.833356975s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 22:14:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:14:44.832034817 +0000 UTC m=+805.788102526" watchObservedRunningTime="2026-09-10 22:14:44.833356975 +0000 UTC m=+805.789424681"
Sep 10 22:14:44.834084 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:14:44.834053    2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7520-e62155\" is forbidden: User \"system:node:ip-10-0-134-144.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-144.ec2.internal' and this object" podUID="3d95e13c-20cc-482d-a221-022236762244" pod="opendatahub/test-curl-7520-e62155"
Sep 10 22:14:45.404554 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:45.404515    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7520-9206c4"]
Sep 10 22:14:45.409207 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:45.409171    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7520-9206c4"]
Sep 10 22:14:45.823804 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:14:45.823778    2582 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7520-9206c4" container="test-curl-7520-9206c4"
Sep 10 22:14:46.161274 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:46.161245    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-d6a4de-openshift-default-6bf96b68c4-6269k"]
Sep 10 22:14:46.161680 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:46.161660    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="769badbb-f1b4-4d68-bb8d-8d757a1e7b2b" containerName="test-curl-7520-9206c4"
Sep 10 22:14:46.161733 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:46.161685    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="769badbb-f1b4-4d68-bb8d-8d757a1e7b2b" containerName="test-curl-7520-9206c4"
Sep 10 22:14:46.161786 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:46.161774    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="769badbb-f1b4-4d68-bb8d-8d757a1e7b2b" containerName="test-curl-7520-9206c4"
Sep 10 22:14:46.164982 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:46.164962    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-d6a4de-openshift-default-6bf96b68c4-6269k"
Sep 10 22:14:46.167047 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:46.167027    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-d6a4de-openshift-default-dockercfg-bx5ld\""
Sep 10 22:14:46.174820 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:46.174798    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-d6a4de-openshift-default-6bf96b68c4-6269k"]
Sep 10 22:14:46.351761 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:46.351728    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78896706-4631-4e69-b205-dfad4be3b549-istiod-ca-cert\") pod \"e2e-autores-d6a4de-openshift-default-6bf96b68c4-6269k\" (UID: \"78896706-4631-4e69-b205-dfad4be3b549\") " pod="openshift-ingress/e2e-autores-d6a4de-openshift-default-6bf96b68c4-6269k"
Sep 10 22:14:46.351937 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:46.351771    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78896706-4631-4e69-b205-dfad4be3b549-istio-envoy\") pod \"e2e-autores-d6a4de-openshift-default-6bf96b68c4-6269k\" (UID: \"78896706-4631-4e69-b205-dfad4be3b549\") " pod="openshift-ingress/e2e-autores-d6a4de-openshift-default-6bf96b68c4-6269k"
Sep 10 22:14:46.351937 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:46.351791    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78896706-4631-4e69-b205-dfad4be3b549-istio-data\") pod \"e2e-autores-d6a4de-openshift-default-6bf96b68c4-6269k\" (UID: \"78896706-4631-4e69-b205-dfad4be3b549\") " pod="openshift-ingress/e2e-autores-d6a4de-openshift-default-6bf96b68c4-6269k"
Sep 10 22:14:46.351937 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:46.351835    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78896706-4631-4e69-b205-dfad4be3b549-workload-socket\") pod \"e2e-autores-d6a4de-openshift-default-6bf96b68c4-6269k\" (UID: \"78896706-4631-4e69-b205-dfad4be3b549\") " pod="openshift-ingress/e2e-autores-d6a4de-openshift-default-6bf96b68c4-6269k"
Sep 10 22:14:46.351937 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:46.351924    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78896706-4631-4e69-b205-dfad4be3b549-istio-podinfo\") pod \"e2e-autores-d6a4de-openshift-default-6bf96b68c4-6269k\" (UID: \"78896706-4631-4e69-b205-dfad4be3b549\") " pod="openshift-ingress/e2e-autores-d6a4de-openshift-default-6bf96b68c4-6269k"
Sep 10 22:14:46.352079 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:46.351953    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78896706-4631-4e69-b205-dfad4be3b549-workload-certs\") pod \"e2e-autores-d6a4de-openshift-default-6bf96b68c4-6269k\" (UID: \"78896706-4631-4e69-b205-dfad4be3b549\") " pod="openshift-ingress/e2e-autores-d6a4de-openshift-default-6bf96b68c4-6269k"
Sep 10 22:14:46.352079 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:46.351968    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjf7x\" (UniqueName: \"kubernetes.io/projected/78896706-4631-4e69-b205-dfad4be3b549-kube-api-access-zjf7x\") pod \"e2e-autores-d6a4de-openshift-default-6bf96b68c4-6269k\" (UID: \"78896706-4631-4e69-b205-dfad4be3b549\") " pod="openshift-ingress/e2e-autores-d6a4de-openshift-default-6bf96b68c4-6269k"
Sep 10 22:14:46.352079 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:46.351987    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78896706-4631-4e69-b205-dfad4be3b549-istio-token\") pod \"e2e-autores-d6a4de-openshift-default-6bf96b68c4-6269k\" (UID: \"78896706-4631-4e69-b205-dfad4be3b549\") " pod="openshift-ingress/e2e-autores-d6a4de-openshift-default-6bf96b68c4-6269k"
Sep 10 22:14:46.352079 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:46.352036    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78896706-4631-4e69-b205-dfad4be3b549-credential-socket\") pod \"e2e-autores-d6a4de-openshift-default-6bf96b68c4-6269k\" (UID: \"78896706-4631-4e69-b205-dfad4be3b549\") " pod="openshift-ingress/e2e-autores-d6a4de-openshift-default-6bf96b68c4-6269k"
Sep 10 22:14:46.452561 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:46.452499    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78896706-4631-4e69-b205-dfad4be3b549-istio-token\") pod \"e2e-autores-d6a4de-openshift-default-6bf96b68c4-6269k\" (UID: \"78896706-4631-4e69-b205-dfad4be3b549\") " pod="openshift-ingress/e2e-autores-d6a4de-openshift-default-6bf96b68c4-6269k"
Sep 10 22:14:46.452561 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:46.452531    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78896706-4631-4e69-b205-dfad4be3b549-credential-socket\") pod \"e2e-autores-d6a4de-openshift-default-6bf96b68c4-6269k\" (UID: \"78896706-4631-4e69-b205-dfad4be3b549\") " pod="openshift-ingress/e2e-autores-d6a4de-openshift-default-6bf96b68c4-6269k"
Sep 10 22:14:46.452561 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:46.452556    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78896706-4631-4e69-b205-dfad4be3b549-istiod-ca-cert\") pod \"e2e-autores-d6a4de-openshift-default-6bf96b68c4-6269k\" (UID: \"78896706-4631-4e69-b205-dfad4be3b549\") " pod="openshift-ingress/e2e-autores-d6a4de-openshift-default-6bf96b68c4-6269k"
Sep 10 22:14:46.452754 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:46.452590    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78896706-4631-4e69-b205-dfad4be3b549-istio-envoy\") pod \"e2e-autores-d6a4de-openshift-default-6bf96b68c4-6269k\" (UID: \"78896706-4631-4e69-b205-dfad4be3b549\") " pod="openshift-ingress/e2e-autores-d6a4de-openshift-default-6bf96b68c4-6269k"
Sep 10 22:14:46.452754 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:46.452617    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78896706-4631-4e69-b205-dfad4be3b549-istio-data\") pod \"e2e-autores-d6a4de-openshift-default-6bf96b68c4-6269k\" (UID: \"78896706-4631-4e69-b205-dfad4be3b549\") " pod="openshift-ingress/e2e-autores-d6a4de-openshift-default-6bf96b68c4-6269k"
Sep 10 22:14:46.452754 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:46.452643    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78896706-4631-4e69-b205-dfad4be3b549-workload-socket\") pod \"e2e-autores-d6a4de-openshift-default-6bf96b68c4-6269k\" (UID: \"78896706-4631-4e69-b205-dfad4be3b549\") " pod="openshift-ingress/e2e-autores-d6a4de-openshift-default-6bf96b68c4-6269k"
Sep 10 22:14:46.452754 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:46.452689    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78896706-4631-4e69-b205-dfad4be3b549-istio-podinfo\") pod \"e2e-autores-d6a4de-openshift-default-6bf96b68c4-6269k\" (UID: \"78896706-4631-4e69-b205-dfad4be3b549\") " pod="openshift-ingress/e2e-autores-d6a4de-openshift-default-6bf96b68c4-6269k"
Sep 10 22:14:46.452754 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:46.452719    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78896706-4631-4e69-b205-dfad4be3b549-workload-certs\") pod \"e2e-autores-d6a4de-openshift-default-6bf96b68c4-6269k\" (UID: \"78896706-4631-4e69-b205-dfad4be3b549\") " pod="openshift-ingress/e2e-autores-d6a4de-openshift-default-6bf96b68c4-6269k"
Sep 10 22:14:46.452754 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:46.452740    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjf7x\" (UniqueName: \"kubernetes.io/projected/78896706-4631-4e69-b205-dfad4be3b549-kube-api-access-zjf7x\") pod \"e2e-autores-d6a4de-openshift-default-6bf96b68c4-6269k\" (UID: \"78896706-4631-4e69-b205-dfad4be3b549\") " pod="openshift-ingress/e2e-autores-d6a4de-openshift-default-6bf96b68c4-6269k"
Sep 10 22:14:46.453085 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:46.453023    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78896706-4631-4e69-b205-dfad4be3b549-credential-socket\") pod \"e2e-autores-d6a4de-openshift-default-6bf96b68c4-6269k\" (UID: \"78896706-4631-4e69-b205-dfad4be3b549\") " pod="openshift-ingress/e2e-autores-d6a4de-openshift-default-6bf96b68c4-6269k"
Sep 10 22:14:46.453085 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:46.453050    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78896706-4631-4e69-b205-dfad4be3b549-istio-data\") pod \"e2e-autores-d6a4de-openshift-default-6bf96b68c4-6269k\" (UID: \"78896706-4631-4e69-b205-dfad4be3b549\") " pod="openshift-ingress/e2e-autores-d6a4de-openshift-default-6bf96b68c4-6269k"
Sep 10 22:14:46.453197 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:46.453141    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78896706-4631-4e69-b205-dfad4be3b549-workload-certs\") pod \"e2e-autores-d6a4de-openshift-default-6bf96b68c4-6269k\" (UID: \"78896706-4631-4e69-b205-dfad4be3b549\") " pod="openshift-ingress/e2e-autores-d6a4de-openshift-default-6bf96b68c4-6269k"
Sep 10 22:14:46.453376 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:46.453353    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78896706-4631-4e69-b205-dfad4be3b549-workload-socket\") pod \"e2e-autores-d6a4de-openshift-default-6bf96b68c4-6269k\" (UID: \"78896706-4631-4e69-b205-dfad4be3b549\") " pod="openshift-ingress/e2e-autores-d6a4de-openshift-default-6bf96b68c4-6269k"
Sep 10 22:14:46.453473 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:46.453452    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78896706-4631-4e69-b205-dfad4be3b549-istiod-ca-cert\") pod \"e2e-autores-d6a4de-openshift-default-6bf96b68c4-6269k\" (UID: \"78896706-4631-4e69-b205-dfad4be3b549\") " pod="openshift-ingress/e2e-autores-d6a4de-openshift-default-6bf96b68c4-6269k"
Sep 10 22:14:46.455020 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:46.455004    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78896706-4631-4e69-b205-dfad4be3b549-istio-envoy\") pod \"e2e-autores-d6a4de-openshift-default-6bf96b68c4-6269k\" (UID: \"78896706-4631-4e69-b205-dfad4be3b549\") " pod="openshift-ingress/e2e-autores-d6a4de-openshift-default-6bf96b68c4-6269k"
Sep 10 22:14:46.455282 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:46.455259    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78896706-4631-4e69-b205-dfad4be3b549-istio-podinfo\") pod \"e2e-autores-d6a4de-openshift-default-6bf96b68c4-6269k\" (UID: \"78896706-4631-4e69-b205-dfad4be3b549\") " pod="openshift-ingress/e2e-autores-d6a4de-openshift-default-6bf96b68c4-6269k"
Sep 10 22:14:46.459973 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:46.459951    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78896706-4631-4e69-b205-dfad4be3b549-istio-token\") pod \"e2e-autores-d6a4de-openshift-default-6bf96b68c4-6269k\" (UID: \"78896706-4631-4e69-b205-dfad4be3b549\") " pod="openshift-ingress/e2e-autores-d6a4de-openshift-default-6bf96b68c4-6269k"
Sep 10 22:14:46.460211 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:46.460195    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjf7x\" (UniqueName: \"kubernetes.io/projected/78896706-4631-4e69-b205-dfad4be3b549-kube-api-access-zjf7x\") pod \"e2e-autores-d6a4de-openshift-default-6bf96b68c4-6269k\" (UID: \"78896706-4631-4e69-b205-dfad4be3b549\") " pod="openshift-ingress/e2e-autores-d6a4de-openshift-default-6bf96b68c4-6269k"
Sep 10 22:14:46.476297 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:46.476274    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-d6a4de-openshift-default-6bf96b68c4-6269k"
Sep 10 22:14:46.753504 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:46.753415    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-d6a4de-openshift-default-6bf96b68c4-6269k"]
Sep 10 22:14:46.755667 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:14:46.755640    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78896706_4631_4e69_b205_dfad4be3b549.slice/crio-ef076a63501faac83d971ffeb069524382d8478c68a663abb27561cf1f8abe66 WatchSource:0}: Error finding container ef076a63501faac83d971ffeb069524382d8478c68a663abb27561cf1f8abe66: Status 404 returned error can't find the container with id ef076a63501faac83d971ffeb069524382d8478c68a663abb27561cf1f8abe66
Sep 10 22:14:46.757931 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:46.757896    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 22:14:46.758008 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:46.757962    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 22:14:46.758008 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:46.757990    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 22:14:46.831404 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:46.831375    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-d6a4de-openshift-default-6bf96b68c4-6269k" event={"ID":"78896706-4631-4e69-b205-dfad4be3b549","Type":"ContainerStarted","Data":"69816f10c217d08dd2575ba64ac215c8fe781713aa566fc6d347f58420a8ca87"}
Sep 10 22:14:46.831742 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:46.831442    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-d6a4de-openshift-default-6bf96b68c4-6269k" event={"ID":"78896706-4631-4e69-b205-dfad4be3b549","Type":"ContainerStarted","Data":"ef076a63501faac83d971ffeb069524382d8478c68a663abb27561cf1f8abe66"}
Sep 10 22:14:46.831742 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:46.831522    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7520-9206c4" podUID="769badbb-f1b4-4d68-bb8d-8d757a1e7b2b" containerName="test-curl-7520-9206c4" containerID="cri-o://be4bb514e6ce0665adc7b0684f4436127d659ea151d7e6a9b410b1f56acb81ca" gracePeriod=2
Sep 10 22:14:46.852598 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:46.852546    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-d6a4de-openshift-default-6bf96b68c4-6269k" podStartSLOduration=0.852523705 podStartE2EDuration="852.523705ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 22:14:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:14:46.849791487 +0000 UTC m=+807.805859194" watchObservedRunningTime="2026-09-10 22:14:46.852523705 +0000 UTC m=+807.808591427"
Sep 10 22:14:46.950173 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:46.950143    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7520-e62155"
Sep 10 22:14:47.057948 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:47.057870    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-22vvh\" (UniqueName: \"kubernetes.io/projected/3d95e13c-20cc-482d-a221-022236762244-kube-api-access-22vvh\") pod \"3d95e13c-20cc-482d-a221-022236762244\" (UID: \"3d95e13c-20cc-482d-a221-022236762244\") "
Sep 10 22:14:47.059989 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:47.059967    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d95e13c-20cc-482d-a221-022236762244-kube-api-access-22vvh" (OuterVolumeSpecName: "kube-api-access-22vvh") pod "3d95e13c-20cc-482d-a221-022236762244" (UID: "3d95e13c-20cc-482d-a221-022236762244"). InnerVolumeSpecName "kube-api-access-22vvh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 22:14:47.159227 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:47.159180    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-22vvh\" (UniqueName: \"kubernetes.io/projected/3d95e13c-20cc-482d-a221-022236762244-kube-api-access-22vvh\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:14:47.380564 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:47.380490    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-2d222ead-openshift-default-84f9ff6b48-8w8gz"]
Sep 10 22:14:47.383658 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:47.383641    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-2d222ead-openshift-default-84f9ff6b48-8w8gz"
Sep 10 22:14:47.387106 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:47.387085    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-2d222ead-openshift-default-dockercfg-4d97l\""
Sep 10 22:14:47.408251 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:47.408222    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-2d222ead-openshift-default-84f9ff6b48-8w8gz"]
Sep 10 22:14:47.476750 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:47.476725    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-d6a4de-openshift-default-6bf96b68c4-6269k"
Sep 10 22:14:47.481138 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:47.481114    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-d6a4de-openshift-default-6bf96b68c4-6269k"
Sep 10 22:14:47.498459 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:47.498433    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3d95e13c-20cc-482d-a221-022236762244" path="/var/lib/kubelet/pods/3d95e13c-20cc-482d-a221-022236762244/volumes"
Sep 10 22:14:47.562674 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:47.562650    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92c884a6-d95f-4e4e-b911-bb848349b39b-istio-envoy\") pod \"e2e-ait-2d222ead-openshift-default-84f9ff6b48-8w8gz\" (UID: \"92c884a6-d95f-4e4e-b911-bb848349b39b\") " pod="openshift-ingress/e2e-ait-2d222ead-openshift-default-84f9ff6b48-8w8gz"
Sep 10 22:14:47.562770 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:47.562683    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chfhd\" (UniqueName: \"kubernetes.io/projected/92c884a6-d95f-4e4e-b911-bb848349b39b-kube-api-access-chfhd\") pod \"e2e-ait-2d222ead-openshift-default-84f9ff6b48-8w8gz\" (UID: \"92c884a6-d95f-4e4e-b911-bb848349b39b\") " pod="openshift-ingress/e2e-ait-2d222ead-openshift-default-84f9ff6b48-8w8gz"
Sep 10 22:14:47.562770 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:47.562710    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92c884a6-d95f-4e4e-b911-bb848349b39b-workload-certs\") pod \"e2e-ait-2d222ead-openshift-default-84f9ff6b48-8w8gz\" (UID: \"92c884a6-d95f-4e4e-b911-bb848349b39b\") " pod="openshift-ingress/e2e-ait-2d222ead-openshift-default-84f9ff6b48-8w8gz"
Sep 10 22:14:47.562770 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:47.562730    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92c884a6-d95f-4e4e-b911-bb848349b39b-istiod-ca-cert\") pod \"e2e-ait-2d222ead-openshift-default-84f9ff6b48-8w8gz\" (UID: \"92c884a6-d95f-4e4e-b911-bb848349b39b\") " pod="openshift-ingress/e2e-ait-2d222ead-openshift-default-84f9ff6b48-8w8gz"
Sep 10 22:14:47.562905 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:47.562812    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92c884a6-d95f-4e4e-b911-bb848349b39b-istio-podinfo\") pod \"e2e-ait-2d222ead-openshift-default-84f9ff6b48-8w8gz\" (UID: \"92c884a6-d95f-4e4e-b911-bb848349b39b\") " pod="openshift-ingress/e2e-ait-2d222ead-openshift-default-84f9ff6b48-8w8gz"
Sep 10 22:14:47.562905 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:47.562844    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92c884a6-d95f-4e4e-b911-bb848349b39b-workload-socket\") pod \"e2e-ait-2d222ead-openshift-default-84f9ff6b48-8w8gz\" (UID: \"92c884a6-d95f-4e4e-b911-bb848349b39b\") " pod="openshift-ingress/e2e-ait-2d222ead-openshift-default-84f9ff6b48-8w8gz"
Sep 10 22:14:47.562905 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:47.562895    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92c884a6-d95f-4e4e-b911-bb848349b39b-istio-data\") pod \"e2e-ait-2d222ead-openshift-default-84f9ff6b48-8w8gz\" (UID: \"92c884a6-d95f-4e4e-b911-bb848349b39b\") " pod="openshift-ingress/e2e-ait-2d222ead-openshift-default-84f9ff6b48-8w8gz"
Sep 10 22:14:47.562998 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:47.562925    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92c884a6-d95f-4e4e-b911-bb848349b39b-credential-socket\") pod \"e2e-ait-2d222ead-openshift-default-84f9ff6b48-8w8gz\" (UID: \"92c884a6-d95f-4e4e-b911-bb848349b39b\") " pod="openshift-ingress/e2e-ait-2d222ead-openshift-default-84f9ff6b48-8w8gz"
Sep 10 22:14:47.562998 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:47.562942    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92c884a6-d95f-4e4e-b911-bb848349b39b-istio-token\") pod \"e2e-ait-2d222ead-openshift-default-84f9ff6b48-8w8gz\" (UID: \"92c884a6-d95f-4e4e-b911-bb848349b39b\") " pod="openshift-ingress/e2e-ait-2d222ead-openshift-default-84f9ff6b48-8w8gz"
Sep 10 22:14:47.664085 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:47.664059    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92c884a6-d95f-4e4e-b911-bb848349b39b-istio-podinfo\") pod \"e2e-ait-2d222ead-openshift-default-84f9ff6b48-8w8gz\" (UID: \"92c884a6-d95f-4e4e-b911-bb848349b39b\") " pod="openshift-ingress/e2e-ait-2d222ead-openshift-default-84f9ff6b48-8w8gz"
Sep 10 22:14:47.664172 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:47.664118    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92c884a6-d95f-4e4e-b911-bb848349b39b-workload-socket\") pod \"e2e-ait-2d222ead-openshift-default-84f9ff6b48-8w8gz\" (UID: \"92c884a6-d95f-4e4e-b911-bb848349b39b\") " pod="openshift-ingress/e2e-ait-2d222ead-openshift-default-84f9ff6b48-8w8gz"
Sep 10 22:14:47.664332 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:47.664312    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92c884a6-d95f-4e4e-b911-bb848349b39b-istio-data\") pod \"e2e-ait-2d222ead-openshift-default-84f9ff6b48-8w8gz\" (UID: \"92c884a6-d95f-4e4e-b911-bb848349b39b\") " pod="openshift-ingress/e2e-ait-2d222ead-openshift-default-84f9ff6b48-8w8gz"
Sep 10 22:14:47.664489 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:47.664470    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92c884a6-d95f-4e4e-b911-bb848349b39b-workload-socket\") pod \"e2e-ait-2d222ead-openshift-default-84f9ff6b48-8w8gz\" (UID: \"92c884a6-d95f-4e4e-b911-bb848349b39b\") " pod="openshift-ingress/e2e-ait-2d222ead-openshift-default-84f9ff6b48-8w8gz"
Sep 10 22:14:47.664555 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:47.664520    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92c884a6-d95f-4e4e-b911-bb848349b39b-credential-socket\") pod \"e2e-ait-2d222ead-openshift-default-84f9ff6b48-8w8gz\" (UID: \"92c884a6-d95f-4e4e-b911-bb848349b39b\") " pod="openshift-ingress/e2e-ait-2d222ead-openshift-default-84f9ff6b48-8w8gz"
Sep 10 22:14:47.664605 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:47.664553    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92c884a6-d95f-4e4e-b911-bb848349b39b-istio-token\") pod \"e2e-ait-2d222ead-openshift-default-84f9ff6b48-8w8gz\" (UID: \"92c884a6-d95f-4e4e-b911-bb848349b39b\") " pod="openshift-ingress/e2e-ait-2d222ead-openshift-default-84f9ff6b48-8w8gz"
Sep 10 22:14:47.664643 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:47.664599    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92c884a6-d95f-4e4e-b911-bb848349b39b-istio-data\") pod \"e2e-ait-2d222ead-openshift-default-84f9ff6b48-8w8gz\" (UID: \"92c884a6-d95f-4e4e-b911-bb848349b39b\") " pod="openshift-ingress/e2e-ait-2d222ead-openshift-default-84f9ff6b48-8w8gz"
Sep 10 22:14:47.664643 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:47.664614    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92c884a6-d95f-4e4e-b911-bb848349b39b-istio-envoy\") pod \"e2e-ait-2d222ead-openshift-default-84f9ff6b48-8w8gz\" (UID: \"92c884a6-d95f-4e4e-b911-bb848349b39b\") " pod="openshift-ingress/e2e-ait-2d222ead-openshift-default-84f9ff6b48-8w8gz"
Sep 10 22:14:47.664720 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:47.664651    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chfhd\" (UniqueName: \"kubernetes.io/projected/92c884a6-d95f-4e4e-b911-bb848349b39b-kube-api-access-chfhd\") pod \"e2e-ait-2d222ead-openshift-default-84f9ff6b48-8w8gz\" (UID: \"92c884a6-d95f-4e4e-b911-bb848349b39b\") " pod="openshift-ingress/e2e-ait-2d222ead-openshift-default-84f9ff6b48-8w8gz"
Sep 10 22:14:47.664720 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:47.664704    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92c884a6-d95f-4e4e-b911-bb848349b39b-workload-certs\") pod \"e2e-ait-2d222ead-openshift-default-84f9ff6b48-8w8gz\" (UID: \"92c884a6-d95f-4e4e-b911-bb848349b39b\") " pod="openshift-ingress/e2e-ait-2d222ead-openshift-default-84f9ff6b48-8w8gz"
Sep 10 22:14:47.664795 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:47.664728    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92c884a6-d95f-4e4e-b911-bb848349b39b-istiod-ca-cert\") pod \"e2e-ait-2d222ead-openshift-default-84f9ff6b48-8w8gz\" (UID: \"92c884a6-d95f-4e4e-b911-bb848349b39b\") " pod="openshift-ingress/e2e-ait-2d222ead-openshift-default-84f9ff6b48-8w8gz"
Sep 10 22:14:47.664842 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:47.664807    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92c884a6-d95f-4e4e-b911-bb848349b39b-credential-socket\") pod \"e2e-ait-2d222ead-openshift-default-84f9ff6b48-8w8gz\" (UID: \"92c884a6-d95f-4e4e-b911-bb848349b39b\") " pod="openshift-ingress/e2e-ait-2d222ead-openshift-default-84f9ff6b48-8w8gz"
Sep 10 22:14:47.665044 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:47.665026    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92c884a6-d95f-4e4e-b911-bb848349b39b-workload-certs\") pod \"e2e-ait-2d222ead-openshift-default-84f9ff6b48-8w8gz\" (UID: \"92c884a6-d95f-4e4e-b911-bb848349b39b\") " pod="openshift-ingress/e2e-ait-2d222ead-openshift-default-84f9ff6b48-8w8gz"
Sep 10 22:14:47.665367 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:47.665346    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92c884a6-d95f-4e4e-b911-bb848349b39b-istiod-ca-cert\") pod \"e2e-ait-2d222ead-openshift-default-84f9ff6b48-8w8gz\" (UID: \"92c884a6-d95f-4e4e-b911-bb848349b39b\") " pod="openshift-ingress/e2e-ait-2d222ead-openshift-default-84f9ff6b48-8w8gz"
Sep 10 22:14:47.667038 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:47.667013    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92c884a6-d95f-4e4e-b911-bb848349b39b-istio-envoy\") pod \"e2e-ait-2d222ead-openshift-default-84f9ff6b48-8w8gz\" (UID: \"92c884a6-d95f-4e4e-b911-bb848349b39b\") " pod="openshift-ingress/e2e-ait-2d222ead-openshift-default-84f9ff6b48-8w8gz"
Sep 10 22:14:47.667232 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:47.667216    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92c884a6-d95f-4e4e-b911-bb848349b39b-istio-podinfo\") pod \"e2e-ait-2d222ead-openshift-default-84f9ff6b48-8w8gz\" (UID: \"92c884a6-d95f-4e4e-b911-bb848349b39b\") " pod="openshift-ingress/e2e-ait-2d222ead-openshift-default-84f9ff6b48-8w8gz"
Sep 10 22:14:47.672739 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:47.672720    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-chfhd\" (UniqueName: \"kubernetes.io/projected/92c884a6-d95f-4e4e-b911-bb848349b39b-kube-api-access-chfhd\") pod \"e2e-ait-2d222ead-openshift-default-84f9ff6b48-8w8gz\" (UID: \"92c884a6-d95f-4e4e-b911-bb848349b39b\") " pod="openshift-ingress/e2e-ait-2d222ead-openshift-default-84f9ff6b48-8w8gz"
Sep 10 22:14:47.672953 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:47.672936    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92c884a6-d95f-4e4e-b911-bb848349b39b-istio-token\") pod \"e2e-ait-2d222ead-openshift-default-84f9ff6b48-8w8gz\" (UID: \"92c884a6-d95f-4e4e-b911-bb848349b39b\") " pod="openshift-ingress/e2e-ait-2d222ead-openshift-default-84f9ff6b48-8w8gz"
Sep 10 22:14:47.693913 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:47.693892    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-2d222ead-openshift-default-84f9ff6b48-8w8gz"
Sep 10 22:14:47.837578 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:47.837551    2582 generic.go:356] "Generic (PLEG): container finished" podID="3d95e13c-20cc-482d-a221-022236762244" containerID="5defbfc6bf843f09fcd31b017babc2f0147eb07fa6ca79856d156a836bac18e9" exitCode=137
Sep 10 22:14:47.837942 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:47.837593    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7520-e62155"
Sep 10 22:14:47.837942 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:47.837649    2582 scope.go:117] "RemoveContainer" containerID="5defbfc6bf843f09fcd31b017babc2f0147eb07fa6ca79856d156a836bac18e9"
Sep 10 22:14:47.837942 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:47.837892    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-d6a4de-openshift-default-6bf96b68c4-6269k"
Sep 10 22:14:47.839166 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:47.839132    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-d6a4de-openshift-default-6bf96b68c4-6269k"
Sep 10 22:14:47.849920 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:47.849894    2582 scope.go:117] "RemoveContainer" containerID="5defbfc6bf843f09fcd31b017babc2f0147eb07fa6ca79856d156a836bac18e9"
Sep 10 22:14:47.850200 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:14:47.850178    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5defbfc6bf843f09fcd31b017babc2f0147eb07fa6ca79856d156a836bac18e9\": container with ID starting with 5defbfc6bf843f09fcd31b017babc2f0147eb07fa6ca79856d156a836bac18e9 not found: ID does not exist" containerID="5defbfc6bf843f09fcd31b017babc2f0147eb07fa6ca79856d156a836bac18e9"
Sep 10 22:14:47.850262 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:47.850208    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5defbfc6bf843f09fcd31b017babc2f0147eb07fa6ca79856d156a836bac18e9"} err="failed to get container status \"5defbfc6bf843f09fcd31b017babc2f0147eb07fa6ca79856d156a836bac18e9\": rpc error: code = NotFound desc = could not find container \"5defbfc6bf843f09fcd31b017babc2f0147eb07fa6ca79856d156a836bac18e9\": container with ID starting with 5defbfc6bf843f09fcd31b017babc2f0147eb07fa6ca79856d156a836bac18e9 not found: ID does not exist"
Sep 10 22:14:47.969265 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:47.969191    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-2d222ead-openshift-default-84f9ff6b48-8w8gz"]
Sep 10 22:14:47.972447 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:14:47.972419    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92c884a6_d95f_4e4e_b911_bb848349b39b.slice/crio-e7872523987bec0ec62cfc606c12378c6554bd84c5a6a645d6d7f376dc80871b WatchSource:0}: Error finding container e7872523987bec0ec62cfc606c12378c6554bd84c5a6a645d6d7f376dc80871b: Status 404 returned error can't find the container with id e7872523987bec0ec62cfc606c12378c6554bd84c5a6a645d6d7f376dc80871b
Sep 10 22:14:47.974569 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:47.974538    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 22:14:47.974633 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:47.974601    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 22:14:47.974633 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:47.974629    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 22:14:48.844085 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:48.843973    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2d222ead-openshift-default-84f9ff6b48-8w8gz" event={"ID":"92c884a6-d95f-4e4e-b911-bb848349b39b","Type":"ContainerStarted","Data":"cbeeef6caed30a638f7c6b25ef75ac084df2f60eb7eb27fbf91db8a9f6a7b979"}
Sep 10 22:14:48.844085 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:48.844029    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2d222ead-openshift-default-84f9ff6b48-8w8gz" event={"ID":"92c884a6-d95f-4e4e-b911-bb848349b39b","Type":"ContainerStarted","Data":"e7872523987bec0ec62cfc606c12378c6554bd84c5a6a645d6d7f376dc80871b"}
Sep 10 22:14:48.864385 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:48.864346    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-2d222ead-openshift-default-84f9ff6b48-8w8gz" podStartSLOduration=1.864330509 podStartE2EDuration="1.864330509s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 22:14:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:14:48.862584412 +0000 UTC m=+809.818652136" watchObservedRunningTime="2026-09-10 22:14:48.864330509 +0000 UTC m=+809.820398282"
Sep 10 22:14:48.978121 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:48.978095    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7520-9206c4"
Sep 10 22:14:49.078189 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:49.078164    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dr8bn\" (UniqueName: \"kubernetes.io/projected/769badbb-f1b4-4d68-bb8d-8d757a1e7b2b-kube-api-access-dr8bn\") pod \"769badbb-f1b4-4d68-bb8d-8d757a1e7b2b\" (UID: \"769badbb-f1b4-4d68-bb8d-8d757a1e7b2b\") "
Sep 10 22:14:49.080198 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:49.080162    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/769badbb-f1b4-4d68-bb8d-8d757a1e7b2b-kube-api-access-dr8bn" (OuterVolumeSpecName: "kube-api-access-dr8bn") pod "769badbb-f1b4-4d68-bb8d-8d757a1e7b2b" (UID: "769badbb-f1b4-4d68-bb8d-8d757a1e7b2b"). InnerVolumeSpecName "kube-api-access-dr8bn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 22:14:49.179586 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:49.179561    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dr8bn\" (UniqueName: \"kubernetes.io/projected/769badbb-f1b4-4d68-bb8d-8d757a1e7b2b-kube-api-access-dr8bn\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:14:49.498769 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:49.498738    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="769badbb-f1b4-4d68-bb8d-8d757a1e7b2b" path="/var/lib/kubelet/pods/769badbb-f1b4-4d68-bb8d-8d757a1e7b2b/volumes"
Sep 10 22:14:49.694161 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:49.694130    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-2d222ead-openshift-default-84f9ff6b48-8w8gz"
Sep 10 22:14:49.699353 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:49.699332    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-2d222ead-openshift-default-84f9ff6b48-8w8gz"
Sep 10 22:14:49.851520 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:49.851428    2582 generic.go:356] "Generic (PLEG): container finished" podID="769badbb-f1b4-4d68-bb8d-8d757a1e7b2b" containerID="be4bb514e6ce0665adc7b0684f4436127d659ea151d7e6a9b410b1f56acb81ca" exitCode=137
Sep 10 22:14:49.851520 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:49.851468    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7520-9206c4"
Sep 10 22:14:49.851520 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:49.851501    2582 scope.go:117] "RemoveContainer" containerID="be4bb514e6ce0665adc7b0684f4436127d659ea151d7e6a9b410b1f56acb81ca"
Sep 10 22:14:49.852104 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:49.851885    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-2d222ead-openshift-default-84f9ff6b48-8w8gz"
Sep 10 22:14:49.853123 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:49.852931    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-2d222ead-openshift-default-84f9ff6b48-8w8gz"
Sep 10 22:14:49.862301 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:49.862275    2582 scope.go:117] "RemoveContainer" containerID="be4bb514e6ce0665adc7b0684f4436127d659ea151d7e6a9b410b1f56acb81ca"
Sep 10 22:14:49.862742 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:14:49.862719    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be4bb514e6ce0665adc7b0684f4436127d659ea151d7e6a9b410b1f56acb81ca\": container with ID starting with be4bb514e6ce0665adc7b0684f4436127d659ea151d7e6a9b410b1f56acb81ca not found: ID does not exist" containerID="be4bb514e6ce0665adc7b0684f4436127d659ea151d7e6a9b410b1f56acb81ca"
Sep 10 22:14:49.862832 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:49.862752    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be4bb514e6ce0665adc7b0684f4436127d659ea151d7e6a9b410b1f56acb81ca"} err="failed to get container status \"be4bb514e6ce0665adc7b0684f4436127d659ea151d7e6a9b410b1f56acb81ca\": rpc error: code = NotFound desc = could not find container \"be4bb514e6ce0665adc7b0684f4436127d659ea151d7e6a9b410b1f56acb81ca\": container with ID starting with be4bb514e6ce0665adc7b0684f4436127d659ea151d7e6a9b410b1f56acb81ca not found: ID does not exist"
Sep 10 22:14:52.722574 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:52.722540    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-2d222ead-8d6745f9-f94wb"]
Sep 10 22:14:52.726950 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:52.726932    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-2d222ead-8d6745f9-f94wb"
Sep 10 22:14:52.729131 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:52.729107    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-2d222ead\""
Sep 10 22:14:52.729232 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:52.729194    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-2d222ead-dockercfg-8vqng\""
Sep 10 22:14:52.737750 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:52.737724    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-2d222ead-8d6745f9-f94wb"]
Sep 10 22:14:52.812738 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:52.812702    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kllfx\" (UniqueName: \"kubernetes.io/projected/3327fdea-2582-480f-8888-5c74d4080412-kube-api-access-kllfx\") pod \"payload-processing-e2e-ait-2d222ead-8d6745f9-f94wb\" (UID: \"3327fdea-2582-480f-8888-5c74d4080412\") " pod="openshift-ingress/payload-processing-e2e-ait-2d222ead-8d6745f9-f94wb"
Sep 10 22:14:52.812965 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:52.812762    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3327fdea-2582-480f-8888-5c74d4080412-plugins-config-volume\") pod \"payload-processing-e2e-ait-2d222ead-8d6745f9-f94wb\" (UID: \"3327fdea-2582-480f-8888-5c74d4080412\") " pod="openshift-ingress/payload-processing-e2e-ait-2d222ead-8d6745f9-f94wb"
Sep 10 22:14:52.913461 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:52.913428    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3327fdea-2582-480f-8888-5c74d4080412-plugins-config-volume\") pod \"payload-processing-e2e-ait-2d222ead-8d6745f9-f94wb\" (UID: \"3327fdea-2582-480f-8888-5c74d4080412\") " pod="openshift-ingress/payload-processing-e2e-ait-2d222ead-8d6745f9-f94wb"
Sep 10 22:14:52.913648 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:52.913559    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kllfx\" (UniqueName: \"kubernetes.io/projected/3327fdea-2582-480f-8888-5c74d4080412-kube-api-access-kllfx\") pod \"payload-processing-e2e-ait-2d222ead-8d6745f9-f94wb\" (UID: \"3327fdea-2582-480f-8888-5c74d4080412\") " pod="openshift-ingress/payload-processing-e2e-ait-2d222ead-8d6745f9-f94wb"
Sep 10 22:14:52.914177 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:52.914157    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3327fdea-2582-480f-8888-5c74d4080412-plugins-config-volume\") pod \"payload-processing-e2e-ait-2d222ead-8d6745f9-f94wb\" (UID: \"3327fdea-2582-480f-8888-5c74d4080412\") " pod="openshift-ingress/payload-processing-e2e-ait-2d222ead-8d6745f9-f94wb"
Sep 10 22:14:52.923926 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:52.923900    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kllfx\" (UniqueName: \"kubernetes.io/projected/3327fdea-2582-480f-8888-5c74d4080412-kube-api-access-kllfx\") pod \"payload-processing-e2e-ait-2d222ead-8d6745f9-f94wb\" (UID: \"3327fdea-2582-480f-8888-5c74d4080412\") " pod="openshift-ingress/payload-processing-e2e-ait-2d222ead-8d6745f9-f94wb"
Sep 10 22:14:53.037458 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:53.037382    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-2d222ead-8d6745f9-f94wb"
Sep 10 22:14:53.321634 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:53.321603    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-2d222ead-8d6745f9-f94wb"]
Sep 10 22:14:53.323578 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:14:53.323551    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3327fdea_2582_480f_8888_5c74d4080412.slice/crio-02f36dfcf72a2d534f22f8110b029f32826ccca5c4020d754e75c197ad11c672 WatchSource:0}: Error finding container 02f36dfcf72a2d534f22f8110b029f32826ccca5c4020d754e75c197ad11c672: Status 404 returned error can't find the container with id 02f36dfcf72a2d534f22f8110b029f32826ccca5c4020d754e75c197ad11c672
Sep 10 22:14:53.873468 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:53.873429    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-2d222ead-8d6745f9-f94wb" event={"ID":"3327fdea-2582-480f-8888-5c74d4080412","Type":"ContainerStarted","Data":"02f36dfcf72a2d534f22f8110b029f32826ccca5c4020d754e75c197ad11c672"}
Sep 10 22:14:55.883555 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:55.883527    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-2d222ead-8d6745f9-f94wb" event={"ID":"3327fdea-2582-480f-8888-5c74d4080412","Type":"ContainerStarted","Data":"97346a13f802922335170280e00598dd1a8b7164212e499b8f68b698112d854e"}
Sep 10 22:14:55.883940 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:55.883666    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-2d222ead-8d6745f9-f94wb"
Sep 10 22:14:55.902703 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:14:55.902662    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-2d222ead-8d6745f9-f94wb" podStartSLOduration=1.5062959949999999 podStartE2EDuration="3.902647021s" totalImagesPullingTime="2.396351026s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 22:14:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:14:55.900149188 +0000 UTC m=+816.856216894" watchObservedRunningTime="2026-09-10 22:14:55.902647021 +0000 UTC m=+816.858714727"
Sep 10 22:15:00.136075 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:00.136041    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-2d222ead-29817975-kmt55"]
Sep 10 22:15:00.139578 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:00.139556    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-2d222ead-29817975-kmt55"
Sep 10 22:15:00.139702 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:00.139669    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e5a2d7-29817975-db5gb"]
Sep 10 22:15:00.141613 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:00.141594    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8hjrv\""
Sep 10 22:15:00.143062 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:00.143047    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e5a2d7-29817975-db5gb"
Sep 10 22:15:00.150172 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:00.150152    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-2d222ead-29817975-kmt55"]
Sep 10 22:15:00.151044 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:00.151027    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-2d222ead-29817975-kmt55"
Sep 10 22:15:00.153315 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:00.153295    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e5a2d7-29817975-db5gb"]
Sep 10 22:15:00.161275 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:00.161251    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e5a2d7-29817975-db5gb"
Sep 10 22:15:00.471770 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:00.471396    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-2d222ead-29817975-kmt55"]
Sep 10 22:15:00.541289 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:00.541140    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e5a2d7-29817975-db5gb"]
Sep 10 22:15:00.543897 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:15:00.543349    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf605e12b_a160_4c45_b2b7_759154729347.slice/crio-ee1d8bdc4ee5081caca7abe9cd6dd3cf50aef39028c442765b560327e157c879 WatchSource:0}: Error finding container ee1d8bdc4ee5081caca7abe9cd6dd3cf50aef39028c442765b560327e157c879: Status 404 returned error can't find the container with id ee1d8bdc4ee5081caca7abe9cd6dd3cf50aef39028c442765b560327e157c879
Sep 10 22:15:00.926240 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:00.926203    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e5a2d7-29817975-db5gb" event={"ID":"f605e12b-a160-4c45-b2b7-759154729347","Type":"ContainerStarted","Data":"ee1d8bdc4ee5081caca7abe9cd6dd3cf50aef39028c442765b560327e157c879"}
Sep 10 22:15:00.927454 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:00.927433    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-2d222ead-29817975-kmt55" event={"ID":"cca563dc-8ed1-45c4-ab45-deec290e50c8","Type":"ContainerStarted","Data":"d565b769b74fb02b30dc1eb7d7b0dec22d27409b60cb4c1a5ee789498baf2ad9"}
Sep 10 22:15:04.955805 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:04.955766    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e5a2d7-29817975-db5gb" event={"ID":"f605e12b-a160-4c45-b2b7-759154729347","Type":"ContainerStarted","Data":"d3e6de3c985d1fdcdc27a17f6adb46474c85e760f9e749a00e1e688decaaa015"}
Sep 10 22:15:04.957747 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:04.957711    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-2d222ead-29817975-kmt55" event={"ID":"cca563dc-8ed1-45c4-ab45-deec290e50c8","Type":"ContainerStarted","Data":"9ed94665562ab3cb5bce48d3e1e4b718ed48595990dbf2cf8be56f3fa6e4ade4"}
Sep 10 22:15:04.972169 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:04.972115    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e5a2d7-29817975-db5gb" podStartSLOduration=1.303690955 podStartE2EDuration="4.972089928s" totalImagesPullingTime="3.668398973s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 22:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:15:04.96959224 +0000 UTC m=+825.925659956" watchObservedRunningTime="2026-09-10 22:15:04.972089928 +0000 UTC m=+825.928157635"
Sep 10 22:15:04.984441 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:04.984381    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-2d222ead-29817975-kmt55" podStartSLOduration=1.269098745 podStartE2EDuration="4.984361017s" totalImagesPullingTime="3.715262272s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 22:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:15:04.983341918 +0000 UTC m=+825.939409649" watchObservedRunningTime="2026-09-10 22:15:04.984361017 +0000 UTC m=+825.940428724"
Sep 10 22:15:05.893011 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:05.892969    2582 patch_prober.go:28] interesting pod/e2e-ait-2d222ead-openshift-default-84f9ff6b48-8w8gz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 10 22:15:05.893158 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:05.893031    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-2d222ead-openshift-default-84f9ff6b48-8w8gz" podUID="92c884a6-d95f-4e4e-b911-bb848349b39b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 10 22:15:05.965746 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:05.965715    2582 generic.go:356] "Generic (PLEG): container finished" podID="cca563dc-8ed1-45c4-ab45-deec290e50c8" containerID="9ed94665562ab3cb5bce48d3e1e4b718ed48595990dbf2cf8be56f3fa6e4ade4" exitCode=0
Sep 10 22:15:05.966192 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:05.965812    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-2d222ead-29817975-kmt55" event={"ID":"cca563dc-8ed1-45c4-ab45-deec290e50c8","Type":"ContainerDied","Data":"9ed94665562ab3cb5bce48d3e1e4b718ed48595990dbf2cf8be56f3fa6e4ade4"}
Sep 10 22:15:05.967688 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:05.967661    2582 generic.go:356] "Generic (PLEG): container finished" podID="f605e12b-a160-4c45-b2b7-759154729347" containerID="d3e6de3c985d1fdcdc27a17f6adb46474c85e760f9e749a00e1e688decaaa015" exitCode=0
Sep 10 22:15:05.967790 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:05.967731    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e5a2d7-29817975-db5gb" event={"ID":"f605e12b-a160-4c45-b2b7-759154729347","Type":"ContainerDied","Data":"d3e6de3c985d1fdcdc27a17f6adb46474c85e760f9e749a00e1e688decaaa015"}
Sep 10 22:15:06.890976 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:06.890946    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-2d222ead-8d6745f9-f94wb"
Sep 10 22:15:07.192066 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:07.138728    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-2d222ead-29817975-kmt55"
Sep 10 22:15:07.192066 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:07.162605    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e5a2d7-29817975-db5gb"
Sep 10 22:15:07.977913 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:07.977882    2582 generic.go:356] "Generic (PLEG): container finished" podID="78896706-4631-4e69-b205-dfad4be3b549" containerID="69816f10c217d08dd2575ba64ac215c8fe781713aa566fc6d347f58420a8ca87" exitCode=0
Sep 10 22:15:07.978104 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:07.977953    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-d6a4de-openshift-default-6bf96b68c4-6269k" event={"ID":"78896706-4631-4e69-b205-dfad4be3b549","Type":"ContainerDied","Data":"69816f10c217d08dd2575ba64ac215c8fe781713aa566fc6d347f58420a8ca87"}
Sep 10 22:15:07.978367 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:07.978349    2582 scope.go:117] "RemoveContainer" containerID="69816f10c217d08dd2575ba64ac215c8fe781713aa566fc6d347f58420a8ca87"
Sep 10 22:15:07.979282 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:07.979265    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-2d222ead-29817975-kmt55"
Sep 10 22:15:07.979354 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:07.979263    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-2d222ead-29817975-kmt55" event={"ID":"cca563dc-8ed1-45c4-ab45-deec290e50c8","Type":"ContainerDied","Data":"d565b769b74fb02b30dc1eb7d7b0dec22d27409b60cb4c1a5ee789498baf2ad9"}
Sep 10 22:15:07.986404 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:07.979696    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d565b769b74fb02b30dc1eb7d7b0dec22d27409b60cb4c1a5ee789498baf2ad9"
Sep 10 22:15:07.986404 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:07.980260    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 22:15:07.986404 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:07.980600    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 22:15:07.986404 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:07.980896    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 22:15:07.986404 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:07.986128    2582 generic.go:356] "Generic (PLEG): container finished" podID="92c884a6-d95f-4e4e-b911-bb848349b39b" containerID="cbeeef6caed30a638f7c6b25ef75ac084df2f60eb7eb27fbf91db8a9f6a7b979" exitCode=0
Sep 10 22:15:07.986404 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:07.986218    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2d222ead-openshift-default-84f9ff6b48-8w8gz" event={"ID":"92c884a6-d95f-4e4e-b911-bb848349b39b","Type":"ContainerDied","Data":"cbeeef6caed30a638f7c6b25ef75ac084df2f60eb7eb27fbf91db8a9f6a7b979"}
Sep 10 22:15:07.987459 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:07.987427    2582 scope.go:117] "RemoveContainer" containerID="cbeeef6caed30a638f7c6b25ef75ac084df2f60eb7eb27fbf91db8a9f6a7b979"
Sep 10 22:15:07.988094 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:07.988081    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e5a2d7-29817975-db5gb"
Sep 10 22:15:07.988152 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:07.988088    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e5a2d7-29817975-db5gb" event={"ID":"f605e12b-a160-4c45-b2b7-759154729347","Type":"ContainerDied","Data":"ee1d8bdc4ee5081caca7abe9cd6dd3cf50aef39028c442765b560327e157c879"}
Sep 10 22:15:07.988152 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:07.988125    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ee1d8bdc4ee5081caca7abe9cd6dd3cf50aef39028c442765b560327e157c879"
Sep 10 22:15:07.990370 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:07.990343    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 22:15:07.990457 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:07.990401    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 22:15:07.990457 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:07.990430    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 22:15:08.998253 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:08.998215    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-d6a4de-openshift-default-6bf96b68c4-6269k" event={"ID":"78896706-4631-4e69-b205-dfad4be3b549","Type":"ContainerStarted","Data":"706d867cbfa19ac386445c30487b59f4c6c6f350bcbadd0a86d542fd46a5a51f"}
Sep 10 22:15:09.002367 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:09.002333    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2d222ead-openshift-default-84f9ff6b48-8w8gz" event={"ID":"92c884a6-d95f-4e4e-b911-bb848349b39b","Type":"ContainerStarted","Data":"8e0f7c14694ee7b11cbc20d141c3c380e379a574b6163bcb0461270ad2b3f834"}
Sep 10 22:15:09.477045 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:09.477005    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-d6a4de-openshift-default-6bf96b68c4-6269k"
Sep 10 22:15:09.477268 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:09.477138    2582 patch_prober.go:28] interesting pod/e2e-autores-d6a4de-openshift-default-6bf96b68c4-6269k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Sep 10 22:15:09.477268 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:09.477185    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-d6a4de-openshift-default-6bf96b68c4-6269k" podUID="78896706-4631-4e69-b205-dfad4be3b549" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Sep 10 22:15:09.697150 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:09.697116    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-2d222ead-openshift-default-84f9ff6b48-8w8gz"
Sep 10 22:15:09.697777 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:09.697750    2582 patch_prober.go:28] interesting pod/e2e-ait-2d222ead-openshift-default-84f9ff6b48-8w8gz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Sep 10 22:15:09.698033 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:09.698004    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-2d222ead-openshift-default-84f9ff6b48-8w8gz" podUID="92c884a6-d95f-4e4e-b911-bb848349b39b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Sep 10 22:15:10.477793 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:10.477754    2582 patch_prober.go:28] interesting pod/e2e-autores-d6a4de-openshift-default-6bf96b68c4-6269k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Sep 10 22:15:10.478411 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:10.478361    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-d6a4de-openshift-default-6bf96b68c4-6269k" podUID="78896706-4631-4e69-b205-dfad4be3b549" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Sep 10 22:15:10.695108 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:10.694624    2582 patch_prober.go:28] interesting pod/e2e-ait-2d222ead-openshift-default-84f9ff6b48-8w8gz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Sep 10 22:15:10.695108 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:10.694689    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-2d222ead-openshift-default-84f9ff6b48-8w8gz" podUID="92c884a6-d95f-4e4e-b911-bb848349b39b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Sep 10 22:15:11.477498 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:11.477434    2582 patch_prober.go:28] interesting pod/e2e-autores-d6a4de-openshift-default-6bf96b68c4-6269k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Sep 10 22:15:11.477731 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:11.477524    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-d6a4de-openshift-default-6bf96b68c4-6269k" podUID="78896706-4631-4e69-b205-dfad4be3b549" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Sep 10 22:15:11.695166 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:11.695121    2582 patch_prober.go:28] interesting pod/e2e-ait-2d222ead-openshift-default-84f9ff6b48-8w8gz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Sep 10 22:15:11.695648 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:11.695189    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-2d222ead-openshift-default-84f9ff6b48-8w8gz" podUID="92c884a6-d95f-4e4e-b911-bb848349b39b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Sep 10 22:15:12.477138 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:12.477077    2582 patch_prober.go:28] interesting pod/e2e-autores-d6a4de-openshift-default-6bf96b68c4-6269k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Sep 10 22:15:12.477472 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:12.477428    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-d6a4de-openshift-default-6bf96b68c4-6269k" podUID="78896706-4631-4e69-b205-dfad4be3b549" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Sep 10 22:15:12.694476 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:12.694440    2582 patch_prober.go:28] interesting pod/e2e-ait-2d222ead-openshift-default-84f9ff6b48-8w8gz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Sep 10 22:15:12.694745 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:12.694716    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-2d222ead-openshift-default-84f9ff6b48-8w8gz" podUID="92c884a6-d95f-4e4e-b911-bb848349b39b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Sep 10 22:15:12.770320 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:12.770234    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-1c252b-76ffcd984c-vqmgv"]
Sep 10 22:15:12.771444 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:12.771419    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cca563dc-8ed1-45c4-ab45-deec290e50c8" containerName="cleanup"
Sep 10 22:15:12.771610 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:12.771596    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="cca563dc-8ed1-45c4-ab45-deec290e50c8" containerName="cleanup"
Sep 10 22:15:12.771723 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:12.771710    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f605e12b-a160-4c45-b2b7-759154729347" containerName="cleanup"
Sep 10 22:15:12.771809 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:12.771799    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="f605e12b-a160-4c45-b2b7-759154729347" containerName="cleanup"
Sep 10 22:15:12.772046 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:12.772030    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="cca563dc-8ed1-45c4-ab45-deec290e50c8" containerName="cleanup"
Sep 10 22:15:12.772138 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:12.772128    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="f605e12b-a160-4c45-b2b7-759154729347" containerName="cleanup"
Sep 10 22:15:12.814283 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:12.814243    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1c252b-76ffcd984c-vqmgv"
Sep 10 22:15:12.819552 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:12.819517    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-spztp\""
Sep 10 22:15:12.820265 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:12.820234    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-1c252b-76ffcd984c-vqmgv"]
Sep 10 22:15:12.821204 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:12.821184    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 10 22:15:12.842220 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:12.821504    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 22:15:12.842220 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:12.821542    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-1c252b-serving-cert\""
Sep 10 22:15:12.842220 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:12.821584    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 10 22:15:12.943518 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:12.943488    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf38af6c-be32-4502-bcb7-05c47ffdce1f-maas-api-tls\") pod \"maas-api-e2e-inf-b-1c252b-76ffcd984c-vqmgv\" (UID: \"cf38af6c-be32-4502-bcb7-05c47ffdce1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1c252b-76ffcd984c-vqmgv"
Sep 10 22:15:12.943695 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:12.943527    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qz6n\" (UniqueName: \"kubernetes.io/projected/cf38af6c-be32-4502-bcb7-05c47ffdce1f-kube-api-access-9qz6n\") pod \"maas-api-e2e-inf-b-1c252b-76ffcd984c-vqmgv\" (UID: \"cf38af6c-be32-4502-bcb7-05c47ffdce1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1c252b-76ffcd984c-vqmgv"
Sep 10 22:15:12.943695 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:12.943553    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf38af6c-be32-4502-bcb7-05c47ffdce1f-openshift-service-ca\") pod \"maas-api-e2e-inf-b-1c252b-76ffcd984c-vqmgv\" (UID: \"cf38af6c-be32-4502-bcb7-05c47ffdce1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1c252b-76ffcd984c-vqmgv"
Sep 10 22:15:12.943695 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:12.943585    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf38af6c-be32-4502-bcb7-05c47ffdce1f-metrics-certs\") pod \"maas-api-e2e-inf-b-1c252b-76ffcd984c-vqmgv\" (UID: \"cf38af6c-be32-4502-bcb7-05c47ffdce1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1c252b-76ffcd984c-vqmgv"
Sep 10 22:15:13.044915 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:13.044816    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf38af6c-be32-4502-bcb7-05c47ffdce1f-maas-api-tls\") pod \"maas-api-e2e-inf-b-1c252b-76ffcd984c-vqmgv\" (UID: \"cf38af6c-be32-4502-bcb7-05c47ffdce1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1c252b-76ffcd984c-vqmgv"
Sep 10 22:15:13.045081 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:15:13.044966    2582 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-1c252b-serving-cert: secret "maas-api-e2e-inf-b-1c252b-serving-cert" not found
Sep 10 22:15:13.045081 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:13.044973    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qz6n\" (UniqueName: \"kubernetes.io/projected/cf38af6c-be32-4502-bcb7-05c47ffdce1f-kube-api-access-9qz6n\") pod \"maas-api-e2e-inf-b-1c252b-76ffcd984c-vqmgv\" (UID: \"cf38af6c-be32-4502-bcb7-05c47ffdce1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1c252b-76ffcd984c-vqmgv"
Sep 10 22:15:13.045081 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:15:13.045039    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf38af6c-be32-4502-bcb7-05c47ffdce1f-maas-api-tls podName:cf38af6c-be32-4502-bcb7-05c47ffdce1f nodeName:}" failed. No retries permitted until 2026-09-10 22:15:13.545023003 +0000 UTC m=+834.501090686 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/cf38af6c-be32-4502-bcb7-05c47ffdce1f-maas-api-tls") pod "maas-api-e2e-inf-b-1c252b-76ffcd984c-vqmgv" (UID: "cf38af6c-be32-4502-bcb7-05c47ffdce1f") : secret "maas-api-e2e-inf-b-1c252b-serving-cert" not found
Sep 10 22:15:13.045081 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:13.045057    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf38af6c-be32-4502-bcb7-05c47ffdce1f-openshift-service-ca\") pod \"maas-api-e2e-inf-b-1c252b-76ffcd984c-vqmgv\" (UID: \"cf38af6c-be32-4502-bcb7-05c47ffdce1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1c252b-76ffcd984c-vqmgv"
Sep 10 22:15:13.045269 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:13.045109    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf38af6c-be32-4502-bcb7-05c47ffdce1f-metrics-certs\") pod \"maas-api-e2e-inf-b-1c252b-76ffcd984c-vqmgv\" (UID: \"cf38af6c-be32-4502-bcb7-05c47ffdce1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1c252b-76ffcd984c-vqmgv"
Sep 10 22:15:13.045808 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:13.045787    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf38af6c-be32-4502-bcb7-05c47ffdce1f-openshift-service-ca\") pod \"maas-api-e2e-inf-b-1c252b-76ffcd984c-vqmgv\" (UID: \"cf38af6c-be32-4502-bcb7-05c47ffdce1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1c252b-76ffcd984c-vqmgv"
Sep 10 22:15:13.047562 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:13.047541    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf38af6c-be32-4502-bcb7-05c47ffdce1f-metrics-certs\") pod \"maas-api-e2e-inf-b-1c252b-76ffcd984c-vqmgv\" (UID: \"cf38af6c-be32-4502-bcb7-05c47ffdce1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1c252b-76ffcd984c-vqmgv"
Sep 10 22:15:13.061390 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:13.061364    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qz6n\" (UniqueName: \"kubernetes.io/projected/cf38af6c-be32-4502-bcb7-05c47ffdce1f-kube-api-access-9qz6n\") pod \"maas-api-e2e-inf-b-1c252b-76ffcd984c-vqmgv\" (UID: \"cf38af6c-be32-4502-bcb7-05c47ffdce1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1c252b-76ffcd984c-vqmgv"
Sep 10 22:15:13.481873 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:13.478174    2582 patch_prober.go:28] interesting pod/e2e-autores-d6a4de-openshift-default-6bf96b68c4-6269k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Sep 10 22:15:13.481873 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:13.478253    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-d6a4de-openshift-default-6bf96b68c4-6269k" podUID="78896706-4631-4e69-b205-dfad4be3b549" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Sep 10 22:15:13.549545 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:13.549501    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf38af6c-be32-4502-bcb7-05c47ffdce1f-maas-api-tls\") pod \"maas-api-e2e-inf-b-1c252b-76ffcd984c-vqmgv\" (UID: \"cf38af6c-be32-4502-bcb7-05c47ffdce1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1c252b-76ffcd984c-vqmgv"
Sep 10 22:15:13.552357 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:13.552324    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf38af6c-be32-4502-bcb7-05c47ffdce1f-maas-api-tls\") pod \"maas-api-e2e-inf-b-1c252b-76ffcd984c-vqmgv\" (UID: \"cf38af6c-be32-4502-bcb7-05c47ffdce1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1c252b-76ffcd984c-vqmgv"
Sep 10 22:15:13.697743 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:13.697265    2582 patch_prober.go:28] interesting pod/e2e-ait-2d222ead-openshift-default-84f9ff6b48-8w8gz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Sep 10 22:15:13.697743 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:13.697349    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-2d222ead-openshift-default-84f9ff6b48-8w8gz" podUID="92c884a6-d95f-4e4e-b911-bb848349b39b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Sep 10 22:15:13.745513 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:13.745427    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1c252b-76ffcd984c-vqmgv"
Sep 10 22:15:14.068493 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:14.068462    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-1c252b-76ffcd984c-vqmgv"]
Sep 10 22:15:14.070593 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:15:14.070561    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf38af6c_be32_4502_bcb7_05c47ffdce1f.slice/crio-09b643abf56796f8f3ea2e2953faa3b6a2c86af6d49626422741290cd33f8f8c WatchSource:0}: Error finding container 09b643abf56796f8f3ea2e2953faa3b6a2c86af6d49626422741290cd33f8f8c: Status 404 returned error can't find the container with id 09b643abf56796f8f3ea2e2953faa3b6a2c86af6d49626422741290cd33f8f8c
Sep 10 22:15:14.700031 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:14.699997    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-2d222ead-openshift-default-84f9ff6b48-8w8gz"
Sep 10 22:15:14.700959 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:14.700931    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-2d222ead-openshift-default-84f9ff6b48-8w8gz"
Sep 10 22:15:14.701645 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:14.701622    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-2d222ead-openshift-default-84f9ff6b48-8w8gz"
Sep 10 22:15:14.761769 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:14.759508    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-d6a4de-openshift-default-6bf96b68c4-6269k"
Sep 10 22:15:14.761769 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:14.760708    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-d6a4de-openshift-default-6bf96b68c4-6269k"
Sep 10 22:15:14.766796 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:14.764227    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-d6a4de-openshift-default-6bf96b68c4-6269k"
Sep 10 22:15:15.043013 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:15.042918    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1c252b-76ffcd984c-vqmgv" event={"ID":"cf38af6c-be32-4502-bcb7-05c47ffdce1f","Type":"ContainerStarted","Data":"09b643abf56796f8f3ea2e2953faa3b6a2c86af6d49626422741290cd33f8f8c"}
Sep 10 22:15:17.054867 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:17.054814    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1c252b-76ffcd984c-vqmgv" event={"ID":"cf38af6c-be32-4502-bcb7-05c47ffdce1f","Type":"ContainerStarted","Data":"478c18d79a85f52f1dde3dfaea199dee00494db34cb1d5aa0713f08e2042fa3d"}
Sep 10 22:15:17.055259 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:17.054914    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1c252b-76ffcd984c-vqmgv"
Sep 10 22:15:17.075968 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:17.075921    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1c252b-76ffcd984c-vqmgv" podStartSLOduration=2.8397587 podStartE2EDuration="5.075904724s" totalImagesPullingTime="2.236146024s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 22:15:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:15:17.074543215 +0000 UTC m=+838.030610922" watchObservedRunningTime="2026-09-10 22:15:17.075904724 +0000 UTC m=+838.031972430"
Sep 10 22:15:23.065836 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:23.065808    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1c252b-76ffcd984c-vqmgv"
Sep 10 22:15:32.219396 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:32.219357    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-2d222ead-29817975-kmt55"]
Sep 10 22:15:32.221490 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:32.221466    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-2d222ead-29817975-kmt55"]
Sep 10 22:15:32.256584 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:32.256557    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-2d222ead-8d6745f9-f94wb"]
Sep 10 22:15:32.256864 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:32.256800    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-2d222ead-8d6745f9-f94wb" podUID="3327fdea-2582-480f-8888-5c74d4080412" containerName="payload-processing" containerID="cri-o://97346a13f802922335170280e00598dd1a8b7164212e499b8f68b698112d854e" gracePeriod=30
Sep 10 22:15:32.497912 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:32.497843    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-2d222ead-8d6745f9-f94wb"
Sep 10 22:15:32.616773 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:32.616737    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kllfx\" (UniqueName: \"kubernetes.io/projected/3327fdea-2582-480f-8888-5c74d4080412-kube-api-access-kllfx\") pod \"3327fdea-2582-480f-8888-5c74d4080412\" (UID: \"3327fdea-2582-480f-8888-5c74d4080412\") "
Sep 10 22:15:32.616980 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:32.616840    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3327fdea-2582-480f-8888-5c74d4080412-plugins-config-volume\") pod \"3327fdea-2582-480f-8888-5c74d4080412\" (UID: \"3327fdea-2582-480f-8888-5c74d4080412\") "
Sep 10 22:15:32.617190 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:32.617159    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3327fdea-2582-480f-8888-5c74d4080412-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3327fdea-2582-480f-8888-5c74d4080412" (UID: "3327fdea-2582-480f-8888-5c74d4080412"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 22:15:32.619237 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:32.619211    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3327fdea-2582-480f-8888-5c74d4080412-kube-api-access-kllfx" (OuterVolumeSpecName: "kube-api-access-kllfx") pod "3327fdea-2582-480f-8888-5c74d4080412" (UID: "3327fdea-2582-480f-8888-5c74d4080412"). InnerVolumeSpecName "kube-api-access-kllfx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 22:15:32.661561 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:32.661537    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-2d222ead-openshift-default-84f9ff6b48-8w8gz"]
Sep 10 22:15:32.661803 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:32.661753    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-2d222ead-openshift-default-84f9ff6b48-8w8gz" podUID="92c884a6-d95f-4e4e-b911-bb848349b39b" containerName="istio-proxy" containerID="cri-o://8e0f7c14694ee7b11cbc20d141c3c380e379a574b6163bcb0461270ad2b3f834" gracePeriod=30
Sep 10 22:15:32.717983 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:32.717958    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3327fdea-2582-480f-8888-5c74d4080412-plugins-config-volume\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:15:32.717983 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:32.717980    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kllfx\" (UniqueName: \"kubernetes.io/projected/3327fdea-2582-480f-8888-5c74d4080412-kube-api-access-kllfx\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:15:33.132091 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:33.132050    2582 generic.go:356] "Generic (PLEG): container finished" podID="3327fdea-2582-480f-8888-5c74d4080412" containerID="97346a13f802922335170280e00598dd1a8b7164212e499b8f68b698112d854e" exitCode=0
Sep 10 22:15:33.132281 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:33.132099    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-2d222ead-8d6745f9-f94wb"
Sep 10 22:15:33.132281 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:33.132142    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-2d222ead-8d6745f9-f94wb" event={"ID":"3327fdea-2582-480f-8888-5c74d4080412","Type":"ContainerDied","Data":"97346a13f802922335170280e00598dd1a8b7164212e499b8f68b698112d854e"}
Sep 10 22:15:33.132281 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:33.132207    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-2d222ead-8d6745f9-f94wb" event={"ID":"3327fdea-2582-480f-8888-5c74d4080412","Type":"ContainerDied","Data":"02f36dfcf72a2d534f22f8110b029f32826ccca5c4020d754e75c197ad11c672"}
Sep 10 22:15:33.132281 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:33.132236    2582 scope.go:117] "RemoveContainer" containerID="97346a13f802922335170280e00598dd1a8b7164212e499b8f68b698112d854e"
Sep 10 22:15:33.142259 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:33.142230    2582 scope.go:117] "RemoveContainer" containerID="97346a13f802922335170280e00598dd1a8b7164212e499b8f68b698112d854e"
Sep 10 22:15:33.142560 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:15:33.142537    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97346a13f802922335170280e00598dd1a8b7164212e499b8f68b698112d854e\": container with ID starting with 97346a13f802922335170280e00598dd1a8b7164212e499b8f68b698112d854e not found: ID does not exist" containerID="97346a13f802922335170280e00598dd1a8b7164212e499b8f68b698112d854e"
Sep 10 22:15:33.142637 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:33.142570    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97346a13f802922335170280e00598dd1a8b7164212e499b8f68b698112d854e"} err="failed to get container status \"97346a13f802922335170280e00598dd1a8b7164212e499b8f68b698112d854e\": rpc error: code = NotFound desc = could not find container \"97346a13f802922335170280e00598dd1a8b7164212e499b8f68b698112d854e\": container with ID starting with 97346a13f802922335170280e00598dd1a8b7164212e499b8f68b698112d854e not found: ID does not exist"
Sep 10 22:15:33.154786 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:33.154764    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-2d222ead-8d6745f9-f94wb"]
Sep 10 22:15:33.162222 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:33.162198    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-2d222ead-8d6745f9-f94wb"]
Sep 10 22:15:33.497860 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:33.497829    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3327fdea-2582-480f-8888-5c74d4080412" path="/var/lib/kubelet/pods/3327fdea-2582-480f-8888-5c74d4080412/volumes"
Sep 10 22:15:33.498204 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:33.498124    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cca563dc-8ed1-45c4-ab45-deec290e50c8" path="/var/lib/kubelet/pods/cca563dc-8ed1-45c4-ab45-deec290e50c8/volumes"
Sep 10 22:15:37.906430 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:37.906401    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-2d222ead-openshift-default-84f9ff6b48-8w8gz"
Sep 10 22:15:38.062763 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:38.062689    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92c884a6-d95f-4e4e-b911-bb848349b39b-istio-envoy\") pod \"92c884a6-d95f-4e4e-b911-bb848349b39b\" (UID: \"92c884a6-d95f-4e4e-b911-bb848349b39b\") "
Sep 10 22:15:38.062763 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:38.062735    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92c884a6-d95f-4e4e-b911-bb848349b39b-istio-podinfo\") pod \"92c884a6-d95f-4e4e-b911-bb848349b39b\" (UID: \"92c884a6-d95f-4e4e-b911-bb848349b39b\") "
Sep 10 22:15:38.063010 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:38.062764    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92c884a6-d95f-4e4e-b911-bb848349b39b-workload-socket\") pod \"92c884a6-d95f-4e4e-b911-bb848349b39b\" (UID: \"92c884a6-d95f-4e4e-b911-bb848349b39b\") "
Sep 10 22:15:38.063010 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:38.062807    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92c884a6-d95f-4e4e-b911-bb848349b39b-istio-data\") pod \"92c884a6-d95f-4e4e-b911-bb848349b39b\" (UID: \"92c884a6-d95f-4e4e-b911-bb848349b39b\") "
Sep 10 22:15:38.063010 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:38.062832    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92c884a6-d95f-4e4e-b911-bb848349b39b-istio-token\") pod \"92c884a6-d95f-4e4e-b911-bb848349b39b\" (UID: \"92c884a6-d95f-4e4e-b911-bb848349b39b\") "
Sep 10 22:15:38.063010 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:38.062887    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92c884a6-d95f-4e4e-b911-bb848349b39b-credential-socket\") pod \"92c884a6-d95f-4e4e-b911-bb848349b39b\" (UID: \"92c884a6-d95f-4e4e-b911-bb848349b39b\") "
Sep 10 22:15:38.063010 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:38.062941    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92c884a6-d95f-4e4e-b911-bb848349b39b-istiod-ca-cert\") pod \"92c884a6-d95f-4e4e-b911-bb848349b39b\" (UID: \"92c884a6-d95f-4e4e-b911-bb848349b39b\") "
Sep 10 22:15:38.063010 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:38.062989    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-chfhd\" (UniqueName: \"kubernetes.io/projected/92c884a6-d95f-4e4e-b911-bb848349b39b-kube-api-access-chfhd\") pod \"92c884a6-d95f-4e4e-b911-bb848349b39b\" (UID: \"92c884a6-d95f-4e4e-b911-bb848349b39b\") "
Sep 10 22:15:38.063302 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:38.063018    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92c884a6-d95f-4e4e-b911-bb848349b39b-workload-certs\") pod \"92c884a6-d95f-4e4e-b911-bb848349b39b\" (UID: \"92c884a6-d95f-4e4e-b911-bb848349b39b\") "
Sep 10 22:15:38.063302 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:38.063115    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92c884a6-d95f-4e4e-b911-bb848349b39b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "92c884a6-d95f-4e4e-b911-bb848349b39b" (UID: "92c884a6-d95f-4e4e-b911-bb848349b39b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 22:15:38.063302 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:38.063127    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92c884a6-d95f-4e4e-b911-bb848349b39b-istio-data" (OuterVolumeSpecName: "istio-data") pod "92c884a6-d95f-4e4e-b911-bb848349b39b" (UID: "92c884a6-d95f-4e4e-b911-bb848349b39b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 22:15:38.063444 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:38.063322    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92c884a6-d95f-4e4e-b911-bb848349b39b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "92c884a6-d95f-4e4e-b911-bb848349b39b" (UID: "92c884a6-d95f-4e4e-b911-bb848349b39b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 22:15:38.063444 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:38.063364    2582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92c884a6-d95f-4e4e-b911-bb848349b39b-workload-socket\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:15:38.063444 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:38.063386    2582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92c884a6-d95f-4e4e-b911-bb848349b39b-istio-data\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:15:38.063561 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:38.063483    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92c884a6-d95f-4e4e-b911-bb848349b39b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "92c884a6-d95f-4e4e-b911-bb848349b39b" (UID: "92c884a6-d95f-4e4e-b911-bb848349b39b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 22:15:38.063605 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:38.063579    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92c884a6-d95f-4e4e-b911-bb848349b39b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "92c884a6-d95f-4e4e-b911-bb848349b39b" (UID: "92c884a6-d95f-4e4e-b911-bb848349b39b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 22:15:38.064921 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:38.064894    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/92c884a6-d95f-4e4e-b911-bb848349b39b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "92c884a6-d95f-4e4e-b911-bb848349b39b" (UID: "92c884a6-d95f-4e4e-b911-bb848349b39b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 22:15:38.065383 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:38.065353    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92c884a6-d95f-4e4e-b911-bb848349b39b-kube-api-access-chfhd" (OuterVolumeSpecName: "kube-api-access-chfhd") pod "92c884a6-d95f-4e4e-b911-bb848349b39b" (UID: "92c884a6-d95f-4e4e-b911-bb848349b39b"). InnerVolumeSpecName "kube-api-access-chfhd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 22:15:38.065480 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:38.065395    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92c884a6-d95f-4e4e-b911-bb848349b39b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "92c884a6-d95f-4e4e-b911-bb848349b39b" (UID: "92c884a6-d95f-4e4e-b911-bb848349b39b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 22:15:38.065570 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:38.065551    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92c884a6-d95f-4e4e-b911-bb848349b39b-istio-token" (OuterVolumeSpecName: "istio-token") pod "92c884a6-d95f-4e4e-b911-bb848349b39b" (UID: "92c884a6-d95f-4e4e-b911-bb848349b39b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 22:15:38.138965 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:38.138940    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-cd1c6d7b-6bd69547fd-xwdx8"]
Sep 10 22:15:38.139243 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:38.139232    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92c884a6-d95f-4e4e-b911-bb848349b39b" containerName="istio-proxy"
Sep 10 22:15:38.139286 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:38.139244    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="92c884a6-d95f-4e4e-b911-bb848349b39b" containerName="istio-proxy"
Sep 10 22:15:38.139286 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:38.139266    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3327fdea-2582-480f-8888-5c74d4080412" containerName="payload-processing"
Sep 10 22:15:38.139286 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:38.139271    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="3327fdea-2582-480f-8888-5c74d4080412" containerName="payload-processing"
Sep 10 22:15:38.139389 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:38.139339    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="92c884a6-d95f-4e4e-b911-bb848349b39b" containerName="istio-proxy"
Sep 10 22:15:38.139389 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:38.139348    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="3327fdea-2582-480f-8888-5c74d4080412" containerName="payload-processing"
Sep 10 22:15:38.139389 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:38.139354    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="92c884a6-d95f-4e4e-b911-bb848349b39b" containerName="istio-proxy"
Sep 10 22:15:38.141671 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:38.141656    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-cd1c6d7b-6bd69547fd-xwdx8"
Sep 10 22:15:38.143765 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:38.143747    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-cd1c6d7b\""
Sep 10 22:15:38.143873 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:38.143748    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-cd1c6d7b-dockercfg-cr8gs\""
Sep 10 22:15:38.157511 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:38.157491    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-cd1c6d7b-6bd69547fd-xwdx8"]
Sep 10 22:15:38.159921 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:38.159897    2582 generic.go:356] "Generic (PLEG): container finished" podID="92c884a6-d95f-4e4e-b911-bb848349b39b" containerID="8e0f7c14694ee7b11cbc20d141c3c380e379a574b6163bcb0461270ad2b3f834" exitCode=0
Sep 10 22:15:38.160038 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:38.159949    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-2d222ead-openshift-default-84f9ff6b48-8w8gz"
Sep 10 22:15:38.160038 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:38.159981    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2d222ead-openshift-default-84f9ff6b48-8w8gz" event={"ID":"92c884a6-d95f-4e4e-b911-bb848349b39b","Type":"ContainerDied","Data":"8e0f7c14694ee7b11cbc20d141c3c380e379a574b6163bcb0461270ad2b3f834"}
Sep 10 22:15:38.160038 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:38.160023    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2d222ead-openshift-default-84f9ff6b48-8w8gz" event={"ID":"92c884a6-d95f-4e4e-b911-bb848349b39b","Type":"ContainerDied","Data":"e7872523987bec0ec62cfc606c12378c6554bd84c5a6a645d6d7f376dc80871b"}
Sep 10 22:15:38.160156 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:38.160042    2582 scope.go:117] "RemoveContainer" containerID="8e0f7c14694ee7b11cbc20d141c3c380e379a574b6163bcb0461270ad2b3f834"
Sep 10 22:15:38.164206 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:38.164181    2582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92c884a6-d95f-4e4e-b911-bb848349b39b-istiod-ca-cert\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:15:38.164325 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:38.164213    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chfhd\" (UniqueName: \"kubernetes.io/projected/92c884a6-d95f-4e4e-b911-bb848349b39b-kube-api-access-chfhd\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:15:38.164325 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:38.164228    2582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92c884a6-d95f-4e4e-b911-bb848349b39b-workload-certs\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:15:38.164325 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:38.164243    2582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92c884a6-d95f-4e4e-b911-bb848349b39b-istio-envoy\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:15:38.164325 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:38.164255    2582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92c884a6-d95f-4e4e-b911-bb848349b39b-istio-podinfo\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:15:38.164325 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:38.164269    2582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92c884a6-d95f-4e4e-b911-bb848349b39b-istio-token\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:15:38.164325 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:38.164282    2582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92c884a6-d95f-4e4e-b911-bb848349b39b-credential-socket\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:15:38.170329 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:38.170315    2582 scope.go:117] "RemoveContainer" containerID="cbeeef6caed30a638f7c6b25ef75ac084df2f60eb7eb27fbf91db8a9f6a7b979"
Sep 10 22:15:38.178160 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:38.178141    2582 scope.go:117] "RemoveContainer" containerID="8e0f7c14694ee7b11cbc20d141c3c380e379a574b6163bcb0461270ad2b3f834"
Sep 10 22:15:38.178424 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:15:38.178405    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e0f7c14694ee7b11cbc20d141c3c380e379a574b6163bcb0461270ad2b3f834\": container with ID starting with 8e0f7c14694ee7b11cbc20d141c3c380e379a574b6163bcb0461270ad2b3f834 not found: ID does not exist" containerID="8e0f7c14694ee7b11cbc20d141c3c380e379a574b6163bcb0461270ad2b3f834"
Sep 10 22:15:38.178494 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:38.178436    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e0f7c14694ee7b11cbc20d141c3c380e379a574b6163bcb0461270ad2b3f834"} err="failed to get container status \"8e0f7c14694ee7b11cbc20d141c3c380e379a574b6163bcb0461270ad2b3f834\": rpc error: code = NotFound desc = could not find container \"8e0f7c14694ee7b11cbc20d141c3c380e379a574b6163bcb0461270ad2b3f834\": container with ID starting with 8e0f7c14694ee7b11cbc20d141c3c380e379a574b6163bcb0461270ad2b3f834 not found: ID does not exist"
Sep 10 22:15:38.178494 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:38.178461    2582 scope.go:117] "RemoveContainer" containerID="cbeeef6caed30a638f7c6b25ef75ac084df2f60eb7eb27fbf91db8a9f6a7b979"
Sep 10 22:15:38.178782 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:15:38.178765    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbeeef6caed30a638f7c6b25ef75ac084df2f60eb7eb27fbf91db8a9f6a7b979\": container with ID starting with cbeeef6caed30a638f7c6b25ef75ac084df2f60eb7eb27fbf91db8a9f6a7b979 not found: ID does not exist" containerID="cbeeef6caed30a638f7c6b25ef75ac084df2f60eb7eb27fbf91db8a9f6a7b979"
Sep 10 22:15:38.178834 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:38.178787    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbeeef6caed30a638f7c6b25ef75ac084df2f60eb7eb27fbf91db8a9f6a7b979"} err="failed to get container status \"cbeeef6caed30a638f7c6b25ef75ac084df2f60eb7eb27fbf91db8a9f6a7b979\": rpc error: code = NotFound desc = could not find container \"cbeeef6caed30a638f7c6b25ef75ac084df2f60eb7eb27fbf91db8a9f6a7b979\": container with ID starting with cbeeef6caed30a638f7c6b25ef75ac084df2f60eb7eb27fbf91db8a9f6a7b979 not found: ID does not exist"
Sep 10 22:15:38.194054 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:38.194030    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-cd1c6d7b-dbf676dd6-wjzbb"]
Sep 10 22:15:38.194437 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:38.194419    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92c884a6-d95f-4e4e-b911-bb848349b39b" containerName="istio-proxy"
Sep 10 22:15:38.194525 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:38.194440    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="92c884a6-d95f-4e4e-b911-bb848349b39b" containerName="istio-proxy"
Sep 10 22:15:38.197911 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:38.197887    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-2d222ead-openshift-default-84f9ff6b48-8w8gz"]
Sep 10 22:15:38.197999 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:38.197961    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-cd1c6d7b-dbf676dd6-wjzbb"
Sep 10 22:15:38.208461 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:38.208434    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-2d222ead-openshift-default-84f9ff6b48-8w8gz"]
Sep 10 22:15:38.212090 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:38.212068    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-cd1c6d7b-dbf676dd6-wjzbb"]
Sep 10 22:15:38.265361 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:38.265330    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9htxt\" (UniqueName: \"kubernetes.io/projected/84c9fe1c-3314-4408-b5f6-895830679145-kube-api-access-9htxt\") pod \"payload-pre-processing-e2e-migrate-cd1c6d7b-6bd69547fd-xwdx8\" (UID: \"84c9fe1c-3314-4408-b5f6-895830679145\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-cd1c6d7b-6bd69547fd-xwdx8"
Sep 10 22:15:38.265489 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:38.265463    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84c9fe1c-3314-4408-b5f6-895830679145-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-cd1c6d7b-6bd69547fd-xwdx8\" (UID: \"84c9fe1c-3314-4408-b5f6-895830679145\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-cd1c6d7b-6bd69547fd-xwdx8"
Sep 10 22:15:38.366776 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:38.366714    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8d2x\" (UniqueName: \"kubernetes.io/projected/54c83d43-1e6e-4865-bf97-2a96baa4a717-kube-api-access-k8d2x\") pod \"payload-processing-e2e-migrate-cd1c6d7b-dbf676dd6-wjzbb\" (UID: \"54c83d43-1e6e-4865-bf97-2a96baa4a717\") " pod="openshift-ingress/payload-processing-e2e-migrate-cd1c6d7b-dbf676dd6-wjzbb"
Sep 10 22:15:38.366776 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:38.366759    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54c83d43-1e6e-4865-bf97-2a96baa4a717-plugins-config-volume\") pod \"payload-processing-e2e-migrate-cd1c6d7b-dbf676dd6-wjzbb\" (UID: \"54c83d43-1e6e-4865-bf97-2a96baa4a717\") " pod="openshift-ingress/payload-processing-e2e-migrate-cd1c6d7b-dbf676dd6-wjzbb"
Sep 10 22:15:38.366970 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:38.366789    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84c9fe1c-3314-4408-b5f6-895830679145-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-cd1c6d7b-6bd69547fd-xwdx8\" (UID: \"84c9fe1c-3314-4408-b5f6-895830679145\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-cd1c6d7b-6bd69547fd-xwdx8"
Sep 10 22:15:38.366970 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:38.366825    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9htxt\" (UniqueName: \"kubernetes.io/projected/84c9fe1c-3314-4408-b5f6-895830679145-kube-api-access-9htxt\") pod \"payload-pre-processing-e2e-migrate-cd1c6d7b-6bd69547fd-xwdx8\" (UID: \"84c9fe1c-3314-4408-b5f6-895830679145\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-cd1c6d7b-6bd69547fd-xwdx8"
Sep 10 22:15:38.367516 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:38.367492    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84c9fe1c-3314-4408-b5f6-895830679145-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-cd1c6d7b-6bd69547fd-xwdx8\" (UID: \"84c9fe1c-3314-4408-b5f6-895830679145\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-cd1c6d7b-6bd69547fd-xwdx8"
Sep 10 22:15:38.375550 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:38.375528    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9htxt\" (UniqueName: \"kubernetes.io/projected/84c9fe1c-3314-4408-b5f6-895830679145-kube-api-access-9htxt\") pod \"payload-pre-processing-e2e-migrate-cd1c6d7b-6bd69547fd-xwdx8\" (UID: \"84c9fe1c-3314-4408-b5f6-895830679145\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-cd1c6d7b-6bd69547fd-xwdx8"
Sep 10 22:15:38.451644 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:38.451618    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-cd1c6d7b-6bd69547fd-xwdx8"
Sep 10 22:15:38.468009 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:38.467986    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8d2x\" (UniqueName: \"kubernetes.io/projected/54c83d43-1e6e-4865-bf97-2a96baa4a717-kube-api-access-k8d2x\") pod \"payload-processing-e2e-migrate-cd1c6d7b-dbf676dd6-wjzbb\" (UID: \"54c83d43-1e6e-4865-bf97-2a96baa4a717\") " pod="openshift-ingress/payload-processing-e2e-migrate-cd1c6d7b-dbf676dd6-wjzbb"
Sep 10 22:15:38.468091 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:38.468028    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54c83d43-1e6e-4865-bf97-2a96baa4a717-plugins-config-volume\") pod \"payload-processing-e2e-migrate-cd1c6d7b-dbf676dd6-wjzbb\" (UID: \"54c83d43-1e6e-4865-bf97-2a96baa4a717\") " pod="openshift-ingress/payload-processing-e2e-migrate-cd1c6d7b-dbf676dd6-wjzbb"
Sep 10 22:15:38.468618 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:38.468601    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54c83d43-1e6e-4865-bf97-2a96baa4a717-plugins-config-volume\") pod \"payload-processing-e2e-migrate-cd1c6d7b-dbf676dd6-wjzbb\" (UID: \"54c83d43-1e6e-4865-bf97-2a96baa4a717\") " pod="openshift-ingress/payload-processing-e2e-migrate-cd1c6d7b-dbf676dd6-wjzbb"
Sep 10 22:15:38.480649 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:38.480627    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8d2x\" (UniqueName: \"kubernetes.io/projected/54c83d43-1e6e-4865-bf97-2a96baa4a717-kube-api-access-k8d2x\") pod \"payload-processing-e2e-migrate-cd1c6d7b-dbf676dd6-wjzbb\" (UID: \"54c83d43-1e6e-4865-bf97-2a96baa4a717\") " pod="openshift-ingress/payload-processing-e2e-migrate-cd1c6d7b-dbf676dd6-wjzbb"
Sep 10 22:15:38.511245 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:38.511222    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-cd1c6d7b-dbf676dd6-wjzbb"
Sep 10 22:15:38.725098 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:38.725075    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-cd1c6d7b-6bd69547fd-xwdx8"]
Sep 10 22:15:38.727259 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:15:38.727236    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84c9fe1c_3314_4408_b5f6_895830679145.slice/crio-9ee7239ffc4a9903cc8d4d03d1769b41ae8947394bfa3792d08ce40d41681424 WatchSource:0}: Error finding container 9ee7239ffc4a9903cc8d4d03d1769b41ae8947394bfa3792d08ce40d41681424: Status 404 returned error can't find the container with id 9ee7239ffc4a9903cc8d4d03d1769b41ae8947394bfa3792d08ce40d41681424
Sep 10 22:15:38.799118 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:38.799076    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-cd1c6d7b-dbf676dd6-wjzbb"]
Sep 10 22:15:38.802189 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:15:38.802162    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54c83d43_1e6e_4865_bf97_2a96baa4a717.slice/crio-208d798d131eedff15a3070d9fd3bb8d28f72a3e907b26c61c1eae55823e71ed WatchSource:0}: Error finding container 208d798d131eedff15a3070d9fd3bb8d28f72a3e907b26c61c1eae55823e71ed: Status 404 returned error can't find the container with id 208d798d131eedff15a3070d9fd3bb8d28f72a3e907b26c61c1eae55823e71ed
Sep 10 22:15:39.167789 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:39.167722    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-cd1c6d7b-dbf676dd6-wjzbb" event={"ID":"54c83d43-1e6e-4865-bf97-2a96baa4a717","Type":"ContainerStarted","Data":"bb533896d6007ab7e7f9b4fa32834174bed06c219e4ba720f9ded5788d1da2ba"}
Sep 10 22:15:39.168286 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:39.167887    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-cd1c6d7b-dbf676dd6-wjzbb"
Sep 10 22:15:39.168286 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:39.167931    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-cd1c6d7b-dbf676dd6-wjzbb" event={"ID":"54c83d43-1e6e-4865-bf97-2a96baa4a717","Type":"ContainerStarted","Data":"208d798d131eedff15a3070d9fd3bb8d28f72a3e907b26c61c1eae55823e71ed"}
Sep 10 22:15:39.169796 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:39.169775    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-cd1c6d7b-6bd69547fd-xwdx8" event={"ID":"84c9fe1c-3314-4408-b5f6-895830679145","Type":"ContainerStarted","Data":"ec9d74434bb75ddc1b12761d204a611fab0b0510607c87538bcd425615b16c47"}
Sep 10 22:15:39.169965 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:39.169812    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-cd1c6d7b-6bd69547fd-xwdx8" event={"ID":"84c9fe1c-3314-4408-b5f6-895830679145","Type":"ContainerStarted","Data":"9ee7239ffc4a9903cc8d4d03d1769b41ae8947394bfa3792d08ce40d41681424"}
Sep 10 22:15:39.169965 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:39.169893    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-cd1c6d7b-6bd69547fd-xwdx8"
Sep 10 22:15:39.193553 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:39.193506    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-cd1c6d7b-dbf676dd6-wjzbb" podStartSLOduration=1.193469362 podStartE2EDuration="1.193469362s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 22:15:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:15:39.193130829 +0000 UTC m=+860.149198535" watchObservedRunningTime="2026-09-10 22:15:39.193469362 +0000 UTC m=+860.149537072"
Sep 10 22:15:39.233712 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:39.233665    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-cd1c6d7b-6bd69547fd-xwdx8" podStartSLOduration=1.233641877 podStartE2EDuration="1.233641877s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 22:15:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:15:39.223400904 +0000 UTC m=+860.179468609" watchObservedRunningTime="2026-09-10 22:15:39.233641877 +0000 UTC m=+860.189709582"
Sep 10 22:15:39.502333 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:39.502256    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="92c884a6-d95f-4e4e-b911-bb848349b39b" path="/var/lib/kubelet/pods/92c884a6-d95f-4e4e-b911-bb848349b39b/volumes"
Sep 10 22:15:50.176385 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:50.176347    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-cd1c6d7b-dbf676dd6-wjzbb"
Sep 10 22:15:50.176768 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:15:50.176445    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-cd1c6d7b-6bd69547fd-xwdx8"
Sep 10 22:16:01.807627 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:01.807591    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cd1c6d7b-kms8b"]
Sep 10 22:16:01.811887 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:01.811868    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cd1c6d7b-kms8b"
Sep 10 22:16:01.820802 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:01.820783    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8hjrv\""
Sep 10 22:16:01.828802 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:01.828775    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cd1c6d7b-kms8b"]
Sep 10 22:16:01.848000 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:01.847967    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/295a9574-4729-41e4-adbb-fee2cfc66a4f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-cd1c6d7b-kms8b\" (UID: \"295a9574-4729-41e4-adbb-fee2cfc66a4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cd1c6d7b-kms8b"
Sep 10 22:16:01.948555 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:01.948526    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/295a9574-4729-41e4-adbb-fee2cfc66a4f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-cd1c6d7b-kms8b\" (UID: \"295a9574-4729-41e4-adbb-fee2cfc66a4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cd1c6d7b-kms8b"
Sep 10 22:16:01.949147 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:01.949125    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/295a9574-4729-41e4-adbb-fee2cfc66a4f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-cd1c6d7b-kms8b\" (UID: \"295a9574-4729-41e4-adbb-fee2cfc66a4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cd1c6d7b-kms8b"
Sep 10 22:16:02.121530 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:02.121476    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cd1c6d7b-kms8b"
Sep 10 22:16:02.430392 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:02.430368    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cd1c6d7b-kms8b"]
Sep 10 22:16:02.432604 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:16:02.432570    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod295a9574_4729_41e4_adbb_fee2cfc66a4f.slice/crio-b02d62ce1af8e43993c1f1579fbddd3ea127e839c6cf24230cc8caa7a80e12bb WatchSource:0}: Error finding container b02d62ce1af8e43993c1f1579fbddd3ea127e839c6cf24230cc8caa7a80e12bb: Status 404 returned error can't find the container with id b02d62ce1af8e43993c1f1579fbddd3ea127e839c6cf24230cc8caa7a80e12bb
Sep 10 22:16:03.286304 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:03.286267    2582 generic.go:356] "Generic (PLEG): container finished" podID="295a9574-4729-41e4-adbb-fee2cfc66a4f" containerID="004ce21bd2aa5ccd1e8f029f17d8d66c43bae2265325ede95af68b1b07d0a1e9" exitCode=0
Sep 10 22:16:03.286670 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:03.286354    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cd1c6d7b-kms8b" event={"ID":"295a9574-4729-41e4-adbb-fee2cfc66a4f","Type":"ContainerDied","Data":"004ce21bd2aa5ccd1e8f029f17d8d66c43bae2265325ede95af68b1b07d0a1e9"}
Sep 10 22:16:03.286670 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:03.286462    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cd1c6d7b-kms8b" event={"ID":"295a9574-4729-41e4-adbb-fee2cfc66a4f","Type":"ContainerStarted","Data":"b02d62ce1af8e43993c1f1579fbddd3ea127e839c6cf24230cc8caa7a80e12bb"}
Sep 10 22:16:03.988241 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:03.988207    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-conflict-a-e5a626b8-kserve-7688fbd-k5z56"]
Sep 10 22:16:03.990491 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:03.990470    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-e5a626b8-kserve-7688fbd-k5z56"
Sep 10 22:16:03.995756 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:03.995732    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-conflict-a-e5a626b8-kserve-self-signed-certs\""
Sep 10 22:16:04.037879 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:04.037838    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-a-e5a626b8-kserve-7688fbd-k5z56"]
Sep 10 22:16:04.064251 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:04.064225    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c423d2e-4cdb-496f-aeb7-3806b801970d-tmp-dir\") pod \"e2e-conflict-a-e5a626b8-kserve-7688fbd-k5z56\" (UID: \"2c423d2e-4cdb-496f-aeb7-3806b801970d\") " pod="llm/e2e-conflict-a-e5a626b8-kserve-7688fbd-k5z56"
Sep 10 22:16:04.064349 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:04.064262    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c423d2e-4cdb-496f-aeb7-3806b801970d-tls-certs\") pod \"e2e-conflict-a-e5a626b8-kserve-7688fbd-k5z56\" (UID: \"2c423d2e-4cdb-496f-aeb7-3806b801970d\") " pod="llm/e2e-conflict-a-e5a626b8-kserve-7688fbd-k5z56"
Sep 10 22:16:04.064349 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:04.064280    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c423d2e-4cdb-496f-aeb7-3806b801970d-home\") pod \"e2e-conflict-a-e5a626b8-kserve-7688fbd-k5z56\" (UID: \"2c423d2e-4cdb-496f-aeb7-3806b801970d\") " pod="llm/e2e-conflict-a-e5a626b8-kserve-7688fbd-k5z56"
Sep 10 22:16:04.064349 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:04.064313    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c423d2e-4cdb-496f-aeb7-3806b801970d-model-cache\") pod \"e2e-conflict-a-e5a626b8-kserve-7688fbd-k5z56\" (UID: \"2c423d2e-4cdb-496f-aeb7-3806b801970d\") " pod="llm/e2e-conflict-a-e5a626b8-kserve-7688fbd-k5z56"
Sep 10 22:16:04.064453 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:04.064348    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csppc\" (UniqueName: \"kubernetes.io/projected/2c423d2e-4cdb-496f-aeb7-3806b801970d-kube-api-access-csppc\") pod \"e2e-conflict-a-e5a626b8-kserve-7688fbd-k5z56\" (UID: \"2c423d2e-4cdb-496f-aeb7-3806b801970d\") " pod="llm/e2e-conflict-a-e5a626b8-kserve-7688fbd-k5z56"
Sep 10 22:16:04.064453 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:04.064406    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c423d2e-4cdb-496f-aeb7-3806b801970d-dshm\") pod \"e2e-conflict-a-e5a626b8-kserve-7688fbd-k5z56\" (UID: \"2c423d2e-4cdb-496f-aeb7-3806b801970d\") " pod="llm/e2e-conflict-a-e5a626b8-kserve-7688fbd-k5z56"
Sep 10 22:16:04.165076 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:04.165055    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c423d2e-4cdb-496f-aeb7-3806b801970d-tmp-dir\") pod \"e2e-conflict-a-e5a626b8-kserve-7688fbd-k5z56\" (UID: \"2c423d2e-4cdb-496f-aeb7-3806b801970d\") " pod="llm/e2e-conflict-a-e5a626b8-kserve-7688fbd-k5z56"
Sep 10 22:16:04.165200 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:04.165092    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c423d2e-4cdb-496f-aeb7-3806b801970d-tls-certs\") pod \"e2e-conflict-a-e5a626b8-kserve-7688fbd-k5z56\" (UID: \"2c423d2e-4cdb-496f-aeb7-3806b801970d\") " pod="llm/e2e-conflict-a-e5a626b8-kserve-7688fbd-k5z56"
Sep 10 22:16:04.165200 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:04.165109    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c423d2e-4cdb-496f-aeb7-3806b801970d-home\") pod \"e2e-conflict-a-e5a626b8-kserve-7688fbd-k5z56\" (UID: \"2c423d2e-4cdb-496f-aeb7-3806b801970d\") " pod="llm/e2e-conflict-a-e5a626b8-kserve-7688fbd-k5z56"
Sep 10 22:16:04.165200 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:04.165124    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c423d2e-4cdb-496f-aeb7-3806b801970d-model-cache\") pod \"e2e-conflict-a-e5a626b8-kserve-7688fbd-k5z56\" (UID: \"2c423d2e-4cdb-496f-aeb7-3806b801970d\") " pod="llm/e2e-conflict-a-e5a626b8-kserve-7688fbd-k5z56"
Sep 10 22:16:04.165200 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:04.165153    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csppc\" (UniqueName: \"kubernetes.io/projected/2c423d2e-4cdb-496f-aeb7-3806b801970d-kube-api-access-csppc\") pod \"e2e-conflict-a-e5a626b8-kserve-7688fbd-k5z56\" (UID: \"2c423d2e-4cdb-496f-aeb7-3806b801970d\") " pod="llm/e2e-conflict-a-e5a626b8-kserve-7688fbd-k5z56"
Sep 10 22:16:04.165404 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:04.165205    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c423d2e-4cdb-496f-aeb7-3806b801970d-dshm\") pod \"e2e-conflict-a-e5a626b8-kserve-7688fbd-k5z56\" (UID: \"2c423d2e-4cdb-496f-aeb7-3806b801970d\") " pod="llm/e2e-conflict-a-e5a626b8-kserve-7688fbd-k5z56"
Sep 10 22:16:04.165520 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:04.165498    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c423d2e-4cdb-496f-aeb7-3806b801970d-home\") pod \"e2e-conflict-a-e5a626b8-kserve-7688fbd-k5z56\" (UID: \"2c423d2e-4cdb-496f-aeb7-3806b801970d\") " pod="llm/e2e-conflict-a-e5a626b8-kserve-7688fbd-k5z56"
Sep 10 22:16:04.165577 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:04.165507    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c423d2e-4cdb-496f-aeb7-3806b801970d-tmp-dir\") pod \"e2e-conflict-a-e5a626b8-kserve-7688fbd-k5z56\" (UID: \"2c423d2e-4cdb-496f-aeb7-3806b801970d\") " pod="llm/e2e-conflict-a-e5a626b8-kserve-7688fbd-k5z56"
Sep 10 22:16:04.165627 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:04.165553    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c423d2e-4cdb-496f-aeb7-3806b801970d-model-cache\") pod \"e2e-conflict-a-e5a626b8-kserve-7688fbd-k5z56\" (UID: \"2c423d2e-4cdb-496f-aeb7-3806b801970d\") " pod="llm/e2e-conflict-a-e5a626b8-kserve-7688fbd-k5z56"
Sep 10 22:16:04.167442 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:04.167424    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c423d2e-4cdb-496f-aeb7-3806b801970d-dshm\") pod \"e2e-conflict-a-e5a626b8-kserve-7688fbd-k5z56\" (UID: \"2c423d2e-4cdb-496f-aeb7-3806b801970d\") " pod="llm/e2e-conflict-a-e5a626b8-kserve-7688fbd-k5z56"
Sep 10 22:16:04.167622 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:04.167607    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c423d2e-4cdb-496f-aeb7-3806b801970d-tls-certs\") pod \"e2e-conflict-a-e5a626b8-kserve-7688fbd-k5z56\" (UID: \"2c423d2e-4cdb-496f-aeb7-3806b801970d\") " pod="llm/e2e-conflict-a-e5a626b8-kserve-7688fbd-k5z56"
Sep 10 22:16:04.176078 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:04.176056    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-csppc\" (UniqueName: \"kubernetes.io/projected/2c423d2e-4cdb-496f-aeb7-3806b801970d-kube-api-access-csppc\") pod \"e2e-conflict-a-e5a626b8-kserve-7688fbd-k5z56\" (UID: \"2c423d2e-4cdb-496f-aeb7-3806b801970d\") " pod="llm/e2e-conflict-a-e5a626b8-kserve-7688fbd-k5z56"
Sep 10 22:16:04.298973 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:04.298952    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-e5a626b8-kserve-7688fbd-k5z56"
Sep 10 22:16:04.424426 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:04.424402    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cd1c6d7b-kms8b"
Sep 10 22:16:04.467277 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:04.467252    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/295a9574-4729-41e4-adbb-fee2cfc66a4f-maas-api-service-ca\") pod \"295a9574-4729-41e4-adbb-fee2cfc66a4f\" (UID: \"295a9574-4729-41e4-adbb-fee2cfc66a4f\") "
Sep 10 22:16:04.467565 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:04.467541    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/295a9574-4729-41e4-adbb-fee2cfc66a4f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "295a9574-4729-41e4-adbb-fee2cfc66a4f" (UID: "295a9574-4729-41e4-adbb-fee2cfc66a4f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 22:16:04.568915 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:04.568833    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/295a9574-4729-41e4-adbb-fee2cfc66a4f-maas-api-service-ca\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:16:04.615383 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:04.615360    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-a-e5a626b8-kserve-7688fbd-k5z56"]
Sep 10 22:16:04.616497 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:16:04.616474    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2c423d2e_4cdb_496f_aeb7_3806b801970d.slice/crio-cc433e6b7c6fb68a3f05fb6f68796627657f9ca673a8587e5054372d7e1d1e65 WatchSource:0}: Error finding container cc433e6b7c6fb68a3f05fb6f68796627657f9ca673a8587e5054372d7e1d1e65: Status 404 returned error can't find the container with id cc433e6b7c6fb68a3f05fb6f68796627657f9ca673a8587e5054372d7e1d1e65
Sep 10 22:16:05.297338 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:05.297309    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cd1c6d7b-kms8b"
Sep 10 22:16:05.297507 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:05.297342    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cd1c6d7b-kms8b" event={"ID":"295a9574-4729-41e4-adbb-fee2cfc66a4f","Type":"ContainerDied","Data":"b02d62ce1af8e43993c1f1579fbddd3ea127e839c6cf24230cc8caa7a80e12bb"}
Sep 10 22:16:05.297507 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:05.297393    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b02d62ce1af8e43993c1f1579fbddd3ea127e839c6cf24230cc8caa7a80e12bb"
Sep 10 22:16:05.298924 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:05.298900    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-e5a626b8-kserve-7688fbd-k5z56" event={"ID":"2c423d2e-4cdb-496f-aeb7-3806b801970d","Type":"ContainerStarted","Data":"89b9dab482a9a4318a316ec0654c9c215c9d9d8d5aa10c260fdd09abca231b5a"}
Sep 10 22:16:05.298924 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:05.298932    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-e5a626b8-kserve-7688fbd-k5z56" event={"ID":"2c423d2e-4cdb-496f-aeb7-3806b801970d","Type":"ContainerStarted","Data":"cc433e6b7c6fb68a3f05fb6f68796627657f9ca673a8587e5054372d7e1d1e65"}
Sep 10 22:16:05.353507 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:05.353456    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-conflict-a-e5a626b8-kserve-7688fbd-k5z56" podStartSLOduration=2.353441972 podStartE2EDuration="2.353441972s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 22:16:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:16:05.350934106 +0000 UTC m=+886.307001811" watchObservedRunningTime="2026-09-10 22:16:05.353441972 +0000 UTC m=+886.309509675"
Sep 10 22:16:09.151406 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:09.151371    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-cd1c6d7b-dbf676dd6-wjzbb"]
Sep 10 22:16:09.151763 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:09.151574    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-cd1c6d7b-dbf676dd6-wjzbb" podUID="54c83d43-1e6e-4865-bf97-2a96baa4a717" containerName="payload-processing" containerID="cri-o://bb533896d6007ab7e7f9b4fa32834174bed06c219e4ba720f9ded5788d1da2ba" gracePeriod=30
Sep 10 22:16:09.154580 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:09.154554    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-cd1c6d7b-6bd69547fd-xwdx8"]
Sep 10 22:16:09.154793 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:09.154766    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-cd1c6d7b-6bd69547fd-xwdx8" podUID="84c9fe1c-3314-4408-b5f6-895830679145" containerName="payload-pre-processing" containerID="cri-o://ec9d74434bb75ddc1b12761d204a611fab0b0510607c87538bcd425615b16c47" gracePeriod=30
Sep 10 22:16:09.319663 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:09.319627    2582 generic.go:356] "Generic (PLEG): container finished" podID="54c83d43-1e6e-4865-bf97-2a96baa4a717" containerID="bb533896d6007ab7e7f9b4fa32834174bed06c219e4ba720f9ded5788d1da2ba" exitCode=0
Sep 10 22:16:09.319834 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:09.319705    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-cd1c6d7b-dbf676dd6-wjzbb" event={"ID":"54c83d43-1e6e-4865-bf97-2a96baa4a717","Type":"ContainerDied","Data":"bb533896d6007ab7e7f9b4fa32834174bed06c219e4ba720f9ded5788d1da2ba"}
Sep 10 22:16:09.328201 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:09.328167    2582 generic.go:356] "Generic (PLEG): container finished" podID="84c9fe1c-3314-4408-b5f6-895830679145" containerID="ec9d74434bb75ddc1b12761d204a611fab0b0510607c87538bcd425615b16c47" exitCode=0
Sep 10 22:16:09.328325 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:09.328212    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-cd1c6d7b-6bd69547fd-xwdx8" event={"ID":"84c9fe1c-3314-4408-b5f6-895830679145","Type":"ContainerDied","Data":"ec9d74434bb75ddc1b12761d204a611fab0b0510607c87538bcd425615b16c47"}
Sep 10 22:16:09.430804 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:09.430778    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-cd1c6d7b-dbf676dd6-wjzbb"
Sep 10 22:16:09.434189 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:09.434162    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-cd1c6d7b-6bd69547fd-xwdx8"
Sep 10 22:16:09.508920 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:09.508892    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9htxt\" (UniqueName: \"kubernetes.io/projected/84c9fe1c-3314-4408-b5f6-895830679145-kube-api-access-9htxt\") pod \"84c9fe1c-3314-4408-b5f6-895830679145\" (UID: \"84c9fe1c-3314-4408-b5f6-895830679145\") "
Sep 10 22:16:09.509082 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:09.508934    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84c9fe1c-3314-4408-b5f6-895830679145-plugins-config-volume\") pod \"84c9fe1c-3314-4408-b5f6-895830679145\" (UID: \"84c9fe1c-3314-4408-b5f6-895830679145\") "
Sep 10 22:16:09.509082 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:09.508957    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54c83d43-1e6e-4865-bf97-2a96baa4a717-plugins-config-volume\") pod \"54c83d43-1e6e-4865-bf97-2a96baa4a717\" (UID: \"54c83d43-1e6e-4865-bf97-2a96baa4a717\") "
Sep 10 22:16:09.509082 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:09.509056    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k8d2x\" (UniqueName: \"kubernetes.io/projected/54c83d43-1e6e-4865-bf97-2a96baa4a717-kube-api-access-k8d2x\") pod \"54c83d43-1e6e-4865-bf97-2a96baa4a717\" (UID: \"54c83d43-1e6e-4865-bf97-2a96baa4a717\") "
Sep 10 22:16:09.509389 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:09.509357    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54c83d43-1e6e-4865-bf97-2a96baa4a717-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "54c83d43-1e6e-4865-bf97-2a96baa4a717" (UID: "54c83d43-1e6e-4865-bf97-2a96baa4a717"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 22:16:09.509506 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:09.509361    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84c9fe1c-3314-4408-b5f6-895830679145-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "84c9fe1c-3314-4408-b5f6-895830679145" (UID: "84c9fe1c-3314-4408-b5f6-895830679145"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 22:16:09.509565 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:09.509534    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84c9fe1c-3314-4408-b5f6-895830679145-plugins-config-volume\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:16:09.509565 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:09.509553    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54c83d43-1e6e-4865-bf97-2a96baa4a717-plugins-config-volume\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:16:09.511248 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:09.511219    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54c83d43-1e6e-4865-bf97-2a96baa4a717-kube-api-access-k8d2x" (OuterVolumeSpecName: "kube-api-access-k8d2x") pod "54c83d43-1e6e-4865-bf97-2a96baa4a717" (UID: "54c83d43-1e6e-4865-bf97-2a96baa4a717"). InnerVolumeSpecName "kube-api-access-k8d2x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 22:16:09.511337 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:09.511278    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84c9fe1c-3314-4408-b5f6-895830679145-kube-api-access-9htxt" (OuterVolumeSpecName: "kube-api-access-9htxt") pod "84c9fe1c-3314-4408-b5f6-895830679145" (UID: "84c9fe1c-3314-4408-b5f6-895830679145"). InnerVolumeSpecName "kube-api-access-9htxt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 22:16:09.610107 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:09.610079    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8d2x\" (UniqueName: \"kubernetes.io/projected/54c83d43-1e6e-4865-bf97-2a96baa4a717-kube-api-access-k8d2x\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:16:09.610107 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:09.610102    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9htxt\" (UniqueName: \"kubernetes.io/projected/84c9fe1c-3314-4408-b5f6-895830679145-kube-api-access-9htxt\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:16:09.664053 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:09.663981    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cd1c6d7b-kms8b"]
Sep 10 22:16:09.684537 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:09.684515    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cd1c6d7b-kms8b"]
Sep 10 22:16:10.333724 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:10.333695    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-cd1c6d7b-dbf676dd6-wjzbb"
Sep 10 22:16:10.333724 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:10.333711    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-cd1c6d7b-dbf676dd6-wjzbb" event={"ID":"54c83d43-1e6e-4865-bf97-2a96baa4a717","Type":"ContainerDied","Data":"208d798d131eedff15a3070d9fd3bb8d28f72a3e907b26c61c1eae55823e71ed"}
Sep 10 22:16:10.334283 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:10.333766    2582 scope.go:117] "RemoveContainer" containerID="bb533896d6007ab7e7f9b4fa32834174bed06c219e4ba720f9ded5788d1da2ba"
Sep 10 22:16:10.335226 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:10.335205    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-cd1c6d7b-6bd69547fd-xwdx8"
Sep 10 22:16:10.335306 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:10.335235    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-cd1c6d7b-6bd69547fd-xwdx8" event={"ID":"84c9fe1c-3314-4408-b5f6-895830679145","Type":"ContainerDied","Data":"9ee7239ffc4a9903cc8d4d03d1769b41ae8947394bfa3792d08ce40d41681424"}
Sep 10 22:16:10.344000 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:10.343983    2582 scope.go:117] "RemoveContainer" containerID="ec9d74434bb75ddc1b12761d204a611fab0b0510607c87538bcd425615b16c47"
Sep 10 22:16:10.369922 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:10.369897    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-cd1c6d7b-6bd69547fd-xwdx8"]
Sep 10 22:16:10.379004 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:10.378984    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-cd1c6d7b-6bd69547fd-xwdx8"]
Sep 10 22:16:10.405442 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:10.405417    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-cd1c6d7b-dbf676dd6-wjzbb"]
Sep 10 22:16:10.420322 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:10.420302    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-cd1c6d7b-dbf676dd6-wjzbb"]
Sep 10 22:16:10.708619 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:10.708595    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-53efb5ef-openshift-default-7fdb9f589d-52p66"]
Sep 10 22:16:10.708958 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:10.708945    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54c83d43-1e6e-4865-bf97-2a96baa4a717" containerName="payload-processing"
Sep 10 22:16:10.709038 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:10.708959    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="54c83d43-1e6e-4865-bf97-2a96baa4a717" containerName="payload-processing"
Sep 10 22:16:10.709038 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:10.708969    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84c9fe1c-3314-4408-b5f6-895830679145" containerName="payload-pre-processing"
Sep 10 22:16:10.709038 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:10.708982    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="84c9fe1c-3314-4408-b5f6-895830679145" containerName="payload-pre-processing"
Sep 10 22:16:10.709038 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:10.708991    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="295a9574-4729-41e4-adbb-fee2cfc66a4f" containerName="revoke-keys"
Sep 10 22:16:10.709038 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:10.708996    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="295a9574-4729-41e4-adbb-fee2cfc66a4f" containerName="revoke-keys"
Sep 10 22:16:10.709226 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:10.709067    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="54c83d43-1e6e-4865-bf97-2a96baa4a717" containerName="payload-processing"
Sep 10 22:16:10.709226 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:10.709076    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="295a9574-4729-41e4-adbb-fee2cfc66a4f" containerName="revoke-keys"
Sep 10 22:16:10.709226 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:10.709083    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="84c9fe1c-3314-4408-b5f6-895830679145" containerName="payload-pre-processing"
Sep 10 22:16:10.713378 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:10.713359    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-53efb5ef-openshift-default-7fdb9f589d-52p66"
Sep 10 22:16:10.723044 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:10.723029    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-53efb5ef-openshift-default-dockercfg-q4fbk\""
Sep 10 22:16:10.733739 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:10.733718    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-53efb5ef-openshift-default-7fdb9f589d-52p66"]
Sep 10 22:16:10.818040 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:10.818006    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34ba450c-a70b-457d-8c7b-0be6ece0be1f-istio-data\") pod \"e2e-ait-53efb5ef-openshift-default-7fdb9f589d-52p66\" (UID: \"34ba450c-a70b-457d-8c7b-0be6ece0be1f\") " pod="openshift-ingress/e2e-ait-53efb5ef-openshift-default-7fdb9f589d-52p66"
Sep 10 22:16:10.818208 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:10.818054    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34ba450c-a70b-457d-8c7b-0be6ece0be1f-istio-envoy\") pod \"e2e-ait-53efb5ef-openshift-default-7fdb9f589d-52p66\" (UID: \"34ba450c-a70b-457d-8c7b-0be6ece0be1f\") " pod="openshift-ingress/e2e-ait-53efb5ef-openshift-default-7fdb9f589d-52p66"
Sep 10 22:16:10.818208 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:10.818111    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34ba450c-a70b-457d-8c7b-0be6ece0be1f-istio-token\") pod \"e2e-ait-53efb5ef-openshift-default-7fdb9f589d-52p66\" (UID: \"34ba450c-a70b-457d-8c7b-0be6ece0be1f\") " pod="openshift-ingress/e2e-ait-53efb5ef-openshift-default-7fdb9f589d-52p66"
Sep 10 22:16:10.818208 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:10.818154    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34ba450c-a70b-457d-8c7b-0be6ece0be1f-workload-socket\") pod \"e2e-ait-53efb5ef-openshift-default-7fdb9f589d-52p66\" (UID: \"34ba450c-a70b-457d-8c7b-0be6ece0be1f\") " pod="openshift-ingress/e2e-ait-53efb5ef-openshift-default-7fdb9f589d-52p66"
Sep 10 22:16:10.818379 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:10.818226    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhrtj\" (UniqueName: \"kubernetes.io/projected/34ba450c-a70b-457d-8c7b-0be6ece0be1f-kube-api-access-mhrtj\") pod \"e2e-ait-53efb5ef-openshift-default-7fdb9f589d-52p66\" (UID: \"34ba450c-a70b-457d-8c7b-0be6ece0be1f\") " pod="openshift-ingress/e2e-ait-53efb5ef-openshift-default-7fdb9f589d-52p66"
Sep 10 22:16:10.818379 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:10.818262    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34ba450c-a70b-457d-8c7b-0be6ece0be1f-credential-socket\") pod \"e2e-ait-53efb5ef-openshift-default-7fdb9f589d-52p66\" (UID: \"34ba450c-a70b-457d-8c7b-0be6ece0be1f\") " pod="openshift-ingress/e2e-ait-53efb5ef-openshift-default-7fdb9f589d-52p66"
Sep 10 22:16:10.818379 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:10.818294    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34ba450c-a70b-457d-8c7b-0be6ece0be1f-istiod-ca-cert\") pod \"e2e-ait-53efb5ef-openshift-default-7fdb9f589d-52p66\" (UID: \"34ba450c-a70b-457d-8c7b-0be6ece0be1f\") " pod="openshift-ingress/e2e-ait-53efb5ef-openshift-default-7fdb9f589d-52p66"
Sep 10 22:16:10.818379 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:10.818354    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34ba450c-a70b-457d-8c7b-0be6ece0be1f-istio-podinfo\") pod \"e2e-ait-53efb5ef-openshift-default-7fdb9f589d-52p66\" (UID: \"34ba450c-a70b-457d-8c7b-0be6ece0be1f\") " pod="openshift-ingress/e2e-ait-53efb5ef-openshift-default-7fdb9f589d-52p66"
Sep 10 22:16:10.818525 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:10.818398    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34ba450c-a70b-457d-8c7b-0be6ece0be1f-workload-certs\") pod \"e2e-ait-53efb5ef-openshift-default-7fdb9f589d-52p66\" (UID: \"34ba450c-a70b-457d-8c7b-0be6ece0be1f\") " pod="openshift-ingress/e2e-ait-53efb5ef-openshift-default-7fdb9f589d-52p66"
Sep 10 22:16:10.919199 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:10.919172    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34ba450c-a70b-457d-8c7b-0be6ece0be1f-istio-data\") pod \"e2e-ait-53efb5ef-openshift-default-7fdb9f589d-52p66\" (UID: \"34ba450c-a70b-457d-8c7b-0be6ece0be1f\") " pod="openshift-ingress/e2e-ait-53efb5ef-openshift-default-7fdb9f589d-52p66"
Sep 10 22:16:10.919317 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:10.919205    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34ba450c-a70b-457d-8c7b-0be6ece0be1f-istio-envoy\") pod \"e2e-ait-53efb5ef-openshift-default-7fdb9f589d-52p66\" (UID: \"34ba450c-a70b-457d-8c7b-0be6ece0be1f\") " pod="openshift-ingress/e2e-ait-53efb5ef-openshift-default-7fdb9f589d-52p66"
Sep 10 22:16:10.919317 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:10.919220    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34ba450c-a70b-457d-8c7b-0be6ece0be1f-istio-token\") pod \"e2e-ait-53efb5ef-openshift-default-7fdb9f589d-52p66\" (UID: \"34ba450c-a70b-457d-8c7b-0be6ece0be1f\") " pod="openshift-ingress/e2e-ait-53efb5ef-openshift-default-7fdb9f589d-52p66"
Sep 10 22:16:10.919317 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:10.919241    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34ba450c-a70b-457d-8c7b-0be6ece0be1f-workload-socket\") pod \"e2e-ait-53efb5ef-openshift-default-7fdb9f589d-52p66\" (UID: \"34ba450c-a70b-457d-8c7b-0be6ece0be1f\") " pod="openshift-ingress/e2e-ait-53efb5ef-openshift-default-7fdb9f589d-52p66"
Sep 10 22:16:10.919317 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:10.919283    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhrtj\" (UniqueName: \"kubernetes.io/projected/34ba450c-a70b-457d-8c7b-0be6ece0be1f-kube-api-access-mhrtj\") pod \"e2e-ait-53efb5ef-openshift-default-7fdb9f589d-52p66\" (UID: \"34ba450c-a70b-457d-8c7b-0be6ece0be1f\") " pod="openshift-ingress/e2e-ait-53efb5ef-openshift-default-7fdb9f589d-52p66"
Sep 10 22:16:10.919317 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:10.919309    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34ba450c-a70b-457d-8c7b-0be6ece0be1f-credential-socket\") pod \"e2e-ait-53efb5ef-openshift-default-7fdb9f589d-52p66\" (UID: \"34ba450c-a70b-457d-8c7b-0be6ece0be1f\") " pod="openshift-ingress/e2e-ait-53efb5ef-openshift-default-7fdb9f589d-52p66"
Sep 10 22:16:10.919569 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:10.919337    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34ba450c-a70b-457d-8c7b-0be6ece0be1f-istiod-ca-cert\") pod \"e2e-ait-53efb5ef-openshift-default-7fdb9f589d-52p66\" (UID: \"34ba450c-a70b-457d-8c7b-0be6ece0be1f\") " pod="openshift-ingress/e2e-ait-53efb5ef-openshift-default-7fdb9f589d-52p66"
Sep 10 22:16:10.919569 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:10.919386    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34ba450c-a70b-457d-8c7b-0be6ece0be1f-istio-podinfo\") pod \"e2e-ait-53efb5ef-openshift-default-7fdb9f589d-52p66\" (UID: \"34ba450c-a70b-457d-8c7b-0be6ece0be1f\") " pod="openshift-ingress/e2e-ait-53efb5ef-openshift-default-7fdb9f589d-52p66"
Sep 10 22:16:10.919569 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:10.919419    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34ba450c-a70b-457d-8c7b-0be6ece0be1f-workload-certs\") pod \"e2e-ait-53efb5ef-openshift-default-7fdb9f589d-52p66\" (UID: \"34ba450c-a70b-457d-8c7b-0be6ece0be1f\") " pod="openshift-ingress/e2e-ait-53efb5ef-openshift-default-7fdb9f589d-52p66"
Sep 10 22:16:10.919682 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:10.919626    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34ba450c-a70b-457d-8c7b-0be6ece0be1f-istio-data\") pod \"e2e-ait-53efb5ef-openshift-default-7fdb9f589d-52p66\" (UID: \"34ba450c-a70b-457d-8c7b-0be6ece0be1f\") " pod="openshift-ingress/e2e-ait-53efb5ef-openshift-default-7fdb9f589d-52p66"
Sep 10 22:16:10.919816 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:10.919785    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34ba450c-a70b-457d-8c7b-0be6ece0be1f-credential-socket\") pod \"e2e-ait-53efb5ef-openshift-default-7fdb9f589d-52p66\" (UID: \"34ba450c-a70b-457d-8c7b-0be6ece0be1f\") " pod="openshift-ingress/e2e-ait-53efb5ef-openshift-default-7fdb9f589d-52p66"
Sep 10 22:16:10.919816 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:10.919801    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34ba450c-a70b-457d-8c7b-0be6ece0be1f-workload-socket\") pod \"e2e-ait-53efb5ef-openshift-default-7fdb9f589d-52p66\" (UID: \"34ba450c-a70b-457d-8c7b-0be6ece0be1f\") " pod="openshift-ingress/e2e-ait-53efb5ef-openshift-default-7fdb9f589d-52p66"
Sep 10 22:16:10.919973 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:10.919904    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34ba450c-a70b-457d-8c7b-0be6ece0be1f-workload-certs\") pod \"e2e-ait-53efb5ef-openshift-default-7fdb9f589d-52p66\" (UID: \"34ba450c-a70b-457d-8c7b-0be6ece0be1f\") " pod="openshift-ingress/e2e-ait-53efb5ef-openshift-default-7fdb9f589d-52p66"
Sep 10 22:16:10.920157 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:10.920137    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34ba450c-a70b-457d-8c7b-0be6ece0be1f-istiod-ca-cert\") pod \"e2e-ait-53efb5ef-openshift-default-7fdb9f589d-52p66\" (UID: \"34ba450c-a70b-457d-8c7b-0be6ece0be1f\") " pod="openshift-ingress/e2e-ait-53efb5ef-openshift-default-7fdb9f589d-52p66"
Sep 10 22:16:10.921683 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:10.921654    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34ba450c-a70b-457d-8c7b-0be6ece0be1f-istio-envoy\") pod \"e2e-ait-53efb5ef-openshift-default-7fdb9f589d-52p66\" (UID: \"34ba450c-a70b-457d-8c7b-0be6ece0be1f\") " pod="openshift-ingress/e2e-ait-53efb5ef-openshift-default-7fdb9f589d-52p66"
Sep 10 22:16:10.921785 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:10.921706    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34ba450c-a70b-457d-8c7b-0be6ece0be1f-istio-podinfo\") pod \"e2e-ait-53efb5ef-openshift-default-7fdb9f589d-52p66\" (UID: \"34ba450c-a70b-457d-8c7b-0be6ece0be1f\") " pod="openshift-ingress/e2e-ait-53efb5ef-openshift-default-7fdb9f589d-52p66"
Sep 10 22:16:10.938317 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:10.938295    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhrtj\" (UniqueName: \"kubernetes.io/projected/34ba450c-a70b-457d-8c7b-0be6ece0be1f-kube-api-access-mhrtj\") pod \"e2e-ait-53efb5ef-openshift-default-7fdb9f589d-52p66\" (UID: \"34ba450c-a70b-457d-8c7b-0be6ece0be1f\") " pod="openshift-ingress/e2e-ait-53efb5ef-openshift-default-7fdb9f589d-52p66"
Sep 10 22:16:10.939629 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:10.939607    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34ba450c-a70b-457d-8c7b-0be6ece0be1f-istio-token\") pod \"e2e-ait-53efb5ef-openshift-default-7fdb9f589d-52p66\" (UID: \"34ba450c-a70b-457d-8c7b-0be6ece0be1f\") " pod="openshift-ingress/e2e-ait-53efb5ef-openshift-default-7fdb9f589d-52p66"
Sep 10 22:16:11.023305 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:11.023254    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-53efb5ef-openshift-default-7fdb9f589d-52p66"
Sep 10 22:16:11.326433 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:11.326408    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-53efb5ef-openshift-default-7fdb9f589d-52p66"]
Sep 10 22:16:11.327860 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:16:11.327819    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34ba450c_a70b_457d_8c7b_0be6ece0be1f.slice/crio-ad6854512e5d22a7e0fe72c01a1e13b8744a68158e06ef1d8a7648077b33ee30 WatchSource:0}: Error finding container ad6854512e5d22a7e0fe72c01a1e13b8744a68158e06ef1d8a7648077b33ee30: Status 404 returned error can't find the container with id ad6854512e5d22a7e0fe72c01a1e13b8744a68158e06ef1d8a7648077b33ee30
Sep 10 22:16:11.329911 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:11.329879    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 22:16:11.329994 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:11.329940    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 22:16:11.329994 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:11.329972    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 22:16:11.341879 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:11.341840    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-53efb5ef-openshift-default-7fdb9f589d-52p66" event={"ID":"34ba450c-a70b-457d-8c7b-0be6ece0be1f","Type":"ContainerStarted","Data":"ad6854512e5d22a7e0fe72c01a1e13b8744a68158e06ef1d8a7648077b33ee30"}
Sep 10 22:16:11.499522 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:11.499488    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="295a9574-4729-41e4-adbb-fee2cfc66a4f" path="/var/lib/kubelet/pods/295a9574-4729-41e4-adbb-fee2cfc66a4f/volumes"
Sep 10 22:16:11.506956 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:11.500420    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="54c83d43-1e6e-4865-bf97-2a96baa4a717" path="/var/lib/kubelet/pods/54c83d43-1e6e-4865-bf97-2a96baa4a717/volumes"
Sep 10 22:16:11.506956 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:11.500995    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="84c9fe1c-3314-4408-b5f6-895830679145" path="/var/lib/kubelet/pods/84c9fe1c-3314-4408-b5f6-895830679145/volumes"
Sep 10 22:16:12.347646 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:12.347565    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-53efb5ef-openshift-default-7fdb9f589d-52p66" event={"ID":"34ba450c-a70b-457d-8c7b-0be6ece0be1f","Type":"ContainerStarted","Data":"143a4abf622aa86c504ae0185b152a10c81517451ad5642730c08f19e40ba50e"}
Sep 10 22:16:12.394036 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:12.393814    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-53efb5ef-openshift-default-7fdb9f589d-52p66" podStartSLOduration=2.393795553 podStartE2EDuration="2.393795553s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 22:16:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:16:12.39167667 +0000 UTC m=+893.347744389" watchObservedRunningTime="2026-09-10 22:16:12.393795553 +0000 UTC m=+893.349863261"
Sep 10 22:16:13.024039 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:13.024000    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-53efb5ef-openshift-default-7fdb9f589d-52p66"
Sep 10 22:16:13.029365 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:13.029340    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-53efb5ef-openshift-default-7fdb9f589d-52p66"
Sep 10 22:16:13.353538 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:13.353445    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-53efb5ef-openshift-default-7fdb9f589d-52p66"
Sep 10 22:16:13.354567 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:13.354540    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-53efb5ef-openshift-default-7fdb9f589d-52p66"
Sep 10 22:16:14.299828 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:14.299795    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-conflict-a-e5a626b8-kserve-7688fbd-k5z56"
Sep 10 22:16:14.300004 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:14.299842    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-conflict-a-e5a626b8-kserve-7688fbd-k5z56"
Sep 10 22:16:14.312212 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:14.312188    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-conflict-a-e5a626b8-kserve-7688fbd-k5z56"
Sep 10 22:16:14.369822 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:14.369800    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-conflict-a-e5a626b8-kserve-7688fbd-k5z56"
Sep 10 22:16:15.464599 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:15.464572    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-53efb5ef-587b54fc89-5fhv5"]
Sep 10 22:16:15.469241 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:15.469217    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-53efb5ef-587b54fc89-5fhv5"
Sep 10 22:16:15.472823 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:15.472797    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-53efb5ef-serving-cert\""
Sep 10 22:16:15.499744 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:15.499722    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-53efb5ef-587b54fc89-5fhv5"]
Sep 10 22:16:15.556352 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:15.556329    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d08d579c-afec-4f8a-bc2c-bdb0c0852fe3-openshift-service-ca\") pod \"maas-api-e2e-ait-53efb5ef-587b54fc89-5fhv5\" (UID: \"d08d579c-afec-4f8a-bc2c-bdb0c0852fe3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-53efb5ef-587b54fc89-5fhv5"
Sep 10 22:16:15.556461 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:15.556370    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrwx4\" (UniqueName: \"kubernetes.io/projected/d08d579c-afec-4f8a-bc2c-bdb0c0852fe3-kube-api-access-xrwx4\") pod \"maas-api-e2e-ait-53efb5ef-587b54fc89-5fhv5\" (UID: \"d08d579c-afec-4f8a-bc2c-bdb0c0852fe3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-53efb5ef-587b54fc89-5fhv5"
Sep 10 22:16:15.556558 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:15.556545    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d08d579c-afec-4f8a-bc2c-bdb0c0852fe3-maas-api-tls\") pod \"maas-api-e2e-ait-53efb5ef-587b54fc89-5fhv5\" (UID: \"d08d579c-afec-4f8a-bc2c-bdb0c0852fe3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-53efb5ef-587b54fc89-5fhv5"
Sep 10 22:16:15.556627 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:15.556563    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d08d579c-afec-4f8a-bc2c-bdb0c0852fe3-metrics-certs\") pod \"maas-api-e2e-ait-53efb5ef-587b54fc89-5fhv5\" (UID: \"d08d579c-afec-4f8a-bc2c-bdb0c0852fe3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-53efb5ef-587b54fc89-5fhv5"
Sep 10 22:16:15.657606 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:15.657577    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d08d579c-afec-4f8a-bc2c-bdb0c0852fe3-openshift-service-ca\") pod \"maas-api-e2e-ait-53efb5ef-587b54fc89-5fhv5\" (UID: \"d08d579c-afec-4f8a-bc2c-bdb0c0852fe3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-53efb5ef-587b54fc89-5fhv5"
Sep 10 22:16:15.657782 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:15.657629    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrwx4\" (UniqueName: \"kubernetes.io/projected/d08d579c-afec-4f8a-bc2c-bdb0c0852fe3-kube-api-access-xrwx4\") pod \"maas-api-e2e-ait-53efb5ef-587b54fc89-5fhv5\" (UID: \"d08d579c-afec-4f8a-bc2c-bdb0c0852fe3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-53efb5ef-587b54fc89-5fhv5"
Sep 10 22:16:15.657782 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:15.657742    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d08d579c-afec-4f8a-bc2c-bdb0c0852fe3-maas-api-tls\") pod \"maas-api-e2e-ait-53efb5ef-587b54fc89-5fhv5\" (UID: \"d08d579c-afec-4f8a-bc2c-bdb0c0852fe3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-53efb5ef-587b54fc89-5fhv5"
Sep 10 22:16:15.657782 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:15.657771    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d08d579c-afec-4f8a-bc2c-bdb0c0852fe3-metrics-certs\") pod \"maas-api-e2e-ait-53efb5ef-587b54fc89-5fhv5\" (UID: \"d08d579c-afec-4f8a-bc2c-bdb0c0852fe3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-53efb5ef-587b54fc89-5fhv5"
Sep 10 22:16:15.658012 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:16:15.657892    2582 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-53efb5ef-serving-cert: secret "maas-api-e2e-ait-53efb5ef-serving-cert" not found
Sep 10 22:16:15.658012 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:16:15.657971    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d08d579c-afec-4f8a-bc2c-bdb0c0852fe3-maas-api-tls podName:d08d579c-afec-4f8a-bc2c-bdb0c0852fe3 nodeName:}" failed. No retries permitted until 2026-09-10 22:16:16.157951329 +0000 UTC m=+897.114019013 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d08d579c-afec-4f8a-bc2c-bdb0c0852fe3-maas-api-tls") pod "maas-api-e2e-ait-53efb5ef-587b54fc89-5fhv5" (UID: "d08d579c-afec-4f8a-bc2c-bdb0c0852fe3") : secret "maas-api-e2e-ait-53efb5ef-serving-cert" not found
Sep 10 22:16:15.658415 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:15.658393    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d08d579c-afec-4f8a-bc2c-bdb0c0852fe3-openshift-service-ca\") pod \"maas-api-e2e-ait-53efb5ef-587b54fc89-5fhv5\" (UID: \"d08d579c-afec-4f8a-bc2c-bdb0c0852fe3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-53efb5ef-587b54fc89-5fhv5"
Sep 10 22:16:15.660079 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:15.660062    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d08d579c-afec-4f8a-bc2c-bdb0c0852fe3-metrics-certs\") pod \"maas-api-e2e-ait-53efb5ef-587b54fc89-5fhv5\" (UID: \"d08d579c-afec-4f8a-bc2c-bdb0c0852fe3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-53efb5ef-587b54fc89-5fhv5"
Sep 10 22:16:15.677354 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:15.677331    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrwx4\" (UniqueName: \"kubernetes.io/projected/d08d579c-afec-4f8a-bc2c-bdb0c0852fe3-kube-api-access-xrwx4\") pod \"maas-api-e2e-ait-53efb5ef-587b54fc89-5fhv5\" (UID: \"d08d579c-afec-4f8a-bc2c-bdb0c0852fe3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-53efb5ef-587b54fc89-5fhv5"
Sep 10 22:16:16.161911 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:16.161882    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d08d579c-afec-4f8a-bc2c-bdb0c0852fe3-maas-api-tls\") pod \"maas-api-e2e-ait-53efb5ef-587b54fc89-5fhv5\" (UID: \"d08d579c-afec-4f8a-bc2c-bdb0c0852fe3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-53efb5ef-587b54fc89-5fhv5"
Sep 10 22:16:16.164075 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:16.164055    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d08d579c-afec-4f8a-bc2c-bdb0c0852fe3-maas-api-tls\") pod \"maas-api-e2e-ait-53efb5ef-587b54fc89-5fhv5\" (UID: \"d08d579c-afec-4f8a-bc2c-bdb0c0852fe3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-53efb5ef-587b54fc89-5fhv5"
Sep 10 22:16:16.378919 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:16.378890    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-53efb5ef-587b54fc89-5fhv5"
Sep 10 22:16:16.665221 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:16.665190    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-53efb5ef-587b54fc89-5fhv5"]
Sep 10 22:16:16.668565 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:16:16.668542    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd08d579c_afec_4f8a_bc2c_bdb0c0852fe3.slice/crio-58797a2a3e5d0f75d1ed83fb3b141da2e1c8002de115efd52ffb913cbf514e4e WatchSource:0}: Error finding container 58797a2a3e5d0f75d1ed83fb3b141da2e1c8002de115efd52ffb913cbf514e4e: Status 404 returned error can't find the container with id 58797a2a3e5d0f75d1ed83fb3b141da2e1c8002de115efd52ffb913cbf514e4e
Sep 10 22:16:16.878730 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:16.878656    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-53efb5ef-595d844fd6-fm22t"]
Sep 10 22:16:16.883014 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:16.882996    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-53efb5ef-595d844fd6-fm22t"
Sep 10 22:16:16.890151 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:16.890130    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-53efb5ef\""
Sep 10 22:16:16.890151 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:16.890141    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-53efb5ef-dockercfg-f7s44\""
Sep 10 22:16:16.906907 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:16.906886    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-53efb5ef-595d844fd6-fm22t"]
Sep 10 22:16:16.968024 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:16.967999    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckxnk\" (UniqueName: \"kubernetes.io/projected/2a7ca5a3-381f-4693-875c-dfe5c9f5f827-kube-api-access-ckxnk\") pod \"payload-pre-processing-e2e-ait-53efb5ef-595d844fd6-fm22t\" (UID: \"2a7ca5a3-381f-4693-875c-dfe5c9f5f827\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-53efb5ef-595d844fd6-fm22t"
Sep 10 22:16:16.968132 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:16.968040    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a7ca5a3-381f-4693-875c-dfe5c9f5f827-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-53efb5ef-595d844fd6-fm22t\" (UID: \"2a7ca5a3-381f-4693-875c-dfe5c9f5f827\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-53efb5ef-595d844fd6-fm22t"
Sep 10 22:16:17.004253 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:17.004221    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-53efb5ef-79f88c7d9-697g5"]
Sep 10 22:16:17.007640 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:17.007618    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-53efb5ef-79f88c7d9-697g5"
Sep 10 22:16:17.036211 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:17.036185    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-53efb5ef-79f88c7d9-697g5"]
Sep 10 22:16:17.069145 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:17.069116    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckxnk\" (UniqueName: \"kubernetes.io/projected/2a7ca5a3-381f-4693-875c-dfe5c9f5f827-kube-api-access-ckxnk\") pod \"payload-pre-processing-e2e-ait-53efb5ef-595d844fd6-fm22t\" (UID: \"2a7ca5a3-381f-4693-875c-dfe5c9f5f827\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-53efb5ef-595d844fd6-fm22t"
Sep 10 22:16:17.069247 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:17.069174    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a7ca5a3-381f-4693-875c-dfe5c9f5f827-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-53efb5ef-595d844fd6-fm22t\" (UID: \"2a7ca5a3-381f-4693-875c-dfe5c9f5f827\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-53efb5ef-595d844fd6-fm22t"
Sep 10 22:16:17.069247 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:17.069194    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d7b58cc-615b-48e2-a4ac-a2d779e57bda-plugins-config-volume\") pod \"payload-processing-e2e-ait-53efb5ef-79f88c7d9-697g5\" (UID: \"5d7b58cc-615b-48e2-a4ac-a2d779e57bda\") " pod="openshift-ingress/payload-processing-e2e-ait-53efb5ef-79f88c7d9-697g5"
Sep 10 22:16:17.069362 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:17.069263    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cv9wh\" (UniqueName: \"kubernetes.io/projected/5d7b58cc-615b-48e2-a4ac-a2d779e57bda-kube-api-access-cv9wh\") pod \"payload-processing-e2e-ait-53efb5ef-79f88c7d9-697g5\" (UID: \"5d7b58cc-615b-48e2-a4ac-a2d779e57bda\") " pod="openshift-ingress/payload-processing-e2e-ait-53efb5ef-79f88c7d9-697g5"
Sep 10 22:16:17.069874 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:17.069823    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a7ca5a3-381f-4693-875c-dfe5c9f5f827-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-53efb5ef-595d844fd6-fm22t\" (UID: \"2a7ca5a3-381f-4693-875c-dfe5c9f5f827\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-53efb5ef-595d844fd6-fm22t"
Sep 10 22:16:17.086271 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:17.086253    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckxnk\" (UniqueName: \"kubernetes.io/projected/2a7ca5a3-381f-4693-875c-dfe5c9f5f827-kube-api-access-ckxnk\") pod \"payload-pre-processing-e2e-ait-53efb5ef-595d844fd6-fm22t\" (UID: \"2a7ca5a3-381f-4693-875c-dfe5c9f5f827\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-53efb5ef-595d844fd6-fm22t"
Sep 10 22:16:17.170535 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:17.170508    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d7b58cc-615b-48e2-a4ac-a2d779e57bda-plugins-config-volume\") pod \"payload-processing-e2e-ait-53efb5ef-79f88c7d9-697g5\" (UID: \"5d7b58cc-615b-48e2-a4ac-a2d779e57bda\") " pod="openshift-ingress/payload-processing-e2e-ait-53efb5ef-79f88c7d9-697g5"
Sep 10 22:16:17.170644 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:17.170586    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cv9wh\" (UniqueName: \"kubernetes.io/projected/5d7b58cc-615b-48e2-a4ac-a2d779e57bda-kube-api-access-cv9wh\") pod \"payload-processing-e2e-ait-53efb5ef-79f88c7d9-697g5\" (UID: \"5d7b58cc-615b-48e2-a4ac-a2d779e57bda\") " pod="openshift-ingress/payload-processing-e2e-ait-53efb5ef-79f88c7d9-697g5"
Sep 10 22:16:17.171155 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:17.171133    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d7b58cc-615b-48e2-a4ac-a2d779e57bda-plugins-config-volume\") pod \"payload-processing-e2e-ait-53efb5ef-79f88c7d9-697g5\" (UID: \"5d7b58cc-615b-48e2-a4ac-a2d779e57bda\") " pod="openshift-ingress/payload-processing-e2e-ait-53efb5ef-79f88c7d9-697g5"
Sep 10 22:16:17.193376 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:17.193356    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-53efb5ef-595d844fd6-fm22t"
Sep 10 22:16:17.208531 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:17.208505    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cv9wh\" (UniqueName: \"kubernetes.io/projected/5d7b58cc-615b-48e2-a4ac-a2d779e57bda-kube-api-access-cv9wh\") pod \"payload-processing-e2e-ait-53efb5ef-79f88c7d9-697g5\" (UID: \"5d7b58cc-615b-48e2-a4ac-a2d779e57bda\") " pod="openshift-ingress/payload-processing-e2e-ait-53efb5ef-79f88c7d9-697g5"
Sep 10 22:16:17.318445 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:17.318414    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-53efb5ef-79f88c7d9-697g5"
Sep 10 22:16:17.377879 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:17.377631    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-53efb5ef-587b54fc89-5fhv5" event={"ID":"d08d579c-afec-4f8a-bc2c-bdb0c0852fe3","Type":"ContainerStarted","Data":"c54ecd83a4ff17b251de2e80bc69d3ca4f693ca0feb27868993a2bfcbf3084c8"}
Sep 10 22:16:17.377879 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:17.377699    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-53efb5ef-587b54fc89-5fhv5" event={"ID":"d08d579c-afec-4f8a-bc2c-bdb0c0852fe3","Type":"ContainerStarted","Data":"58797a2a3e5d0f75d1ed83fb3b141da2e1c8002de115efd52ffb913cbf514e4e"}
Sep 10 22:16:17.378600 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:17.378441    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-53efb5ef-587b54fc89-5fhv5"
Sep 10 22:16:17.432010 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:17.431931    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-53efb5ef-587b54fc89-5fhv5" podStartSLOduration=2.042291535 podStartE2EDuration="2.431915932s" totalImagesPullingTime="389.624397ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 22:16:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:16:17.430455972 +0000 UTC m=+898.386523676" watchObservedRunningTime="2026-09-10 22:16:17.431915932 +0000 UTC m=+898.387983637"
Sep 10 22:16:17.489835 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:17.489815    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-53efb5ef-595d844fd6-fm22t"]
Sep 10 22:16:17.491145 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:16:17.491126    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a7ca5a3_381f_4693_875c_dfe5c9f5f827.slice/crio-f6259fcd7afd248fe4bf92a228f1dc4c3ce5aa049b1b3d2fb23c7c3e4a8fe72b WatchSource:0}: Error finding container f6259fcd7afd248fe4bf92a228f1dc4c3ce5aa049b1b3d2fb23c7c3e4a8fe72b: Status 404 returned error can't find the container with id f6259fcd7afd248fe4bf92a228f1dc4c3ce5aa049b1b3d2fb23c7c3e4a8fe72b
Sep 10 22:16:17.625041 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:17.625016    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-53efb5ef-79f88c7d9-697g5"]
Sep 10 22:16:17.627508 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:16:17.627481    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d7b58cc_615b_48e2_a4ac_a2d779e57bda.slice/crio-445a025b7c87fdf9249cf58591e966c17edc008f680779d998f142c02488ea5d WatchSource:0}: Error finding container 445a025b7c87fdf9249cf58591e966c17edc008f680779d998f142c02488ea5d: Status 404 returned error can't find the container with id 445a025b7c87fdf9249cf58591e966c17edc008f680779d998f142c02488ea5d
Sep 10 22:16:18.384745 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:18.384701    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-53efb5ef-79f88c7d9-697g5" event={"ID":"5d7b58cc-615b-48e2-a4ac-a2d779e57bda","Type":"ContainerStarted","Data":"13fd04b674d16064e466304775f959851360cf1f54e24927d77ab86e30e563b3"}
Sep 10 22:16:18.385265 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:18.384782    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-53efb5ef-79f88c7d9-697g5"
Sep 10 22:16:18.385265 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:18.384806    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-53efb5ef-79f88c7d9-697g5" event={"ID":"5d7b58cc-615b-48e2-a4ac-a2d779e57bda","Type":"ContainerStarted","Data":"445a025b7c87fdf9249cf58591e966c17edc008f680779d998f142c02488ea5d"}
Sep 10 22:16:18.386434 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:18.386410    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-53efb5ef-595d844fd6-fm22t" event={"ID":"2a7ca5a3-381f-4693-875c-dfe5c9f5f827","Type":"ContainerStarted","Data":"b35d83b6e1602b12a066bcb7d40196c3cc7ed0d30f678ef8f143d96ef0324e04"}
Sep 10 22:16:18.386548 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:18.386452    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-53efb5ef-595d844fd6-fm22t" event={"ID":"2a7ca5a3-381f-4693-875c-dfe5c9f5f827","Type":"ContainerStarted","Data":"f6259fcd7afd248fe4bf92a228f1dc4c3ce5aa049b1b3d2fb23c7c3e4a8fe72b"}
Sep 10 22:16:18.386548 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:18.386489    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-53efb5ef-595d844fd6-fm22t"
Sep 10 22:16:18.405789 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:18.405747    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-53efb5ef-79f88c7d9-697g5" podStartSLOduration=2.405736266 podStartE2EDuration="2.405736266s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 22:16:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:16:18.404007151 +0000 UTC m=+899.360074860" watchObservedRunningTime="2026-09-10 22:16:18.405736266 +0000 UTC m=+899.361803972"
Sep 10 22:16:18.431353 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:18.431315    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-53efb5ef-595d844fd6-fm22t" podStartSLOduration=2.431302543 podStartE2EDuration="2.431302543s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 22:16:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:16:18.430176361 +0000 UTC m=+899.386244068" watchObservedRunningTime="2026-09-10 22:16:18.431302543 +0000 UTC m=+899.387370248"
Sep 10 22:16:19.475550 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:19.475520    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-w6vlb_8517c9a7-382a-40fd-bc52-e6da90ebc584/console-operator/2.log"
Sep 10 22:16:19.476015 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:19.475630    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-w6vlb_8517c9a7-382a-40fd-bc52-e6da90ebc584/console-operator/2.log"
Sep 10 22:16:24.399658 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:24.399627    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-53efb5ef-587b54fc89-5fhv5"
Sep 10 22:16:29.394833 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:29.394800    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-53efb5ef-595d844fd6-fm22t"
Sep 10 22:16:29.395828 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:29.395806    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-53efb5ef-79f88c7d9-697g5"
Sep 10 22:16:33.459870 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:33.459814    2582 generic.go:356] "Generic (PLEG): container finished" podID="34ba450c-a70b-457d-8c7b-0be6ece0be1f" containerID="143a4abf622aa86c504ae0185b152a10c81517451ad5642730c08f19e40ba50e" exitCode=0
Sep 10 22:16:33.460300 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:33.459890    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-53efb5ef-openshift-default-7fdb9f589d-52p66" event={"ID":"34ba450c-a70b-457d-8c7b-0be6ece0be1f","Type":"ContainerDied","Data":"143a4abf622aa86c504ae0185b152a10c81517451ad5642730c08f19e40ba50e"}
Sep 10 22:16:33.460351 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:33.460307    2582 scope.go:117] "RemoveContainer" containerID="143a4abf622aa86c504ae0185b152a10c81517451ad5642730c08f19e40ba50e"
Sep 10 22:16:33.461432 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:33.461402    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 22:16:33.461499 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:33.461484    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 22:16:33.461538 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:33.461521    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 22:16:34.466282 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:34.466249    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-53efb5ef-openshift-default-7fdb9f589d-52p66" event={"ID":"34ba450c-a70b-457d-8c7b-0be6ece0be1f","Type":"ContainerStarted","Data":"d1b6477c495b8e0eeed868956d748287aeb14bef31a14fb9b6bcbfa3a2a86ffb"}
Sep 10 22:16:35.023376 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:35.023344    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-53efb5ef-openshift-default-7fdb9f589d-52p66"
Sep 10 22:16:35.023699 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:35.023669    2582 patch_prober.go:28] interesting pod/e2e-ait-53efb5ef-openshift-default-7fdb9f589d-52p66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Sep 10 22:16:35.023810 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:35.023734    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-53efb5ef-openshift-default-7fdb9f589d-52p66" podUID="34ba450c-a70b-457d-8c7b-0be6ece0be1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Sep 10 22:16:36.024027 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:36.023996    2582 patch_prober.go:28] interesting pod/e2e-ait-53efb5ef-openshift-default-7fdb9f589d-52p66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Sep 10 22:16:36.024399 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:36.024051    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-53efb5ef-openshift-default-7fdb9f589d-52p66" podUID="34ba450c-a70b-457d-8c7b-0be6ece0be1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Sep 10 22:16:37.024471 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:37.024435    2582 patch_prober.go:28] interesting pod/e2e-ait-53efb5ef-openshift-default-7fdb9f589d-52p66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Sep 10 22:16:37.024867 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:37.024492    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-53efb5ef-openshift-default-7fdb9f589d-52p66" podUID="34ba450c-a70b-457d-8c7b-0be6ece0be1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Sep 10 22:16:38.024268 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:38.024230    2582 patch_prober.go:28] interesting pod/e2e-ait-53efb5ef-openshift-default-7fdb9f589d-52p66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Sep 10 22:16:38.024462 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:38.024295    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-53efb5ef-openshift-default-7fdb9f589d-52p66" podUID="34ba450c-a70b-457d-8c7b-0be6ece0be1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Sep 10 22:16:39.023699 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:39.023667    2582 patch_prober.go:28] interesting pod/e2e-ait-53efb5ef-openshift-default-7fdb9f589d-52p66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Sep 10 22:16:39.024178 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:39.023723    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-53efb5ef-openshift-default-7fdb9f589d-52p66" podUID="34ba450c-a70b-457d-8c7b-0be6ece0be1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Sep 10 22:16:40.028137 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:40.028108    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-53efb5ef-openshift-default-7fdb9f589d-52p66"
Sep 10 22:16:40.028524 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:40.028499    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-53efb5ef-openshift-default-7fdb9f589d-52p66"
Sep 10 22:16:40.029210 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:40.029193    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-53efb5ef-openshift-default-7fdb9f589d-52p66"
Sep 10 22:16:45.125998 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:45.125968    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-e5a2d7/test-model-e5a2d7-kserve-57bbd95c65-xjc9m"]
Sep 10 22:16:45.128504 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:45.128488    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-e5a2d7/test-model-e5a2d7-kserve-57bbd95c65-xjc9m"
Sep 10 22:16:45.165968 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:45.165938    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-e5a2d7\"/\"default-dockercfg-xc76k\""
Sep 10 22:16:45.165968 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:45.165964    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-e5a2d7\"/\"openshift-service-ca.crt\""
Sep 10 22:16:45.167574 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:45.167557    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-e5a2d7\"/\"test-model-e5a2d7-kserve-self-signed-certs\""
Sep 10 22:16:45.189787 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:45.189765    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-e5a2d7\"/\"kube-root-ca.crt\""
Sep 10 22:16:45.226632 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:45.226606    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d91987a5-bc23-4976-a1c7-c10448df0e0e-tls-certs\") pod \"test-model-e5a2d7-kserve-57bbd95c65-xjc9m\" (UID: \"d91987a5-bc23-4976-a1c7-c10448df0e0e\") " pod="ai-tenant-e2e-inf-a-e5a2d7/test-model-e5a2d7-kserve-57bbd95c65-xjc9m"
Sep 10 22:16:45.226767 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:45.226648    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d91987a5-bc23-4976-a1c7-c10448df0e0e-dshm\") pod \"test-model-e5a2d7-kserve-57bbd95c65-xjc9m\" (UID: \"d91987a5-bc23-4976-a1c7-c10448df0e0e\") " pod="ai-tenant-e2e-inf-a-e5a2d7/test-model-e5a2d7-kserve-57bbd95c65-xjc9m"
Sep 10 22:16:45.226767 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:45.226690    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d91987a5-bc23-4976-a1c7-c10448df0e0e-home\") pod \"test-model-e5a2d7-kserve-57bbd95c65-xjc9m\" (UID: \"d91987a5-bc23-4976-a1c7-c10448df0e0e\") " pod="ai-tenant-e2e-inf-a-e5a2d7/test-model-e5a2d7-kserve-57bbd95c65-xjc9m"
Sep 10 22:16:45.226767 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:45.226711    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d91987a5-bc23-4976-a1c7-c10448df0e0e-model-cache\") pod \"test-model-e5a2d7-kserve-57bbd95c65-xjc9m\" (UID: \"d91987a5-bc23-4976-a1c7-c10448df0e0e\") " pod="ai-tenant-e2e-inf-a-e5a2d7/test-model-e5a2d7-kserve-57bbd95c65-xjc9m"
Sep 10 22:16:45.226767 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:45.226729    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d91987a5-bc23-4976-a1c7-c10448df0e0e-tmp-dir\") pod \"test-model-e5a2d7-kserve-57bbd95c65-xjc9m\" (UID: \"d91987a5-bc23-4976-a1c7-c10448df0e0e\") " pod="ai-tenant-e2e-inf-a-e5a2d7/test-model-e5a2d7-kserve-57bbd95c65-xjc9m"
Sep 10 22:16:45.226767 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:45.226749    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljml4\" (UniqueName: \"kubernetes.io/projected/d91987a5-bc23-4976-a1c7-c10448df0e0e-kube-api-access-ljml4\") pod \"test-model-e5a2d7-kserve-57bbd95c65-xjc9m\" (UID: \"d91987a5-bc23-4976-a1c7-c10448df0e0e\") " pod="ai-tenant-e2e-inf-a-e5a2d7/test-model-e5a2d7-kserve-57bbd95c65-xjc9m"
Sep 10 22:16:45.259367 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:45.259328    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-e5a2d7/test-model-e5a2d7-kserve-57bbd95c65-xjc9m"]
Sep 10 22:16:45.327283 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:45.327242    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d91987a5-bc23-4976-a1c7-c10448df0e0e-home\") pod \"test-model-e5a2d7-kserve-57bbd95c65-xjc9m\" (UID: \"d91987a5-bc23-4976-a1c7-c10448df0e0e\") " pod="ai-tenant-e2e-inf-a-e5a2d7/test-model-e5a2d7-kserve-57bbd95c65-xjc9m"
Sep 10 22:16:45.327283 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:45.327289    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d91987a5-bc23-4976-a1c7-c10448df0e0e-model-cache\") pod \"test-model-e5a2d7-kserve-57bbd95c65-xjc9m\" (UID: \"d91987a5-bc23-4976-a1c7-c10448df0e0e\") " pod="ai-tenant-e2e-inf-a-e5a2d7/test-model-e5a2d7-kserve-57bbd95c65-xjc9m"
Sep 10 22:16:45.327542 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:45.327308    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d91987a5-bc23-4976-a1c7-c10448df0e0e-tmp-dir\") pod \"test-model-e5a2d7-kserve-57bbd95c65-xjc9m\" (UID: \"d91987a5-bc23-4976-a1c7-c10448df0e0e\") " pod="ai-tenant-e2e-inf-a-e5a2d7/test-model-e5a2d7-kserve-57bbd95c65-xjc9m"
Sep 10 22:16:45.327542 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:45.327434    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljml4\" (UniqueName: \"kubernetes.io/projected/d91987a5-bc23-4976-a1c7-c10448df0e0e-kube-api-access-ljml4\") pod \"test-model-e5a2d7-kserve-57bbd95c65-xjc9m\" (UID: \"d91987a5-bc23-4976-a1c7-c10448df0e0e\") " pod="ai-tenant-e2e-inf-a-e5a2d7/test-model-e5a2d7-kserve-57bbd95c65-xjc9m"
Sep 10 22:16:45.327644 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:45.327545    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d91987a5-bc23-4976-a1c7-c10448df0e0e-tls-certs\") pod \"test-model-e5a2d7-kserve-57bbd95c65-xjc9m\" (UID: \"d91987a5-bc23-4976-a1c7-c10448df0e0e\") " pod="ai-tenant-e2e-inf-a-e5a2d7/test-model-e5a2d7-kserve-57bbd95c65-xjc9m"
Sep 10 22:16:45.327644 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:45.327601    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d91987a5-bc23-4976-a1c7-c10448df0e0e-dshm\") pod \"test-model-e5a2d7-kserve-57bbd95c65-xjc9m\" (UID: \"d91987a5-bc23-4976-a1c7-c10448df0e0e\") " pod="ai-tenant-e2e-inf-a-e5a2d7/test-model-e5a2d7-kserve-57bbd95c65-xjc9m"
Sep 10 22:16:45.327732 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:45.327657    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d91987a5-bc23-4976-a1c7-c10448df0e0e-model-cache\") pod \"test-model-e5a2d7-kserve-57bbd95c65-xjc9m\" (UID: \"d91987a5-bc23-4976-a1c7-c10448df0e0e\") " pod="ai-tenant-e2e-inf-a-e5a2d7/test-model-e5a2d7-kserve-57bbd95c65-xjc9m"
Sep 10 22:16:45.327732 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:45.327705    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d91987a5-bc23-4976-a1c7-c10448df0e0e-tmp-dir\") pod \"test-model-e5a2d7-kserve-57bbd95c65-xjc9m\" (UID: \"d91987a5-bc23-4976-a1c7-c10448df0e0e\") " pod="ai-tenant-e2e-inf-a-e5a2d7/test-model-e5a2d7-kserve-57bbd95c65-xjc9m"
Sep 10 22:16:45.327799 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:45.327742    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d91987a5-bc23-4976-a1c7-c10448df0e0e-home\") pod \"test-model-e5a2d7-kserve-57bbd95c65-xjc9m\" (UID: \"d91987a5-bc23-4976-a1c7-c10448df0e0e\") " pod="ai-tenant-e2e-inf-a-e5a2d7/test-model-e5a2d7-kserve-57bbd95c65-xjc9m"
Sep 10 22:16:45.329882 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:45.329836    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d91987a5-bc23-4976-a1c7-c10448df0e0e-dshm\") pod \"test-model-e5a2d7-kserve-57bbd95c65-xjc9m\" (UID: \"d91987a5-bc23-4976-a1c7-c10448df0e0e\") " pod="ai-tenant-e2e-inf-a-e5a2d7/test-model-e5a2d7-kserve-57bbd95c65-xjc9m"
Sep 10 22:16:45.330109 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:45.330089    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d91987a5-bc23-4976-a1c7-c10448df0e0e-tls-certs\") pod \"test-model-e5a2d7-kserve-57bbd95c65-xjc9m\" (UID: \"d91987a5-bc23-4976-a1c7-c10448df0e0e\") " pod="ai-tenant-e2e-inf-a-e5a2d7/test-model-e5a2d7-kserve-57bbd95c65-xjc9m"
Sep 10 22:16:45.371925 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:45.371892    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljml4\" (UniqueName: \"kubernetes.io/projected/d91987a5-bc23-4976-a1c7-c10448df0e0e-kube-api-access-ljml4\") pod \"test-model-e5a2d7-kserve-57bbd95c65-xjc9m\" (UID: \"d91987a5-bc23-4976-a1c7-c10448df0e0e\") " pod="ai-tenant-e2e-inf-a-e5a2d7/test-model-e5a2d7-kserve-57bbd95c65-xjc9m"
Sep 10 22:16:45.436896 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:45.436842    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-e5a2d7/test-model-e5a2d7-kserve-57bbd95c65-xjc9m"
Sep 10 22:16:45.733118 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:45.733092    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-e5a2d7/test-model-e5a2d7-kserve-57bbd95c65-xjc9m"]
Sep 10 22:16:45.735625 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:16:45.735592    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd91987a5_bc23_4976_a1c7_c10448df0e0e.slice/crio-b8f645dd39e5c42dde5b598c43079cd9ae7259eb0f764a2653077169dd0a36eb WatchSource:0}: Error finding container b8f645dd39e5c42dde5b598c43079cd9ae7259eb0f764a2653077169dd0a36eb: Status 404 returned error can't find the container with id b8f645dd39e5c42dde5b598c43079cd9ae7259eb0f764a2653077169dd0a36eb
Sep 10 22:16:46.533275 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:46.533237    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-e5a2d7/test-model-e5a2d7-kserve-57bbd95c65-xjc9m" event={"ID":"d91987a5-bc23-4976-a1c7-c10448df0e0e","Type":"ContainerStarted","Data":"06c34155530371088ab7322d2cf6ac0f10b322f655d33a084fae7d09af2bb6f8"}
Sep 10 22:16:46.533713 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:46.533286    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-e5a2d7/test-model-e5a2d7-kserve-57bbd95c65-xjc9m" event={"ID":"d91987a5-bc23-4976-a1c7-c10448df0e0e","Type":"ContainerStarted","Data":"b8f645dd39e5c42dde5b598c43079cd9ae7259eb0f764a2653077169dd0a36eb"}
Sep 10 22:16:46.569472 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:46.569418    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-e5a2d7/test-model-e5a2d7-kserve-57bbd95c65-xjc9m" podStartSLOduration=1.569403249 podStartE2EDuration="1.569403249s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 22:16:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:16:46.568479325 +0000 UTC m=+927.524547031" watchObservedRunningTime="2026-09-10 22:16:46.569403249 +0000 UTC m=+927.525470955"
Sep 10 22:16:55.437563 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:55.437512    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-e5a2d7/test-model-e5a2d7-kserve-57bbd95c65-xjc9m"
Sep 10 22:16:55.438078 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:55.437578    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-e5a2d7/test-model-e5a2d7-kserve-57bbd95c65-xjc9m"
Sep 10 22:16:55.450367 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:55.450346    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-e5a2d7/test-model-e5a2d7-kserve-57bbd95c65-xjc9m"
Sep 10 22:16:55.585382 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:55.585352    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-e5a2d7/test-model-e5a2d7-kserve-57bbd95c65-xjc9m"
Sep 10 22:16:56.302418 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:56.302388    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-53efb5ef-587b54fc89-5fhv5"]
Sep 10 22:16:56.302648 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:56.302626    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-53efb5ef-587b54fc89-5fhv5" podUID="d08d579c-afec-4f8a-bc2c-bdb0c0852fe3" containerName="maas-api" containerID="cri-o://c54ecd83a4ff17b251de2e80bc69d3ca4f693ca0feb27868993a2bfcbf3084c8" gracePeriod=30
Sep 10 22:16:56.359559 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:56.359529    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-53efb5ef-79f88c7d9-697g5"]
Sep 10 22:16:56.359782 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:56.359758    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-53efb5ef-79f88c7d9-697g5" podUID="5d7b58cc-615b-48e2-a4ac-a2d779e57bda" containerName="payload-processing" containerID="cri-o://13fd04b674d16064e466304775f959851360cf1f54e24927d77ab86e30e563b3" gracePeriod=30
Sep 10 22:16:56.366162 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:56.366130    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-53efb5ef-595d844fd6-fm22t"]
Sep 10 22:16:56.366378 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:56.366352    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-53efb5ef-595d844fd6-fm22t" podUID="2a7ca5a3-381f-4693-875c-dfe5c9f5f827" containerName="payload-pre-processing" containerID="cri-o://b35d83b6e1602b12a066bcb7d40196c3cc7ed0d30f678ef8f143d96ef0324e04" gracePeriod=30
Sep 10 22:16:56.584414 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:56.584385    2582 generic.go:356] "Generic (PLEG): container finished" podID="d08d579c-afec-4f8a-bc2c-bdb0c0852fe3" containerID="c54ecd83a4ff17b251de2e80bc69d3ca4f693ca0feb27868993a2bfcbf3084c8" exitCode=0
Sep 10 22:16:56.584832 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:56.584454    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-53efb5ef-587b54fc89-5fhv5" event={"ID":"d08d579c-afec-4f8a-bc2c-bdb0c0852fe3","Type":"ContainerDied","Data":"c54ecd83a4ff17b251de2e80bc69d3ca4f693ca0feb27868993a2bfcbf3084c8"}
Sep 10 22:16:56.586430 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:56.586399    2582 generic.go:356] "Generic (PLEG): container finished" podID="5d7b58cc-615b-48e2-a4ac-a2d779e57bda" containerID="13fd04b674d16064e466304775f959851360cf1f54e24927d77ab86e30e563b3" exitCode=0
Sep 10 22:16:56.586555 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:56.586435    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-53efb5ef-79f88c7d9-697g5" event={"ID":"5d7b58cc-615b-48e2-a4ac-a2d779e57bda","Type":"ContainerDied","Data":"13fd04b674d16064e466304775f959851360cf1f54e24927d77ab86e30e563b3"}
Sep 10 22:16:56.588582 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:56.588549    2582 generic.go:356] "Generic (PLEG): container finished" podID="2a7ca5a3-381f-4693-875c-dfe5c9f5f827" containerID="b35d83b6e1602b12a066bcb7d40196c3cc7ed0d30f678ef8f143d96ef0324e04" exitCode=0
Sep 10 22:16:56.588686 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:56.588607    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-53efb5ef-595d844fd6-fm22t" event={"ID":"2a7ca5a3-381f-4693-875c-dfe5c9f5f827","Type":"ContainerDied","Data":"b35d83b6e1602b12a066bcb7d40196c3cc7ed0d30f678ef8f143d96ef0324e04"}
Sep 10 22:16:56.621098 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:56.621071    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-53efb5ef-79f88c7d9-697g5"
Sep 10 22:16:56.631449 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:56.631428    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-53efb5ef-587b54fc89-5fhv5"
Sep 10 22:16:56.639144 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:56.639118    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-53efb5ef-595d844fd6-fm22t"
Sep 10 22:16:56.722343 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:56.722308    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d08d579c-afec-4f8a-bc2c-bdb0c0852fe3-metrics-certs\") pod \"d08d579c-afec-4f8a-bc2c-bdb0c0852fe3\" (UID: \"d08d579c-afec-4f8a-bc2c-bdb0c0852fe3\") "
Sep 10 22:16:56.722343 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:56.722349    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d08d579c-afec-4f8a-bc2c-bdb0c0852fe3-openshift-service-ca\") pod \"d08d579c-afec-4f8a-bc2c-bdb0c0852fe3\" (UID: \"d08d579c-afec-4f8a-bc2c-bdb0c0852fe3\") "
Sep 10 22:16:56.722576 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:56.722368    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d7b58cc-615b-48e2-a4ac-a2d779e57bda-plugins-config-volume\") pod \"5d7b58cc-615b-48e2-a4ac-a2d779e57bda\" (UID: \"5d7b58cc-615b-48e2-a4ac-a2d779e57bda\") "
Sep 10 22:16:56.722576 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:56.722403    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d08d579c-afec-4f8a-bc2c-bdb0c0852fe3-maas-api-tls\") pod \"d08d579c-afec-4f8a-bc2c-bdb0c0852fe3\" (UID: \"d08d579c-afec-4f8a-bc2c-bdb0c0852fe3\") "
Sep 10 22:16:56.722576 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:56.722427    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a7ca5a3-381f-4693-875c-dfe5c9f5f827-plugins-config-volume\") pod \"2a7ca5a3-381f-4693-875c-dfe5c9f5f827\" (UID: \"2a7ca5a3-381f-4693-875c-dfe5c9f5f827\") "
Sep 10 22:16:56.722576 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:56.722458    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ckxnk\" (UniqueName: \"kubernetes.io/projected/2a7ca5a3-381f-4693-875c-dfe5c9f5f827-kube-api-access-ckxnk\") pod \"2a7ca5a3-381f-4693-875c-dfe5c9f5f827\" (UID: \"2a7ca5a3-381f-4693-875c-dfe5c9f5f827\") "
Sep 10 22:16:56.722576 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:56.722485    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xrwx4\" (UniqueName: \"kubernetes.io/projected/d08d579c-afec-4f8a-bc2c-bdb0c0852fe3-kube-api-access-xrwx4\") pod \"d08d579c-afec-4f8a-bc2c-bdb0c0852fe3\" (UID: \"d08d579c-afec-4f8a-bc2c-bdb0c0852fe3\") "
Sep 10 22:16:56.722576 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:56.722561    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cv9wh\" (UniqueName: \"kubernetes.io/projected/5d7b58cc-615b-48e2-a4ac-a2d779e57bda-kube-api-access-cv9wh\") pod \"5d7b58cc-615b-48e2-a4ac-a2d779e57bda\" (UID: \"5d7b58cc-615b-48e2-a4ac-a2d779e57bda\") "
Sep 10 22:16:56.722933 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:56.722759    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d08d579c-afec-4f8a-bc2c-bdb0c0852fe3-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d08d579c-afec-4f8a-bc2c-bdb0c0852fe3" (UID: "d08d579c-afec-4f8a-bc2c-bdb0c0852fe3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 22:16:56.722933 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:56.722799    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a7ca5a3-381f-4693-875c-dfe5c9f5f827-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2a7ca5a3-381f-4693-875c-dfe5c9f5f827" (UID: "2a7ca5a3-381f-4693-875c-dfe5c9f5f827"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 22:16:56.723048 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:56.722937    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a7ca5a3-381f-4693-875c-dfe5c9f5f827-plugins-config-volume\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:16:56.723048 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:56.722959    2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d08d579c-afec-4f8a-bc2c-bdb0c0852fe3-openshift-service-ca\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:16:56.723221 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:56.723190    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d7b58cc-615b-48e2-a4ac-a2d779e57bda-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5d7b58cc-615b-48e2-a4ac-a2d779e57bda" (UID: "5d7b58cc-615b-48e2-a4ac-a2d779e57bda"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 22:16:56.724948 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:56.724920    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d08d579c-afec-4f8a-bc2c-bdb0c0852fe3-kube-api-access-xrwx4" (OuterVolumeSpecName: "kube-api-access-xrwx4") pod "d08d579c-afec-4f8a-bc2c-bdb0c0852fe3" (UID: "d08d579c-afec-4f8a-bc2c-bdb0c0852fe3"). InnerVolumeSpecName "kube-api-access-xrwx4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 22:16:56.725112 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:56.725085    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d08d579c-afec-4f8a-bc2c-bdb0c0852fe3-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "d08d579c-afec-4f8a-bc2c-bdb0c0852fe3" (UID: "d08d579c-afec-4f8a-bc2c-bdb0c0852fe3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 22:16:56.725112 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:56.725094    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d08d579c-afec-4f8a-bc2c-bdb0c0852fe3-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d08d579c-afec-4f8a-bc2c-bdb0c0852fe3" (UID: "d08d579c-afec-4f8a-bc2c-bdb0c0852fe3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 22:16:56.725238 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:56.725144    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d7b58cc-615b-48e2-a4ac-a2d779e57bda-kube-api-access-cv9wh" (OuterVolumeSpecName: "kube-api-access-cv9wh") pod "5d7b58cc-615b-48e2-a4ac-a2d779e57bda" (UID: "5d7b58cc-615b-48e2-a4ac-a2d779e57bda"). InnerVolumeSpecName "kube-api-access-cv9wh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 22:16:56.725238 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:56.725174    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a7ca5a3-381f-4693-875c-dfe5c9f5f827-kube-api-access-ckxnk" (OuterVolumeSpecName: "kube-api-access-ckxnk") pod "2a7ca5a3-381f-4693-875c-dfe5c9f5f827" (UID: "2a7ca5a3-381f-4693-875c-dfe5c9f5f827"). InnerVolumeSpecName "kube-api-access-ckxnk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 22:16:56.824157 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:56.824125    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d08d579c-afec-4f8a-bc2c-bdb0c0852fe3-maas-api-tls\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:16:56.824157 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:56.824154    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ckxnk\" (UniqueName: \"kubernetes.io/projected/2a7ca5a3-381f-4693-875c-dfe5c9f5f827-kube-api-access-ckxnk\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:16:56.824157 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:56.824164    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrwx4\" (UniqueName: \"kubernetes.io/projected/d08d579c-afec-4f8a-bc2c-bdb0c0852fe3-kube-api-access-xrwx4\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:16:56.824351 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:56.824174    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cv9wh\" (UniqueName: \"kubernetes.io/projected/5d7b58cc-615b-48e2-a4ac-a2d779e57bda-kube-api-access-cv9wh\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:16:56.824351 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:56.824183    2582 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d08d579c-afec-4f8a-bc2c-bdb0c0852fe3-metrics-certs\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:16:56.824351 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:56.824191    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d7b58cc-615b-48e2-a4ac-a2d779e57bda-plugins-config-volume\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:16:57.595867 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:57.595761    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-53efb5ef-595d844fd6-fm22t"
Sep 10 22:16:57.596316 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:57.595760    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-53efb5ef-595d844fd6-fm22t" event={"ID":"2a7ca5a3-381f-4693-875c-dfe5c9f5f827","Type":"ContainerDied","Data":"f6259fcd7afd248fe4bf92a228f1dc4c3ce5aa049b1b3d2fb23c7c3e4a8fe72b"}
Sep 10 22:16:57.596316 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:57.595991    2582 scope.go:117] "RemoveContainer" containerID="b35d83b6e1602b12a066bcb7d40196c3cc7ed0d30f678ef8f143d96ef0324e04"
Sep 10 22:16:57.598240 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:57.598213    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-53efb5ef-587b54fc89-5fhv5" event={"ID":"d08d579c-afec-4f8a-bc2c-bdb0c0852fe3","Type":"ContainerDied","Data":"58797a2a3e5d0f75d1ed83fb3b141da2e1c8002de115efd52ffb913cbf514e4e"}
Sep 10 22:16:57.598366 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:57.598350    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-53efb5ef-587b54fc89-5fhv5"
Sep 10 22:16:57.600217 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:57.600189    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-53efb5ef-79f88c7d9-697g5" event={"ID":"5d7b58cc-615b-48e2-a4ac-a2d779e57bda","Type":"ContainerDied","Data":"445a025b7c87fdf9249cf58591e966c17edc008f680779d998f142c02488ea5d"}
Sep 10 22:16:57.600531 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:57.600509    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-53efb5ef-79f88c7d9-697g5"
Sep 10 22:16:57.606768 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:57.606736    2582 scope.go:117] "RemoveContainer" containerID="c54ecd83a4ff17b251de2e80bc69d3ca4f693ca0feb27868993a2bfcbf3084c8"
Sep 10 22:16:57.611166 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:57.611142    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-53efb5ef-openshift-default-7fdb9f589d-52p66"]
Sep 10 22:16:57.611369 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:57.611347    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-53efb5ef-openshift-default-7fdb9f589d-52p66" podUID="34ba450c-a70b-457d-8c7b-0be6ece0be1f" containerName="istio-proxy" containerID="cri-o://d1b6477c495b8e0eeed868956d748287aeb14bef31a14fb9b6bcbfa3a2a86ffb" gracePeriod=30
Sep 10 22:16:57.616444 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:57.616425    2582 scope.go:117] "RemoveContainer" containerID="13fd04b674d16064e466304775f959851360cf1f54e24927d77ab86e30e563b3"
Sep 10 22:16:57.631314 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:57.631219    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-53efb5ef-595d844fd6-fm22t"]
Sep 10 22:16:57.633612 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:57.633588    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-53efb5ef-595d844fd6-fm22t"]
Sep 10 22:16:57.667352 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:57.667324    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-53efb5ef-587b54fc89-5fhv5"]
Sep 10 22:16:57.671956 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:57.671931    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-53efb5ef-587b54fc89-5fhv5"]
Sep 10 22:16:57.682867 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:57.682823    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-53efb5ef-79f88c7d9-697g5"]
Sep 10 22:16:57.686609 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:57.686586    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-53efb5ef-79f88c7d9-697g5"]
Sep 10 22:16:59.499885 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:59.499833    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2a7ca5a3-381f-4693-875c-dfe5c9f5f827" path="/var/lib/kubelet/pods/2a7ca5a3-381f-4693-875c-dfe5c9f5f827/volumes"
Sep 10 22:16:59.500243 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:59.500208    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5d7b58cc-615b-48e2-a4ac-a2d779e57bda" path="/var/lib/kubelet/pods/5d7b58cc-615b-48e2-a4ac-a2d779e57bda/volumes"
Sep 10 22:16:59.500547 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:16:59.500534    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d08d579c-afec-4f8a-bc2c-bdb0c0852fe3" path="/var/lib/kubelet/pods/d08d579c-afec-4f8a-bc2c-bdb0c0852fe3/volumes"
Sep 10 22:17:02.866139 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:02.866111    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-53efb5ef-openshift-default-7fdb9f589d-52p66"
Sep 10 22:17:02.978676 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:02.978635    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34ba450c-a70b-457d-8c7b-0be6ece0be1f-istio-envoy\") pod \"34ba450c-a70b-457d-8c7b-0be6ece0be1f\" (UID: \"34ba450c-a70b-457d-8c7b-0be6ece0be1f\") "
Sep 10 22:17:02.978913 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:02.978693    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34ba450c-a70b-457d-8c7b-0be6ece0be1f-workload-socket\") pod \"34ba450c-a70b-457d-8c7b-0be6ece0be1f\" (UID: \"34ba450c-a70b-457d-8c7b-0be6ece0be1f\") "
Sep 10 22:17:02.978913 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:02.978718    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34ba450c-a70b-457d-8c7b-0be6ece0be1f-istio-token\") pod \"34ba450c-a70b-457d-8c7b-0be6ece0be1f\" (UID: \"34ba450c-a70b-457d-8c7b-0be6ece0be1f\") "
Sep 10 22:17:02.978913 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:02.978751    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34ba450c-a70b-457d-8c7b-0be6ece0be1f-istio-data\") pod \"34ba450c-a70b-457d-8c7b-0be6ece0be1f\" (UID: \"34ba450c-a70b-457d-8c7b-0be6ece0be1f\") "
Sep 10 22:17:02.978913 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:02.978766    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mhrtj\" (UniqueName: \"kubernetes.io/projected/34ba450c-a70b-457d-8c7b-0be6ece0be1f-kube-api-access-mhrtj\") pod \"34ba450c-a70b-457d-8c7b-0be6ece0be1f\" (UID: \"34ba450c-a70b-457d-8c7b-0be6ece0be1f\") "
Sep 10 22:17:02.979151 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:02.978931    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34ba450c-a70b-457d-8c7b-0be6ece0be1f-istio-podinfo\") pod \"34ba450c-a70b-457d-8c7b-0be6ece0be1f\" (UID: \"34ba450c-a70b-457d-8c7b-0be6ece0be1f\") "
Sep 10 22:17:02.979151 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:02.979036    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34ba450c-a70b-457d-8c7b-0be6ece0be1f-credential-socket\") pod \"34ba450c-a70b-457d-8c7b-0be6ece0be1f\" (UID: \"34ba450c-a70b-457d-8c7b-0be6ece0be1f\") "
Sep 10 22:17:02.979151 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:02.979072    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34ba450c-a70b-457d-8c7b-0be6ece0be1f-istiod-ca-cert\") pod \"34ba450c-a70b-457d-8c7b-0be6ece0be1f\" (UID: \"34ba450c-a70b-457d-8c7b-0be6ece0be1f\") "
Sep 10 22:17:02.979151 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:02.979097    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34ba450c-a70b-457d-8c7b-0be6ece0be1f-workload-certs\") pod \"34ba450c-a70b-457d-8c7b-0be6ece0be1f\" (UID: \"34ba450c-a70b-457d-8c7b-0be6ece0be1f\") "
Sep 10 22:17:02.979393 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:02.979087    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34ba450c-a70b-457d-8c7b-0be6ece0be1f-istio-data" (OuterVolumeSpecName: "istio-data") pod "34ba450c-a70b-457d-8c7b-0be6ece0be1f" (UID: "34ba450c-a70b-457d-8c7b-0be6ece0be1f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 22:17:02.979436 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:02.979247    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34ba450c-a70b-457d-8c7b-0be6ece0be1f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "34ba450c-a70b-457d-8c7b-0be6ece0be1f" (UID: "34ba450c-a70b-457d-8c7b-0be6ece0be1f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 22:17:02.979436 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:02.979280    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34ba450c-a70b-457d-8c7b-0be6ece0be1f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "34ba450c-a70b-457d-8c7b-0be6ece0be1f" (UID: "34ba450c-a70b-457d-8c7b-0be6ece0be1f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 22:17:02.979436 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:02.979349    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34ba450c-a70b-457d-8c7b-0be6ece0be1f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "34ba450c-a70b-457d-8c7b-0be6ece0be1f" (UID: "34ba450c-a70b-457d-8c7b-0be6ece0be1f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 22:17:02.979559 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:02.979435    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34ba450c-a70b-457d-8c7b-0be6ece0be1f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "34ba450c-a70b-457d-8c7b-0be6ece0be1f" (UID: "34ba450c-a70b-457d-8c7b-0be6ece0be1f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 22:17:02.981176 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:02.981138    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34ba450c-a70b-457d-8c7b-0be6ece0be1f-istio-token" (OuterVolumeSpecName: "istio-token") pod "34ba450c-a70b-457d-8c7b-0be6ece0be1f" (UID: "34ba450c-a70b-457d-8c7b-0be6ece0be1f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 22:17:02.981295 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:02.981248    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/34ba450c-a70b-457d-8c7b-0be6ece0be1f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "34ba450c-a70b-457d-8c7b-0be6ece0be1f" (UID: "34ba450c-a70b-457d-8c7b-0be6ece0be1f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 22:17:02.981420 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:02.981400    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34ba450c-a70b-457d-8c7b-0be6ece0be1f-kube-api-access-mhrtj" (OuterVolumeSpecName: "kube-api-access-mhrtj") pod "34ba450c-a70b-457d-8c7b-0be6ece0be1f" (UID: "34ba450c-a70b-457d-8c7b-0be6ece0be1f"). InnerVolumeSpecName "kube-api-access-mhrtj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 22:17:02.981472 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:02.981445    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34ba450c-a70b-457d-8c7b-0be6ece0be1f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "34ba450c-a70b-457d-8c7b-0be6ece0be1f" (UID: "34ba450c-a70b-457d-8c7b-0be6ece0be1f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 22:17:03.078941 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:03.078905    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-4149405d-7d474f89f4-c7sfd"]
Sep 10 22:17:03.079469 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:03.079452    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d08d579c-afec-4f8a-bc2c-bdb0c0852fe3" containerName="maas-api"
Sep 10 22:17:03.079523 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:03.079473    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="d08d579c-afec-4f8a-bc2c-bdb0c0852fe3" containerName="maas-api"
Sep 10 22:17:03.079523 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:03.079487    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34ba450c-a70b-457d-8c7b-0be6ece0be1f" containerName="istio-proxy"
Sep 10 22:17:03.079523 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:03.079495    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="34ba450c-a70b-457d-8c7b-0be6ece0be1f" containerName="istio-proxy"
Sep 10 22:17:03.079523 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:03.079511    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34ba450c-a70b-457d-8c7b-0be6ece0be1f" containerName="istio-proxy"
Sep 10 22:17:03.079523 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:03.079519    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="34ba450c-a70b-457d-8c7b-0be6ece0be1f" containerName="istio-proxy"
Sep 10 22:17:03.079680 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:03.079536    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a7ca5a3-381f-4693-875c-dfe5c9f5f827" containerName="payload-pre-processing"
Sep 10 22:17:03.079680 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:03.079545    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a7ca5a3-381f-4693-875c-dfe5c9f5f827" containerName="payload-pre-processing"
Sep 10 22:17:03.079680 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:03.079555    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d7b58cc-615b-48e2-a4ac-a2d779e57bda" containerName="payload-processing"
Sep 10 22:17:03.079680 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:03.079564    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d7b58cc-615b-48e2-a4ac-a2d779e57bda" containerName="payload-processing"
Sep 10 22:17:03.079680 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:03.079639    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="34ba450c-a70b-457d-8c7b-0be6ece0be1f" containerName="istio-proxy"
Sep 10 22:17:03.079680 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:03.079651    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a7ca5a3-381f-4693-875c-dfe5c9f5f827" containerName="payload-pre-processing"
Sep 10 22:17:03.079680 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:03.079659    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d7b58cc-615b-48e2-a4ac-a2d779e57bda" containerName="payload-processing"
Sep 10 22:17:03.079680 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:03.079668    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="34ba450c-a70b-457d-8c7b-0be6ece0be1f" containerName="istio-proxy"
Sep 10 22:17:03.079680 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:03.079678    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="d08d579c-afec-4f8a-bc2c-bdb0c0852fe3" containerName="maas-api"
Sep 10 22:17:03.080655 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:03.080627    2582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34ba450c-a70b-457d-8c7b-0be6ece0be1f-workload-socket\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:17:03.080655 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:03.080651    2582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34ba450c-a70b-457d-8c7b-0be6ece0be1f-istio-token\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:17:03.080655 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:03.080660    2582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34ba450c-a70b-457d-8c7b-0be6ece0be1f-istio-data\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:17:03.080934 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:03.080668    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhrtj\" (UniqueName: \"kubernetes.io/projected/34ba450c-a70b-457d-8c7b-0be6ece0be1f-kube-api-access-mhrtj\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:17:03.080934 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:03.080677    2582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34ba450c-a70b-457d-8c7b-0be6ece0be1f-istio-podinfo\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:17:03.080934 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:03.080685    2582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34ba450c-a70b-457d-8c7b-0be6ece0be1f-credential-socket\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:17:03.080934 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:03.080695    2582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34ba450c-a70b-457d-8c7b-0be6ece0be1f-istiod-ca-cert\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:17:03.080934 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:03.080703    2582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34ba450c-a70b-457d-8c7b-0be6ece0be1f-workload-certs\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:17:03.080934 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:03.080711    2582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34ba450c-a70b-457d-8c7b-0be6ece0be1f-istio-envoy\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:17:03.082240 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:03.082218    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4149405d-7d474f89f4-c7sfd"
Sep 10 22:17:03.089691 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:03.089666    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-4149405d-serving-cert\""
Sep 10 22:17:03.118130 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:03.118104    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-4149405d-7d474f89f4-c7sfd"]
Sep 10 22:17:03.181969 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:03.181934    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sr8x6\" (UniqueName: \"kubernetes.io/projected/726a426b-3549-486f-b8c8-ba1542f8557c-kube-api-access-sr8x6\") pod \"maas-api-e2e-derive-4149405d-7d474f89f4-c7sfd\" (UID: \"726a426b-3549-486f-b8c8-ba1542f8557c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4149405d-7d474f89f4-c7sfd"
Sep 10 22:17:03.182138 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:03.181983    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/726a426b-3549-486f-b8c8-ba1542f8557c-metrics-certs\") pod \"maas-api-e2e-derive-4149405d-7d474f89f4-c7sfd\" (UID: \"726a426b-3549-486f-b8c8-ba1542f8557c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4149405d-7d474f89f4-c7sfd"
Sep 10 22:17:03.182138 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:03.182090    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/726a426b-3549-486f-b8c8-ba1542f8557c-maas-api-tls\") pod \"maas-api-e2e-derive-4149405d-7d474f89f4-c7sfd\" (UID: \"726a426b-3549-486f-b8c8-ba1542f8557c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4149405d-7d474f89f4-c7sfd"
Sep 10 22:17:03.182231 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:03.182136    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/726a426b-3549-486f-b8c8-ba1542f8557c-openshift-service-ca\") pod \"maas-api-e2e-derive-4149405d-7d474f89f4-c7sfd\" (UID: \"726a426b-3549-486f-b8c8-ba1542f8557c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4149405d-7d474f89f4-c7sfd"
Sep 10 22:17:03.283165 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:03.283072    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/726a426b-3549-486f-b8c8-ba1542f8557c-metrics-certs\") pod \"maas-api-e2e-derive-4149405d-7d474f89f4-c7sfd\" (UID: \"726a426b-3549-486f-b8c8-ba1542f8557c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4149405d-7d474f89f4-c7sfd"
Sep 10 22:17:03.283165 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:03.283136    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/726a426b-3549-486f-b8c8-ba1542f8557c-maas-api-tls\") pod \"maas-api-e2e-derive-4149405d-7d474f89f4-c7sfd\" (UID: \"726a426b-3549-486f-b8c8-ba1542f8557c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4149405d-7d474f89f4-c7sfd"
Sep 10 22:17:03.283165 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:03.283167    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/726a426b-3549-486f-b8c8-ba1542f8557c-openshift-service-ca\") pod \"maas-api-e2e-derive-4149405d-7d474f89f4-c7sfd\" (UID: \"726a426b-3549-486f-b8c8-ba1542f8557c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4149405d-7d474f89f4-c7sfd"
Sep 10 22:17:03.283386 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:03.283199    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sr8x6\" (UniqueName: \"kubernetes.io/projected/726a426b-3549-486f-b8c8-ba1542f8557c-kube-api-access-sr8x6\") pod \"maas-api-e2e-derive-4149405d-7d474f89f4-c7sfd\" (UID: \"726a426b-3549-486f-b8c8-ba1542f8557c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4149405d-7d474f89f4-c7sfd"
Sep 10 22:17:03.283386 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:17:03.283304    2582 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-4149405d-serving-cert: secret "maas-api-e2e-derive-4149405d-serving-cert" not found
Sep 10 22:17:03.283386 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:17:03.283381    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/726a426b-3549-486f-b8c8-ba1542f8557c-maas-api-tls podName:726a426b-3549-486f-b8c8-ba1542f8557c nodeName:}" failed. No retries permitted until 2026-09-10 22:17:03.783364203 +0000 UTC m=+944.739431889 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/726a426b-3549-486f-b8c8-ba1542f8557c-maas-api-tls") pod "maas-api-e2e-derive-4149405d-7d474f89f4-c7sfd" (UID: "726a426b-3549-486f-b8c8-ba1542f8557c") : secret "maas-api-e2e-derive-4149405d-serving-cert" not found
Sep 10 22:17:03.283906 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:03.283889    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/726a426b-3549-486f-b8c8-ba1542f8557c-openshift-service-ca\") pod \"maas-api-e2e-derive-4149405d-7d474f89f4-c7sfd\" (UID: \"726a426b-3549-486f-b8c8-ba1542f8557c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4149405d-7d474f89f4-c7sfd"
Sep 10 22:17:03.285497 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:03.285479    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/726a426b-3549-486f-b8c8-ba1542f8557c-metrics-certs\") pod \"maas-api-e2e-derive-4149405d-7d474f89f4-c7sfd\" (UID: \"726a426b-3549-486f-b8c8-ba1542f8557c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4149405d-7d474f89f4-c7sfd"
Sep 10 22:17:03.294002 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:03.293979    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sr8x6\" (UniqueName: \"kubernetes.io/projected/726a426b-3549-486f-b8c8-ba1542f8557c-kube-api-access-sr8x6\") pod \"maas-api-e2e-derive-4149405d-7d474f89f4-c7sfd\" (UID: \"726a426b-3549-486f-b8c8-ba1542f8557c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4149405d-7d474f89f4-c7sfd"
Sep 10 22:17:03.638071 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:03.637987    2582 generic.go:356] "Generic (PLEG): container finished" podID="34ba450c-a70b-457d-8c7b-0be6ece0be1f" containerID="d1b6477c495b8e0eeed868956d748287aeb14bef31a14fb9b6bcbfa3a2a86ffb" exitCode=0
Sep 10 22:17:03.638071 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:03.638041    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-53efb5ef-openshift-default-7fdb9f589d-52p66"
Sep 10 22:17:03.638259 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:03.638076    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-53efb5ef-openshift-default-7fdb9f589d-52p66" event={"ID":"34ba450c-a70b-457d-8c7b-0be6ece0be1f","Type":"ContainerDied","Data":"d1b6477c495b8e0eeed868956d748287aeb14bef31a14fb9b6bcbfa3a2a86ffb"}
Sep 10 22:17:03.638259 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:03.638132    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-53efb5ef-openshift-default-7fdb9f589d-52p66" event={"ID":"34ba450c-a70b-457d-8c7b-0be6ece0be1f","Type":"ContainerDied","Data":"ad6854512e5d22a7e0fe72c01a1e13b8744a68158e06ef1d8a7648077b33ee30"}
Sep 10 22:17:03.638259 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:03.638153    2582 scope.go:117] "RemoveContainer" containerID="d1b6477c495b8e0eeed868956d748287aeb14bef31a14fb9b6bcbfa3a2a86ffb"
Sep 10 22:17:03.647159 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:03.647113    2582 scope.go:117] "RemoveContainer" containerID="143a4abf622aa86c504ae0185b152a10c81517451ad5642730c08f19e40ba50e"
Sep 10 22:17:03.654800 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:03.654782    2582 scope.go:117] "RemoveContainer" containerID="d1b6477c495b8e0eeed868956d748287aeb14bef31a14fb9b6bcbfa3a2a86ffb"
Sep 10 22:17:03.655132 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:17:03.655113    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1b6477c495b8e0eeed868956d748287aeb14bef31a14fb9b6bcbfa3a2a86ffb\": container with ID starting with d1b6477c495b8e0eeed868956d748287aeb14bef31a14fb9b6bcbfa3a2a86ffb not found: ID does not exist" containerID="d1b6477c495b8e0eeed868956d748287aeb14bef31a14fb9b6bcbfa3a2a86ffb"
Sep 10 22:17:03.655195 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:03.655141    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1b6477c495b8e0eeed868956d748287aeb14bef31a14fb9b6bcbfa3a2a86ffb"} err="failed to get container status \"d1b6477c495b8e0eeed868956d748287aeb14bef31a14fb9b6bcbfa3a2a86ffb\": rpc error: code = NotFound desc = could not find container \"d1b6477c495b8e0eeed868956d748287aeb14bef31a14fb9b6bcbfa3a2a86ffb\": container with ID starting with d1b6477c495b8e0eeed868956d748287aeb14bef31a14fb9b6bcbfa3a2a86ffb not found: ID does not exist"
Sep 10 22:17:03.655195 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:03.655165    2582 scope.go:117] "RemoveContainer" containerID="143a4abf622aa86c504ae0185b152a10c81517451ad5642730c08f19e40ba50e"
Sep 10 22:17:03.655425 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:17:03.655402    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"143a4abf622aa86c504ae0185b152a10c81517451ad5642730c08f19e40ba50e\": container with ID starting with 143a4abf622aa86c504ae0185b152a10c81517451ad5642730c08f19e40ba50e not found: ID does not exist" containerID="143a4abf622aa86c504ae0185b152a10c81517451ad5642730c08f19e40ba50e"
Sep 10 22:17:03.655529 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:03.655430    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"143a4abf622aa86c504ae0185b152a10c81517451ad5642730c08f19e40ba50e"} err="failed to get container status \"143a4abf622aa86c504ae0185b152a10c81517451ad5642730c08f19e40ba50e\": rpc error: code = NotFound desc = could not find container \"143a4abf622aa86c504ae0185b152a10c81517451ad5642730c08f19e40ba50e\": container with ID starting with 143a4abf622aa86c504ae0185b152a10c81517451ad5642730c08f19e40ba50e not found: ID does not exist"
Sep 10 22:17:03.656548 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:03.656529    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-53efb5ef-openshift-default-7fdb9f589d-52p66"]
Sep 10 22:17:03.661987 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:03.661966    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-53efb5ef-openshift-default-7fdb9f589d-52p66"]
Sep 10 22:17:03.788046 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:03.788011    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/726a426b-3549-486f-b8c8-ba1542f8557c-maas-api-tls\") pod \"maas-api-e2e-derive-4149405d-7d474f89f4-c7sfd\" (UID: \"726a426b-3549-486f-b8c8-ba1542f8557c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4149405d-7d474f89f4-c7sfd"
Sep 10 22:17:03.790669 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:03.790638    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/726a426b-3549-486f-b8c8-ba1542f8557c-maas-api-tls\") pod \"maas-api-e2e-derive-4149405d-7d474f89f4-c7sfd\" (UID: \"726a426b-3549-486f-b8c8-ba1542f8557c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4149405d-7d474f89f4-c7sfd"
Sep 10 22:17:03.900784 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:03.900699    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-4149405d-6b5dcf9dcc-dxbf4"]
Sep 10 22:17:03.904175 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:03.904151    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-4149405d-6b5dcf9dcc-dxbf4"
Sep 10 22:17:03.906802 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:03.906783    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-4149405d-dockercfg-6shs9\""
Sep 10 22:17:03.907128 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:03.907110    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-4149405d\""
Sep 10 22:17:03.919912 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:03.919888    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-4149405d-6b5dcf9dcc-dxbf4"]
Sep 10 22:17:03.952201 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:03.952171    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-4149405d-5bbcdcdfc4-btfqc"]
Sep 10 22:17:03.954555 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:03.954540    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-4149405d-5bbcdcdfc4-btfqc"
Sep 10 22:17:03.966447 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:03.966423    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-4149405d-5bbcdcdfc4-btfqc"]
Sep 10 22:17:03.990790 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:03.990757    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f043662-4ea5-4b65-b3be-ab66d5d3696b-plugins-config-volume\") pod \"payload-processing-e2e-derive-4149405d-6b5dcf9dcc-dxbf4\" (UID: \"6f043662-4ea5-4b65-b3be-ab66d5d3696b\") " pod="openshift-ingress/payload-processing-e2e-derive-4149405d-6b5dcf9dcc-dxbf4"
Sep 10 22:17:03.990955 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:03.990815    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03b96f6e-5021-4459-8031-015a0f3aef66-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-4149405d-5bbcdcdfc4-btfqc\" (UID: \"03b96f6e-5021-4459-8031-015a0f3aef66\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-4149405d-5bbcdcdfc4-btfqc"
Sep 10 22:17:03.990955 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:03.990913    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8tlq\" (UniqueName: \"kubernetes.io/projected/6f043662-4ea5-4b65-b3be-ab66d5d3696b-kube-api-access-m8tlq\") pod \"payload-processing-e2e-derive-4149405d-6b5dcf9dcc-dxbf4\" (UID: \"6f043662-4ea5-4b65-b3be-ab66d5d3696b\") " pod="openshift-ingress/payload-processing-e2e-derive-4149405d-6b5dcf9dcc-dxbf4"
Sep 10 22:17:03.991037 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:03.990978    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2m6k\" (UniqueName: \"kubernetes.io/projected/03b96f6e-5021-4459-8031-015a0f3aef66-kube-api-access-b2m6k\") pod \"payload-pre-processing-e2e-derive-4149405d-5bbcdcdfc4-btfqc\" (UID: \"03b96f6e-5021-4459-8031-015a0f3aef66\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-4149405d-5bbcdcdfc4-btfqc"
Sep 10 22:17:03.994563 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:03.994534    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4149405d-7d474f89f4-c7sfd"
Sep 10 22:17:04.091663 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:04.091620    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2m6k\" (UniqueName: \"kubernetes.io/projected/03b96f6e-5021-4459-8031-015a0f3aef66-kube-api-access-b2m6k\") pod \"payload-pre-processing-e2e-derive-4149405d-5bbcdcdfc4-btfqc\" (UID: \"03b96f6e-5021-4459-8031-015a0f3aef66\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-4149405d-5bbcdcdfc4-btfqc"
Sep 10 22:17:04.091881 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:04.091718    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f043662-4ea5-4b65-b3be-ab66d5d3696b-plugins-config-volume\") pod \"payload-processing-e2e-derive-4149405d-6b5dcf9dcc-dxbf4\" (UID: \"6f043662-4ea5-4b65-b3be-ab66d5d3696b\") " pod="openshift-ingress/payload-processing-e2e-derive-4149405d-6b5dcf9dcc-dxbf4"
Sep 10 22:17:04.091881 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:04.091752    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03b96f6e-5021-4459-8031-015a0f3aef66-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-4149405d-5bbcdcdfc4-btfqc\" (UID: \"03b96f6e-5021-4459-8031-015a0f3aef66\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-4149405d-5bbcdcdfc4-btfqc"
Sep 10 22:17:04.091881 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:04.091777    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8tlq\" (UniqueName: \"kubernetes.io/projected/6f043662-4ea5-4b65-b3be-ab66d5d3696b-kube-api-access-m8tlq\") pod \"payload-processing-e2e-derive-4149405d-6b5dcf9dcc-dxbf4\" (UID: \"6f043662-4ea5-4b65-b3be-ab66d5d3696b\") " pod="openshift-ingress/payload-processing-e2e-derive-4149405d-6b5dcf9dcc-dxbf4"
Sep 10 22:17:04.092539 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:04.092516    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03b96f6e-5021-4459-8031-015a0f3aef66-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-4149405d-5bbcdcdfc4-btfqc\" (UID: \"03b96f6e-5021-4459-8031-015a0f3aef66\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-4149405d-5bbcdcdfc4-btfqc"
Sep 10 22:17:04.092629 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:04.092572    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f043662-4ea5-4b65-b3be-ab66d5d3696b-plugins-config-volume\") pod \"payload-processing-e2e-derive-4149405d-6b5dcf9dcc-dxbf4\" (UID: \"6f043662-4ea5-4b65-b3be-ab66d5d3696b\") " pod="openshift-ingress/payload-processing-e2e-derive-4149405d-6b5dcf9dcc-dxbf4"
Sep 10 22:17:04.101120 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:04.101092    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2m6k\" (UniqueName: \"kubernetes.io/projected/03b96f6e-5021-4459-8031-015a0f3aef66-kube-api-access-b2m6k\") pod \"payload-pre-processing-e2e-derive-4149405d-5bbcdcdfc4-btfqc\" (UID: \"03b96f6e-5021-4459-8031-015a0f3aef66\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-4149405d-5bbcdcdfc4-btfqc"
Sep 10 22:17:04.101405 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:04.101380    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8tlq\" (UniqueName: \"kubernetes.io/projected/6f043662-4ea5-4b65-b3be-ab66d5d3696b-kube-api-access-m8tlq\") pod \"payload-processing-e2e-derive-4149405d-6b5dcf9dcc-dxbf4\" (UID: \"6f043662-4ea5-4b65-b3be-ab66d5d3696b\") " pod="openshift-ingress/payload-processing-e2e-derive-4149405d-6b5dcf9dcc-dxbf4"
Sep 10 22:17:04.214541 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:04.214511    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-4149405d-6b5dcf9dcc-dxbf4"
Sep 10 22:17:04.263716 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:04.263681    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-4149405d-5bbcdcdfc4-btfqc"
Sep 10 22:17:04.266315 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:04.265925    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-4149405d-7d474f89f4-c7sfd"]
Sep 10 22:17:04.271311 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:17:04.271282    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod726a426b_3549_486f_b8c8_ba1542f8557c.slice/crio-72c88a2bddc0720b4b42780ec961f1056479126859e4400c7a1eb6019fc75091 WatchSource:0}: Error finding container 72c88a2bddc0720b4b42780ec961f1056479126859e4400c7a1eb6019fc75091: Status 404 returned error can't find the container with id 72c88a2bddc0720b4b42780ec961f1056479126859e4400c7a1eb6019fc75091
Sep 10 22:17:04.486069 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:04.486045    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-4149405d-6b5dcf9dcc-dxbf4"]
Sep 10 22:17:04.487682 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:17:04.487653    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f043662_4ea5_4b65_b3be_ab66d5d3696b.slice/crio-153744fc95ac68a64274ef766f1c464ad76e4140963a93776b5a0d165f332484 WatchSource:0}: Error finding container 153744fc95ac68a64274ef766f1c464ad76e4140963a93776b5a0d165f332484: Status 404 returned error can't find the container with id 153744fc95ac68a64274ef766f1c464ad76e4140963a93776b5a0d165f332484
Sep 10 22:17:04.545122 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:04.545097    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-4149405d-5bbcdcdfc4-btfqc"]
Sep 10 22:17:04.547157 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:17:04.547133    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03b96f6e_5021_4459_8031_015a0f3aef66.slice/crio-6ac08428e9e1a5a4f9db6f4265b82fc5ec32d1da3e9455ccef6aa48503158fa4 WatchSource:0}: Error finding container 6ac08428e9e1a5a4f9db6f4265b82fc5ec32d1da3e9455ccef6aa48503158fa4: Status 404 returned error can't find the container with id 6ac08428e9e1a5a4f9db6f4265b82fc5ec32d1da3e9455ccef6aa48503158fa4
Sep 10 22:17:04.648840 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:04.648802    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-4149405d-6b5dcf9dcc-dxbf4" event={"ID":"6f043662-4ea5-4b65-b3be-ab66d5d3696b","Type":"ContainerStarted","Data":"60331a501c0aa756f7ce3d49dff6d0ebcdf62b0efd708045593dc15805dbf11c"}
Sep 10 22:17:04.649043 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:04.648881    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-4149405d-6b5dcf9dcc-dxbf4" event={"ID":"6f043662-4ea5-4b65-b3be-ab66d5d3696b","Type":"ContainerStarted","Data":"153744fc95ac68a64274ef766f1c464ad76e4140963a93776b5a0d165f332484"}
Sep 10 22:17:04.649043 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:04.649002    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-4149405d-6b5dcf9dcc-dxbf4"
Sep 10 22:17:04.650422 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:04.650396    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-4149405d-5bbcdcdfc4-btfqc" event={"ID":"03b96f6e-5021-4459-8031-015a0f3aef66","Type":"ContainerStarted","Data":"94e4ec84ed633500999198aee1e220b8704755eee22d06b338c2c8f1880421bb"}
Sep 10 22:17:04.650561 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:04.650448    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-4149405d-5bbcdcdfc4-btfqc"
Sep 10 22:17:04.650561 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:04.650473    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-4149405d-5bbcdcdfc4-btfqc" event={"ID":"03b96f6e-5021-4459-8031-015a0f3aef66","Type":"ContainerStarted","Data":"6ac08428e9e1a5a4f9db6f4265b82fc5ec32d1da3e9455ccef6aa48503158fa4"}
Sep 10 22:17:04.651643 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:04.651619    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4149405d-7d474f89f4-c7sfd" event={"ID":"726a426b-3549-486f-b8c8-ba1542f8557c","Type":"ContainerStarted","Data":"72c88a2bddc0720b4b42780ec961f1056479126859e4400c7a1eb6019fc75091"}
Sep 10 22:17:04.667783 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:04.667741    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-4149405d-6b5dcf9dcc-dxbf4" podStartSLOduration=1.66772638 podStartE2EDuration="1.66772638s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 22:17:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:17:04.665504277 +0000 UTC m=+945.621572020" watchObservedRunningTime="2026-09-10 22:17:04.66772638 +0000 UTC m=+945.623794086"
Sep 10 22:17:04.680343 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:04.680184    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-4149405d-5bbcdcdfc4-btfqc" podStartSLOduration=1.6801670560000002 podStartE2EDuration="1.680167056s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 22:17:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:17:04.680049166 +0000 UTC m=+945.636116873" watchObservedRunningTime="2026-09-10 22:17:04.680167056 +0000 UTC m=+945.636234762"
Sep 10 22:17:05.500124 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:05.500092    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="34ba450c-a70b-457d-8c7b-0be6ece0be1f" path="/var/lib/kubelet/pods/34ba450c-a70b-457d-8c7b-0be6ece0be1f/volumes"
Sep 10 22:17:05.657900 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:05.657840    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4149405d-7d474f89f4-c7sfd" event={"ID":"726a426b-3549-486f-b8c8-ba1542f8557c","Type":"ContainerStarted","Data":"2aef7cdbae6913a5c3a2fd2547b7dfb3ae502f2300fe78fdbda8996b4331d099"}
Sep 10 22:17:05.658052 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:05.657925    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4149405d-7d474f89f4-c7sfd"
Sep 10 22:17:05.675045 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:05.674986    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4149405d-7d474f89f4-c7sfd" podStartSLOduration=2.266494321 podStartE2EDuration="2.674968526s" totalImagesPullingTime="408.474205ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 22:17:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:17:05.673942069 +0000 UTC m=+946.630009775" watchObservedRunningTime="2026-09-10 22:17:05.674968526 +0000 UTC m=+946.631036232"
Sep 10 22:17:11.670007 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:11.669977    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4149405d-7d474f89f4-c7sfd"
Sep 10 22:17:13.300543 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:13.300507    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-conflict-b-e5a626b8-kserve-66559ccb54-b44ch"]
Sep 10 22:17:13.304712 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:13.304691    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-e5a626b8-kserve-66559ccb54-b44ch"
Sep 10 22:17:13.306803 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:13.306775    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-conflict-b-e5a626b8-kserve-self-signed-certs\""
Sep 10 22:17:13.311210 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:13.311186    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-b-e5a626b8-kserve-66559ccb54-b44ch"]
Sep 10 22:17:13.378281 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:13.378242    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee68b40d-907e-45fb-a27b-d8b79eae19bc-home\") pod \"e2e-conflict-b-e5a626b8-kserve-66559ccb54-b44ch\" (UID: \"ee68b40d-907e-45fb-a27b-d8b79eae19bc\") " pod="llm/e2e-conflict-b-e5a626b8-kserve-66559ccb54-b44ch"
Sep 10 22:17:13.378281 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:13.378284    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9c6l\" (UniqueName: \"kubernetes.io/projected/ee68b40d-907e-45fb-a27b-d8b79eae19bc-kube-api-access-j9c6l\") pod \"e2e-conflict-b-e5a626b8-kserve-66559ccb54-b44ch\" (UID: \"ee68b40d-907e-45fb-a27b-d8b79eae19bc\") " pod="llm/e2e-conflict-b-e5a626b8-kserve-66559ccb54-b44ch"
Sep 10 22:17:13.378503 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:13.378348    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee68b40d-907e-45fb-a27b-d8b79eae19bc-tmp-dir\") pod \"e2e-conflict-b-e5a626b8-kserve-66559ccb54-b44ch\" (UID: \"ee68b40d-907e-45fb-a27b-d8b79eae19bc\") " pod="llm/e2e-conflict-b-e5a626b8-kserve-66559ccb54-b44ch"
Sep 10 22:17:13.378503 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:13.378386    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee68b40d-907e-45fb-a27b-d8b79eae19bc-model-cache\") pod \"e2e-conflict-b-e5a626b8-kserve-66559ccb54-b44ch\" (UID: \"ee68b40d-907e-45fb-a27b-d8b79eae19bc\") " pod="llm/e2e-conflict-b-e5a626b8-kserve-66559ccb54-b44ch"
Sep 10 22:17:13.378503 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:13.378453    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee68b40d-907e-45fb-a27b-d8b79eae19bc-dshm\") pod \"e2e-conflict-b-e5a626b8-kserve-66559ccb54-b44ch\" (UID: \"ee68b40d-907e-45fb-a27b-d8b79eae19bc\") " pod="llm/e2e-conflict-b-e5a626b8-kserve-66559ccb54-b44ch"
Sep 10 22:17:13.378503 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:13.378482    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee68b40d-907e-45fb-a27b-d8b79eae19bc-tls-certs\") pod \"e2e-conflict-b-e5a626b8-kserve-66559ccb54-b44ch\" (UID: \"ee68b40d-907e-45fb-a27b-d8b79eae19bc\") " pod="llm/e2e-conflict-b-e5a626b8-kserve-66559ccb54-b44ch"
Sep 10 22:17:13.479141 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:13.479101    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee68b40d-907e-45fb-a27b-d8b79eae19bc-model-cache\") pod \"e2e-conflict-b-e5a626b8-kserve-66559ccb54-b44ch\" (UID: \"ee68b40d-907e-45fb-a27b-d8b79eae19bc\") " pod="llm/e2e-conflict-b-e5a626b8-kserve-66559ccb54-b44ch"
Sep 10 22:17:13.479317 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:13.479233    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee68b40d-907e-45fb-a27b-d8b79eae19bc-dshm\") pod \"e2e-conflict-b-e5a626b8-kserve-66559ccb54-b44ch\" (UID: \"ee68b40d-907e-45fb-a27b-d8b79eae19bc\") " pod="llm/e2e-conflict-b-e5a626b8-kserve-66559ccb54-b44ch"
Sep 10 22:17:13.479317 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:13.479266    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee68b40d-907e-45fb-a27b-d8b79eae19bc-tls-certs\") pod \"e2e-conflict-b-e5a626b8-kserve-66559ccb54-b44ch\" (UID: \"ee68b40d-907e-45fb-a27b-d8b79eae19bc\") " pod="llm/e2e-conflict-b-e5a626b8-kserve-66559ccb54-b44ch"
Sep 10 22:17:13.479426 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:13.479404    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee68b40d-907e-45fb-a27b-d8b79eae19bc-home\") pod \"e2e-conflict-b-e5a626b8-kserve-66559ccb54-b44ch\" (UID: \"ee68b40d-907e-45fb-a27b-d8b79eae19bc\") " pod="llm/e2e-conflict-b-e5a626b8-kserve-66559ccb54-b44ch"
Sep 10 22:17:13.479463 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:13.479449    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9c6l\" (UniqueName: \"kubernetes.io/projected/ee68b40d-907e-45fb-a27b-d8b79eae19bc-kube-api-access-j9c6l\") pod \"e2e-conflict-b-e5a626b8-kserve-66559ccb54-b44ch\" (UID: \"ee68b40d-907e-45fb-a27b-d8b79eae19bc\") " pod="llm/e2e-conflict-b-e5a626b8-kserve-66559ccb54-b44ch"
Sep 10 22:17:13.479524 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:13.479510    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee68b40d-907e-45fb-a27b-d8b79eae19bc-tmp-dir\") pod \"e2e-conflict-b-e5a626b8-kserve-66559ccb54-b44ch\" (UID: \"ee68b40d-907e-45fb-a27b-d8b79eae19bc\") " pod="llm/e2e-conflict-b-e5a626b8-kserve-66559ccb54-b44ch"
Sep 10 22:17:13.479573 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:13.479520    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee68b40d-907e-45fb-a27b-d8b79eae19bc-model-cache\") pod \"e2e-conflict-b-e5a626b8-kserve-66559ccb54-b44ch\" (UID: \"ee68b40d-907e-45fb-a27b-d8b79eae19bc\") " pod="llm/e2e-conflict-b-e5a626b8-kserve-66559ccb54-b44ch"
Sep 10 22:17:13.479761 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:13.479736    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee68b40d-907e-45fb-a27b-d8b79eae19bc-home\") pod \"e2e-conflict-b-e5a626b8-kserve-66559ccb54-b44ch\" (UID: \"ee68b40d-907e-45fb-a27b-d8b79eae19bc\") " pod="llm/e2e-conflict-b-e5a626b8-kserve-66559ccb54-b44ch"
Sep 10 22:17:13.479926 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:13.479905    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee68b40d-907e-45fb-a27b-d8b79eae19bc-tmp-dir\") pod \"e2e-conflict-b-e5a626b8-kserve-66559ccb54-b44ch\" (UID: \"ee68b40d-907e-45fb-a27b-d8b79eae19bc\") " pod="llm/e2e-conflict-b-e5a626b8-kserve-66559ccb54-b44ch"
Sep 10 22:17:13.481536 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:13.481508    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee68b40d-907e-45fb-a27b-d8b79eae19bc-dshm\") pod \"e2e-conflict-b-e5a626b8-kserve-66559ccb54-b44ch\" (UID: \"ee68b40d-907e-45fb-a27b-d8b79eae19bc\") " pod="llm/e2e-conflict-b-e5a626b8-kserve-66559ccb54-b44ch"
Sep 10 22:17:13.481875 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:13.481829    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee68b40d-907e-45fb-a27b-d8b79eae19bc-tls-certs\") pod \"e2e-conflict-b-e5a626b8-kserve-66559ccb54-b44ch\" (UID: \"ee68b40d-907e-45fb-a27b-d8b79eae19bc\") " pod="llm/e2e-conflict-b-e5a626b8-kserve-66559ccb54-b44ch"
Sep 10 22:17:13.487167 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:13.487143    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9c6l\" (UniqueName: \"kubernetes.io/projected/ee68b40d-907e-45fb-a27b-d8b79eae19bc-kube-api-access-j9c6l\") pod \"e2e-conflict-b-e5a626b8-kserve-66559ccb54-b44ch\" (UID: \"ee68b40d-907e-45fb-a27b-d8b79eae19bc\") " pod="llm/e2e-conflict-b-e5a626b8-kserve-66559ccb54-b44ch"
Sep 10 22:17:13.615345 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:13.615258    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-e5a626b8-kserve-66559ccb54-b44ch"
Sep 10 22:17:13.886673 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:13.886589    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-b-e5a626b8-kserve-66559ccb54-b44ch"]
Sep 10 22:17:13.889805 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:17:13.889772    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podee68b40d_907e_45fb_a27b_d8b79eae19bc.slice/crio-509e1d5460dcb3d4e7aeed6f9c5aa6de00b37e55da5a99fbeaeceb1f832f5bcf WatchSource:0}: Error finding container 509e1d5460dcb3d4e7aeed6f9c5aa6de00b37e55da5a99fbeaeceb1f832f5bcf: Status 404 returned error can't find the container with id 509e1d5460dcb3d4e7aeed6f9c5aa6de00b37e55da5a99fbeaeceb1f832f5bcf
Sep 10 22:17:14.704550 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:14.704509    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-e5a626b8-kserve-66559ccb54-b44ch" event={"ID":"ee68b40d-907e-45fb-a27b-d8b79eae19bc","Type":"ContainerStarted","Data":"9480dc11d6895ac72ba51d5048580751add59be09d2532c7cf9519b82057b09d"}
Sep 10 22:17:14.705039 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:14.704558    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-e5a626b8-kserve-66559ccb54-b44ch" event={"ID":"ee68b40d-907e-45fb-a27b-d8b79eae19bc","Type":"ContainerStarted","Data":"509e1d5460dcb3d4e7aeed6f9c5aa6de00b37e55da5a99fbeaeceb1f832f5bcf"}
Sep 10 22:17:14.720495 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:14.720438    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-conflict-b-e5a626b8-kserve-66559ccb54-b44ch" podStartSLOduration=1.720419713 podStartE2EDuration="1.720419713s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 22:17:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:17:14.719356393 +0000 UTC m=+955.675424100" watchObservedRunningTime="2026-09-10 22:17:14.720419713 +0000 UTC m=+955.676487448"
Sep 10 22:17:15.660349 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:15.660313    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-4149405d-5bbcdcdfc4-btfqc"
Sep 10 22:17:15.660514 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:15.660483    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-4149405d-6b5dcf9dcc-dxbf4"
Sep 10 22:17:23.615641 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:23.615611    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-conflict-b-e5a626b8-kserve-66559ccb54-b44ch"
Sep 10 22:17:23.616036 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:23.615657    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-conflict-b-e5a626b8-kserve-66559ccb54-b44ch"
Sep 10 22:17:23.628000 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:23.627976    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-conflict-b-e5a626b8-kserve-66559ccb54-b44ch"
Sep 10 22:17:23.759364 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:23.759336    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-conflict-b-e5a626b8-kserve-66559ccb54-b44ch"
Sep 10 22:17:33.035694 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:33.035660    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-4149405d-7d474f89f4-c7sfd"]
Sep 10 22:17:33.036190 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:33.035892    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4149405d-7d474f89f4-c7sfd" podUID="726a426b-3549-486f-b8c8-ba1542f8557c" containerName="maas-api" containerID="cri-o://2aef7cdbae6913a5c3a2fd2547b7dfb3ae502f2300fe78fdbda8996b4331d099" gracePeriod=30
Sep 10 22:17:33.097420 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:33.097378    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-4149405d-6b5dcf9dcc-dxbf4"]
Sep 10 22:17:33.097654 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:33.097608    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-4149405d-6b5dcf9dcc-dxbf4" podUID="6f043662-4ea5-4b65-b3be-ab66d5d3696b" containerName="payload-processing" containerID="cri-o://60331a501c0aa756f7ce3d49dff6d0ebcdf62b0efd708045593dc15805dbf11c" gracePeriod=30
Sep 10 22:17:33.101943 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:33.101917    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-4149405d-5bbcdcdfc4-btfqc"]
Sep 10 22:17:33.102303 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:33.102258    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-4149405d-5bbcdcdfc4-btfqc" podUID="03b96f6e-5021-4459-8031-015a0f3aef66" containerName="payload-pre-processing" containerID="cri-o://94e4ec84ed633500999198aee1e220b8704755eee22d06b338c2c8f1880421bb" gracePeriod=30
Sep 10 22:17:33.389487 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:33.389404    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-4149405d-6b5dcf9dcc-dxbf4"
Sep 10 22:17:33.392685 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:33.392662    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-4149405d-5bbcdcdfc4-btfqc"
Sep 10 22:17:33.401930 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:33.401907    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4149405d-7d474f89f4-c7sfd"
Sep 10 22:17:33.554213 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:33.554133    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/726a426b-3549-486f-b8c8-ba1542f8557c-metrics-certs\") pod \"726a426b-3549-486f-b8c8-ba1542f8557c\" (UID: \"726a426b-3549-486f-b8c8-ba1542f8557c\") "
Sep 10 22:17:33.554213 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:33.554165    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/726a426b-3549-486f-b8c8-ba1542f8557c-maas-api-tls\") pod \"726a426b-3549-486f-b8c8-ba1542f8557c\" (UID: \"726a426b-3549-486f-b8c8-ba1542f8557c\") "
Sep 10 22:17:33.554213 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:33.554210    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b2m6k\" (UniqueName: \"kubernetes.io/projected/03b96f6e-5021-4459-8031-015a0f3aef66-kube-api-access-b2m6k\") pod \"03b96f6e-5021-4459-8031-015a0f3aef66\" (UID: \"03b96f6e-5021-4459-8031-015a0f3aef66\") "
Sep 10 22:17:33.554472 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:33.554250    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/726a426b-3549-486f-b8c8-ba1542f8557c-openshift-service-ca\") pod \"726a426b-3549-486f-b8c8-ba1542f8557c\" (UID: \"726a426b-3549-486f-b8c8-ba1542f8557c\") "
Sep 10 22:17:33.554472 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:33.554280    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sr8x6\" (UniqueName: \"kubernetes.io/projected/726a426b-3549-486f-b8c8-ba1542f8557c-kube-api-access-sr8x6\") pod \"726a426b-3549-486f-b8c8-ba1542f8557c\" (UID: \"726a426b-3549-486f-b8c8-ba1542f8557c\") "
Sep 10 22:17:33.554472 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:33.554317    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f043662-4ea5-4b65-b3be-ab66d5d3696b-plugins-config-volume\") pod \"6f043662-4ea5-4b65-b3be-ab66d5d3696b\" (UID: \"6f043662-4ea5-4b65-b3be-ab66d5d3696b\") "
Sep 10 22:17:33.554472 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:33.554340    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m8tlq\" (UniqueName: \"kubernetes.io/projected/6f043662-4ea5-4b65-b3be-ab66d5d3696b-kube-api-access-m8tlq\") pod \"6f043662-4ea5-4b65-b3be-ab66d5d3696b\" (UID: \"6f043662-4ea5-4b65-b3be-ab66d5d3696b\") "
Sep 10 22:17:33.554472 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:33.554421    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03b96f6e-5021-4459-8031-015a0f3aef66-plugins-config-volume\") pod \"03b96f6e-5021-4459-8031-015a0f3aef66\" (UID: \"03b96f6e-5021-4459-8031-015a0f3aef66\") "
Sep 10 22:17:33.554743 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:33.554672    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/726a426b-3549-486f-b8c8-ba1542f8557c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "726a426b-3549-486f-b8c8-ba1542f8557c" (UID: "726a426b-3549-486f-b8c8-ba1542f8557c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 22:17:33.554804 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:33.554749    2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/726a426b-3549-486f-b8c8-ba1542f8557c-openshift-service-ca\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:17:33.555135 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:33.555078    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f043662-4ea5-4b65-b3be-ab66d5d3696b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6f043662-4ea5-4b65-b3be-ab66d5d3696b" (UID: "6f043662-4ea5-4b65-b3be-ab66d5d3696b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 22:17:33.555135 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:33.555099    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03b96f6e-5021-4459-8031-015a0f3aef66-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "03b96f6e-5021-4459-8031-015a0f3aef66" (UID: "03b96f6e-5021-4459-8031-015a0f3aef66"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 22:17:33.556905 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:33.556880    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/726a426b-3549-486f-b8c8-ba1542f8557c-kube-api-access-sr8x6" (OuterVolumeSpecName: "kube-api-access-sr8x6") pod "726a426b-3549-486f-b8c8-ba1542f8557c" (UID: "726a426b-3549-486f-b8c8-ba1542f8557c"). InnerVolumeSpecName "kube-api-access-sr8x6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 22:17:33.557057 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:33.556912    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03b96f6e-5021-4459-8031-015a0f3aef66-kube-api-access-b2m6k" (OuterVolumeSpecName: "kube-api-access-b2m6k") pod "03b96f6e-5021-4459-8031-015a0f3aef66" (UID: "03b96f6e-5021-4459-8031-015a0f3aef66"). InnerVolumeSpecName "kube-api-access-b2m6k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 22:17:33.557133 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:33.557093    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f043662-4ea5-4b65-b3be-ab66d5d3696b-kube-api-access-m8tlq" (OuterVolumeSpecName: "kube-api-access-m8tlq") pod "6f043662-4ea5-4b65-b3be-ab66d5d3696b" (UID: "6f043662-4ea5-4b65-b3be-ab66d5d3696b"). InnerVolumeSpecName "kube-api-access-m8tlq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 22:17:33.557564 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:33.557545    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/726a426b-3549-486f-b8c8-ba1542f8557c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "726a426b-3549-486f-b8c8-ba1542f8557c" (UID: "726a426b-3549-486f-b8c8-ba1542f8557c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 22:17:33.558301 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:33.558275    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/726a426b-3549-486f-b8c8-ba1542f8557c-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "726a426b-3549-486f-b8c8-ba1542f8557c" (UID: "726a426b-3549-486f-b8c8-ba1542f8557c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 22:17:33.656237 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:33.656209    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03b96f6e-5021-4459-8031-015a0f3aef66-plugins-config-volume\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:17:33.656237 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:33.656232    2582 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/726a426b-3549-486f-b8c8-ba1542f8557c-metrics-certs\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:17:33.656237 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:33.656242    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/726a426b-3549-486f-b8c8-ba1542f8557c-maas-api-tls\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:17:33.656449 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:33.656250    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b2m6k\" (UniqueName: \"kubernetes.io/projected/03b96f6e-5021-4459-8031-015a0f3aef66-kube-api-access-b2m6k\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:17:33.656449 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:33.656259    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sr8x6\" (UniqueName: \"kubernetes.io/projected/726a426b-3549-486f-b8c8-ba1542f8557c-kube-api-access-sr8x6\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:17:33.656449 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:33.656268    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f043662-4ea5-4b65-b3be-ab66d5d3696b-plugins-config-volume\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:17:33.656449 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:33.656276    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8tlq\" (UniqueName: \"kubernetes.io/projected/6f043662-4ea5-4b65-b3be-ab66d5d3696b-kube-api-access-m8tlq\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:17:33.808084 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:33.808008    2582 generic.go:356] "Generic (PLEG): container finished" podID="6f043662-4ea5-4b65-b3be-ab66d5d3696b" containerID="60331a501c0aa756f7ce3d49dff6d0ebcdf62b0efd708045593dc15805dbf11c" exitCode=0
Sep 10 22:17:33.808084 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:33.808064    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-4149405d-6b5dcf9dcc-dxbf4"
Sep 10 22:17:33.808265 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:33.808099    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-4149405d-6b5dcf9dcc-dxbf4" event={"ID":"6f043662-4ea5-4b65-b3be-ab66d5d3696b","Type":"ContainerDied","Data":"60331a501c0aa756f7ce3d49dff6d0ebcdf62b0efd708045593dc15805dbf11c"}
Sep 10 22:17:33.808265 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:33.808161    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-4149405d-6b5dcf9dcc-dxbf4" event={"ID":"6f043662-4ea5-4b65-b3be-ab66d5d3696b","Type":"ContainerDied","Data":"153744fc95ac68a64274ef766f1c464ad76e4140963a93776b5a0d165f332484"}
Sep 10 22:17:33.808265 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:33.808195    2582 scope.go:117] "RemoveContainer" containerID="60331a501c0aa756f7ce3d49dff6d0ebcdf62b0efd708045593dc15805dbf11c"
Sep 10 22:17:33.809473 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:33.809451    2582 generic.go:356] "Generic (PLEG): container finished" podID="03b96f6e-5021-4459-8031-015a0f3aef66" containerID="94e4ec84ed633500999198aee1e220b8704755eee22d06b338c2c8f1880421bb" exitCode=0
Sep 10 22:17:33.809559 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:33.809496    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-4149405d-5bbcdcdfc4-btfqc"
Sep 10 22:17:33.809559 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:33.809548    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-4149405d-5bbcdcdfc4-btfqc" event={"ID":"03b96f6e-5021-4459-8031-015a0f3aef66","Type":"ContainerDied","Data":"94e4ec84ed633500999198aee1e220b8704755eee22d06b338c2c8f1880421bb"}
Sep 10 22:17:33.809674 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:33.809586    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-4149405d-5bbcdcdfc4-btfqc" event={"ID":"03b96f6e-5021-4459-8031-015a0f3aef66","Type":"ContainerDied","Data":"6ac08428e9e1a5a4f9db6f4265b82fc5ec32d1da3e9455ccef6aa48503158fa4"}
Sep 10 22:17:33.811093 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:33.811063    2582 generic.go:356] "Generic (PLEG): container finished" podID="726a426b-3549-486f-b8c8-ba1542f8557c" containerID="2aef7cdbae6913a5c3a2fd2547b7dfb3ae502f2300fe78fdbda8996b4331d099" exitCode=0
Sep 10 22:17:33.811198 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:33.811106    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4149405d-7d474f89f4-c7sfd"
Sep 10 22:17:33.811198 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:33.811120    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4149405d-7d474f89f4-c7sfd" event={"ID":"726a426b-3549-486f-b8c8-ba1542f8557c","Type":"ContainerDied","Data":"2aef7cdbae6913a5c3a2fd2547b7dfb3ae502f2300fe78fdbda8996b4331d099"}
Sep 10 22:17:33.811198 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:33.811160    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4149405d-7d474f89f4-c7sfd" event={"ID":"726a426b-3549-486f-b8c8-ba1542f8557c","Type":"ContainerDied","Data":"72c88a2bddc0720b4b42780ec961f1056479126859e4400c7a1eb6019fc75091"}
Sep 10 22:17:33.818875 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:33.818834    2582 scope.go:117] "RemoveContainer" containerID="60331a501c0aa756f7ce3d49dff6d0ebcdf62b0efd708045593dc15805dbf11c"
Sep 10 22:17:33.819184 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:17:33.819160    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60331a501c0aa756f7ce3d49dff6d0ebcdf62b0efd708045593dc15805dbf11c\": container with ID starting with 60331a501c0aa756f7ce3d49dff6d0ebcdf62b0efd708045593dc15805dbf11c not found: ID does not exist" containerID="60331a501c0aa756f7ce3d49dff6d0ebcdf62b0efd708045593dc15805dbf11c"
Sep 10 22:17:33.819251 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:33.819194    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60331a501c0aa756f7ce3d49dff6d0ebcdf62b0efd708045593dc15805dbf11c"} err="failed to get container status \"60331a501c0aa756f7ce3d49dff6d0ebcdf62b0efd708045593dc15805dbf11c\": rpc error: code = NotFound desc = could not find container \"60331a501c0aa756f7ce3d49dff6d0ebcdf62b0efd708045593dc15805dbf11c\": container with ID starting with 60331a501c0aa756f7ce3d49dff6d0ebcdf62b0efd708045593dc15805dbf11c not found: ID does not exist"
Sep 10 22:17:33.819251 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:33.819218    2582 scope.go:117] "RemoveContainer" containerID="94e4ec84ed633500999198aee1e220b8704755eee22d06b338c2c8f1880421bb"
Sep 10 22:17:33.826893 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:33.826873    2582 scope.go:117] "RemoveContainer" containerID="94e4ec84ed633500999198aee1e220b8704755eee22d06b338c2c8f1880421bb"
Sep 10 22:17:33.827145 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:17:33.827128    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94e4ec84ed633500999198aee1e220b8704755eee22d06b338c2c8f1880421bb\": container with ID starting with 94e4ec84ed633500999198aee1e220b8704755eee22d06b338c2c8f1880421bb not found: ID does not exist" containerID="94e4ec84ed633500999198aee1e220b8704755eee22d06b338c2c8f1880421bb"
Sep 10 22:17:33.827186 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:33.827153    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94e4ec84ed633500999198aee1e220b8704755eee22d06b338c2c8f1880421bb"} err="failed to get container status \"94e4ec84ed633500999198aee1e220b8704755eee22d06b338c2c8f1880421bb\": rpc error: code = NotFound desc = could not find container \"94e4ec84ed633500999198aee1e220b8704755eee22d06b338c2c8f1880421bb\": container with ID starting with 94e4ec84ed633500999198aee1e220b8704755eee22d06b338c2c8f1880421bb not found: ID does not exist"
Sep 10 22:17:33.827186 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:33.827170    2582 scope.go:117] "RemoveContainer" containerID="2aef7cdbae6913a5c3a2fd2547b7dfb3ae502f2300fe78fdbda8996b4331d099"
Sep 10 22:17:33.831352 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:33.831332    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-4149405d-6b5dcf9dcc-dxbf4"]
Sep 10 22:17:33.834563 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:33.834545    2582 scope.go:117] "RemoveContainer" containerID="2aef7cdbae6913a5c3a2fd2547b7dfb3ae502f2300fe78fdbda8996b4331d099"
Sep 10 22:17:33.834825 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:17:33.834808    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2aef7cdbae6913a5c3a2fd2547b7dfb3ae502f2300fe78fdbda8996b4331d099\": container with ID starting with 2aef7cdbae6913a5c3a2fd2547b7dfb3ae502f2300fe78fdbda8996b4331d099 not found: ID does not exist" containerID="2aef7cdbae6913a5c3a2fd2547b7dfb3ae502f2300fe78fdbda8996b4331d099"
Sep 10 22:17:33.834912 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:33.834831    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2aef7cdbae6913a5c3a2fd2547b7dfb3ae502f2300fe78fdbda8996b4331d099"} err="failed to get container status \"2aef7cdbae6913a5c3a2fd2547b7dfb3ae502f2300fe78fdbda8996b4331d099\": rpc error: code = NotFound desc = could not find container \"2aef7cdbae6913a5c3a2fd2547b7dfb3ae502f2300fe78fdbda8996b4331d099\": container with ID starting with 2aef7cdbae6913a5c3a2fd2547b7dfb3ae502f2300fe78fdbda8996b4331d099 not found: ID does not exist"
Sep 10 22:17:33.837658 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:33.837636    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-4149405d-6b5dcf9dcc-dxbf4"]
Sep 10 22:17:33.847214 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:33.847192    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-4149405d-7d474f89f4-c7sfd"]
Sep 10 22:17:33.851216 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:33.851197    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-4149405d-7d474f89f4-c7sfd"]
Sep 10 22:17:33.862670 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:33.862649    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-4149405d-5bbcdcdfc4-btfqc"]
Sep 10 22:17:33.865211 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:33.865192    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-4149405d-5bbcdcdfc4-btfqc"]
Sep 10 22:17:35.497440 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:35.497405    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="03b96f6e-5021-4459-8031-015a0f3aef66" path="/var/lib/kubelet/pods/03b96f6e-5021-4459-8031-015a0f3aef66/volumes"
Sep 10 22:17:35.497776 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:35.497752    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6f043662-4ea5-4b65-b3be-ab66d5d3696b" path="/var/lib/kubelet/pods/6f043662-4ea5-4b65-b3be-ab66d5d3696b/volumes"
Sep 10 22:17:35.498146 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:35.498131    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="726a426b-3549-486f-b8c8-ba1542f8557c" path="/var/lib/kubelet/pods/726a426b-3549-486f-b8c8-ba1542f8557c/volumes"
Sep 10 22:17:47.113741 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:47.113661    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-d6a4de-openshift-default-6bf96b68c4-6269k"]
Sep 10 22:17:47.114235 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:47.113889    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-d6a4de-openshift-default-6bf96b68c4-6269k" podUID="78896706-4631-4e69-b205-dfad4be3b549" containerName="istio-proxy" containerID="cri-o://706d867cbfa19ac386445c30487b59f4c6c6f350bcbadd0a86d542fd46a5a51f" gracePeriod=30
Sep 10 22:17:50.874204 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:50.874170    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-a-e5a626b8-kserve-7688fbd-k5z56"]
Sep 10 22:17:51.680604 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:51.680563    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-b-e5a626b8-kserve-66559ccb54-b44ch"]
Sep 10 22:17:52.375513 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:52.375452    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-d6a4de-openshift-default-6bf96b68c4-6269k"
Sep 10 22:17:52.418427 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:52.418397    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78896706-4631-4e69-b205-dfad4be3b549-istio-token\") pod \"78896706-4631-4e69-b205-dfad4be3b549\" (UID: \"78896706-4631-4e69-b205-dfad4be3b549\") "
Sep 10 22:17:52.418597 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:52.418466    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zjf7x\" (UniqueName: \"kubernetes.io/projected/78896706-4631-4e69-b205-dfad4be3b549-kube-api-access-zjf7x\") pod \"78896706-4631-4e69-b205-dfad4be3b549\" (UID: \"78896706-4631-4e69-b205-dfad4be3b549\") "
Sep 10 22:17:52.418597 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:52.418493    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78896706-4631-4e69-b205-dfad4be3b549-istio-data\") pod \"78896706-4631-4e69-b205-dfad4be3b549\" (UID: \"78896706-4631-4e69-b205-dfad4be3b549\") "
Sep 10 22:17:52.418721 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:52.418590    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78896706-4631-4e69-b205-dfad4be3b549-istiod-ca-cert\") pod \"78896706-4631-4e69-b205-dfad4be3b549\" (UID: \"78896706-4631-4e69-b205-dfad4be3b549\") "
Sep 10 22:17:52.418721 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:52.418638    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78896706-4631-4e69-b205-dfad4be3b549-istio-envoy\") pod \"78896706-4631-4e69-b205-dfad4be3b549\" (UID: \"78896706-4631-4e69-b205-dfad4be3b549\") "
Sep 10 22:17:52.418721 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:52.418661    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78896706-4631-4e69-b205-dfad4be3b549-istio-podinfo\") pod \"78896706-4631-4e69-b205-dfad4be3b549\" (UID: \"78896706-4631-4e69-b205-dfad4be3b549\") "
Sep 10 22:17:52.418721 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:52.418684    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78896706-4631-4e69-b205-dfad4be3b549-credential-socket\") pod \"78896706-4631-4e69-b205-dfad4be3b549\" (UID: \"78896706-4631-4e69-b205-dfad4be3b549\") "
Sep 10 22:17:52.418721 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:52.418711    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78896706-4631-4e69-b205-dfad4be3b549-workload-socket\") pod \"78896706-4631-4e69-b205-dfad4be3b549\" (UID: \"78896706-4631-4e69-b205-dfad4be3b549\") "
Sep 10 22:17:52.418945 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:52.418741    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78896706-4631-4e69-b205-dfad4be3b549-istio-data" (OuterVolumeSpecName: "istio-data") pod "78896706-4631-4e69-b205-dfad4be3b549" (UID: "78896706-4631-4e69-b205-dfad4be3b549"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 22:17:52.418945 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:52.418753    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78896706-4631-4e69-b205-dfad4be3b549-workload-certs\") pod \"78896706-4631-4e69-b205-dfad4be3b549\" (UID: \"78896706-4631-4e69-b205-dfad4be3b549\") "
Sep 10 22:17:52.419055 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:52.418953    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78896706-4631-4e69-b205-dfad4be3b549-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "78896706-4631-4e69-b205-dfad4be3b549" (UID: "78896706-4631-4e69-b205-dfad4be3b549"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 22:17:52.419119 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:52.419049    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78896706-4631-4e69-b205-dfad4be3b549-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "78896706-4631-4e69-b205-dfad4be3b549" (UID: "78896706-4631-4e69-b205-dfad4be3b549"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 22:17:52.419229 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:52.419147    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78896706-4631-4e69-b205-dfad4be3b549-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "78896706-4631-4e69-b205-dfad4be3b549" (UID: "78896706-4631-4e69-b205-dfad4be3b549"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 22:17:52.419229 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:52.419174    2582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78896706-4631-4e69-b205-dfad4be3b549-istio-data\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:17:52.419229 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:52.419198    2582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78896706-4631-4e69-b205-dfad4be3b549-credential-socket\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:17:52.419229 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:52.419213    2582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78896706-4631-4e69-b205-dfad4be3b549-workload-socket\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:17:52.419389 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:52.419358    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78896706-4631-4e69-b205-dfad4be3b549-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "78896706-4631-4e69-b205-dfad4be3b549" (UID: "78896706-4631-4e69-b205-dfad4be3b549"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 22:17:52.420934 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:52.420901    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78896706-4631-4e69-b205-dfad4be3b549-kube-api-access-zjf7x" (OuterVolumeSpecName: "kube-api-access-zjf7x") pod "78896706-4631-4e69-b205-dfad4be3b549" (UID: "78896706-4631-4e69-b205-dfad4be3b549"). InnerVolumeSpecName "kube-api-access-zjf7x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 22:17:52.421342 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:52.421315    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78896706-4631-4e69-b205-dfad4be3b549-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "78896706-4631-4e69-b205-dfad4be3b549" (UID: "78896706-4631-4e69-b205-dfad4be3b549"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 22:17:52.421431 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:52.421379    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78896706-4631-4e69-b205-dfad4be3b549-istio-token" (OuterVolumeSpecName: "istio-token") pod "78896706-4631-4e69-b205-dfad4be3b549" (UID: "78896706-4631-4e69-b205-dfad4be3b549"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 22:17:52.421509 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:52.421493    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/78896706-4631-4e69-b205-dfad4be3b549-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "78896706-4631-4e69-b205-dfad4be3b549" (UID: "78896706-4631-4e69-b205-dfad4be3b549"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 22:17:52.520021 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:52.519981    2582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78896706-4631-4e69-b205-dfad4be3b549-istio-envoy\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:17:52.520021 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:52.520016    2582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78896706-4631-4e69-b205-dfad4be3b549-istio-podinfo\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:17:52.520021 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:52.520025    2582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78896706-4631-4e69-b205-dfad4be3b549-workload-certs\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:17:52.520021 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:52.520034    2582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78896706-4631-4e69-b205-dfad4be3b549-istio-token\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:17:52.520308 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:52.520042    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjf7x\" (UniqueName: \"kubernetes.io/projected/78896706-4631-4e69-b205-dfad4be3b549-kube-api-access-zjf7x\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:17:52.520308 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:52.520051    2582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78896706-4631-4e69-b205-dfad4be3b549-istiod-ca-cert\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:17:52.908960 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:52.908921    2582 generic.go:356] "Generic (PLEG): container finished" podID="78896706-4631-4e69-b205-dfad4be3b549" containerID="706d867cbfa19ac386445c30487b59f4c6c6f350bcbadd0a86d542fd46a5a51f" exitCode=0
Sep 10 22:17:52.909153 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:52.908992    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-d6a4de-openshift-default-6bf96b68c4-6269k"
Sep 10 22:17:52.909153 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:52.909006    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-d6a4de-openshift-default-6bf96b68c4-6269k" event={"ID":"78896706-4631-4e69-b205-dfad4be3b549","Type":"ContainerDied","Data":"706d867cbfa19ac386445c30487b59f4c6c6f350bcbadd0a86d542fd46a5a51f"}
Sep 10 22:17:52.909153 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:52.909067    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-d6a4de-openshift-default-6bf96b68c4-6269k" event={"ID":"78896706-4631-4e69-b205-dfad4be3b549","Type":"ContainerDied","Data":"ef076a63501faac83d971ffeb069524382d8478c68a663abb27561cf1f8abe66"}
Sep 10 22:17:52.909153 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:52.909088    2582 scope.go:117] "RemoveContainer" containerID="706d867cbfa19ac386445c30487b59f4c6c6f350bcbadd0a86d542fd46a5a51f"
Sep 10 22:17:52.919110 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:52.918904    2582 scope.go:117] "RemoveContainer" containerID="69816f10c217d08dd2575ba64ac215c8fe781713aa566fc6d347f58420a8ca87"
Sep 10 22:17:52.927064 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:52.927047    2582 scope.go:117] "RemoveContainer" containerID="706d867cbfa19ac386445c30487b59f4c6c6f350bcbadd0a86d542fd46a5a51f"
Sep 10 22:17:52.927362 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:17:52.927343    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"706d867cbfa19ac386445c30487b59f4c6c6f350bcbadd0a86d542fd46a5a51f\": container with ID starting with 706d867cbfa19ac386445c30487b59f4c6c6f350bcbadd0a86d542fd46a5a51f not found: ID does not exist" containerID="706d867cbfa19ac386445c30487b59f4c6c6f350bcbadd0a86d542fd46a5a51f"
Sep 10 22:17:52.927412 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:52.927372    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"706d867cbfa19ac386445c30487b59f4c6c6f350bcbadd0a86d542fd46a5a51f"} err="failed to get container status \"706d867cbfa19ac386445c30487b59f4c6c6f350bcbadd0a86d542fd46a5a51f\": rpc error: code = NotFound desc = could not find container \"706d867cbfa19ac386445c30487b59f4c6c6f350bcbadd0a86d542fd46a5a51f\": container with ID starting with 706d867cbfa19ac386445c30487b59f4c6c6f350bcbadd0a86d542fd46a5a51f not found: ID does not exist"
Sep 10 22:17:52.927412 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:52.927390    2582 scope.go:117] "RemoveContainer" containerID="69816f10c217d08dd2575ba64ac215c8fe781713aa566fc6d347f58420a8ca87"
Sep 10 22:17:52.927662 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:17:52.927643    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69816f10c217d08dd2575ba64ac215c8fe781713aa566fc6d347f58420a8ca87\": container with ID starting with 69816f10c217d08dd2575ba64ac215c8fe781713aa566fc6d347f58420a8ca87 not found: ID does not exist" containerID="69816f10c217d08dd2575ba64ac215c8fe781713aa566fc6d347f58420a8ca87"
Sep 10 22:17:52.927704 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:52.927668    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69816f10c217d08dd2575ba64ac215c8fe781713aa566fc6d347f58420a8ca87"} err="failed to get container status \"69816f10c217d08dd2575ba64ac215c8fe781713aa566fc6d347f58420a8ca87\": rpc error: code = NotFound desc = could not find container \"69816f10c217d08dd2575ba64ac215c8fe781713aa566fc6d347f58420a8ca87\": container with ID starting with 69816f10c217d08dd2575ba64ac215c8fe781713aa566fc6d347f58420a8ca87 not found: ID does not exist"
Sep 10 22:17:52.932155 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:52.932130    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-d6a4de-openshift-default-6bf96b68c4-6269k"]
Sep 10 22:17:52.938457 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:52.938435    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-d6a4de-openshift-default-6bf96b68c4-6269k"]
Sep 10 22:17:52.979555 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:52.979522    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-8f9fc7-8597dd45f8-phpd9"]
Sep 10 22:17:52.979857 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:52.979835    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03b96f6e-5021-4459-8031-015a0f3aef66" containerName="payload-pre-processing"
Sep 10 22:17:52.979906 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:52.979859    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="03b96f6e-5021-4459-8031-015a0f3aef66" containerName="payload-pre-processing"
Sep 10 22:17:52.979906 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:52.979879    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78896706-4631-4e69-b205-dfad4be3b549" containerName="istio-proxy"
Sep 10 22:17:52.979906 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:52.979884    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="78896706-4631-4e69-b205-dfad4be3b549" containerName="istio-proxy"
Sep 10 22:17:52.979906 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:52.979892    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f043662-4ea5-4b65-b3be-ab66d5d3696b" containerName="payload-processing"
Sep 10 22:17:52.979906 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:52.979899    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f043662-4ea5-4b65-b3be-ab66d5d3696b" containerName="payload-processing"
Sep 10 22:17:52.979906 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:52.979907    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78896706-4631-4e69-b205-dfad4be3b549" containerName="istio-proxy"
Sep 10 22:17:52.980097 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:52.979913    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="78896706-4631-4e69-b205-dfad4be3b549" containerName="istio-proxy"
Sep 10 22:17:52.980097 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:52.979921    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="726a426b-3549-486f-b8c8-ba1542f8557c" containerName="maas-api"
Sep 10 22:17:52.980097 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:52.979926    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="726a426b-3549-486f-b8c8-ba1542f8557c" containerName="maas-api"
Sep 10 22:17:52.980097 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:52.979980    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="78896706-4631-4e69-b205-dfad4be3b549" containerName="istio-proxy"
Sep 10 22:17:52.980097 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:52.979988    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f043662-4ea5-4b65-b3be-ab66d5d3696b" containerName="payload-processing"
Sep 10 22:17:52.980097 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:52.979995    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="03b96f6e-5021-4459-8031-015a0f3aef66" containerName="payload-pre-processing"
Sep 10 22:17:52.980097 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:52.980002    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="726a426b-3549-486f-b8c8-ba1542f8557c" containerName="maas-api"
Sep 10 22:17:52.984314 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:52.984297    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-8f9fc7-8597dd45f8-phpd9"
Sep 10 22:17:52.999671 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:52.999653    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-8f9fc7-dockercfg-jkcmc\""
Sep 10 22:17:53.001396 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:53.001371    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-8f9fc7\""
Sep 10 22:17:53.003067 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:53.003046    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-8f9fc7-8597dd45f8-phpd9"]
Sep 10 22:17:53.024414 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:53.024384    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbfct\" (UniqueName: \"kubernetes.io/projected/e0718e77-8ff0-4b70-b172-16959e5c353b-kube-api-access-lbfct\") pod \"payload-pre-processing-e2e-explicit-8f9fc7-8597dd45f8-phpd9\" (UID: \"e0718e77-8ff0-4b70-b172-16959e5c353b\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-8f9fc7-8597dd45f8-phpd9"
Sep 10 22:17:53.024559 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:53.024421    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0718e77-8ff0-4b70-b172-16959e5c353b-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-8f9fc7-8597dd45f8-phpd9\" (UID: \"e0718e77-8ff0-4b70-b172-16959e5c353b\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-8f9fc7-8597dd45f8-phpd9"
Sep 10 22:17:53.027954 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:53.027931    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-8f9fc7-b4cffd49c-np2zh"]
Sep 10 22:17:53.028342 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:53.028330    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="78896706-4631-4e69-b205-dfad4be3b549" containerName="istio-proxy"
Sep 10 22:17:53.031146 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:53.031131    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-8f9fc7-b4cffd49c-np2zh"
Sep 10 22:17:53.040357 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:53.040331    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-8f9fc7-b4cffd49c-np2zh"]
Sep 10 22:17:53.125597 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:53.125565    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zf9n\" (UniqueName: \"kubernetes.io/projected/80516e33-181b-465f-b897-9cd78cdda578-kube-api-access-7zf9n\") pod \"payload-processing-e2e-explicit-8f9fc7-b4cffd49c-np2zh\" (UID: \"80516e33-181b-465f-b897-9cd78cdda578\") " pod="openshift-ingress/payload-processing-e2e-explicit-8f9fc7-b4cffd49c-np2zh"
Sep 10 22:17:53.125597 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:53.125601    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80516e33-181b-465f-b897-9cd78cdda578-plugins-config-volume\") pod \"payload-processing-e2e-explicit-8f9fc7-b4cffd49c-np2zh\" (UID: \"80516e33-181b-465f-b897-9cd78cdda578\") " pod="openshift-ingress/payload-processing-e2e-explicit-8f9fc7-b4cffd49c-np2zh"
Sep 10 22:17:53.125810 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:53.125726    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbfct\" (UniqueName: \"kubernetes.io/projected/e0718e77-8ff0-4b70-b172-16959e5c353b-kube-api-access-lbfct\") pod \"payload-pre-processing-e2e-explicit-8f9fc7-8597dd45f8-phpd9\" (UID: \"e0718e77-8ff0-4b70-b172-16959e5c353b\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-8f9fc7-8597dd45f8-phpd9"
Sep 10 22:17:53.125810 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:53.125781    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0718e77-8ff0-4b70-b172-16959e5c353b-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-8f9fc7-8597dd45f8-phpd9\" (UID: \"e0718e77-8ff0-4b70-b172-16959e5c353b\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-8f9fc7-8597dd45f8-phpd9"
Sep 10 22:17:53.126453 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:53.126435    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0718e77-8ff0-4b70-b172-16959e5c353b-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-8f9fc7-8597dd45f8-phpd9\" (UID: \"e0718e77-8ff0-4b70-b172-16959e5c353b\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-8f9fc7-8597dd45f8-phpd9"
Sep 10 22:17:53.138686 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:53.138658    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbfct\" (UniqueName: \"kubernetes.io/projected/e0718e77-8ff0-4b70-b172-16959e5c353b-kube-api-access-lbfct\") pod \"payload-pre-processing-e2e-explicit-8f9fc7-8597dd45f8-phpd9\" (UID: \"e0718e77-8ff0-4b70-b172-16959e5c353b\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-8f9fc7-8597dd45f8-phpd9"
Sep 10 22:17:53.226784 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:53.226704    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zf9n\" (UniqueName: \"kubernetes.io/projected/80516e33-181b-465f-b897-9cd78cdda578-kube-api-access-7zf9n\") pod \"payload-processing-e2e-explicit-8f9fc7-b4cffd49c-np2zh\" (UID: \"80516e33-181b-465f-b897-9cd78cdda578\") " pod="openshift-ingress/payload-processing-e2e-explicit-8f9fc7-b4cffd49c-np2zh"
Sep 10 22:17:53.226784 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:53.226737    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80516e33-181b-465f-b897-9cd78cdda578-plugins-config-volume\") pod \"payload-processing-e2e-explicit-8f9fc7-b4cffd49c-np2zh\" (UID: \"80516e33-181b-465f-b897-9cd78cdda578\") " pod="openshift-ingress/payload-processing-e2e-explicit-8f9fc7-b4cffd49c-np2zh"
Sep 10 22:17:53.227395 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:53.227379    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80516e33-181b-465f-b897-9cd78cdda578-plugins-config-volume\") pod \"payload-processing-e2e-explicit-8f9fc7-b4cffd49c-np2zh\" (UID: \"80516e33-181b-465f-b897-9cd78cdda578\") " pod="openshift-ingress/payload-processing-e2e-explicit-8f9fc7-b4cffd49c-np2zh"
Sep 10 22:17:53.235952 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:53.235922    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zf9n\" (UniqueName: \"kubernetes.io/projected/80516e33-181b-465f-b897-9cd78cdda578-kube-api-access-7zf9n\") pod \"payload-processing-e2e-explicit-8f9fc7-b4cffd49c-np2zh\" (UID: \"80516e33-181b-465f-b897-9cd78cdda578\") " pod="openshift-ingress/payload-processing-e2e-explicit-8f9fc7-b4cffd49c-np2zh"
Sep 10 22:17:53.293781 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:53.293746    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-8f9fc7-8597dd45f8-phpd9"
Sep 10 22:17:53.343037 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:53.342223    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-8f9fc7-b4cffd49c-np2zh"
Sep 10 22:17:53.498909 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:53.498827    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="78896706-4631-4e69-b205-dfad4be3b549" path="/var/lib/kubelet/pods/78896706-4631-4e69-b205-dfad4be3b549/volumes"
Sep 10 22:17:53.568260 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:53.568228    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-8f9fc7-8597dd45f8-phpd9"]
Sep 10 22:17:53.571440 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:17:53.571408    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0718e77_8ff0_4b70_b172_16959e5c353b.slice/crio-c9ad0fade46b63cb62a6da7eb66b21d43a666c92c4061d138e3acf67e33d2f1b WatchSource:0}: Error finding container c9ad0fade46b63cb62a6da7eb66b21d43a666c92c4061d138e3acf67e33d2f1b: Status 404 returned error can't find the container with id c9ad0fade46b63cb62a6da7eb66b21d43a666c92c4061d138e3acf67e33d2f1b
Sep 10 22:17:53.623262 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:53.623131    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-8f9fc7-b4cffd49c-np2zh"]
Sep 10 22:17:53.626841 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:17:53.626812    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80516e33_181b_465f_b897_9cd78cdda578.slice/crio-3e030e7b89b365030f10bd5d980cc1ec11c29d7275ee2744e76027a432aab99f WatchSource:0}: Error finding container 3e030e7b89b365030f10bd5d980cc1ec11c29d7275ee2744e76027a432aab99f: Status 404 returned error can't find the container with id 3e030e7b89b365030f10bd5d980cc1ec11c29d7275ee2744e76027a432aab99f
Sep 10 22:17:53.915262 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:53.915222    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-8f9fc7-b4cffd49c-np2zh" event={"ID":"80516e33-181b-465f-b897-9cd78cdda578","Type":"ContainerStarted","Data":"5cde55e1512c0203cbf987ca43db8165938afe5684cee8c37ee3d539b6844672"}
Sep 10 22:17:53.915478 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:53.915291    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-8f9fc7-b4cffd49c-np2zh" event={"ID":"80516e33-181b-465f-b897-9cd78cdda578","Type":"ContainerStarted","Data":"3e030e7b89b365030f10bd5d980cc1ec11c29d7275ee2744e76027a432aab99f"}
Sep 10 22:17:53.915478 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:53.915323    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-8f9fc7-b4cffd49c-np2zh"
Sep 10 22:17:53.916941 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:53.916903    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-8f9fc7-8597dd45f8-phpd9" event={"ID":"e0718e77-8ff0-4b70-b172-16959e5c353b","Type":"ContainerStarted","Data":"ed28b5d2f7faedcb3db613bda3d09d51bfcb75c80fb303f8ee2433d5903f00ad"}
Sep 10 22:17:53.917071 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:53.916963    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-8f9fc7-8597dd45f8-phpd9" event={"ID":"e0718e77-8ff0-4b70-b172-16959e5c353b","Type":"ContainerStarted","Data":"c9ad0fade46b63cb62a6da7eb66b21d43a666c92c4061d138e3acf67e33d2f1b"}
Sep 10 22:17:53.917071 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:53.916994    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-8f9fc7-8597dd45f8-phpd9"
Sep 10 22:17:53.936477 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:53.936429    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-8f9fc7-b4cffd49c-np2zh" podStartSLOduration=1.936414569 podStartE2EDuration="1.936414569s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 22:17:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:17:53.934761991 +0000 UTC m=+994.890829697" watchObservedRunningTime="2026-09-10 22:17:53.936414569 +0000 UTC m=+994.892482274"
Sep 10 22:17:53.953795 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:17:53.953747    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-8f9fc7-8597dd45f8-phpd9" podStartSLOduration=1.953732993 podStartE2EDuration="1.953732993s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 22:17:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:17:53.952485456 +0000 UTC m=+994.908553172" watchObservedRunningTime="2026-09-10 22:17:53.953732993 +0000 UTC m=+994.909800698"
Sep 10 22:18:04.926047 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:04.926013    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-8f9fc7-b4cffd49c-np2zh"
Sep 10 22:18:04.926457 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:04.926080    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-8f9fc7-8597dd45f8-phpd9"
Sep 10 22:18:05.895555 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:05.895500    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-conflict-a-e5a626b8-kserve-7688fbd-k5z56" podUID="2c423d2e-4cdb-496f-aeb7-3806b801970d" containerName="main" containerID="cri-o://89b9dab482a9a4318a316ec0654c9c215c9d9d8d5aa10c260fdd09abca231b5a" gracePeriod=45
Sep 10 22:18:06.132572 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:06.132549    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-e5a626b8-kserve-7688fbd-k5z56"
Sep 10 22:18:06.245090 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:06.245061    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c423d2e-4cdb-496f-aeb7-3806b801970d-home\") pod \"2c423d2e-4cdb-496f-aeb7-3806b801970d\" (UID: \"2c423d2e-4cdb-496f-aeb7-3806b801970d\") "
Sep 10 22:18:06.245290 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:06.245107    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c423d2e-4cdb-496f-aeb7-3806b801970d-tmp-dir\") pod \"2c423d2e-4cdb-496f-aeb7-3806b801970d\" (UID: \"2c423d2e-4cdb-496f-aeb7-3806b801970d\") "
Sep 10 22:18:06.245290 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:06.245143    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c423d2e-4cdb-496f-aeb7-3806b801970d-model-cache\") pod \"2c423d2e-4cdb-496f-aeb7-3806b801970d\" (UID: \"2c423d2e-4cdb-496f-aeb7-3806b801970d\") "
Sep 10 22:18:06.245290 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:06.245159    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-csppc\" (UniqueName: \"kubernetes.io/projected/2c423d2e-4cdb-496f-aeb7-3806b801970d-kube-api-access-csppc\") pod \"2c423d2e-4cdb-496f-aeb7-3806b801970d\" (UID: \"2c423d2e-4cdb-496f-aeb7-3806b801970d\") "
Sep 10 22:18:06.245290 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:06.245190    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c423d2e-4cdb-496f-aeb7-3806b801970d-tls-certs\") pod \"2c423d2e-4cdb-496f-aeb7-3806b801970d\" (UID: \"2c423d2e-4cdb-496f-aeb7-3806b801970d\") "
Sep 10 22:18:06.245290 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:06.245216    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c423d2e-4cdb-496f-aeb7-3806b801970d-dshm\") pod \"2c423d2e-4cdb-496f-aeb7-3806b801970d\" (UID: \"2c423d2e-4cdb-496f-aeb7-3806b801970d\") "
Sep 10 22:18:06.245544 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:06.245392    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c423d2e-4cdb-496f-aeb7-3806b801970d-home" (OuterVolumeSpecName: "home") pod "2c423d2e-4cdb-496f-aeb7-3806b801970d" (UID: "2c423d2e-4cdb-496f-aeb7-3806b801970d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 22:18:06.245544 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:06.245446    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c423d2e-4cdb-496f-aeb7-3806b801970d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2c423d2e-4cdb-496f-aeb7-3806b801970d" (UID: "2c423d2e-4cdb-496f-aeb7-3806b801970d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 22:18:06.245544 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:06.245463    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c423d2e-4cdb-496f-aeb7-3806b801970d-model-cache" (OuterVolumeSpecName: "model-cache") pod "2c423d2e-4cdb-496f-aeb7-3806b801970d" (UID: "2c423d2e-4cdb-496f-aeb7-3806b801970d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 22:18:06.245688 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:06.245575    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c423d2e-4cdb-496f-aeb7-3806b801970d-home\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:18:06.245688 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:06.245592    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c423d2e-4cdb-496f-aeb7-3806b801970d-tmp-dir\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:18:06.245688 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:06.245606    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c423d2e-4cdb-496f-aeb7-3806b801970d-model-cache\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:18:06.247496 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:06.247464    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c423d2e-4cdb-496f-aeb7-3806b801970d-dshm" (OuterVolumeSpecName: "dshm") pod "2c423d2e-4cdb-496f-aeb7-3806b801970d" (UID: "2c423d2e-4cdb-496f-aeb7-3806b801970d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 22:18:06.247619 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:06.247525    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c423d2e-4cdb-496f-aeb7-3806b801970d-kube-api-access-csppc" (OuterVolumeSpecName: "kube-api-access-csppc") pod "2c423d2e-4cdb-496f-aeb7-3806b801970d" (UID: "2c423d2e-4cdb-496f-aeb7-3806b801970d"). InnerVolumeSpecName "kube-api-access-csppc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 22:18:06.247679 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:06.247610    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c423d2e-4cdb-496f-aeb7-3806b801970d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2c423d2e-4cdb-496f-aeb7-3806b801970d" (UID: "2c423d2e-4cdb-496f-aeb7-3806b801970d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 22:18:06.346483 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:06.346449    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-csppc\" (UniqueName: \"kubernetes.io/projected/2c423d2e-4cdb-496f-aeb7-3806b801970d-kube-api-access-csppc\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:18:06.346483 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:06.346479    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c423d2e-4cdb-496f-aeb7-3806b801970d-tls-certs\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:18:06.346483 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:06.346489    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c423d2e-4cdb-496f-aeb7-3806b801970d-dshm\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:18:06.699481 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:06.699432    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-conflict-b-e5a626b8-kserve-66559ccb54-b44ch" podUID="ee68b40d-907e-45fb-a27b-d8b79eae19bc" containerName="main" containerID="cri-o://9480dc11d6895ac72ba51d5048580751add59be09d2532c7cf9519b82057b09d" gracePeriod=45
Sep 10 22:18:06.942567 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:06.942542    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-e5a626b8-kserve-66559ccb54-b44ch"
Sep 10 22:18:06.983410 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:06.983327    2582 generic.go:356] "Generic (PLEG): container finished" podID="2c423d2e-4cdb-496f-aeb7-3806b801970d" containerID="89b9dab482a9a4318a316ec0654c9c215c9d9d8d5aa10c260fdd09abca231b5a" exitCode=0
Sep 10 22:18:06.983410 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:06.983369    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-e5a626b8-kserve-7688fbd-k5z56"
Sep 10 22:18:06.983596 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:06.983410    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-e5a626b8-kserve-7688fbd-k5z56" event={"ID":"2c423d2e-4cdb-496f-aeb7-3806b801970d","Type":"ContainerDied","Data":"89b9dab482a9a4318a316ec0654c9c215c9d9d8d5aa10c260fdd09abca231b5a"}
Sep 10 22:18:06.983596 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:06.983457    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-e5a626b8-kserve-7688fbd-k5z56" event={"ID":"2c423d2e-4cdb-496f-aeb7-3806b801970d","Type":"ContainerDied","Data":"cc433e6b7c6fb68a3f05fb6f68796627657f9ca673a8587e5054372d7e1d1e65"}
Sep 10 22:18:06.983596 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:06.983484    2582 scope.go:117] "RemoveContainer" containerID="89b9dab482a9a4318a316ec0654c9c215c9d9d8d5aa10c260fdd09abca231b5a"
Sep 10 22:18:06.985041 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:06.985013    2582 generic.go:356] "Generic (PLEG): container finished" podID="ee68b40d-907e-45fb-a27b-d8b79eae19bc" containerID="9480dc11d6895ac72ba51d5048580751add59be09d2532c7cf9519b82057b09d" exitCode=0
Sep 10 22:18:06.985163 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:06.985053    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-e5a626b8-kserve-66559ccb54-b44ch" event={"ID":"ee68b40d-907e-45fb-a27b-d8b79eae19bc","Type":"ContainerDied","Data":"9480dc11d6895ac72ba51d5048580751add59be09d2532c7cf9519b82057b09d"}
Sep 10 22:18:06.985163 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:06.985100    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-e5a626b8-kserve-66559ccb54-b44ch" event={"ID":"ee68b40d-907e-45fb-a27b-d8b79eae19bc","Type":"ContainerDied","Data":"509e1d5460dcb3d4e7aeed6f9c5aa6de00b37e55da5a99fbeaeceb1f832f5bcf"}
Sep 10 22:18:06.985163 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:06.985062    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-e5a626b8-kserve-66559ccb54-b44ch"
Sep 10 22:18:06.992275 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:06.992255    2582 scope.go:117] "RemoveContainer" containerID="89b9dab482a9a4318a316ec0654c9c215c9d9d8d5aa10c260fdd09abca231b5a"
Sep 10 22:18:06.992533 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:18:06.992504    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89b9dab482a9a4318a316ec0654c9c215c9d9d8d5aa10c260fdd09abca231b5a\": container with ID starting with 89b9dab482a9a4318a316ec0654c9c215c9d9d8d5aa10c260fdd09abca231b5a not found: ID does not exist" containerID="89b9dab482a9a4318a316ec0654c9c215c9d9d8d5aa10c260fdd09abca231b5a"
Sep 10 22:18:06.992594 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:06.992529    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89b9dab482a9a4318a316ec0654c9c215c9d9d8d5aa10c260fdd09abca231b5a"} err="failed to get container status \"89b9dab482a9a4318a316ec0654c9c215c9d9d8d5aa10c260fdd09abca231b5a\": rpc error: code = NotFound desc = could not find container \"89b9dab482a9a4318a316ec0654c9c215c9d9d8d5aa10c260fdd09abca231b5a\": container with ID starting with 89b9dab482a9a4318a316ec0654c9c215c9d9d8d5aa10c260fdd09abca231b5a not found: ID does not exist"
Sep 10 22:18:06.992594 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:06.992545    2582 scope.go:117] "RemoveContainer" containerID="9480dc11d6895ac72ba51d5048580751add59be09d2532c7cf9519b82057b09d"
Sep 10 22:18:07.000154 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:07.000129    2582 scope.go:117] "RemoveContainer" containerID="9480dc11d6895ac72ba51d5048580751add59be09d2532c7cf9519b82057b09d"
Sep 10 22:18:07.000400 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:18:07.000380    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9480dc11d6895ac72ba51d5048580751add59be09d2532c7cf9519b82057b09d\": container with ID starting with 9480dc11d6895ac72ba51d5048580751add59be09d2532c7cf9519b82057b09d not found: ID does not exist" containerID="9480dc11d6895ac72ba51d5048580751add59be09d2532c7cf9519b82057b09d"
Sep 10 22:18:07.000454 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:07.000406    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9480dc11d6895ac72ba51d5048580751add59be09d2532c7cf9519b82057b09d"} err="failed to get container status \"9480dc11d6895ac72ba51d5048580751add59be09d2532c7cf9519b82057b09d\": rpc error: code = NotFound desc = could not find container \"9480dc11d6895ac72ba51d5048580751add59be09d2532c7cf9519b82057b09d\": container with ID starting with 9480dc11d6895ac72ba51d5048580751add59be09d2532c7cf9519b82057b09d not found: ID does not exist"
Sep 10 22:18:07.004075 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:07.004055    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-a-e5a626b8-kserve-7688fbd-k5z56"]
Sep 10 22:18:07.009627 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:07.009605    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-conflict-a-e5a626b8-kserve-7688fbd-k5z56"]
Sep 10 22:18:07.052245 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:07.052223    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee68b40d-907e-45fb-a27b-d8b79eae19bc-home\") pod \"ee68b40d-907e-45fb-a27b-d8b79eae19bc\" (UID: \"ee68b40d-907e-45fb-a27b-d8b79eae19bc\") "
Sep 10 22:18:07.052338 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:07.052292    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee68b40d-907e-45fb-a27b-d8b79eae19bc-tmp-dir\") pod \"ee68b40d-907e-45fb-a27b-d8b79eae19bc\" (UID: \"ee68b40d-907e-45fb-a27b-d8b79eae19bc\") "
Sep 10 22:18:07.052409 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:07.052348    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j9c6l\" (UniqueName: \"kubernetes.io/projected/ee68b40d-907e-45fb-a27b-d8b79eae19bc-kube-api-access-j9c6l\") pod \"ee68b40d-907e-45fb-a27b-d8b79eae19bc\" (UID: \"ee68b40d-907e-45fb-a27b-d8b79eae19bc\") "
Sep 10 22:18:07.052409 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:07.052379    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee68b40d-907e-45fb-a27b-d8b79eae19bc-model-cache\") pod \"ee68b40d-907e-45fb-a27b-d8b79eae19bc\" (UID: \"ee68b40d-907e-45fb-a27b-d8b79eae19bc\") "
Sep 10 22:18:07.052409 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:07.052393    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee68b40d-907e-45fb-a27b-d8b79eae19bc-dshm\") pod \"ee68b40d-907e-45fb-a27b-d8b79eae19bc\" (UID: \"ee68b40d-907e-45fb-a27b-d8b79eae19bc\") "
Sep 10 22:18:07.052409 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:07.052407    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee68b40d-907e-45fb-a27b-d8b79eae19bc-tls-certs\") pod \"ee68b40d-907e-45fb-a27b-d8b79eae19bc\" (UID: \"ee68b40d-907e-45fb-a27b-d8b79eae19bc\") "
Sep 10 22:18:07.052611 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:07.052504    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee68b40d-907e-45fb-a27b-d8b79eae19bc-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ee68b40d-907e-45fb-a27b-d8b79eae19bc" (UID: "ee68b40d-907e-45fb-a27b-d8b79eae19bc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 22:18:07.052611 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:07.052520    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee68b40d-907e-45fb-a27b-d8b79eae19bc-home" (OuterVolumeSpecName: "home") pod "ee68b40d-907e-45fb-a27b-d8b79eae19bc" (UID: "ee68b40d-907e-45fb-a27b-d8b79eae19bc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 22:18:07.052682 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:07.052635    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee68b40d-907e-45fb-a27b-d8b79eae19bc-model-cache" (OuterVolumeSpecName: "model-cache") pod "ee68b40d-907e-45fb-a27b-d8b79eae19bc" (UID: "ee68b40d-907e-45fb-a27b-d8b79eae19bc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 22:18:07.052682 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:07.052667    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee68b40d-907e-45fb-a27b-d8b79eae19bc-tmp-dir\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:18:07.052747 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:07.052687    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee68b40d-907e-45fb-a27b-d8b79eae19bc-home\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:18:07.054570 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:07.054546    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee68b40d-907e-45fb-a27b-d8b79eae19bc-dshm" (OuterVolumeSpecName: "dshm") pod "ee68b40d-907e-45fb-a27b-d8b79eae19bc" (UID: "ee68b40d-907e-45fb-a27b-d8b79eae19bc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 22:18:07.054570 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:07.054559    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee68b40d-907e-45fb-a27b-d8b79eae19bc-kube-api-access-j9c6l" (OuterVolumeSpecName: "kube-api-access-j9c6l") pod "ee68b40d-907e-45fb-a27b-d8b79eae19bc" (UID: "ee68b40d-907e-45fb-a27b-d8b79eae19bc"). InnerVolumeSpecName "kube-api-access-j9c6l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 22:18:07.054681 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:07.054546    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee68b40d-907e-45fb-a27b-d8b79eae19bc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ee68b40d-907e-45fb-a27b-d8b79eae19bc" (UID: "ee68b40d-907e-45fb-a27b-d8b79eae19bc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 22:18:07.153248 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:07.153207    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee68b40d-907e-45fb-a27b-d8b79eae19bc-model-cache\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:18:07.153248 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:07.153244    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee68b40d-907e-45fb-a27b-d8b79eae19bc-dshm\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:18:07.153248 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:07.153253    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee68b40d-907e-45fb-a27b-d8b79eae19bc-tls-certs\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:18:07.153248 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:07.153262    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9c6l\" (UniqueName: \"kubernetes.io/projected/ee68b40d-907e-45fb-a27b-d8b79eae19bc-kube-api-access-j9c6l\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:18:07.304889 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:07.304843    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-b-e5a626b8-kserve-66559ccb54-b44ch"]
Sep 10 22:18:07.309129 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:07.309104    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-conflict-b-e5a626b8-kserve-66559ccb54-b44ch"]
Sep 10 22:18:07.498466 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:07.498434    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2c423d2e-4cdb-496f-aeb7-3806b801970d" path="/var/lib/kubelet/pods/2c423d2e-4cdb-496f-aeb7-3806b801970d/volumes"
Sep 10 22:18:07.498821 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:07.498807    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ee68b40d-907e-45fb-a27b-d8b79eae19bc" path="/var/lib/kubelet/pods/ee68b40d-907e-45fb-a27b-d8b79eae19bc/volumes"
Sep 10 22:18:17.887422 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:17.887388    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-1c252b/test-model-1c252b-kserve-9b5ffcc55-j8qpx"]
Sep 10 22:18:17.887803 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:17.887703    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c423d2e-4cdb-496f-aeb7-3806b801970d" containerName="main"
Sep 10 22:18:17.887803 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:17.887713    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c423d2e-4cdb-496f-aeb7-3806b801970d" containerName="main"
Sep 10 22:18:17.887803 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:17.887731    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee68b40d-907e-45fb-a27b-d8b79eae19bc" containerName="main"
Sep 10 22:18:17.887803 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:17.887737    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee68b40d-907e-45fb-a27b-d8b79eae19bc" containerName="main"
Sep 10 22:18:17.887803 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:17.887789    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c423d2e-4cdb-496f-aeb7-3806b801970d" containerName="main"
Sep 10 22:18:17.887803 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:17.887796    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee68b40d-907e-45fb-a27b-d8b79eae19bc" containerName="main"
Sep 10 22:18:17.890584 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:17.890566    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-1c252b/test-model-1c252b-kserve-9b5ffcc55-j8qpx"
Sep 10 22:18:17.893405 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:17.893382    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-1c252b\"/\"openshift-service-ca.crt\""
Sep 10 22:18:17.893544 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:17.893409    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-1c252b\"/\"default-dockercfg-7bvv7\""
Sep 10 22:18:17.893544 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:17.893414    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-1c252b\"/\"kube-root-ca.crt\""
Sep 10 22:18:17.893544 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:17.893502    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-1c252b\"/\"test-model-1c252b-kserve-self-signed-certs\""
Sep 10 22:18:17.900566 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:17.900544    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-1c252b/test-model-1c252b-kserve-9b5ffcc55-j8qpx"]
Sep 10 22:18:18.043787 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:18.043761    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba783dae-1685-4cb9-b045-dc838e0a596c-model-cache\") pod \"test-model-1c252b-kserve-9b5ffcc55-j8qpx\" (UID: \"ba783dae-1685-4cb9-b045-dc838e0a596c\") " pod="ai-tenant-e2e-inf-b-1c252b/test-model-1c252b-kserve-9b5ffcc55-j8qpx"
Sep 10 22:18:18.044007 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:18.043795    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba783dae-1685-4cb9-b045-dc838e0a596c-tls-certs\") pod \"test-model-1c252b-kserve-9b5ffcc55-j8qpx\" (UID: \"ba783dae-1685-4cb9-b045-dc838e0a596c\") " pod="ai-tenant-e2e-inf-b-1c252b/test-model-1c252b-kserve-9b5ffcc55-j8qpx"
Sep 10 22:18:18.044007 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:18.043934    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba783dae-1685-4cb9-b045-dc838e0a596c-tmp-dir\") pod \"test-model-1c252b-kserve-9b5ffcc55-j8qpx\" (UID: \"ba783dae-1685-4cb9-b045-dc838e0a596c\") " pod="ai-tenant-e2e-inf-b-1c252b/test-model-1c252b-kserve-9b5ffcc55-j8qpx"
Sep 10 22:18:18.044007 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:18.043966    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwm6v\" (UniqueName: \"kubernetes.io/projected/ba783dae-1685-4cb9-b045-dc838e0a596c-kube-api-access-pwm6v\") pod \"test-model-1c252b-kserve-9b5ffcc55-j8qpx\" (UID: \"ba783dae-1685-4cb9-b045-dc838e0a596c\") " pod="ai-tenant-e2e-inf-b-1c252b/test-model-1c252b-kserve-9b5ffcc55-j8qpx"
Sep 10 22:18:18.044007 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:18.043994    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba783dae-1685-4cb9-b045-dc838e0a596c-home\") pod \"test-model-1c252b-kserve-9b5ffcc55-j8qpx\" (UID: \"ba783dae-1685-4cb9-b045-dc838e0a596c\") " pod="ai-tenant-e2e-inf-b-1c252b/test-model-1c252b-kserve-9b5ffcc55-j8qpx"
Sep 10 22:18:18.044175 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:18.044018    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba783dae-1685-4cb9-b045-dc838e0a596c-dshm\") pod \"test-model-1c252b-kserve-9b5ffcc55-j8qpx\" (UID: \"ba783dae-1685-4cb9-b045-dc838e0a596c\") " pod="ai-tenant-e2e-inf-b-1c252b/test-model-1c252b-kserve-9b5ffcc55-j8qpx"
Sep 10 22:18:18.144696 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:18.144634    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwm6v\" (UniqueName: \"kubernetes.io/projected/ba783dae-1685-4cb9-b045-dc838e0a596c-kube-api-access-pwm6v\") pod \"test-model-1c252b-kserve-9b5ffcc55-j8qpx\" (UID: \"ba783dae-1685-4cb9-b045-dc838e0a596c\") " pod="ai-tenant-e2e-inf-b-1c252b/test-model-1c252b-kserve-9b5ffcc55-j8qpx"
Sep 10 22:18:18.144696 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:18.144666    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba783dae-1685-4cb9-b045-dc838e0a596c-home\") pod \"test-model-1c252b-kserve-9b5ffcc55-j8qpx\" (UID: \"ba783dae-1685-4cb9-b045-dc838e0a596c\") " pod="ai-tenant-e2e-inf-b-1c252b/test-model-1c252b-kserve-9b5ffcc55-j8qpx"
Sep 10 22:18:18.144696 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:18.144688    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba783dae-1685-4cb9-b045-dc838e0a596c-dshm\") pod \"test-model-1c252b-kserve-9b5ffcc55-j8qpx\" (UID: \"ba783dae-1685-4cb9-b045-dc838e0a596c\") " pod="ai-tenant-e2e-inf-b-1c252b/test-model-1c252b-kserve-9b5ffcc55-j8qpx"
Sep 10 22:18:18.144945 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:18.144715    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba783dae-1685-4cb9-b045-dc838e0a596c-model-cache\") pod \"test-model-1c252b-kserve-9b5ffcc55-j8qpx\" (UID: \"ba783dae-1685-4cb9-b045-dc838e0a596c\") " pod="ai-tenant-e2e-inf-b-1c252b/test-model-1c252b-kserve-9b5ffcc55-j8qpx"
Sep 10 22:18:18.144945 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:18.144741    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba783dae-1685-4cb9-b045-dc838e0a596c-tls-certs\") pod \"test-model-1c252b-kserve-9b5ffcc55-j8qpx\" (UID: \"ba783dae-1685-4cb9-b045-dc838e0a596c\") " pod="ai-tenant-e2e-inf-b-1c252b/test-model-1c252b-kserve-9b5ffcc55-j8qpx"
Sep 10 22:18:18.144945 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:18.144805    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba783dae-1685-4cb9-b045-dc838e0a596c-tmp-dir\") pod \"test-model-1c252b-kserve-9b5ffcc55-j8qpx\" (UID: \"ba783dae-1685-4cb9-b045-dc838e0a596c\") " pod="ai-tenant-e2e-inf-b-1c252b/test-model-1c252b-kserve-9b5ffcc55-j8qpx"
Sep 10 22:18:18.145116 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:18.145006    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba783dae-1685-4cb9-b045-dc838e0a596c-home\") pod \"test-model-1c252b-kserve-9b5ffcc55-j8qpx\" (UID: \"ba783dae-1685-4cb9-b045-dc838e0a596c\") " pod="ai-tenant-e2e-inf-b-1c252b/test-model-1c252b-kserve-9b5ffcc55-j8qpx"
Sep 10 22:18:18.145177 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:18.145134    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba783dae-1685-4cb9-b045-dc838e0a596c-tmp-dir\") pod \"test-model-1c252b-kserve-9b5ffcc55-j8qpx\" (UID: \"ba783dae-1685-4cb9-b045-dc838e0a596c\") " pod="ai-tenant-e2e-inf-b-1c252b/test-model-1c252b-kserve-9b5ffcc55-j8qpx"
Sep 10 22:18:18.145177 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:18.145165    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba783dae-1685-4cb9-b045-dc838e0a596c-model-cache\") pod \"test-model-1c252b-kserve-9b5ffcc55-j8qpx\" (UID: \"ba783dae-1685-4cb9-b045-dc838e0a596c\") " pod="ai-tenant-e2e-inf-b-1c252b/test-model-1c252b-kserve-9b5ffcc55-j8qpx"
Sep 10 22:18:18.146895 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:18.146863    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba783dae-1685-4cb9-b045-dc838e0a596c-dshm\") pod \"test-model-1c252b-kserve-9b5ffcc55-j8qpx\" (UID: \"ba783dae-1685-4cb9-b045-dc838e0a596c\") " pod="ai-tenant-e2e-inf-b-1c252b/test-model-1c252b-kserve-9b5ffcc55-j8qpx"
Sep 10 22:18:18.147129 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:18.147114    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba783dae-1685-4cb9-b045-dc838e0a596c-tls-certs\") pod \"test-model-1c252b-kserve-9b5ffcc55-j8qpx\" (UID: \"ba783dae-1685-4cb9-b045-dc838e0a596c\") " pod="ai-tenant-e2e-inf-b-1c252b/test-model-1c252b-kserve-9b5ffcc55-j8qpx"
Sep 10 22:18:18.152655 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:18.152638    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwm6v\" (UniqueName: \"kubernetes.io/projected/ba783dae-1685-4cb9-b045-dc838e0a596c-kube-api-access-pwm6v\") pod \"test-model-1c252b-kserve-9b5ffcc55-j8qpx\" (UID: \"ba783dae-1685-4cb9-b045-dc838e0a596c\") " pod="ai-tenant-e2e-inf-b-1c252b/test-model-1c252b-kserve-9b5ffcc55-j8qpx"
Sep 10 22:18:18.200652 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:18.200626    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-1c252b/test-model-1c252b-kserve-9b5ffcc55-j8qpx"
Sep 10 22:18:18.479982 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:18.479951    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-1c252b/test-model-1c252b-kserve-9b5ffcc55-j8qpx"]
Sep 10 22:18:18.483422 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:18:18.483395    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podba783dae_1685_4cb9_b045_dc838e0a596c.slice/crio-9f378010871bc2285a1bc76370b3dc80c044c6d2b75f6063487d18765125d4c6 WatchSource:0}: Error finding container 9f378010871bc2285a1bc76370b3dc80c044c6d2b75f6063487d18765125d4c6: Status 404 returned error can't find the container with id 9f378010871bc2285a1bc76370b3dc80c044c6d2b75f6063487d18765125d4c6
Sep 10 22:18:19.042482 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:19.042446    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-1c252b/test-model-1c252b-kserve-9b5ffcc55-j8qpx" event={"ID":"ba783dae-1685-4cb9-b045-dc838e0a596c","Type":"ContainerStarted","Data":"c22dcab9d429f9da9d313f07a06eac95b830113297986312a3f7c3791376ce23"}
Sep 10 22:18:19.042840 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:19.042493    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-1c252b/test-model-1c252b-kserve-9b5ffcc55-j8qpx" event={"ID":"ba783dae-1685-4cb9-b045-dc838e0a596c","Type":"ContainerStarted","Data":"9f378010871bc2285a1bc76370b3dc80c044c6d2b75f6063487d18765125d4c6"}
Sep 10 22:18:19.069787 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:19.069742    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-1c252b/test-model-1c252b-kserve-9b5ffcc55-j8qpx" podStartSLOduration=2.069722571 podStartE2EDuration="2.069722571s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 22:18:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:18:19.068903328 +0000 UTC m=+1020.024971034" watchObservedRunningTime="2026-09-10 22:18:19.069722571 +0000 UTC m=+1020.025790277"
Sep 10 22:18:28.201205 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:28.201171    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-1c252b/test-model-1c252b-kserve-9b5ffcc55-j8qpx"
Sep 10 22:18:28.201622 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:28.201219    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-1c252b/test-model-1c252b-kserve-9b5ffcc55-j8qpx"
Sep 10 22:18:28.214281 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:28.214256    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-1c252b/test-model-1c252b-kserve-9b5ffcc55-j8qpx"
Sep 10 22:18:29.099585 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:29.099556    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-1c252b/test-model-1c252b-kserve-9b5ffcc55-j8qpx"
Sep 10 22:18:37.710443 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:37.710407    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-8f9fc7/explicit-model-8f9fc7-kserve-6cb5757c8c-bnn5f"]
Sep 10 22:18:37.715438 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:37.715421    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-8f9fc7/explicit-model-8f9fc7-kserve-6cb5757c8c-bnn5f"
Sep 10 22:18:37.718464 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:37.718440    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-8f9fc7\"/\"openshift-service-ca.crt\""
Sep 10 22:18:37.718601 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:37.718483    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-8f9fc7\"/\"default-dockercfg-4ntkt\""
Sep 10 22:18:37.718601 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:37.718530    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-8f9fc7\"/\"explicit-model-8f9fc7-kserve-self-signed-certs\""
Sep 10 22:18:37.718699 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:37.718611    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-8f9fc7\"/\"kube-root-ca.crt\""
Sep 10 22:18:37.725186 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:37.725166    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-8f9fc7/explicit-model-8f9fc7-kserve-6cb5757c8c-bnn5f"]
Sep 10 22:18:37.798768 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:37.798741    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f69lv\" (UniqueName: \"kubernetes.io/projected/d9cf63f3-ca6e-4bed-a508-5054cb0daf9f-kube-api-access-f69lv\") pod \"explicit-model-8f9fc7-kserve-6cb5757c8c-bnn5f\" (UID: \"d9cf63f3-ca6e-4bed-a508-5054cb0daf9f\") " pod="ai-tenant-e2e-explicit-8f9fc7/explicit-model-8f9fc7-kserve-6cb5757c8c-bnn5f"
Sep 10 22:18:37.798948 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:37.798800    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9cf63f3-ca6e-4bed-a508-5054cb0daf9f-tls-certs\") pod \"explicit-model-8f9fc7-kserve-6cb5757c8c-bnn5f\" (UID: \"d9cf63f3-ca6e-4bed-a508-5054cb0daf9f\") " pod="ai-tenant-e2e-explicit-8f9fc7/explicit-model-8f9fc7-kserve-6cb5757c8c-bnn5f"
Sep 10 22:18:37.798948 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:37.798842    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9cf63f3-ca6e-4bed-a508-5054cb0daf9f-dshm\") pod \"explicit-model-8f9fc7-kserve-6cb5757c8c-bnn5f\" (UID: \"d9cf63f3-ca6e-4bed-a508-5054cb0daf9f\") " pod="ai-tenant-e2e-explicit-8f9fc7/explicit-model-8f9fc7-kserve-6cb5757c8c-bnn5f"
Sep 10 22:18:37.798948 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:37.798933    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9cf63f3-ca6e-4bed-a508-5054cb0daf9f-tmp-dir\") pod \"explicit-model-8f9fc7-kserve-6cb5757c8c-bnn5f\" (UID: \"d9cf63f3-ca6e-4bed-a508-5054cb0daf9f\") " pod="ai-tenant-e2e-explicit-8f9fc7/explicit-model-8f9fc7-kserve-6cb5757c8c-bnn5f"
Sep 10 22:18:37.799108 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:37.798968    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9cf63f3-ca6e-4bed-a508-5054cb0daf9f-model-cache\") pod \"explicit-model-8f9fc7-kserve-6cb5757c8c-bnn5f\" (UID: \"d9cf63f3-ca6e-4bed-a508-5054cb0daf9f\") " pod="ai-tenant-e2e-explicit-8f9fc7/explicit-model-8f9fc7-kserve-6cb5757c8c-bnn5f"
Sep 10 22:18:37.799108 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:37.798996    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9cf63f3-ca6e-4bed-a508-5054cb0daf9f-home\") pod \"explicit-model-8f9fc7-kserve-6cb5757c8c-bnn5f\" (UID: \"d9cf63f3-ca6e-4bed-a508-5054cb0daf9f\") " pod="ai-tenant-e2e-explicit-8f9fc7/explicit-model-8f9fc7-kserve-6cb5757c8c-bnn5f"
Sep 10 22:18:37.899562 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:37.899536    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9cf63f3-ca6e-4bed-a508-5054cb0daf9f-tls-certs\") pod \"explicit-model-8f9fc7-kserve-6cb5757c8c-bnn5f\" (UID: \"d9cf63f3-ca6e-4bed-a508-5054cb0daf9f\") " pod="ai-tenant-e2e-explicit-8f9fc7/explicit-model-8f9fc7-kserve-6cb5757c8c-bnn5f"
Sep 10 22:18:37.899668 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:37.899573    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9cf63f3-ca6e-4bed-a508-5054cb0daf9f-dshm\") pod \"explicit-model-8f9fc7-kserve-6cb5757c8c-bnn5f\" (UID: \"d9cf63f3-ca6e-4bed-a508-5054cb0daf9f\") " pod="ai-tenant-e2e-explicit-8f9fc7/explicit-model-8f9fc7-kserve-6cb5757c8c-bnn5f"
Sep 10 22:18:37.899668 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:37.899600    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9cf63f3-ca6e-4bed-a508-5054cb0daf9f-tmp-dir\") pod \"explicit-model-8f9fc7-kserve-6cb5757c8c-bnn5f\" (UID: \"d9cf63f3-ca6e-4bed-a508-5054cb0daf9f\") " pod="ai-tenant-e2e-explicit-8f9fc7/explicit-model-8f9fc7-kserve-6cb5757c8c-bnn5f"
Sep 10 22:18:37.899668 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:37.899617    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9cf63f3-ca6e-4bed-a508-5054cb0daf9f-model-cache\") pod \"explicit-model-8f9fc7-kserve-6cb5757c8c-bnn5f\" (UID: \"d9cf63f3-ca6e-4bed-a508-5054cb0daf9f\") " pod="ai-tenant-e2e-explicit-8f9fc7/explicit-model-8f9fc7-kserve-6cb5757c8c-bnn5f"
Sep 10 22:18:37.899668 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:37.899636    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9cf63f3-ca6e-4bed-a508-5054cb0daf9f-home\") pod \"explicit-model-8f9fc7-kserve-6cb5757c8c-bnn5f\" (UID: \"d9cf63f3-ca6e-4bed-a508-5054cb0daf9f\") " pod="ai-tenant-e2e-explicit-8f9fc7/explicit-model-8f9fc7-kserve-6cb5757c8c-bnn5f"
Sep 10 22:18:37.899668 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:37.899669    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f69lv\" (UniqueName: \"kubernetes.io/projected/d9cf63f3-ca6e-4bed-a508-5054cb0daf9f-kube-api-access-f69lv\") pod \"explicit-model-8f9fc7-kserve-6cb5757c8c-bnn5f\" (UID: \"d9cf63f3-ca6e-4bed-a508-5054cb0daf9f\") " pod="ai-tenant-e2e-explicit-8f9fc7/explicit-model-8f9fc7-kserve-6cb5757c8c-bnn5f"
Sep 10 22:18:37.900076 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:37.900022    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9cf63f3-ca6e-4bed-a508-5054cb0daf9f-tmp-dir\") pod \"explicit-model-8f9fc7-kserve-6cb5757c8c-bnn5f\" (UID: \"d9cf63f3-ca6e-4bed-a508-5054cb0daf9f\") " pod="ai-tenant-e2e-explicit-8f9fc7/explicit-model-8f9fc7-kserve-6cb5757c8c-bnn5f"
Sep 10 22:18:37.900076 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:37.900038    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9cf63f3-ca6e-4bed-a508-5054cb0daf9f-model-cache\") pod \"explicit-model-8f9fc7-kserve-6cb5757c8c-bnn5f\" (UID: \"d9cf63f3-ca6e-4bed-a508-5054cb0daf9f\") " pod="ai-tenant-e2e-explicit-8f9fc7/explicit-model-8f9fc7-kserve-6cb5757c8c-bnn5f"
Sep 10 22:18:37.900243 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:37.900095    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9cf63f3-ca6e-4bed-a508-5054cb0daf9f-home\") pod \"explicit-model-8f9fc7-kserve-6cb5757c8c-bnn5f\" (UID: \"d9cf63f3-ca6e-4bed-a508-5054cb0daf9f\") " pod="ai-tenant-e2e-explicit-8f9fc7/explicit-model-8f9fc7-kserve-6cb5757c8c-bnn5f"
Sep 10 22:18:37.901944 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:37.901918    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9cf63f3-ca6e-4bed-a508-5054cb0daf9f-dshm\") pod \"explicit-model-8f9fc7-kserve-6cb5757c8c-bnn5f\" (UID: \"d9cf63f3-ca6e-4bed-a508-5054cb0daf9f\") " pod="ai-tenant-e2e-explicit-8f9fc7/explicit-model-8f9fc7-kserve-6cb5757c8c-bnn5f"
Sep 10 22:18:37.902192 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:37.902176    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9cf63f3-ca6e-4bed-a508-5054cb0daf9f-tls-certs\") pod \"explicit-model-8f9fc7-kserve-6cb5757c8c-bnn5f\" (UID: \"d9cf63f3-ca6e-4bed-a508-5054cb0daf9f\") " pod="ai-tenant-e2e-explicit-8f9fc7/explicit-model-8f9fc7-kserve-6cb5757c8c-bnn5f"
Sep 10 22:18:37.907468 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:37.907447    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f69lv\" (UniqueName: \"kubernetes.io/projected/d9cf63f3-ca6e-4bed-a508-5054cb0daf9f-kube-api-access-f69lv\") pod \"explicit-model-8f9fc7-kserve-6cb5757c8c-bnn5f\" (UID: \"d9cf63f3-ca6e-4bed-a508-5054cb0daf9f\") " pod="ai-tenant-e2e-explicit-8f9fc7/explicit-model-8f9fc7-kserve-6cb5757c8c-bnn5f"
Sep 10 22:18:38.025464 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:38.025405    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-8f9fc7/explicit-model-8f9fc7-kserve-6cb5757c8c-bnn5f"
Sep 10 22:18:38.310180 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:38.310153    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-8f9fc7/explicit-model-8f9fc7-kserve-6cb5757c8c-bnn5f"]
Sep 10 22:18:38.312376 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:18:38.312350    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd9cf63f3_ca6e_4bed_a508_5054cb0daf9f.slice/crio-e701e83e7d76d1e5009801952961828a9c753c6f17e580dbd97e74fb89004653 WatchSource:0}: Error finding container e701e83e7d76d1e5009801952961828a9c753c6f17e580dbd97e74fb89004653: Status 404 returned error can't find the container with id e701e83e7d76d1e5009801952961828a9c753c6f17e580dbd97e74fb89004653
Sep 10 22:18:39.144670 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:39.144628    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-8f9fc7/explicit-model-8f9fc7-kserve-6cb5757c8c-bnn5f" event={"ID":"d9cf63f3-ca6e-4bed-a508-5054cb0daf9f","Type":"ContainerStarted","Data":"9b6168412382ea59c71727a957166b8d2ac86440653c06e79b9a081b1c712222"}
Sep 10 22:18:39.144670 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:39.144674    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-8f9fc7/explicit-model-8f9fc7-kserve-6cb5757c8c-bnn5f" event={"ID":"d9cf63f3-ca6e-4bed-a508-5054cb0daf9f","Type":"ContainerStarted","Data":"e701e83e7d76d1e5009801952961828a9c753c6f17e580dbd97e74fb89004653"}
Sep 10 22:18:39.160284 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:39.160238    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-8f9fc7/explicit-model-8f9fc7-kserve-6cb5757c8c-bnn5f" podStartSLOduration=2.160222037 podStartE2EDuration="2.160222037s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 22:18:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:18:39.159128828 +0000 UTC m=+1040.115196544" watchObservedRunningTime="2026-09-10 22:18:39.160222037 +0000 UTC m=+1040.116289740"
Sep 10 22:18:45.876940 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:45.876903    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-8f9fc7/explicit-model-8f9fc7-kserve-6cb5757c8c-bnn5f"]
Sep 10 22:18:45.991976 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:45.991940    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-8f9fc7-xph2r"]
Sep 10 22:18:45.995499 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:45.995482    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-8f9fc7-xph2r"
Sep 10 22:18:45.997454 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:45.997434    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8hjrv\""
Sep 10 22:18:46.002293 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:46.002234    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-8f9fc7-xph2r"]
Sep 10 22:18:46.057723 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:46.057690    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6885f5f2-9325-455d-a4dc-4ffad3333e96-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-8f9fc7-xph2r\" (UID: \"6885f5f2-9325-455d-a4dc-4ffad3333e96\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-8f9fc7-xph2r"
Sep 10 22:18:46.158192 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:46.158165    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6885f5f2-9325-455d-a4dc-4ffad3333e96-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-8f9fc7-xph2r\" (UID: \"6885f5f2-9325-455d-a4dc-4ffad3333e96\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-8f9fc7-xph2r"
Sep 10 22:18:46.158732 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:46.158713    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6885f5f2-9325-455d-a4dc-4ffad3333e96-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-8f9fc7-xph2r\" (UID: \"6885f5f2-9325-455d-a4dc-4ffad3333e96\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-8f9fc7-xph2r"
Sep 10 22:18:46.307547 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:46.307493    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-8f9fc7-xph2r"
Sep 10 22:18:46.578639 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:46.578559    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-8f9fc7-xph2r"]
Sep 10 22:18:46.582183 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:18:46.582158    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6885f5f2_9325_455d_a4dc_4ffad3333e96.slice/crio-91d7a807efc5442824bbba5b53d6d3cfef1f22781163ade8b0fd3f522eaadc18 WatchSource:0}: Error finding container 91d7a807efc5442824bbba5b53d6d3cfef1f22781163ade8b0fd3f522eaadc18: Status 404 returned error can't find the container with id 91d7a807efc5442824bbba5b53d6d3cfef1f22781163ade8b0fd3f522eaadc18
Sep 10 22:18:47.183916 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:47.183881    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-8f9fc7-xph2r" event={"ID":"6885f5f2-9325-455d-a4dc-4ffad3333e96","Type":"ContainerStarted","Data":"5aa75f0d4719bd6f78754819960a0fe33230b680242e54528118f6950e663959"}
Sep 10 22:18:47.184379 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:47.183952    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-8f9fc7-xph2r" event={"ID":"6885f5f2-9325-455d-a4dc-4ffad3333e96","Type":"ContainerStarted","Data":"91d7a807efc5442824bbba5b53d6d3cfef1f22781163ade8b0fd3f522eaadc18"}
Sep 10 22:18:47.201461 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:47.201417    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-8f9fc7-xph2r" podStartSLOduration=2.201401456 podStartE2EDuration="2.201401456s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 22:18:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:18:47.200450602 +0000 UTC m=+1048.156518309" watchObservedRunningTime="2026-09-10 22:18:47.201401456 +0000 UTC m=+1048.157469162"
Sep 10 22:18:48.026263 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:48.026230    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-8f9fc7/explicit-model-8f9fc7-kserve-6cb5757c8c-bnn5f"
Sep 10 22:18:48.189645 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:48.189613    2582 generic.go:356] "Generic (PLEG): container finished" podID="6885f5f2-9325-455d-a4dc-4ffad3333e96" containerID="5aa75f0d4719bd6f78754819960a0fe33230b680242e54528118f6950e663959" exitCode=0
Sep 10 22:18:48.190042 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:48.189658    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-8f9fc7-xph2r" event={"ID":"6885f5f2-9325-455d-a4dc-4ffad3333e96","Type":"ContainerDied","Data":"5aa75f0d4719bd6f78754819960a0fe33230b680242e54528118f6950e663959"}
Sep 10 22:18:49.320959 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:49.320934    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-8f9fc7-xph2r"
Sep 10 22:18:49.386280 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:49.386247    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6885f5f2-9325-455d-a4dc-4ffad3333e96-maas-api-service-ca\") pod \"6885f5f2-9325-455d-a4dc-4ffad3333e96\" (UID: \"6885f5f2-9325-455d-a4dc-4ffad3333e96\") "
Sep 10 22:18:49.386623 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:49.386599    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6885f5f2-9325-455d-a4dc-4ffad3333e96-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6885f5f2-9325-455d-a4dc-4ffad3333e96" (UID: "6885f5f2-9325-455d-a4dc-4ffad3333e96"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 22:18:49.486906 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:49.486827    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6885f5f2-9325-455d-a4dc-4ffad3333e96-maas-api-service-ca\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:18:50.200688 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:50.200658    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-8f9fc7-xph2r"
Sep 10 22:18:50.200888 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:50.200662    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-8f9fc7-xph2r" event={"ID":"6885f5f2-9325-455d-a4dc-4ffad3333e96","Type":"ContainerDied","Data":"91d7a807efc5442824bbba5b53d6d3cfef1f22781163ade8b0fd3f522eaadc18"}
Sep 10 22:18:50.200888 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:50.200809    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="91d7a807efc5442824bbba5b53d6d3cfef1f22781163ade8b0fd3f522eaadc18"
Sep 10 22:18:56.589753 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:56.589719    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-8f9fc7-b4cffd49c-np2zh"]
Sep 10 22:18:56.590192 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:56.589978    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-8f9fc7-b4cffd49c-np2zh" podUID="80516e33-181b-465f-b897-9cd78cdda578" containerName="payload-processing" containerID="cri-o://5cde55e1512c0203cbf987ca43db8165938afe5684cee8c37ee3d539b6844672" gracePeriod=30
Sep 10 22:18:56.596482 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:56.596402    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-8f9fc7-8597dd45f8-phpd9"]
Sep 10 22:18:56.596648 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:56.596618    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-8f9fc7-8597dd45f8-phpd9" podUID="e0718e77-8ff0-4b70-b172-16959e5c353b" containerName="payload-pre-processing" containerID="cri-o://ed28b5d2f7faedcb3db613bda3d09d51bfcb75c80fb303f8ee2433d5903f00ad" gracePeriod=30
Sep 10 22:18:56.768772 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:56.768689    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-8f9fc7-xph2r"]
Sep 10 22:18:56.772172 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:56.772128    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-8f9fc7-xph2r"]
Sep 10 22:18:56.840968 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:56.840907    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-8f9fc7-b4cffd49c-np2zh"
Sep 10 22:18:56.872457 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:56.872435    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-8f9fc7-8597dd45f8-phpd9"
Sep 10 22:18:56.950451 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:56.950423    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7zf9n\" (UniqueName: \"kubernetes.io/projected/80516e33-181b-465f-b897-9cd78cdda578-kube-api-access-7zf9n\") pod \"80516e33-181b-465f-b897-9cd78cdda578\" (UID: \"80516e33-181b-465f-b897-9cd78cdda578\") "
Sep 10 22:18:56.950636 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:56.950494    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80516e33-181b-465f-b897-9cd78cdda578-plugins-config-volume\") pod \"80516e33-181b-465f-b897-9cd78cdda578\" (UID: \"80516e33-181b-465f-b897-9cd78cdda578\") "
Sep 10 22:18:56.950636 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:56.950547    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0718e77-8ff0-4b70-b172-16959e5c353b-plugins-config-volume\") pod \"e0718e77-8ff0-4b70-b172-16959e5c353b\" (UID: \"e0718e77-8ff0-4b70-b172-16959e5c353b\") "
Sep 10 22:18:56.950636 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:56.950602    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lbfct\" (UniqueName: \"kubernetes.io/projected/e0718e77-8ff0-4b70-b172-16959e5c353b-kube-api-access-lbfct\") pod \"e0718e77-8ff0-4b70-b172-16959e5c353b\" (UID: \"e0718e77-8ff0-4b70-b172-16959e5c353b\") "
Sep 10 22:18:56.950917 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:56.950887    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80516e33-181b-465f-b897-9cd78cdda578-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "80516e33-181b-465f-b897-9cd78cdda578" (UID: "80516e33-181b-465f-b897-9cd78cdda578"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 22:18:56.950997 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:56.950935    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0718e77-8ff0-4b70-b172-16959e5c353b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e0718e77-8ff0-4b70-b172-16959e5c353b" (UID: "e0718e77-8ff0-4b70-b172-16959e5c353b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 22:18:56.952674 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:56.952650    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80516e33-181b-465f-b897-9cd78cdda578-kube-api-access-7zf9n" (OuterVolumeSpecName: "kube-api-access-7zf9n") pod "80516e33-181b-465f-b897-9cd78cdda578" (UID: "80516e33-181b-465f-b897-9cd78cdda578"). InnerVolumeSpecName "kube-api-access-7zf9n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 22:18:56.952744 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:56.952683    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0718e77-8ff0-4b70-b172-16959e5c353b-kube-api-access-lbfct" (OuterVolumeSpecName: "kube-api-access-lbfct") pod "e0718e77-8ff0-4b70-b172-16959e5c353b" (UID: "e0718e77-8ff0-4b70-b172-16959e5c353b"). InnerVolumeSpecName "kube-api-access-lbfct". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 22:18:57.051893 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:57.051862    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zf9n\" (UniqueName: \"kubernetes.io/projected/80516e33-181b-465f-b897-9cd78cdda578-kube-api-access-7zf9n\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:18:57.051893 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:57.051886    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80516e33-181b-465f-b897-9cd78cdda578-plugins-config-volume\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:18:57.051893 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:57.051896    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0718e77-8ff0-4b70-b172-16959e5c353b-plugins-config-volume\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:18:57.052090 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:57.051904    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbfct\" (UniqueName: \"kubernetes.io/projected/e0718e77-8ff0-4b70-b172-16959e5c353b-kube-api-access-lbfct\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:18:57.236203 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:57.236164    2582 generic.go:356] "Generic (PLEG): container finished" podID="80516e33-181b-465f-b897-9cd78cdda578" containerID="5cde55e1512c0203cbf987ca43db8165938afe5684cee8c37ee3d539b6844672" exitCode=0
Sep 10 22:18:57.236406 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:57.236212    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-8f9fc7-b4cffd49c-np2zh"
Sep 10 22:18:57.236406 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:57.236253    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-8f9fc7-b4cffd49c-np2zh" event={"ID":"80516e33-181b-465f-b897-9cd78cdda578","Type":"ContainerDied","Data":"5cde55e1512c0203cbf987ca43db8165938afe5684cee8c37ee3d539b6844672"}
Sep 10 22:18:57.236406 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:57.236315    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-8f9fc7-b4cffd49c-np2zh" event={"ID":"80516e33-181b-465f-b897-9cd78cdda578","Type":"ContainerDied","Data":"3e030e7b89b365030f10bd5d980cc1ec11c29d7275ee2744e76027a432aab99f"}
Sep 10 22:18:57.236406 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:57.236344    2582 scope.go:117] "RemoveContainer" containerID="5cde55e1512c0203cbf987ca43db8165938afe5684cee8c37ee3d539b6844672"
Sep 10 22:18:57.237729 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:57.237695    2582 generic.go:356] "Generic (PLEG): container finished" podID="e0718e77-8ff0-4b70-b172-16959e5c353b" containerID="ed28b5d2f7faedcb3db613bda3d09d51bfcb75c80fb303f8ee2433d5903f00ad" exitCode=0
Sep 10 22:18:57.237823 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:57.237727    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-8f9fc7-8597dd45f8-phpd9" event={"ID":"e0718e77-8ff0-4b70-b172-16959e5c353b","Type":"ContainerDied","Data":"ed28b5d2f7faedcb3db613bda3d09d51bfcb75c80fb303f8ee2433d5903f00ad"}
Sep 10 22:18:57.237823 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:57.237769    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-8f9fc7-8597dd45f8-phpd9" event={"ID":"e0718e77-8ff0-4b70-b172-16959e5c353b","Type":"ContainerDied","Data":"c9ad0fade46b63cb62a6da7eb66b21d43a666c92c4061d138e3acf67e33d2f1b"}
Sep 10 22:18:57.237823 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:57.237797    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-8f9fc7-8597dd45f8-phpd9"
Sep 10 22:18:57.246362 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:57.246344    2582 scope.go:117] "RemoveContainer" containerID="5cde55e1512c0203cbf987ca43db8165938afe5684cee8c37ee3d539b6844672"
Sep 10 22:18:57.246630 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:18:57.246609    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5cde55e1512c0203cbf987ca43db8165938afe5684cee8c37ee3d539b6844672\": container with ID starting with 5cde55e1512c0203cbf987ca43db8165938afe5684cee8c37ee3d539b6844672 not found: ID does not exist" containerID="5cde55e1512c0203cbf987ca43db8165938afe5684cee8c37ee3d539b6844672"
Sep 10 22:18:57.246677 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:57.246639    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5cde55e1512c0203cbf987ca43db8165938afe5684cee8c37ee3d539b6844672"} err="failed to get container status \"5cde55e1512c0203cbf987ca43db8165938afe5684cee8c37ee3d539b6844672\": rpc error: code = NotFound desc = could not find container \"5cde55e1512c0203cbf987ca43db8165938afe5684cee8c37ee3d539b6844672\": container with ID starting with 5cde55e1512c0203cbf987ca43db8165938afe5684cee8c37ee3d539b6844672 not found: ID does not exist"
Sep 10 22:18:57.246677 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:57.246656    2582 scope.go:117] "RemoveContainer" containerID="ed28b5d2f7faedcb3db613bda3d09d51bfcb75c80fb303f8ee2433d5903f00ad"
Sep 10 22:18:57.254063 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:57.254047    2582 scope.go:117] "RemoveContainer" containerID="ed28b5d2f7faedcb3db613bda3d09d51bfcb75c80fb303f8ee2433d5903f00ad"
Sep 10 22:18:57.254307 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:18:57.254289    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed28b5d2f7faedcb3db613bda3d09d51bfcb75c80fb303f8ee2433d5903f00ad\": container with ID starting with ed28b5d2f7faedcb3db613bda3d09d51bfcb75c80fb303f8ee2433d5903f00ad not found: ID does not exist" containerID="ed28b5d2f7faedcb3db613bda3d09d51bfcb75c80fb303f8ee2433d5903f00ad"
Sep 10 22:18:57.254349 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:57.254316    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed28b5d2f7faedcb3db613bda3d09d51bfcb75c80fb303f8ee2433d5903f00ad"} err="failed to get container status \"ed28b5d2f7faedcb3db613bda3d09d51bfcb75c80fb303f8ee2433d5903f00ad\": rpc error: code = NotFound desc = could not find container \"ed28b5d2f7faedcb3db613bda3d09d51bfcb75c80fb303f8ee2433d5903f00ad\": container with ID starting with ed28b5d2f7faedcb3db613bda3d09d51bfcb75c80fb303f8ee2433d5903f00ad not found: ID does not exist"
Sep 10 22:18:57.257659 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:57.257640    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-8f9fc7-b4cffd49c-np2zh"]
Sep 10 22:18:57.264204 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:57.264184    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-8f9fc7-b4cffd49c-np2zh"]
Sep 10 22:18:57.273443 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:57.273422    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-8f9fc7-8597dd45f8-phpd9"]
Sep 10 22:18:57.277520 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:57.277500    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-8f9fc7-8597dd45f8-phpd9"]
Sep 10 22:18:57.498201 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:57.498121    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6885f5f2-9325-455d-a4dc-4ffad3333e96" path="/var/lib/kubelet/pods/6885f5f2-9325-455d-a4dc-4ffad3333e96/volumes"
Sep 10 22:18:57.498495 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:57.498482    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="80516e33-181b-465f-b897-9cd78cdda578" path="/var/lib/kubelet/pods/80516e33-181b-465f-b897-9cd78cdda578/volumes"
Sep 10 22:18:57.498815 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:18:57.498801    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e0718e77-8ff0-4b70-b172-16959e5c353b" path="/var/lib/kubelet/pods/e0718e77-8ff0-4b70-b172-16959e5c353b/volumes"
Sep 10 22:19:00.896105 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:00.896055    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-8f9fc7/explicit-model-8f9fc7-kserve-6cb5757c8c-bnn5f" podUID="d9cf63f3-ca6e-4bed-a508-5054cb0daf9f" containerName="main" containerID="cri-o://9b6168412382ea59c71727a957166b8d2ac86440653c06e79b9a081b1c712222" gracePeriod=45
Sep 10 22:19:01.151479 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:01.151424    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-8f9fc7/explicit-model-8f9fc7-kserve-6cb5757c8c-bnn5f"
Sep 10 22:19:01.184579 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:01.184538    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9cf63f3-ca6e-4bed-a508-5054cb0daf9f-dshm\") pod \"d9cf63f3-ca6e-4bed-a508-5054cb0daf9f\" (UID: \"d9cf63f3-ca6e-4bed-a508-5054cb0daf9f\") "
Sep 10 22:19:01.184746 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:01.184596    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9cf63f3-ca6e-4bed-a508-5054cb0daf9f-tls-certs\") pod \"d9cf63f3-ca6e-4bed-a508-5054cb0daf9f\" (UID: \"d9cf63f3-ca6e-4bed-a508-5054cb0daf9f\") "
Sep 10 22:19:01.184746 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:01.184617    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9cf63f3-ca6e-4bed-a508-5054cb0daf9f-model-cache\") pod \"d9cf63f3-ca6e-4bed-a508-5054cb0daf9f\" (UID: \"d9cf63f3-ca6e-4bed-a508-5054cb0daf9f\") "
Sep 10 22:19:01.184923 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:01.184773    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9cf63f3-ca6e-4bed-a508-5054cb0daf9f-home\") pod \"d9cf63f3-ca6e-4bed-a508-5054cb0daf9f\" (UID: \"d9cf63f3-ca6e-4bed-a508-5054cb0daf9f\") "
Sep 10 22:19:01.184923 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:01.184882    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9cf63f3-ca6e-4bed-a508-5054cb0daf9f-model-cache" (OuterVolumeSpecName: "model-cache") pod "d9cf63f3-ca6e-4bed-a508-5054cb0daf9f" (UID: "d9cf63f3-ca6e-4bed-a508-5054cb0daf9f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 22:19:01.184923 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:01.184892    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f69lv\" (UniqueName: \"kubernetes.io/projected/d9cf63f3-ca6e-4bed-a508-5054cb0daf9f-kube-api-access-f69lv\") pod \"d9cf63f3-ca6e-4bed-a508-5054cb0daf9f\" (UID: \"d9cf63f3-ca6e-4bed-a508-5054cb0daf9f\") "
Sep 10 22:19:01.185091 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:01.184975    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9cf63f3-ca6e-4bed-a508-5054cb0daf9f-tmp-dir\") pod \"d9cf63f3-ca6e-4bed-a508-5054cb0daf9f\" (UID: \"d9cf63f3-ca6e-4bed-a508-5054cb0daf9f\") "
Sep 10 22:19:01.185091 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:01.185063    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9cf63f3-ca6e-4bed-a508-5054cb0daf9f-home" (OuterVolumeSpecName: "home") pod "d9cf63f3-ca6e-4bed-a508-5054cb0daf9f" (UID: "d9cf63f3-ca6e-4bed-a508-5054cb0daf9f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 22:19:01.185272 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:01.185242    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9cf63f3-ca6e-4bed-a508-5054cb0daf9f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d9cf63f3-ca6e-4bed-a508-5054cb0daf9f" (UID: "d9cf63f3-ca6e-4bed-a508-5054cb0daf9f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 22:19:01.185398 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:01.185347    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9cf63f3-ca6e-4bed-a508-5054cb0daf9f-tmp-dir\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:19:01.185398 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:01.185368    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9cf63f3-ca6e-4bed-a508-5054cb0daf9f-model-cache\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:19:01.185398 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:01.185381    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9cf63f3-ca6e-4bed-a508-5054cb0daf9f-home\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:19:01.187085 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:01.187058    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9cf63f3-ca6e-4bed-a508-5054cb0daf9f-kube-api-access-f69lv" (OuterVolumeSpecName: "kube-api-access-f69lv") pod "d9cf63f3-ca6e-4bed-a508-5054cb0daf9f" (UID: "d9cf63f3-ca6e-4bed-a508-5054cb0daf9f"). InnerVolumeSpecName "kube-api-access-f69lv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 22:19:01.187281 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:01.187262    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9cf63f3-ca6e-4bed-a508-5054cb0daf9f-dshm" (OuterVolumeSpecName: "dshm") pod "d9cf63f3-ca6e-4bed-a508-5054cb0daf9f" (UID: "d9cf63f3-ca6e-4bed-a508-5054cb0daf9f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 22:19:01.187330 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:01.187272    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9cf63f3-ca6e-4bed-a508-5054cb0daf9f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d9cf63f3-ca6e-4bed-a508-5054cb0daf9f" (UID: "d9cf63f3-ca6e-4bed-a508-5054cb0daf9f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 22:19:01.258441 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:01.258408    2582 generic.go:356] "Generic (PLEG): container finished" podID="d9cf63f3-ca6e-4bed-a508-5054cb0daf9f" containerID="9b6168412382ea59c71727a957166b8d2ac86440653c06e79b9a081b1c712222" exitCode=0
Sep 10 22:19:01.258441 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:01.258446    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-8f9fc7/explicit-model-8f9fc7-kserve-6cb5757c8c-bnn5f" event={"ID":"d9cf63f3-ca6e-4bed-a508-5054cb0daf9f","Type":"ContainerDied","Data":"9b6168412382ea59c71727a957166b8d2ac86440653c06e79b9a081b1c712222"}
Sep 10 22:19:01.258681 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:01.258453    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-8f9fc7/explicit-model-8f9fc7-kserve-6cb5757c8c-bnn5f"
Sep 10 22:19:01.258681 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:01.258474    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-8f9fc7/explicit-model-8f9fc7-kserve-6cb5757c8c-bnn5f" event={"ID":"d9cf63f3-ca6e-4bed-a508-5054cb0daf9f","Type":"ContainerDied","Data":"e701e83e7d76d1e5009801952961828a9c753c6f17e580dbd97e74fb89004653"}
Sep 10 22:19:01.258681 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:01.258491    2582 scope.go:117] "RemoveContainer" containerID="9b6168412382ea59c71727a957166b8d2ac86440653c06e79b9a081b1c712222"
Sep 10 22:19:01.267257 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:01.267237    2582 scope.go:117] "RemoveContainer" containerID="9b6168412382ea59c71727a957166b8d2ac86440653c06e79b9a081b1c712222"
Sep 10 22:19:01.267540 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:19:01.267522    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b6168412382ea59c71727a957166b8d2ac86440653c06e79b9a081b1c712222\": container with ID starting with 9b6168412382ea59c71727a957166b8d2ac86440653c06e79b9a081b1c712222 not found: ID does not exist" containerID="9b6168412382ea59c71727a957166b8d2ac86440653c06e79b9a081b1c712222"
Sep 10 22:19:01.267584 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:01.267547    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b6168412382ea59c71727a957166b8d2ac86440653c06e79b9a081b1c712222"} err="failed to get container status \"9b6168412382ea59c71727a957166b8d2ac86440653c06e79b9a081b1c712222\": rpc error: code = NotFound desc = could not find container \"9b6168412382ea59c71727a957166b8d2ac86440653c06e79b9a081b1c712222\": container with ID starting with 9b6168412382ea59c71727a957166b8d2ac86440653c06e79b9a081b1c712222 not found: ID does not exist"
Sep 10 22:19:01.279191 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:01.279168    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-8f9fc7/explicit-model-8f9fc7-kserve-6cb5757c8c-bnn5f"]
Sep 10 22:19:01.283027 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:01.283006    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-8f9fc7/explicit-model-8f9fc7-kserve-6cb5757c8c-bnn5f"]
Sep 10 22:19:01.286655 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:01.286640    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9cf63f3-ca6e-4bed-a508-5054cb0daf9f-dshm\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:19:01.286717 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:01.286657    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9cf63f3-ca6e-4bed-a508-5054cb0daf9f-tls-certs\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:19:01.286717 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:01.286666    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f69lv\" (UniqueName: \"kubernetes.io/projected/d9cf63f3-ca6e-4bed-a508-5054cb0daf9f-kube-api-access-f69lv\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:19:01.498299 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:01.498268    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d9cf63f3-ca6e-4bed-a508-5054cb0daf9f" path="/var/lib/kubelet/pods/d9cf63f3-ca6e-4bed-a508-5054cb0daf9f/volumes"
Sep 10 22:19:04.691808 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:04.691771    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-361ccf-openshift-default-758565f796-f4jhq"]
Sep 10 22:19:04.692342 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:04.692179    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80516e33-181b-465f-b897-9cd78cdda578" containerName="payload-processing"
Sep 10 22:19:04.692342 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:04.692199    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="80516e33-181b-465f-b897-9cd78cdda578" containerName="payload-processing"
Sep 10 22:19:04.692342 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:04.692215    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9cf63f3-ca6e-4bed-a508-5054cb0daf9f" containerName="main"
Sep 10 22:19:04.692342 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:04.692221    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9cf63f3-ca6e-4bed-a508-5054cb0daf9f" containerName="main"
Sep 10 22:19:04.692342 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:04.692235    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6885f5f2-9325-455d-a4dc-4ffad3333e96" containerName="revoke-keys"
Sep 10 22:19:04.692342 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:04.692241    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="6885f5f2-9325-455d-a4dc-4ffad3333e96" containerName="revoke-keys"
Sep 10 22:19:04.692342 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:04.692247    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0718e77-8ff0-4b70-b172-16959e5c353b" containerName="payload-pre-processing"
Sep 10 22:19:04.692342 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:04.692252    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0718e77-8ff0-4b70-b172-16959e5c353b" containerName="payload-pre-processing"
Sep 10 22:19:04.692342 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:04.692315    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="80516e33-181b-465f-b897-9cd78cdda578" containerName="payload-processing"
Sep 10 22:19:04.692342 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:04.692322    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9cf63f3-ca6e-4bed-a508-5054cb0daf9f" containerName="main"
Sep 10 22:19:04.692342 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:04.692330    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="6885f5f2-9325-455d-a4dc-4ffad3333e96" containerName="revoke-keys"
Sep 10 22:19:04.692342 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:04.692337    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0718e77-8ff0-4b70-b172-16959e5c353b" containerName="payload-pre-processing"
Sep 10 22:19:04.696315 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:04.696293    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-361ccf-openshift-default-758565f796-f4jhq"
Sep 10 22:19:04.699378 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:04.699359    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-361ccf-openshift-default-dockercfg-zd9gs\""
Sep 10 22:19:04.710735 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:04.710713    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-361ccf-openshift-default-758565f796-f4jhq"]
Sep 10 22:19:04.813774 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:04.813734    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23fb1677-9564-480a-a2bd-2aec7f9b81bc-workload-certs\") pod \"e2e-noten-361ccf-openshift-default-758565f796-f4jhq\" (UID: \"23fb1677-9564-480a-a2bd-2aec7f9b81bc\") " pod="openshift-ingress/e2e-noten-361ccf-openshift-default-758565f796-f4jhq"
Sep 10 22:19:04.813774 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:04.813774    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23fb1677-9564-480a-a2bd-2aec7f9b81bc-istiod-ca-cert\") pod \"e2e-noten-361ccf-openshift-default-758565f796-f4jhq\" (UID: \"23fb1677-9564-480a-a2bd-2aec7f9b81bc\") " pod="openshift-ingress/e2e-noten-361ccf-openshift-default-758565f796-f4jhq"
Sep 10 22:19:04.813997 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:04.813831    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23fb1677-9564-480a-a2bd-2aec7f9b81bc-istio-envoy\") pod \"e2e-noten-361ccf-openshift-default-758565f796-f4jhq\" (UID: \"23fb1677-9564-480a-a2bd-2aec7f9b81bc\") " pod="openshift-ingress/e2e-noten-361ccf-openshift-default-758565f796-f4jhq"
Sep 10 22:19:04.813997 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:04.813885    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23fb1677-9564-480a-a2bd-2aec7f9b81bc-istio-podinfo\") pod \"e2e-noten-361ccf-openshift-default-758565f796-f4jhq\" (UID: \"23fb1677-9564-480a-a2bd-2aec7f9b81bc\") " pod="openshift-ingress/e2e-noten-361ccf-openshift-default-758565f796-f4jhq"
Sep 10 22:19:04.813997 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:04.813914    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23fb1677-9564-480a-a2bd-2aec7f9b81bc-workload-socket\") pod \"e2e-noten-361ccf-openshift-default-758565f796-f4jhq\" (UID: \"23fb1677-9564-480a-a2bd-2aec7f9b81bc\") " pod="openshift-ingress/e2e-noten-361ccf-openshift-default-758565f796-f4jhq"
Sep 10 22:19:04.813997 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:04.813928    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23fb1677-9564-480a-a2bd-2aec7f9b81bc-credential-socket\") pod \"e2e-noten-361ccf-openshift-default-758565f796-f4jhq\" (UID: \"23fb1677-9564-480a-a2bd-2aec7f9b81bc\") " pod="openshift-ingress/e2e-noten-361ccf-openshift-default-758565f796-f4jhq"
Sep 10 22:19:04.813997 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:04.813947    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23fb1677-9564-480a-a2bd-2aec7f9b81bc-istio-data\") pod \"e2e-noten-361ccf-openshift-default-758565f796-f4jhq\" (UID: \"23fb1677-9564-480a-a2bd-2aec7f9b81bc\") " pod="openshift-ingress/e2e-noten-361ccf-openshift-default-758565f796-f4jhq"
Sep 10 22:19:04.814271 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:04.814086    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzmc2\" (UniqueName: \"kubernetes.io/projected/23fb1677-9564-480a-a2bd-2aec7f9b81bc-kube-api-access-vzmc2\") pod \"e2e-noten-361ccf-openshift-default-758565f796-f4jhq\" (UID: \"23fb1677-9564-480a-a2bd-2aec7f9b81bc\") " pod="openshift-ingress/e2e-noten-361ccf-openshift-default-758565f796-f4jhq"
Sep 10 22:19:04.814271 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:04.814151    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23fb1677-9564-480a-a2bd-2aec7f9b81bc-istio-token\") pod \"e2e-noten-361ccf-openshift-default-758565f796-f4jhq\" (UID: \"23fb1677-9564-480a-a2bd-2aec7f9b81bc\") " pod="openshift-ingress/e2e-noten-361ccf-openshift-default-758565f796-f4jhq"
Sep 10 22:19:04.914533 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:04.914508    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23fb1677-9564-480a-a2bd-2aec7f9b81bc-istio-envoy\") pod \"e2e-noten-361ccf-openshift-default-758565f796-f4jhq\" (UID: \"23fb1677-9564-480a-a2bd-2aec7f9b81bc\") " pod="openshift-ingress/e2e-noten-361ccf-openshift-default-758565f796-f4jhq"
Sep 10 22:19:04.914690 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:04.914535    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23fb1677-9564-480a-a2bd-2aec7f9b81bc-istio-podinfo\") pod \"e2e-noten-361ccf-openshift-default-758565f796-f4jhq\" (UID: \"23fb1677-9564-480a-a2bd-2aec7f9b81bc\") " pod="openshift-ingress/e2e-noten-361ccf-openshift-default-758565f796-f4jhq"
Sep 10 22:19:04.914690 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:04.914559    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23fb1677-9564-480a-a2bd-2aec7f9b81bc-workload-socket\") pod \"e2e-noten-361ccf-openshift-default-758565f796-f4jhq\" (UID: \"23fb1677-9564-480a-a2bd-2aec7f9b81bc\") " pod="openshift-ingress/e2e-noten-361ccf-openshift-default-758565f796-f4jhq"
Sep 10 22:19:04.914690 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:04.914574    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23fb1677-9564-480a-a2bd-2aec7f9b81bc-credential-socket\") pod \"e2e-noten-361ccf-openshift-default-758565f796-f4jhq\" (UID: \"23fb1677-9564-480a-a2bd-2aec7f9b81bc\") " pod="openshift-ingress/e2e-noten-361ccf-openshift-default-758565f796-f4jhq"
Sep 10 22:19:04.914690 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:04.914591    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23fb1677-9564-480a-a2bd-2aec7f9b81bc-istio-data\") pod \"e2e-noten-361ccf-openshift-default-758565f796-f4jhq\" (UID: \"23fb1677-9564-480a-a2bd-2aec7f9b81bc\") " pod="openshift-ingress/e2e-noten-361ccf-openshift-default-758565f796-f4jhq"
Sep 10 22:19:04.914690 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:04.914669    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzmc2\" (UniqueName: \"kubernetes.io/projected/23fb1677-9564-480a-a2bd-2aec7f9b81bc-kube-api-access-vzmc2\") pod \"e2e-noten-361ccf-openshift-default-758565f796-f4jhq\" (UID: \"23fb1677-9564-480a-a2bd-2aec7f9b81bc\") " pod="openshift-ingress/e2e-noten-361ccf-openshift-default-758565f796-f4jhq"
Sep 10 22:19:04.915002 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:04.914731    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23fb1677-9564-480a-a2bd-2aec7f9b81bc-istio-token\") pod \"e2e-noten-361ccf-openshift-default-758565f796-f4jhq\" (UID: \"23fb1677-9564-480a-a2bd-2aec7f9b81bc\") " pod="openshift-ingress/e2e-noten-361ccf-openshift-default-758565f796-f4jhq"
Sep 10 22:19:04.915002 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:04.914762    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23fb1677-9564-480a-a2bd-2aec7f9b81bc-workload-certs\") pod \"e2e-noten-361ccf-openshift-default-758565f796-f4jhq\" (UID: \"23fb1677-9564-480a-a2bd-2aec7f9b81bc\") " pod="openshift-ingress/e2e-noten-361ccf-openshift-default-758565f796-f4jhq"
Sep 10 22:19:04.915002 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:04.914785    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23fb1677-9564-480a-a2bd-2aec7f9b81bc-istiod-ca-cert\") pod \"e2e-noten-361ccf-openshift-default-758565f796-f4jhq\" (UID: \"23fb1677-9564-480a-a2bd-2aec7f9b81bc\") " pod="openshift-ingress/e2e-noten-361ccf-openshift-default-758565f796-f4jhq"
Sep 10 22:19:04.915002 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:04.914968    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23fb1677-9564-480a-a2bd-2aec7f9b81bc-credential-socket\") pod \"e2e-noten-361ccf-openshift-default-758565f796-f4jhq\" (UID: \"23fb1677-9564-480a-a2bd-2aec7f9b81bc\") " pod="openshift-ingress/e2e-noten-361ccf-openshift-default-758565f796-f4jhq"
Sep 10 22:19:04.915273 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:04.915008    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23fb1677-9564-480a-a2bd-2aec7f9b81bc-workload-socket\") pod \"e2e-noten-361ccf-openshift-default-758565f796-f4jhq\" (UID: \"23fb1677-9564-480a-a2bd-2aec7f9b81bc\") " pod="openshift-ingress/e2e-noten-361ccf-openshift-default-758565f796-f4jhq"
Sep 10 22:19:04.915333 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:04.915312    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23fb1677-9564-480a-a2bd-2aec7f9b81bc-workload-certs\") pod \"e2e-noten-361ccf-openshift-default-758565f796-f4jhq\" (UID: \"23fb1677-9564-480a-a2bd-2aec7f9b81bc\") " pod="openshift-ingress/e2e-noten-361ccf-openshift-default-758565f796-f4jhq"
Sep 10 22:19:04.915393 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:04.915338    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23fb1677-9564-480a-a2bd-2aec7f9b81bc-istio-data\") pod \"e2e-noten-361ccf-openshift-default-758565f796-f4jhq\" (UID: \"23fb1677-9564-480a-a2bd-2aec7f9b81bc\") " pod="openshift-ingress/e2e-noten-361ccf-openshift-default-758565f796-f4jhq"
Sep 10 22:19:04.915471 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:04.915453    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23fb1677-9564-480a-a2bd-2aec7f9b81bc-istiod-ca-cert\") pod \"e2e-noten-361ccf-openshift-default-758565f796-f4jhq\" (UID: \"23fb1677-9564-480a-a2bd-2aec7f9b81bc\") " pod="openshift-ingress/e2e-noten-361ccf-openshift-default-758565f796-f4jhq"
Sep 10 22:19:04.916946 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:04.916926    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23fb1677-9564-480a-a2bd-2aec7f9b81bc-istio-envoy\") pod \"e2e-noten-361ccf-openshift-default-758565f796-f4jhq\" (UID: \"23fb1677-9564-480a-a2bd-2aec7f9b81bc\") " pod="openshift-ingress/e2e-noten-361ccf-openshift-default-758565f796-f4jhq"
Sep 10 22:19:04.917182 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:04.917164    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23fb1677-9564-480a-a2bd-2aec7f9b81bc-istio-podinfo\") pod \"e2e-noten-361ccf-openshift-default-758565f796-f4jhq\" (UID: \"23fb1677-9564-480a-a2bd-2aec7f9b81bc\") " pod="openshift-ingress/e2e-noten-361ccf-openshift-default-758565f796-f4jhq"
Sep 10 22:19:04.921874 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:04.921831    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23fb1677-9564-480a-a2bd-2aec7f9b81bc-istio-token\") pod \"e2e-noten-361ccf-openshift-default-758565f796-f4jhq\" (UID: \"23fb1677-9564-480a-a2bd-2aec7f9b81bc\") " pod="openshift-ingress/e2e-noten-361ccf-openshift-default-758565f796-f4jhq"
Sep 10 22:19:04.921953 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:04.921913    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzmc2\" (UniqueName: \"kubernetes.io/projected/23fb1677-9564-480a-a2bd-2aec7f9b81bc-kube-api-access-vzmc2\") pod \"e2e-noten-361ccf-openshift-default-758565f796-f4jhq\" (UID: \"23fb1677-9564-480a-a2bd-2aec7f9b81bc\") " pod="openshift-ingress/e2e-noten-361ccf-openshift-default-758565f796-f4jhq"
Sep 10 22:19:05.007331 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:05.007278    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-361ccf-openshift-default-758565f796-f4jhq"
Sep 10 22:19:05.280097 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:05.280073    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-361ccf-openshift-default-758565f796-f4jhq"]
Sep 10 22:19:05.282055 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:19:05.282023    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23fb1677_9564_480a_a2bd_2aec7f9b81bc.slice/crio-d4d23f94e671ff32782a3bc40f6bbbffeb4d3fc810c8a3b3d27337007656a874 WatchSource:0}: Error finding container d4d23f94e671ff32782a3bc40f6bbbffeb4d3fc810c8a3b3d27337007656a874: Status 404 returned error can't find the container with id d4d23f94e671ff32782a3bc40f6bbbffeb4d3fc810c8a3b3d27337007656a874
Sep 10 22:19:05.284270 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:05.284233    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 22:19:05.284353 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:05.284302    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 22:19:05.284393 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:05.284351    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 22:19:06.283512 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:06.283478    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-361ccf-openshift-default-758565f796-f4jhq" event={"ID":"23fb1677-9564-480a-a2bd-2aec7f9b81bc","Type":"ContainerStarted","Data":"dd5400686c299df50527ce42f00db2748800f7eb27479ce368e2c29ccaf9d64e"}
Sep 10 22:19:06.283898 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:06.283536    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-361ccf-openshift-default-758565f796-f4jhq" event={"ID":"23fb1677-9564-480a-a2bd-2aec7f9b81bc","Type":"ContainerStarted","Data":"d4d23f94e671ff32782a3bc40f6bbbffeb4d3fc810c8a3b3d27337007656a874"}
Sep 10 22:19:06.304476 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:06.304414    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-361ccf-openshift-default-758565f796-f4jhq" podStartSLOduration=2.304390667 podStartE2EDuration="2.304390667s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 22:19:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:19:06.301694621 +0000 UTC m=+1067.257762326" watchObservedRunningTime="2026-09-10 22:19:06.304390667 +0000 UTC m=+1067.260458374"
Sep 10 22:19:07.007626 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:07.007588    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-361ccf-openshift-default-758565f796-f4jhq"
Sep 10 22:19:07.012665 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:07.012642    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-361ccf-openshift-default-758565f796-f4jhq"
Sep 10 22:19:07.288710 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:07.288621    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-361ccf-openshift-default-758565f796-f4jhq"
Sep 10 22:19:07.289537 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:07.289521    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-361ccf-openshift-default-758565f796-f4jhq"
Sep 10 22:19:14.855620 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:14.855531    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d98bb58d-54fdbf4cf5-4bldz"]
Sep 10 22:19:14.859829 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:14.859799    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d98bb58d-54fdbf4cf5-4bldz"
Sep 10 22:19:14.861756 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:14.861734    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-d98bb58d-serving-cert\""
Sep 10 22:19:14.869677 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:14.869651    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d98bb58d-54fdbf4cf5-4bldz"]
Sep 10 22:19:14.998177 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:14.998138    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d97d86ed-a521-4792-a760-0d95aa5ee6e7-openshift-service-ca\") pod \"maas-api-e2e-mt-d98bb58d-54fdbf4cf5-4bldz\" (UID: \"d97d86ed-a521-4792-a760-0d95aa5ee6e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d98bb58d-54fdbf4cf5-4bldz"
Sep 10 22:19:14.998177 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:14.998184    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrfkr\" (UniqueName: \"kubernetes.io/projected/d97d86ed-a521-4792-a760-0d95aa5ee6e7-kube-api-access-mrfkr\") pod \"maas-api-e2e-mt-d98bb58d-54fdbf4cf5-4bldz\" (UID: \"d97d86ed-a521-4792-a760-0d95aa5ee6e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d98bb58d-54fdbf4cf5-4bldz"
Sep 10 22:19:14.998425 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:14.998217    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d97d86ed-a521-4792-a760-0d95aa5ee6e7-maas-api-tls\") pod \"maas-api-e2e-mt-d98bb58d-54fdbf4cf5-4bldz\" (UID: \"d97d86ed-a521-4792-a760-0d95aa5ee6e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d98bb58d-54fdbf4cf5-4bldz"
Sep 10 22:19:14.998425 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:14.998289    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d97d86ed-a521-4792-a760-0d95aa5ee6e7-metrics-certs\") pod \"maas-api-e2e-mt-d98bb58d-54fdbf4cf5-4bldz\" (UID: \"d97d86ed-a521-4792-a760-0d95aa5ee6e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d98bb58d-54fdbf4cf5-4bldz"
Sep 10 22:19:15.099532 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:15.099495    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d97d86ed-a521-4792-a760-0d95aa5ee6e7-metrics-certs\") pod \"maas-api-e2e-mt-d98bb58d-54fdbf4cf5-4bldz\" (UID: \"d97d86ed-a521-4792-a760-0d95aa5ee6e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d98bb58d-54fdbf4cf5-4bldz"
Sep 10 22:19:15.099701 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:15.099585    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d97d86ed-a521-4792-a760-0d95aa5ee6e7-openshift-service-ca\") pod \"maas-api-e2e-mt-d98bb58d-54fdbf4cf5-4bldz\" (UID: \"d97d86ed-a521-4792-a760-0d95aa5ee6e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d98bb58d-54fdbf4cf5-4bldz"
Sep 10 22:19:15.099701 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:15.099620    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrfkr\" (UniqueName: \"kubernetes.io/projected/d97d86ed-a521-4792-a760-0d95aa5ee6e7-kube-api-access-mrfkr\") pod \"maas-api-e2e-mt-d98bb58d-54fdbf4cf5-4bldz\" (UID: \"d97d86ed-a521-4792-a760-0d95aa5ee6e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d98bb58d-54fdbf4cf5-4bldz"
Sep 10 22:19:15.099701 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:15.099663    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d97d86ed-a521-4792-a760-0d95aa5ee6e7-maas-api-tls\") pod \"maas-api-e2e-mt-d98bb58d-54fdbf4cf5-4bldz\" (UID: \"d97d86ed-a521-4792-a760-0d95aa5ee6e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d98bb58d-54fdbf4cf5-4bldz"
Sep 10 22:19:15.099912 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:19:15.099768    2582 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-d98bb58d-serving-cert: secret "maas-api-e2e-mt-d98bb58d-serving-cert" not found
Sep 10 22:19:15.099912 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:19:15.099838    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d97d86ed-a521-4792-a760-0d95aa5ee6e7-maas-api-tls podName:d97d86ed-a521-4792-a760-0d95aa5ee6e7 nodeName:}" failed. No retries permitted until 2026-09-10 22:19:15.59981541 +0000 UTC m=+1076.555883111 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d97d86ed-a521-4792-a760-0d95aa5ee6e7-maas-api-tls") pod "maas-api-e2e-mt-d98bb58d-54fdbf4cf5-4bldz" (UID: "d97d86ed-a521-4792-a760-0d95aa5ee6e7") : secret "maas-api-e2e-mt-d98bb58d-serving-cert" not found
Sep 10 22:19:15.100232 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:15.100213    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d97d86ed-a521-4792-a760-0d95aa5ee6e7-openshift-service-ca\") pod \"maas-api-e2e-mt-d98bb58d-54fdbf4cf5-4bldz\" (UID: \"d97d86ed-a521-4792-a760-0d95aa5ee6e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d98bb58d-54fdbf4cf5-4bldz"
Sep 10 22:19:15.101876 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:15.101841    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d97d86ed-a521-4792-a760-0d95aa5ee6e7-metrics-certs\") pod \"maas-api-e2e-mt-d98bb58d-54fdbf4cf5-4bldz\" (UID: \"d97d86ed-a521-4792-a760-0d95aa5ee6e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d98bb58d-54fdbf4cf5-4bldz"
Sep 10 22:19:15.107823 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:15.107773    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrfkr\" (UniqueName: \"kubernetes.io/projected/d97d86ed-a521-4792-a760-0d95aa5ee6e7-kube-api-access-mrfkr\") pod \"maas-api-e2e-mt-d98bb58d-54fdbf4cf5-4bldz\" (UID: \"d97d86ed-a521-4792-a760-0d95aa5ee6e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d98bb58d-54fdbf4cf5-4bldz"
Sep 10 22:19:15.604588 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:15.604547    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d97d86ed-a521-4792-a760-0d95aa5ee6e7-maas-api-tls\") pod \"maas-api-e2e-mt-d98bb58d-54fdbf4cf5-4bldz\" (UID: \"d97d86ed-a521-4792-a760-0d95aa5ee6e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d98bb58d-54fdbf4cf5-4bldz"
Sep 10 22:19:15.607004 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:15.606976    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d97d86ed-a521-4792-a760-0d95aa5ee6e7-maas-api-tls\") pod \"maas-api-e2e-mt-d98bb58d-54fdbf4cf5-4bldz\" (UID: \"d97d86ed-a521-4792-a760-0d95aa5ee6e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d98bb58d-54fdbf4cf5-4bldz"
Sep 10 22:19:15.609233 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:15.609207    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d98bb58d-5dc747fd5d-gvbsd"]
Sep 10 22:19:15.612502 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:15.612484    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-d98bb58d-5dc747fd5d-gvbsd"
Sep 10 22:19:15.614640 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:15.614624    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-d98bb58d-dockercfg-l6lkv\""
Sep 10 22:19:15.614992 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:15.614977    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-d98bb58d\""
Sep 10 22:19:15.622749 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:15.622725    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d98bb58d-5dc747fd5d-gvbsd"]
Sep 10 22:19:15.706172 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:15.706141    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwlj6\" (UniqueName: \"kubernetes.io/projected/be6caaba-5c60-4d62-b0db-512db79382a8-kube-api-access-pwlj6\") pod \"payload-pre-processing-e2e-mt-d98bb58d-5dc747fd5d-gvbsd\" (UID: \"be6caaba-5c60-4d62-b0db-512db79382a8\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d98bb58d-5dc747fd5d-gvbsd"
Sep 10 22:19:15.706349 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:15.706193    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be6caaba-5c60-4d62-b0db-512db79382a8-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-d98bb58d-5dc747fd5d-gvbsd\" (UID: \"be6caaba-5c60-4d62-b0db-512db79382a8\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d98bb58d-5dc747fd5d-gvbsd"
Sep 10 22:19:15.772832 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:15.772803    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d98bb58d-54fdbf4cf5-4bldz"
Sep 10 22:19:15.807112 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:15.807081    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be6caaba-5c60-4d62-b0db-512db79382a8-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-d98bb58d-5dc747fd5d-gvbsd\" (UID: \"be6caaba-5c60-4d62-b0db-512db79382a8\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d98bb58d-5dc747fd5d-gvbsd"
Sep 10 22:19:15.807227 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:15.807191    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwlj6\" (UniqueName: \"kubernetes.io/projected/be6caaba-5c60-4d62-b0db-512db79382a8-kube-api-access-pwlj6\") pod \"payload-pre-processing-e2e-mt-d98bb58d-5dc747fd5d-gvbsd\" (UID: \"be6caaba-5c60-4d62-b0db-512db79382a8\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d98bb58d-5dc747fd5d-gvbsd"
Sep 10 22:19:15.807741 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:15.807721    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be6caaba-5c60-4d62-b0db-512db79382a8-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-d98bb58d-5dc747fd5d-gvbsd\" (UID: \"be6caaba-5c60-4d62-b0db-512db79382a8\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d98bb58d-5dc747fd5d-gvbsd"
Sep 10 22:19:15.815265 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:15.815245    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwlj6\" (UniqueName: \"kubernetes.io/projected/be6caaba-5c60-4d62-b0db-512db79382a8-kube-api-access-pwlj6\") pod \"payload-pre-processing-e2e-mt-d98bb58d-5dc747fd5d-gvbsd\" (UID: \"be6caaba-5c60-4d62-b0db-512db79382a8\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d98bb58d-5dc747fd5d-gvbsd"
Sep 10 22:19:15.924546 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:15.924520    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-d98bb58d-5dc747fd5d-gvbsd"
Sep 10 22:19:16.043922 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:16.043888    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d98bb58d-54fdbf4cf5-4bldz"]
Sep 10 22:19:16.045680 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:19:16.045646    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd97d86ed_a521_4792_a760_0d95aa5ee6e7.slice/crio-c5b707abb8fff9f0dc02f5ee3d7d76e4ea35cc013c989d4323fefc9cbe276c36 WatchSource:0}: Error finding container c5b707abb8fff9f0dc02f5ee3d7d76e4ea35cc013c989d4323fefc9cbe276c36: Status 404 returned error can't find the container with id c5b707abb8fff9f0dc02f5ee3d7d76e4ea35cc013c989d4323fefc9cbe276c36
Sep 10 22:19:16.047078 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:16.047053    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 22:19:16.194001 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:16.193953    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d98bb58d-5dc747fd5d-gvbsd"]
Sep 10 22:19:16.196280 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:19:16.196253    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe6caaba_5c60_4d62_b0db_512db79382a8.slice/crio-39d17a17677ce0e8b390d46eb4f15c4c62227b6b1e5311435d9d08c19deb629b WatchSource:0}: Error finding container 39d17a17677ce0e8b390d46eb4f15c4c62227b6b1e5311435d9d08c19deb629b: Status 404 returned error can't find the container with id 39d17a17677ce0e8b390d46eb4f15c4c62227b6b1e5311435d9d08c19deb629b
Sep 10 22:19:16.330376 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:16.330344    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-d98bb58d-5dc747fd5d-gvbsd" event={"ID":"be6caaba-5c60-4d62-b0db-512db79382a8","Type":"ContainerStarted","Data":"0dbc09f7b33a88a9afe5fdd79b945f8a4c1cee1d8208b7029be7d2ee9cb2169c"}
Sep 10 22:19:16.330542 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:16.330506    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-d98bb58d-5dc747fd5d-gvbsd"
Sep 10 22:19:16.330606 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:16.330549    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-d98bb58d-5dc747fd5d-gvbsd" event={"ID":"be6caaba-5c60-4d62-b0db-512db79382a8","Type":"ContainerStarted","Data":"39d17a17677ce0e8b390d46eb4f15c4c62227b6b1e5311435d9d08c19deb629b"}
Sep 10 22:19:16.331636 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:16.331609    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d98bb58d-54fdbf4cf5-4bldz" event={"ID":"d97d86ed-a521-4792-a760-0d95aa5ee6e7","Type":"ContainerStarted","Data":"c5b707abb8fff9f0dc02f5ee3d7d76e4ea35cc013c989d4323fefc9cbe276c36"}
Sep 10 22:19:16.351036 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:16.350992    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-d98bb58d-5dc747fd5d-gvbsd" podStartSLOduration=1.35097658 podStartE2EDuration="1.35097658s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 22:19:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:19:16.349604323 +0000 UTC m=+1077.305672029" watchObservedRunningTime="2026-09-10 22:19:16.35097658 +0000 UTC m=+1077.307044287"
Sep 10 22:19:17.338073 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:17.338037    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d98bb58d-54fdbf4cf5-4bldz" event={"ID":"d97d86ed-a521-4792-a760-0d95aa5ee6e7","Type":"ContainerStarted","Data":"a3468e07481a60e4afed00c1d4e3bc4c397a255af0ff84ae4b4bcf4383c110f3"}
Sep 10 22:19:17.338540 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:17.338295    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d98bb58d-54fdbf4cf5-4bldz"
Sep 10 22:19:17.356736 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:17.356691    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d98bb58d-54fdbf4cf5-4bldz" podStartSLOduration=2.920393563 podStartE2EDuration="3.356673444s" totalImagesPullingTime="436.279881ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 22:19:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:19:17.356161105 +0000 UTC m=+1078.312228838" watchObservedRunningTime="2026-09-10 22:19:17.356673444 +0000 UTC m=+1078.312741214"
Sep 10 22:19:23.349162 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:23.349129    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d98bb58d-54fdbf4cf5-4bldz"
Sep 10 22:19:27.340918 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:27.340883    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-d98bb58d-5dc747fd5d-gvbsd"
Sep 10 22:19:41.097521 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:41.097478    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-361ccf/noten-model-361ccf-kserve-59f79fb88c-7cjgz"]
Sep 10 22:19:41.102509 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:41.102485    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-361ccf/noten-model-361ccf-kserve-59f79fb88c-7cjgz"
Sep 10 22:19:41.105806 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:41.105782    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-361ccf\"/\"default-dockercfg-jv4rg\""
Sep 10 22:19:41.106134 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:41.106110    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-361ccf\"/\"noten-model-361ccf-kserve-self-signed-certs\""
Sep 10 22:19:41.106134 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:41.106128    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-361ccf\"/\"openshift-service-ca.crt\""
Sep 10 22:19:41.106275 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:41.106133    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-361ccf\"/\"kube-root-ca.crt\""
Sep 10 22:19:41.115341 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:41.115318    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-361ccf/noten-model-361ccf-kserve-59f79fb88c-7cjgz"]
Sep 10 22:19:41.127237 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:41.127213    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae629c41-66d4-4754-b4f6-df23857b31de-tls-certs\") pod \"noten-model-361ccf-kserve-59f79fb88c-7cjgz\" (UID: \"ae629c41-66d4-4754-b4f6-df23857b31de\") " pod="ai-tenant-e2e-noten-361ccf/noten-model-361ccf-kserve-59f79fb88c-7cjgz"
Sep 10 22:19:41.127365 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:41.127241    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdlz5\" (UniqueName: \"kubernetes.io/projected/ae629c41-66d4-4754-b4f6-df23857b31de-kube-api-access-hdlz5\") pod \"noten-model-361ccf-kserve-59f79fb88c-7cjgz\" (UID: \"ae629c41-66d4-4754-b4f6-df23857b31de\") " pod="ai-tenant-e2e-noten-361ccf/noten-model-361ccf-kserve-59f79fb88c-7cjgz"
Sep 10 22:19:41.127365 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:41.127271    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae629c41-66d4-4754-b4f6-df23857b31de-home\") pod \"noten-model-361ccf-kserve-59f79fb88c-7cjgz\" (UID: \"ae629c41-66d4-4754-b4f6-df23857b31de\") " pod="ai-tenant-e2e-noten-361ccf/noten-model-361ccf-kserve-59f79fb88c-7cjgz"
Sep 10 22:19:41.127365 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:41.127326    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae629c41-66d4-4754-b4f6-df23857b31de-model-cache\") pod \"noten-model-361ccf-kserve-59f79fb88c-7cjgz\" (UID: \"ae629c41-66d4-4754-b4f6-df23857b31de\") " pod="ai-tenant-e2e-noten-361ccf/noten-model-361ccf-kserve-59f79fb88c-7cjgz"
Sep 10 22:19:41.127478 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:41.127379    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae629c41-66d4-4754-b4f6-df23857b31de-tmp-dir\") pod \"noten-model-361ccf-kserve-59f79fb88c-7cjgz\" (UID: \"ae629c41-66d4-4754-b4f6-df23857b31de\") " pod="ai-tenant-e2e-noten-361ccf/noten-model-361ccf-kserve-59f79fb88c-7cjgz"
Sep 10 22:19:41.127478 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:41.127404    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae629c41-66d4-4754-b4f6-df23857b31de-dshm\") pod \"noten-model-361ccf-kserve-59f79fb88c-7cjgz\" (UID: \"ae629c41-66d4-4754-b4f6-df23857b31de\") " pod="ai-tenant-e2e-noten-361ccf/noten-model-361ccf-kserve-59f79fb88c-7cjgz"
Sep 10 22:19:41.228351 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:41.228315    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae629c41-66d4-4754-b4f6-df23857b31de-model-cache\") pod \"noten-model-361ccf-kserve-59f79fb88c-7cjgz\" (UID: \"ae629c41-66d4-4754-b4f6-df23857b31de\") " pod="ai-tenant-e2e-noten-361ccf/noten-model-361ccf-kserve-59f79fb88c-7cjgz"
Sep 10 22:19:41.228530 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:41.228360    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae629c41-66d4-4754-b4f6-df23857b31de-tmp-dir\") pod \"noten-model-361ccf-kserve-59f79fb88c-7cjgz\" (UID: \"ae629c41-66d4-4754-b4f6-df23857b31de\") " pod="ai-tenant-e2e-noten-361ccf/noten-model-361ccf-kserve-59f79fb88c-7cjgz"
Sep 10 22:19:41.228530 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:41.228385    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae629c41-66d4-4754-b4f6-df23857b31de-dshm\") pod \"noten-model-361ccf-kserve-59f79fb88c-7cjgz\" (UID: \"ae629c41-66d4-4754-b4f6-df23857b31de\") " pod="ai-tenant-e2e-noten-361ccf/noten-model-361ccf-kserve-59f79fb88c-7cjgz"
Sep 10 22:19:41.228530 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:41.228433    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae629c41-66d4-4754-b4f6-df23857b31de-tls-certs\") pod \"noten-model-361ccf-kserve-59f79fb88c-7cjgz\" (UID: \"ae629c41-66d4-4754-b4f6-df23857b31de\") " pod="ai-tenant-e2e-noten-361ccf/noten-model-361ccf-kserve-59f79fb88c-7cjgz"
Sep 10 22:19:41.228530 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:41.228450    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdlz5\" (UniqueName: \"kubernetes.io/projected/ae629c41-66d4-4754-b4f6-df23857b31de-kube-api-access-hdlz5\") pod \"noten-model-361ccf-kserve-59f79fb88c-7cjgz\" (UID: \"ae629c41-66d4-4754-b4f6-df23857b31de\") " pod="ai-tenant-e2e-noten-361ccf/noten-model-361ccf-kserve-59f79fb88c-7cjgz"
Sep 10 22:19:41.228530 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:41.228479    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae629c41-66d4-4754-b4f6-df23857b31de-home\") pod \"noten-model-361ccf-kserve-59f79fb88c-7cjgz\" (UID: \"ae629c41-66d4-4754-b4f6-df23857b31de\") " pod="ai-tenant-e2e-noten-361ccf/noten-model-361ccf-kserve-59f79fb88c-7cjgz"
Sep 10 22:19:41.228812 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:41.228709    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae629c41-66d4-4754-b4f6-df23857b31de-model-cache\") pod \"noten-model-361ccf-kserve-59f79fb88c-7cjgz\" (UID: \"ae629c41-66d4-4754-b4f6-df23857b31de\") " pod="ai-tenant-e2e-noten-361ccf/noten-model-361ccf-kserve-59f79fb88c-7cjgz"
Sep 10 22:19:41.228812 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:41.228759    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae629c41-66d4-4754-b4f6-df23857b31de-tmp-dir\") pod \"noten-model-361ccf-kserve-59f79fb88c-7cjgz\" (UID: \"ae629c41-66d4-4754-b4f6-df23857b31de\") " pod="ai-tenant-e2e-noten-361ccf/noten-model-361ccf-kserve-59f79fb88c-7cjgz"
Sep 10 22:19:41.228929 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:41.228888    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae629c41-66d4-4754-b4f6-df23857b31de-home\") pod \"noten-model-361ccf-kserve-59f79fb88c-7cjgz\" (UID: \"ae629c41-66d4-4754-b4f6-df23857b31de\") " pod="ai-tenant-e2e-noten-361ccf/noten-model-361ccf-kserve-59f79fb88c-7cjgz"
Sep 10 22:19:41.230770 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:41.230745    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae629c41-66d4-4754-b4f6-df23857b31de-dshm\") pod \"noten-model-361ccf-kserve-59f79fb88c-7cjgz\" (UID: \"ae629c41-66d4-4754-b4f6-df23857b31de\") " pod="ai-tenant-e2e-noten-361ccf/noten-model-361ccf-kserve-59f79fb88c-7cjgz"
Sep 10 22:19:41.231053 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:41.231037    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae629c41-66d4-4754-b4f6-df23857b31de-tls-certs\") pod \"noten-model-361ccf-kserve-59f79fb88c-7cjgz\" (UID: \"ae629c41-66d4-4754-b4f6-df23857b31de\") " pod="ai-tenant-e2e-noten-361ccf/noten-model-361ccf-kserve-59f79fb88c-7cjgz"
Sep 10 22:19:41.235927 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:41.235908    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdlz5\" (UniqueName: \"kubernetes.io/projected/ae629c41-66d4-4754-b4f6-df23857b31de-kube-api-access-hdlz5\") pod \"noten-model-361ccf-kserve-59f79fb88c-7cjgz\" (UID: \"ae629c41-66d4-4754-b4f6-df23857b31de\") " pod="ai-tenant-e2e-noten-361ccf/noten-model-361ccf-kserve-59f79fb88c-7cjgz"
Sep 10 22:19:41.411962 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:41.411929    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-361ccf/noten-model-361ccf-kserve-59f79fb88c-7cjgz"
Sep 10 22:19:41.692085 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:41.692059    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-361ccf/noten-model-361ccf-kserve-59f79fb88c-7cjgz"]
Sep 10 22:19:41.694232 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:19:41.694205    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podae629c41_66d4_4754_b4f6_df23857b31de.slice/crio-d8af910e2ed91089f51fc722400d9569d661a56ea1726cb75e7d62926a58a3a8 WatchSource:0}: Error finding container d8af910e2ed91089f51fc722400d9569d661a56ea1726cb75e7d62926a58a3a8: Status 404 returned error can't find the container with id d8af910e2ed91089f51fc722400d9569d661a56ea1726cb75e7d62926a58a3a8
Sep 10 22:19:42.462796 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:42.462756    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-361ccf/noten-model-361ccf-kserve-59f79fb88c-7cjgz" event={"ID":"ae629c41-66d4-4754-b4f6-df23857b31de","Type":"ContainerStarted","Data":"4aa47b9931d7869ceb614167154af8a8b86f0d15e96a66f1b1925d1b08c0a7bc"}
Sep 10 22:19:42.462796 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:42.462804    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-361ccf/noten-model-361ccf-kserve-59f79fb88c-7cjgz" event={"ID":"ae629c41-66d4-4754-b4f6-df23857b31de","Type":"ContainerStarted","Data":"d8af910e2ed91089f51fc722400d9569d661a56ea1726cb75e7d62926a58a3a8"}
Sep 10 22:19:42.479348 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:42.479304    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-noten-361ccf/noten-model-361ccf-kserve-59f79fb88c-7cjgz" podStartSLOduration=1.479290861 podStartE2EDuration="1.479290861s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 22:19:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:19:42.478456504 +0000 UTC m=+1103.434524212" watchObservedRunningTime="2026-09-10 22:19:42.479290861 +0000 UTC m=+1103.435358567"
Sep 10 22:19:44.136514 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:44.136481    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-e5a2d7/test-model-e5a2d7-kserve-57bbd95c65-xjc9m"]
Sep 10 22:19:44.982051 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:44.982016    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-361ccf/noten-model-361ccf-kserve-59f79fb88c-7cjgz"]
Sep 10 22:19:45.101421 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:45.101383    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-361ccf-openshift-default-758565f796-f4jhq"]
Sep 10 22:19:45.101633 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:45.101601    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-361ccf-openshift-default-758565f796-f4jhq" podUID="23fb1677-9564-480a-a2bd-2aec7f9b81bc" containerName="istio-proxy" containerID="cri-o://dd5400686c299df50527ce42f00db2748800f7eb27479ce368e2c29ccaf9d64e" gracePeriod=30
Sep 10 22:19:45.993195 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:45.993162    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-1c252b/test-model-1c252b-kserve-9b5ffcc55-j8qpx"]
Sep 10 22:19:50.344115 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:50.344092    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-361ccf-openshift-default-758565f796-f4jhq"
Sep 10 22:19:50.408919 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:50.408895    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23fb1677-9564-480a-a2bd-2aec7f9b81bc-istiod-ca-cert\") pod \"23fb1677-9564-480a-a2bd-2aec7f9b81bc\" (UID: \"23fb1677-9564-480a-a2bd-2aec7f9b81bc\") "
Sep 10 22:19:50.409034 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:50.408961    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23fb1677-9564-480a-a2bd-2aec7f9b81bc-istio-podinfo\") pod \"23fb1677-9564-480a-a2bd-2aec7f9b81bc\" (UID: \"23fb1677-9564-480a-a2bd-2aec7f9b81bc\") "
Sep 10 22:19:50.409034 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:50.409003    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23fb1677-9564-480a-a2bd-2aec7f9b81bc-credential-socket\") pod \"23fb1677-9564-480a-a2bd-2aec7f9b81bc\" (UID: \"23fb1677-9564-480a-a2bd-2aec7f9b81bc\") "
Sep 10 22:19:50.409136 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:50.409040    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23fb1677-9564-480a-a2bd-2aec7f9b81bc-workload-socket\") pod \"23fb1677-9564-480a-a2bd-2aec7f9b81bc\" (UID: \"23fb1677-9564-480a-a2bd-2aec7f9b81bc\") "
Sep 10 22:19:50.409136 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:50.409064    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23fb1677-9564-480a-a2bd-2aec7f9b81bc-istio-token\") pod \"23fb1677-9564-480a-a2bd-2aec7f9b81bc\" (UID: \"23fb1677-9564-480a-a2bd-2aec7f9b81bc\") "
Sep 10 22:19:50.409136 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:50.409091    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23fb1677-9564-480a-a2bd-2aec7f9b81bc-istio-envoy\") pod \"23fb1677-9564-480a-a2bd-2aec7f9b81bc\" (UID: \"23fb1677-9564-480a-a2bd-2aec7f9b81bc\") "
Sep 10 22:19:50.409136 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:50.409130    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vzmc2\" (UniqueName: \"kubernetes.io/projected/23fb1677-9564-480a-a2bd-2aec7f9b81bc-kube-api-access-vzmc2\") pod \"23fb1677-9564-480a-a2bd-2aec7f9b81bc\" (UID: \"23fb1677-9564-480a-a2bd-2aec7f9b81bc\") "
Sep 10 22:19:50.409336 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:50.409159    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23fb1677-9564-480a-a2bd-2aec7f9b81bc-istio-data\") pod \"23fb1677-9564-480a-a2bd-2aec7f9b81bc\" (UID: \"23fb1677-9564-480a-a2bd-2aec7f9b81bc\") "
Sep 10 22:19:50.409336 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:50.409193    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23fb1677-9564-480a-a2bd-2aec7f9b81bc-workload-certs\") pod \"23fb1677-9564-480a-a2bd-2aec7f9b81bc\" (UID: \"23fb1677-9564-480a-a2bd-2aec7f9b81bc\") "
Sep 10 22:19:50.409336 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:50.409318    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23fb1677-9564-480a-a2bd-2aec7f9b81bc-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "23fb1677-9564-480a-a2bd-2aec7f9b81bc" (UID: "23fb1677-9564-480a-a2bd-2aec7f9b81bc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 22:19:50.409502 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:50.409345    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23fb1677-9564-480a-a2bd-2aec7f9b81bc-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "23fb1677-9564-480a-a2bd-2aec7f9b81bc" (UID: "23fb1677-9564-480a-a2bd-2aec7f9b81bc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 22:19:50.409502 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:50.409398    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23fb1677-9564-480a-a2bd-2aec7f9b81bc-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "23fb1677-9564-480a-a2bd-2aec7f9b81bc" (UID: "23fb1677-9564-480a-a2bd-2aec7f9b81bc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 22:19:50.409614 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:50.409538    2582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23fb1677-9564-480a-a2bd-2aec7f9b81bc-istiod-ca-cert\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:19:50.409614 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:50.409524    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23fb1677-9564-480a-a2bd-2aec7f9b81bc-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "23fb1677-9564-480a-a2bd-2aec7f9b81bc" (UID: "23fb1677-9564-480a-a2bd-2aec7f9b81bc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 22:19:50.409614 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:50.409558    2582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23fb1677-9564-480a-a2bd-2aec7f9b81bc-credential-socket\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:19:50.409614 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:50.409575    2582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23fb1677-9564-480a-a2bd-2aec7f9b81bc-workload-socket\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:19:50.409614 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:50.409572    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23fb1677-9564-480a-a2bd-2aec7f9b81bc-istio-data" (OuterVolumeSpecName: "istio-data") pod "23fb1677-9564-480a-a2bd-2aec7f9b81bc" (UID: "23fb1677-9564-480a-a2bd-2aec7f9b81bc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 22:19:50.411224 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:50.411192    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23fb1677-9564-480a-a2bd-2aec7f9b81bc-istio-token" (OuterVolumeSpecName: "istio-token") pod "23fb1677-9564-480a-a2bd-2aec7f9b81bc" (UID: "23fb1677-9564-480a-a2bd-2aec7f9b81bc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 22:19:50.411342 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:50.411239    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23fb1677-9564-480a-a2bd-2aec7f9b81bc-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "23fb1677-9564-480a-a2bd-2aec7f9b81bc" (UID: "23fb1677-9564-480a-a2bd-2aec7f9b81bc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 22:19:50.411342 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:50.411285    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/23fb1677-9564-480a-a2bd-2aec7f9b81bc-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "23fb1677-9564-480a-a2bd-2aec7f9b81bc" (UID: "23fb1677-9564-480a-a2bd-2aec7f9b81bc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 22:19:50.411427 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:50.411366    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23fb1677-9564-480a-a2bd-2aec7f9b81bc-kube-api-access-vzmc2" (OuterVolumeSpecName: "kube-api-access-vzmc2") pod "23fb1677-9564-480a-a2bd-2aec7f9b81bc" (UID: "23fb1677-9564-480a-a2bd-2aec7f9b81bc"). InnerVolumeSpecName "kube-api-access-vzmc2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 22:19:50.502032 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:50.502005    2582 generic.go:356] "Generic (PLEG): container finished" podID="23fb1677-9564-480a-a2bd-2aec7f9b81bc" containerID="dd5400686c299df50527ce42f00db2748800f7eb27479ce368e2c29ccaf9d64e" exitCode=0
Sep 10 22:19:50.502129 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:50.502046    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-361ccf-openshift-default-758565f796-f4jhq"
Sep 10 22:19:50.502129 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:50.502058    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-361ccf-openshift-default-758565f796-f4jhq" event={"ID":"23fb1677-9564-480a-a2bd-2aec7f9b81bc","Type":"ContainerDied","Data":"dd5400686c299df50527ce42f00db2748800f7eb27479ce368e2c29ccaf9d64e"}
Sep 10 22:19:50.502129 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:50.502097    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-361ccf-openshift-default-758565f796-f4jhq" event={"ID":"23fb1677-9564-480a-a2bd-2aec7f9b81bc","Type":"ContainerDied","Data":"d4d23f94e671ff32782a3bc40f6bbbffeb4d3fc810c8a3b3d27337007656a874"}
Sep 10 22:19:50.502129 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:50.502118    2582 scope.go:117] "RemoveContainer" containerID="dd5400686c299df50527ce42f00db2748800f7eb27479ce368e2c29ccaf9d64e"
Sep 10 22:19:50.510117 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:50.510075    2582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23fb1677-9564-480a-a2bd-2aec7f9b81bc-istio-token\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:19:50.510117 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:50.510098    2582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23fb1677-9564-480a-a2bd-2aec7f9b81bc-istio-envoy\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:19:50.510117 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:50.510113    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vzmc2\" (UniqueName: \"kubernetes.io/projected/23fb1677-9564-480a-a2bd-2aec7f9b81bc-kube-api-access-vzmc2\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:19:50.510331 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:50.510128    2582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23fb1677-9564-480a-a2bd-2aec7f9b81bc-istio-data\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:19:50.510331 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:50.510140    2582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23fb1677-9564-480a-a2bd-2aec7f9b81bc-workload-certs\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:19:50.510331 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:50.510148    2582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23fb1677-9564-480a-a2bd-2aec7f9b81bc-istio-podinfo\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:19:50.511210 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:50.511194    2582 scope.go:117] "RemoveContainer" containerID="dd5400686c299df50527ce42f00db2748800f7eb27479ce368e2c29ccaf9d64e"
Sep 10 22:19:50.511454 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:19:50.511432    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd5400686c299df50527ce42f00db2748800f7eb27479ce368e2c29ccaf9d64e\": container with ID starting with dd5400686c299df50527ce42f00db2748800f7eb27479ce368e2c29ccaf9d64e not found: ID does not exist" containerID="dd5400686c299df50527ce42f00db2748800f7eb27479ce368e2c29ccaf9d64e"
Sep 10 22:19:50.511507 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:50.511469    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd5400686c299df50527ce42f00db2748800f7eb27479ce368e2c29ccaf9d64e"} err="failed to get container status \"dd5400686c299df50527ce42f00db2748800f7eb27479ce368e2c29ccaf9d64e\": rpc error: code = NotFound desc = could not find container \"dd5400686c299df50527ce42f00db2748800f7eb27479ce368e2c29ccaf9d64e\": container with ID starting with dd5400686c299df50527ce42f00db2748800f7eb27479ce368e2c29ccaf9d64e not found: ID does not exist"
Sep 10 22:19:50.530950 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:50.530927    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-361ccf-openshift-default-758565f796-f4jhq"]
Sep 10 22:19:50.537699 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:50.537673    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-361ccf-openshift-default-758565f796-f4jhq"]
Sep 10 22:19:51.412997 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:51.412966    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-361ccf/noten-model-361ccf-kserve-59f79fb88c-7cjgz"
Sep 10 22:19:51.498508 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:51.498476    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="23fb1677-9564-480a-a2bd-2aec7f9b81bc" path="/var/lib/kubelet/pods/23fb1677-9564-480a-a2bd-2aec7f9b81bc/volumes"
Sep 10 22:19:56.628228 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:56.628192    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d98bb58d-54fdbf4cf5-4bldz"]
Sep 10 22:19:56.628824 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:56.628445    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d98bb58d-54fdbf4cf5-4bldz" podUID="d97d86ed-a521-4792-a760-0d95aa5ee6e7" containerName="maas-api" containerID="cri-o://a3468e07481a60e4afed00c1d4e3bc4c397a255af0ff84ae4b4bcf4383c110f3" gracePeriod=30
Sep 10 22:19:56.689741 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:56.689696    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d98bb58d-5dc747fd5d-gvbsd"]
Sep 10 22:19:56.690010 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:56.689977    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-d98bb58d-5dc747fd5d-gvbsd" podUID="be6caaba-5c60-4d62-b0db-512db79382a8" containerName="payload-pre-processing" containerID="cri-o://0dbc09f7b33a88a9afe5fdd79b945f8a4c1cee1d8208b7029be7d2ee9cb2169c" gracePeriod=30
Sep 10 22:19:56.843012 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:56.842967    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e5a2d7-29817975-db5gb"]
Sep 10 22:19:56.845785 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:56.845755    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e5a2d7-29817975-db5gb"]
Sep 10 22:19:56.949451 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:56.949424    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d98bb58d-54fdbf4cf5-4bldz"
Sep 10 22:19:56.978196 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:56.978165    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-d98bb58d-5dc747fd5d-gvbsd"
Sep 10 22:19:57.060869 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:57.060816    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d97d86ed-a521-4792-a760-0d95aa5ee6e7-openshift-service-ca\") pod \"d97d86ed-a521-4792-a760-0d95aa5ee6e7\" (UID: \"d97d86ed-a521-4792-a760-0d95aa5ee6e7\") "
Sep 10 22:19:57.061020 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:57.060876    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be6caaba-5c60-4d62-b0db-512db79382a8-plugins-config-volume\") pod \"be6caaba-5c60-4d62-b0db-512db79382a8\" (UID: \"be6caaba-5c60-4d62-b0db-512db79382a8\") "
Sep 10 22:19:57.061020 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:57.060918    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mrfkr\" (UniqueName: \"kubernetes.io/projected/d97d86ed-a521-4792-a760-0d95aa5ee6e7-kube-api-access-mrfkr\") pod \"d97d86ed-a521-4792-a760-0d95aa5ee6e7\" (UID: \"d97d86ed-a521-4792-a760-0d95aa5ee6e7\") "
Sep 10 22:19:57.061020 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:57.060983    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pwlj6\" (UniqueName: \"kubernetes.io/projected/be6caaba-5c60-4d62-b0db-512db79382a8-kube-api-access-pwlj6\") pod \"be6caaba-5c60-4d62-b0db-512db79382a8\" (UID: \"be6caaba-5c60-4d62-b0db-512db79382a8\") "
Sep 10 22:19:57.061160 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:57.061035    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d97d86ed-a521-4792-a760-0d95aa5ee6e7-metrics-certs\") pod \"d97d86ed-a521-4792-a760-0d95aa5ee6e7\" (UID: \"d97d86ed-a521-4792-a760-0d95aa5ee6e7\") "
Sep 10 22:19:57.061160 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:57.061085    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d97d86ed-a521-4792-a760-0d95aa5ee6e7-maas-api-tls\") pod \"d97d86ed-a521-4792-a760-0d95aa5ee6e7\" (UID: \"d97d86ed-a521-4792-a760-0d95aa5ee6e7\") "
Sep 10 22:19:57.061278 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:57.061247    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d97d86ed-a521-4792-a760-0d95aa5ee6e7-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d97d86ed-a521-4792-a760-0d95aa5ee6e7" (UID: "d97d86ed-a521-4792-a760-0d95aa5ee6e7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 22:19:57.061278 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:57.061252    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be6caaba-5c60-4d62-b0db-512db79382a8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "be6caaba-5c60-4d62-b0db-512db79382a8" (UID: "be6caaba-5c60-4d62-b0db-512db79382a8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 22:19:57.061413 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:57.061385    2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d97d86ed-a521-4792-a760-0d95aa5ee6e7-openshift-service-ca\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:19:57.061413 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:57.061408    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be6caaba-5c60-4d62-b0db-512db79382a8-plugins-config-volume\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:19:57.063354 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:57.063321    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d97d86ed-a521-4792-a760-0d95aa5ee6e7-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "d97d86ed-a521-4792-a760-0d95aa5ee6e7" (UID: "d97d86ed-a521-4792-a760-0d95aa5ee6e7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 22:19:57.063462 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:57.063373    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d97d86ed-a521-4792-a760-0d95aa5ee6e7-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d97d86ed-a521-4792-a760-0d95aa5ee6e7" (UID: "d97d86ed-a521-4792-a760-0d95aa5ee6e7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 22:19:57.063605 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:57.063586    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d97d86ed-a521-4792-a760-0d95aa5ee6e7-kube-api-access-mrfkr" (OuterVolumeSpecName: "kube-api-access-mrfkr") pod "d97d86ed-a521-4792-a760-0d95aa5ee6e7" (UID: "d97d86ed-a521-4792-a760-0d95aa5ee6e7"). InnerVolumeSpecName "kube-api-access-mrfkr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 22:19:57.063651 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:57.063596    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be6caaba-5c60-4d62-b0db-512db79382a8-kube-api-access-pwlj6" (OuterVolumeSpecName: "kube-api-access-pwlj6") pod "be6caaba-5c60-4d62-b0db-512db79382a8" (UID: "be6caaba-5c60-4d62-b0db-512db79382a8"). InnerVolumeSpecName "kube-api-access-pwlj6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 22:19:57.162744 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:57.162711    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mrfkr\" (UniqueName: \"kubernetes.io/projected/d97d86ed-a521-4792-a760-0d95aa5ee6e7-kube-api-access-mrfkr\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:19:57.162744 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:57.162736    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwlj6\" (UniqueName: \"kubernetes.io/projected/be6caaba-5c60-4d62-b0db-512db79382a8-kube-api-access-pwlj6\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:19:57.162744 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:57.162746    2582 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d97d86ed-a521-4792-a760-0d95aa5ee6e7-metrics-certs\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:19:57.162982 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:57.162755    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d97d86ed-a521-4792-a760-0d95aa5ee6e7-maas-api-tls\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:19:57.498659 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:57.498622    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f605e12b-a160-4c45-b2b7-759154729347" path="/var/lib/kubelet/pods/f605e12b-a160-4c45-b2b7-759154729347/volumes"
Sep 10 22:19:57.533964 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:57.533930    2582 generic.go:356] "Generic (PLEG): container finished" podID="d97d86ed-a521-4792-a760-0d95aa5ee6e7" containerID="a3468e07481a60e4afed00c1d4e3bc4c397a255af0ff84ae4b4bcf4383c110f3" exitCode=0
Sep 10 22:19:57.534098 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:57.533967    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d98bb58d-54fdbf4cf5-4bldz"
Sep 10 22:19:57.534098 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:57.534022    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d98bb58d-54fdbf4cf5-4bldz" event={"ID":"d97d86ed-a521-4792-a760-0d95aa5ee6e7","Type":"ContainerDied","Data":"a3468e07481a60e4afed00c1d4e3bc4c397a255af0ff84ae4b4bcf4383c110f3"}
Sep 10 22:19:57.534098 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:57.534073    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d98bb58d-54fdbf4cf5-4bldz" event={"ID":"d97d86ed-a521-4792-a760-0d95aa5ee6e7","Type":"ContainerDied","Data":"c5b707abb8fff9f0dc02f5ee3d7d76e4ea35cc013c989d4323fefc9cbe276c36"}
Sep 10 22:19:57.534098 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:57.534097    2582 scope.go:117] "RemoveContainer" containerID="a3468e07481a60e4afed00c1d4e3bc4c397a255af0ff84ae4b4bcf4383c110f3"
Sep 10 22:19:57.535541 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:57.535518    2582 generic.go:356] "Generic (PLEG): container finished" podID="be6caaba-5c60-4d62-b0db-512db79382a8" containerID="0dbc09f7b33a88a9afe5fdd79b945f8a4c1cee1d8208b7029be7d2ee9cb2169c" exitCode=0
Sep 10 22:19:57.535661 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:57.535553    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-d98bb58d-5dc747fd5d-gvbsd" event={"ID":"be6caaba-5c60-4d62-b0db-512db79382a8","Type":"ContainerDied","Data":"0dbc09f7b33a88a9afe5fdd79b945f8a4c1cee1d8208b7029be7d2ee9cb2169c"}
Sep 10 22:19:57.535661 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:57.535577    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-d98bb58d-5dc747fd5d-gvbsd"
Sep 10 22:19:57.535661 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:57.535603    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-d98bb58d-5dc747fd5d-gvbsd" event={"ID":"be6caaba-5c60-4d62-b0db-512db79382a8","Type":"ContainerDied","Data":"39d17a17677ce0e8b390d46eb4f15c4c62227b6b1e5311435d9d08c19deb629b"}
Sep 10 22:19:57.543011 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:57.542994    2582 scope.go:117] "RemoveContainer" containerID="a3468e07481a60e4afed00c1d4e3bc4c397a255af0ff84ae4b4bcf4383c110f3"
Sep 10 22:19:57.543328 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:19:57.543309    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3468e07481a60e4afed00c1d4e3bc4c397a255af0ff84ae4b4bcf4383c110f3\": container with ID starting with a3468e07481a60e4afed00c1d4e3bc4c397a255af0ff84ae4b4bcf4383c110f3 not found: ID does not exist" containerID="a3468e07481a60e4afed00c1d4e3bc4c397a255af0ff84ae4b4bcf4383c110f3"
Sep 10 22:19:57.543399 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:57.543336    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3468e07481a60e4afed00c1d4e3bc4c397a255af0ff84ae4b4bcf4383c110f3"} err="failed to get container status \"a3468e07481a60e4afed00c1d4e3bc4c397a255af0ff84ae4b4bcf4383c110f3\": rpc error: code = NotFound desc = could not find container \"a3468e07481a60e4afed00c1d4e3bc4c397a255af0ff84ae4b4bcf4383c110f3\": container with ID starting with a3468e07481a60e4afed00c1d4e3bc4c397a255af0ff84ae4b4bcf4383c110f3 not found: ID does not exist"
Sep 10 22:19:57.543399 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:57.543351    2582 scope.go:117] "RemoveContainer" containerID="0dbc09f7b33a88a9afe5fdd79b945f8a4c1cee1d8208b7029be7d2ee9cb2169c"
Sep 10 22:19:57.550798 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:57.550781    2582 scope.go:117] "RemoveContainer" containerID="0dbc09f7b33a88a9afe5fdd79b945f8a4c1cee1d8208b7029be7d2ee9cb2169c"
Sep 10 22:19:57.551147 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:19:57.551126    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0dbc09f7b33a88a9afe5fdd79b945f8a4c1cee1d8208b7029be7d2ee9cb2169c\": container with ID starting with 0dbc09f7b33a88a9afe5fdd79b945f8a4c1cee1d8208b7029be7d2ee9cb2169c not found: ID does not exist" containerID="0dbc09f7b33a88a9afe5fdd79b945f8a4c1cee1d8208b7029be7d2ee9cb2169c"
Sep 10 22:19:57.551231 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:57.551155    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0dbc09f7b33a88a9afe5fdd79b945f8a4c1cee1d8208b7029be7d2ee9cb2169c"} err="failed to get container status \"0dbc09f7b33a88a9afe5fdd79b945f8a4c1cee1d8208b7029be7d2ee9cb2169c\": rpc error: code = NotFound desc = could not find container \"0dbc09f7b33a88a9afe5fdd79b945f8a4c1cee1d8208b7029be7d2ee9cb2169c\": container with ID starting with 0dbc09f7b33a88a9afe5fdd79b945f8a4c1cee1d8208b7029be7d2ee9cb2169c not found: ID does not exist"
Sep 10 22:19:57.554895 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:57.554873    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d98bb58d-54fdbf4cf5-4bldz"]
Sep 10 22:19:57.558588 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:57.558568    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d98bb58d-54fdbf4cf5-4bldz"]
Sep 10 22:19:57.567581 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:57.567559    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d98bb58d-5dc747fd5d-gvbsd"]
Sep 10 22:19:57.571741 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:57.571722    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d98bb58d-5dc747fd5d-gvbsd"]
Sep 10 22:19:59.155734 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:59.155694    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-e5a2d7/test-model-e5a2d7-kserve-57bbd95c65-xjc9m" podUID="d91987a5-bc23-4976-a1c7-c10448df0e0e" containerName="main" containerID="cri-o://06c34155530371088ab7322d2cf6ac0f10b322f655d33a084fae7d09af2bb6f8" gracePeriod=45
Sep 10 22:19:59.396193 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:59.396163    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-e5a2d7/test-model-e5a2d7-kserve-57bbd95c65-xjc9m"
Sep 10 22:19:59.481905 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:59.481822    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d91987a5-bc23-4976-a1c7-c10448df0e0e-dshm\") pod \"d91987a5-bc23-4976-a1c7-c10448df0e0e\" (UID: \"d91987a5-bc23-4976-a1c7-c10448df0e0e\") "
Sep 10 22:19:59.481905 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:59.481878    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d91987a5-bc23-4976-a1c7-c10448df0e0e-model-cache\") pod \"d91987a5-bc23-4976-a1c7-c10448df0e0e\" (UID: \"d91987a5-bc23-4976-a1c7-c10448df0e0e\") "
Sep 10 22:19:59.482095 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:59.481939    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ljml4\" (UniqueName: \"kubernetes.io/projected/d91987a5-bc23-4976-a1c7-c10448df0e0e-kube-api-access-ljml4\") pod \"d91987a5-bc23-4976-a1c7-c10448df0e0e\" (UID: \"d91987a5-bc23-4976-a1c7-c10448df0e0e\") "
Sep 10 22:19:59.482095 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:59.481970    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d91987a5-bc23-4976-a1c7-c10448df0e0e-tmp-dir\") pod \"d91987a5-bc23-4976-a1c7-c10448df0e0e\" (UID: \"d91987a5-bc23-4976-a1c7-c10448df0e0e\") "
Sep 10 22:19:59.482208 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:59.482178    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d91987a5-bc23-4976-a1c7-c10448df0e0e-tls-certs\") pod \"d91987a5-bc23-4976-a1c7-c10448df0e0e\" (UID: \"d91987a5-bc23-4976-a1c7-c10448df0e0e\") "
Sep 10 22:19:59.482250 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:59.482178    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d91987a5-bc23-4976-a1c7-c10448df0e0e-model-cache" (OuterVolumeSpecName: "model-cache") pod "d91987a5-bc23-4976-a1c7-c10448df0e0e" (UID: "d91987a5-bc23-4976-a1c7-c10448df0e0e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 22:19:59.482250 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:59.482211    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d91987a5-bc23-4976-a1c7-c10448df0e0e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d91987a5-bc23-4976-a1c7-c10448df0e0e" (UID: "d91987a5-bc23-4976-a1c7-c10448df0e0e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 22:19:59.482349 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:59.482252    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d91987a5-bc23-4976-a1c7-c10448df0e0e-home\") pod \"d91987a5-bc23-4976-a1c7-c10448df0e0e\" (UID: \"d91987a5-bc23-4976-a1c7-c10448df0e0e\") "
Sep 10 22:19:59.482455 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:59.482436    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d91987a5-bc23-4976-a1c7-c10448df0e0e-home" (OuterVolumeSpecName: "home") pod "d91987a5-bc23-4976-a1c7-c10448df0e0e" (UID: "d91987a5-bc23-4976-a1c7-c10448df0e0e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 22:19:59.482518 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:59.482481    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d91987a5-bc23-4976-a1c7-c10448df0e0e-model-cache\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:19:59.482518 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:59.482492    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d91987a5-bc23-4976-a1c7-c10448df0e0e-tmp-dir\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:19:59.484164 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:59.484140    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d91987a5-bc23-4976-a1c7-c10448df0e0e-dshm" (OuterVolumeSpecName: "dshm") pod "d91987a5-bc23-4976-a1c7-c10448df0e0e" (UID: "d91987a5-bc23-4976-a1c7-c10448df0e0e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 22:19:59.484164 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:59.484156    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d91987a5-bc23-4976-a1c7-c10448df0e0e-kube-api-access-ljml4" (OuterVolumeSpecName: "kube-api-access-ljml4") pod "d91987a5-bc23-4976-a1c7-c10448df0e0e" (UID: "d91987a5-bc23-4976-a1c7-c10448df0e0e"). InnerVolumeSpecName "kube-api-access-ljml4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 22:19:59.484298 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:59.484171    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d91987a5-bc23-4976-a1c7-c10448df0e0e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d91987a5-bc23-4976-a1c7-c10448df0e0e" (UID: "d91987a5-bc23-4976-a1c7-c10448df0e0e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 22:19:59.498062 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:59.498038    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="be6caaba-5c60-4d62-b0db-512db79382a8" path="/var/lib/kubelet/pods/be6caaba-5c60-4d62-b0db-512db79382a8/volumes"
Sep 10 22:19:59.498422 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:59.498409    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d97d86ed-a521-4792-a760-0d95aa5ee6e7" path="/var/lib/kubelet/pods/d97d86ed-a521-4792-a760-0d95aa5ee6e7/volumes"
Sep 10 22:19:59.546952 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:59.546924    2582 generic.go:356] "Generic (PLEG): container finished" podID="d91987a5-bc23-4976-a1c7-c10448df0e0e" containerID="06c34155530371088ab7322d2cf6ac0f10b322f655d33a084fae7d09af2bb6f8" exitCode=0
Sep 10 22:19:59.547029 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:59.546963    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-e5a2d7/test-model-e5a2d7-kserve-57bbd95c65-xjc9m"
Sep 10 22:19:59.547029 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:59.547014    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-e5a2d7/test-model-e5a2d7-kserve-57bbd95c65-xjc9m" event={"ID":"d91987a5-bc23-4976-a1c7-c10448df0e0e","Type":"ContainerDied","Data":"06c34155530371088ab7322d2cf6ac0f10b322f655d33a084fae7d09af2bb6f8"}
Sep 10 22:19:59.547120 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:59.547047    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-e5a2d7/test-model-e5a2d7-kserve-57bbd95c65-xjc9m" event={"ID":"d91987a5-bc23-4976-a1c7-c10448df0e0e","Type":"ContainerDied","Data":"b8f645dd39e5c42dde5b598c43079cd9ae7259eb0f764a2653077169dd0a36eb"}
Sep 10 22:19:59.547120 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:59.547064    2582 scope.go:117] "RemoveContainer" containerID="06c34155530371088ab7322d2cf6ac0f10b322f655d33a084fae7d09af2bb6f8"
Sep 10 22:19:59.554986 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:59.554967    2582 scope.go:117] "RemoveContainer" containerID="06c34155530371088ab7322d2cf6ac0f10b322f655d33a084fae7d09af2bb6f8"
Sep 10 22:19:59.555276 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:19:59.555254    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06c34155530371088ab7322d2cf6ac0f10b322f655d33a084fae7d09af2bb6f8\": container with ID starting with 06c34155530371088ab7322d2cf6ac0f10b322f655d33a084fae7d09af2bb6f8 not found: ID does not exist" containerID="06c34155530371088ab7322d2cf6ac0f10b322f655d33a084fae7d09af2bb6f8"
Sep 10 22:19:59.555327 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:59.555283    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06c34155530371088ab7322d2cf6ac0f10b322f655d33a084fae7d09af2bb6f8"} err="failed to get container status \"06c34155530371088ab7322d2cf6ac0f10b322f655d33a084fae7d09af2bb6f8\": rpc error: code = NotFound desc = could not find container \"06c34155530371088ab7322d2cf6ac0f10b322f655d33a084fae7d09af2bb6f8\": container with ID starting with 06c34155530371088ab7322d2cf6ac0f10b322f655d33a084fae7d09af2bb6f8 not found: ID does not exist"
Sep 10 22:19:59.572044 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:59.572019    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-e5a2d7/test-model-e5a2d7-kserve-57bbd95c65-xjc9m"]
Sep 10 22:19:59.576313 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:59.576295    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-e5a2d7/test-model-e5a2d7-kserve-57bbd95c65-xjc9m"]
Sep 10 22:19:59.583058 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:59.583040    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljml4\" (UniqueName: \"kubernetes.io/projected/d91987a5-bc23-4976-a1c7-c10448df0e0e-kube-api-access-ljml4\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:19:59.583120 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:59.583060    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d91987a5-bc23-4976-a1c7-c10448df0e0e-tls-certs\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:19:59.583120 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:59.583072    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d91987a5-bc23-4976-a1c7-c10448df0e0e-home\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:19:59.583120 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:19:59.583079    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d91987a5-bc23-4976-a1c7-c10448df0e0e-dshm\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:20:00.002529 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:00.002493    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-noten-361ccf/noten-model-361ccf-kserve-59f79fb88c-7cjgz" podUID="ae629c41-66d4-4754-b4f6-df23857b31de" containerName="main" containerID="cri-o://4aa47b9931d7869ceb614167154af8a8b86f0d15e96a66f1b1925d1b08c0a7bc" gracePeriod=45
Sep 10 22:20:00.247887 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:00.247865    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-361ccf/noten-model-361ccf-kserve-59f79fb88c-7cjgz"
Sep 10 22:20:00.290038 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:00.289968    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hdlz5\" (UniqueName: \"kubernetes.io/projected/ae629c41-66d4-4754-b4f6-df23857b31de-kube-api-access-hdlz5\") pod \"ae629c41-66d4-4754-b4f6-df23857b31de\" (UID: \"ae629c41-66d4-4754-b4f6-df23857b31de\") "
Sep 10 22:20:00.290038 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:00.290030    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae629c41-66d4-4754-b4f6-df23857b31de-home\") pod \"ae629c41-66d4-4754-b4f6-df23857b31de\" (UID: \"ae629c41-66d4-4754-b4f6-df23857b31de\") "
Sep 10 22:20:00.290246 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:00.290052    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae629c41-66d4-4754-b4f6-df23857b31de-tmp-dir\") pod \"ae629c41-66d4-4754-b4f6-df23857b31de\" (UID: \"ae629c41-66d4-4754-b4f6-df23857b31de\") "
Sep 10 22:20:00.290246 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:00.290077    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae629c41-66d4-4754-b4f6-df23857b31de-tls-certs\") pod \"ae629c41-66d4-4754-b4f6-df23857b31de\" (UID: \"ae629c41-66d4-4754-b4f6-df23857b31de\") "
Sep 10 22:20:00.290246 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:00.290101    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae629c41-66d4-4754-b4f6-df23857b31de-model-cache\") pod \"ae629c41-66d4-4754-b4f6-df23857b31de\" (UID: \"ae629c41-66d4-4754-b4f6-df23857b31de\") "
Sep 10 22:20:00.290246 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:00.290178    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae629c41-66d4-4754-b4f6-df23857b31de-dshm\") pod \"ae629c41-66d4-4754-b4f6-df23857b31de\" (UID: \"ae629c41-66d4-4754-b4f6-df23857b31de\") "
Sep 10 22:20:00.290457 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:00.290301    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae629c41-66d4-4754-b4f6-df23857b31de-home" (OuterVolumeSpecName: "home") pod "ae629c41-66d4-4754-b4f6-df23857b31de" (UID: "ae629c41-66d4-4754-b4f6-df23857b31de"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 22:20:00.290457 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:00.290356    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae629c41-66d4-4754-b4f6-df23857b31de-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ae629c41-66d4-4754-b4f6-df23857b31de" (UID: "ae629c41-66d4-4754-b4f6-df23857b31de"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 22:20:00.290457 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:00.290445    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae629c41-66d4-4754-b4f6-df23857b31de-model-cache" (OuterVolumeSpecName: "model-cache") pod "ae629c41-66d4-4754-b4f6-df23857b31de" (UID: "ae629c41-66d4-4754-b4f6-df23857b31de"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 22:20:00.290575 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:00.290469    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae629c41-66d4-4754-b4f6-df23857b31de-home\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:20:00.290575 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:00.290489    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae629c41-66d4-4754-b4f6-df23857b31de-tmp-dir\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:20:00.292380 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:00.292357    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae629c41-66d4-4754-b4f6-df23857b31de-dshm" (OuterVolumeSpecName: "dshm") pod "ae629c41-66d4-4754-b4f6-df23857b31de" (UID: "ae629c41-66d4-4754-b4f6-df23857b31de"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 22:20:00.292670 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:00.292649    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae629c41-66d4-4754-b4f6-df23857b31de-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ae629c41-66d4-4754-b4f6-df23857b31de" (UID: "ae629c41-66d4-4754-b4f6-df23857b31de"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 22:20:00.292670 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:00.292658    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae629c41-66d4-4754-b4f6-df23857b31de-kube-api-access-hdlz5" (OuterVolumeSpecName: "kube-api-access-hdlz5") pod "ae629c41-66d4-4754-b4f6-df23857b31de" (UID: "ae629c41-66d4-4754-b4f6-df23857b31de"). InnerVolumeSpecName "kube-api-access-hdlz5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 22:20:00.391029 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:00.390995    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae629c41-66d4-4754-b4f6-df23857b31de-dshm\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:20:00.391029 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:00.391024    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hdlz5\" (UniqueName: \"kubernetes.io/projected/ae629c41-66d4-4754-b4f6-df23857b31de-kube-api-access-hdlz5\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:20:00.391029 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:00.391035    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae629c41-66d4-4754-b4f6-df23857b31de-tls-certs\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:20:00.391262 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:00.391045    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae629c41-66d4-4754-b4f6-df23857b31de-model-cache\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:20:00.555353 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:00.555256    2582 generic.go:356] "Generic (PLEG): container finished" podID="ae629c41-66d4-4754-b4f6-df23857b31de" containerID="4aa47b9931d7869ceb614167154af8a8b86f0d15e96a66f1b1925d1b08c0a7bc" exitCode=0
Sep 10 22:20:00.555353 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:00.555297    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-361ccf/noten-model-361ccf-kserve-59f79fb88c-7cjgz"
Sep 10 22:20:00.555353 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:00.555344    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-361ccf/noten-model-361ccf-kserve-59f79fb88c-7cjgz" event={"ID":"ae629c41-66d4-4754-b4f6-df23857b31de","Type":"ContainerDied","Data":"4aa47b9931d7869ceb614167154af8a8b86f0d15e96a66f1b1925d1b08c0a7bc"}
Sep 10 22:20:00.555599 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:00.555381    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-361ccf/noten-model-361ccf-kserve-59f79fb88c-7cjgz" event={"ID":"ae629c41-66d4-4754-b4f6-df23857b31de","Type":"ContainerDied","Data":"d8af910e2ed91089f51fc722400d9569d661a56ea1726cb75e7d62926a58a3a8"}
Sep 10 22:20:00.555599 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:00.555401    2582 scope.go:117] "RemoveContainer" containerID="4aa47b9931d7869ceb614167154af8a8b86f0d15e96a66f1b1925d1b08c0a7bc"
Sep 10 22:20:00.564065 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:00.564048    2582 scope.go:117] "RemoveContainer" containerID="4aa47b9931d7869ceb614167154af8a8b86f0d15e96a66f1b1925d1b08c0a7bc"
Sep 10 22:20:00.564322 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:20:00.564307    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4aa47b9931d7869ceb614167154af8a8b86f0d15e96a66f1b1925d1b08c0a7bc\": container with ID starting with 4aa47b9931d7869ceb614167154af8a8b86f0d15e96a66f1b1925d1b08c0a7bc not found: ID does not exist" containerID="4aa47b9931d7869ceb614167154af8a8b86f0d15e96a66f1b1925d1b08c0a7bc"
Sep 10 22:20:00.564368 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:00.564331    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4aa47b9931d7869ceb614167154af8a8b86f0d15e96a66f1b1925d1b08c0a7bc"} err="failed to get container status \"4aa47b9931d7869ceb614167154af8a8b86f0d15e96a66f1b1925d1b08c0a7bc\": rpc error: code = NotFound desc = could not find container \"4aa47b9931d7869ceb614167154af8a8b86f0d15e96a66f1b1925d1b08c0a7bc\": container with ID starting with 4aa47b9931d7869ceb614167154af8a8b86f0d15e96a66f1b1925d1b08c0a7bc not found: ID does not exist"
Sep 10 22:20:00.584350 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:00.584322    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-361ccf/noten-model-361ccf-kserve-59f79fb88c-7cjgz"]
Sep 10 22:20:00.588279 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:00.588255    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-361ccf/noten-model-361ccf-kserve-59f79fb88c-7cjgz"]
Sep 10 22:20:01.011768 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:01.011733    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-1c252b/test-model-1c252b-kserve-9b5ffcc55-j8qpx" podUID="ba783dae-1685-4cb9-b045-dc838e0a596c" containerName="main" containerID="cri-o://c22dcab9d429f9da9d313f07a06eac95b830113297986312a3f7c3791376ce23" gracePeriod=45
Sep 10 22:20:01.255652 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:01.255624    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-1c252b/test-model-1c252b-kserve-9b5ffcc55-j8qpx"
Sep 10 22:20:01.299488 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:01.299413    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba783dae-1685-4cb9-b045-dc838e0a596c-tmp-dir\") pod \"ba783dae-1685-4cb9-b045-dc838e0a596c\" (UID: \"ba783dae-1685-4cb9-b045-dc838e0a596c\") "
Sep 10 22:20:01.299488 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:01.299459    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba783dae-1685-4cb9-b045-dc838e0a596c-home\") pod \"ba783dae-1685-4cb9-b045-dc838e0a596c\" (UID: \"ba783dae-1685-4cb9-b045-dc838e0a596c\") "
Sep 10 22:20:01.299663 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:01.299548    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba783dae-1685-4cb9-b045-dc838e0a596c-dshm\") pod \"ba783dae-1685-4cb9-b045-dc838e0a596c\" (UID: \"ba783dae-1685-4cb9-b045-dc838e0a596c\") "
Sep 10 22:20:01.299663 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:01.299586    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba783dae-1685-4cb9-b045-dc838e0a596c-model-cache\") pod \"ba783dae-1685-4cb9-b045-dc838e0a596c\" (UID: \"ba783dae-1685-4cb9-b045-dc838e0a596c\") "
Sep 10 22:20:01.299663 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:01.299623    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pwm6v\" (UniqueName: \"kubernetes.io/projected/ba783dae-1685-4cb9-b045-dc838e0a596c-kube-api-access-pwm6v\") pod \"ba783dae-1685-4cb9-b045-dc838e0a596c\" (UID: \"ba783dae-1685-4cb9-b045-dc838e0a596c\") "
Sep 10 22:20:01.299663 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:01.299655    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba783dae-1685-4cb9-b045-dc838e0a596c-tls-certs\") pod \"ba783dae-1685-4cb9-b045-dc838e0a596c\" (UID: \"ba783dae-1685-4cb9-b045-dc838e0a596c\") "
Sep 10 22:20:01.299897 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:01.299707    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba783dae-1685-4cb9-b045-dc838e0a596c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ba783dae-1685-4cb9-b045-dc838e0a596c" (UID: "ba783dae-1685-4cb9-b045-dc838e0a596c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 22:20:01.299897 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:01.299749    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba783dae-1685-4cb9-b045-dc838e0a596c-home" (OuterVolumeSpecName: "home") pod "ba783dae-1685-4cb9-b045-dc838e0a596c" (UID: "ba783dae-1685-4cb9-b045-dc838e0a596c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 22:20:01.299897 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:01.299841    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba783dae-1685-4cb9-b045-dc838e0a596c-model-cache" (OuterVolumeSpecName: "model-cache") pod "ba783dae-1685-4cb9-b045-dc838e0a596c" (UID: "ba783dae-1685-4cb9-b045-dc838e0a596c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 22:20:01.300043 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:01.299958    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba783dae-1685-4cb9-b045-dc838e0a596c-model-cache\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:20:01.300043 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:01.299976    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba783dae-1685-4cb9-b045-dc838e0a596c-tmp-dir\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:20:01.300043 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:01.299987    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba783dae-1685-4cb9-b045-dc838e0a596c-home\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:20:01.301818 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:01.301792    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba783dae-1685-4cb9-b045-dc838e0a596c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ba783dae-1685-4cb9-b045-dc838e0a596c" (UID: "ba783dae-1685-4cb9-b045-dc838e0a596c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 22:20:01.301940 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:01.301843    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba783dae-1685-4cb9-b045-dc838e0a596c-dshm" (OuterVolumeSpecName: "dshm") pod "ba783dae-1685-4cb9-b045-dc838e0a596c" (UID: "ba783dae-1685-4cb9-b045-dc838e0a596c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 22:20:01.301940 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:01.301918    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba783dae-1685-4cb9-b045-dc838e0a596c-kube-api-access-pwm6v" (OuterVolumeSpecName: "kube-api-access-pwm6v") pod "ba783dae-1685-4cb9-b045-dc838e0a596c" (UID: "ba783dae-1685-4cb9-b045-dc838e0a596c"). InnerVolumeSpecName "kube-api-access-pwm6v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 22:20:01.400533 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:01.400499    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba783dae-1685-4cb9-b045-dc838e0a596c-dshm\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:20:01.400533 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:01.400527    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwm6v\" (UniqueName: \"kubernetes.io/projected/ba783dae-1685-4cb9-b045-dc838e0a596c-kube-api-access-pwm6v\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:20:01.400533 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:01.400538    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba783dae-1685-4cb9-b045-dc838e0a596c-tls-certs\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:20:01.498409 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:01.498365    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ae629c41-66d4-4754-b4f6-df23857b31de" path="/var/lib/kubelet/pods/ae629c41-66d4-4754-b4f6-df23857b31de/volumes"
Sep 10 22:20:01.498873 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:01.498827    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d91987a5-bc23-4976-a1c7-c10448df0e0e" path="/var/lib/kubelet/pods/d91987a5-bc23-4976-a1c7-c10448df0e0e/volumes"
Sep 10 22:20:01.561262 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:01.561199    2582 generic.go:356] "Generic (PLEG): container finished" podID="ba783dae-1685-4cb9-b045-dc838e0a596c" containerID="c22dcab9d429f9da9d313f07a06eac95b830113297986312a3f7c3791376ce23" exitCode=0
Sep 10 22:20:01.561262 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:01.561242    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-1c252b/test-model-1c252b-kserve-9b5ffcc55-j8qpx"
Sep 10 22:20:01.561427 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:01.561287    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-1c252b/test-model-1c252b-kserve-9b5ffcc55-j8qpx" event={"ID":"ba783dae-1685-4cb9-b045-dc838e0a596c","Type":"ContainerDied","Data":"c22dcab9d429f9da9d313f07a06eac95b830113297986312a3f7c3791376ce23"}
Sep 10 22:20:01.561427 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:01.561331    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-1c252b/test-model-1c252b-kserve-9b5ffcc55-j8qpx" event={"ID":"ba783dae-1685-4cb9-b045-dc838e0a596c","Type":"ContainerDied","Data":"9f378010871bc2285a1bc76370b3dc80c044c6d2b75f6063487d18765125d4c6"}
Sep 10 22:20:01.561427 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:01.561355    2582 scope.go:117] "RemoveContainer" containerID="c22dcab9d429f9da9d313f07a06eac95b830113297986312a3f7c3791376ce23"
Sep 10 22:20:01.569913 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:01.569897    2582 scope.go:117] "RemoveContainer" containerID="c22dcab9d429f9da9d313f07a06eac95b830113297986312a3f7c3791376ce23"
Sep 10 22:20:01.570157 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:20:01.570138    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c22dcab9d429f9da9d313f07a06eac95b830113297986312a3f7c3791376ce23\": container with ID starting with c22dcab9d429f9da9d313f07a06eac95b830113297986312a3f7c3791376ce23 not found: ID does not exist" containerID="c22dcab9d429f9da9d313f07a06eac95b830113297986312a3f7c3791376ce23"
Sep 10 22:20:01.570219 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:01.570166    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c22dcab9d429f9da9d313f07a06eac95b830113297986312a3f7c3791376ce23"} err="failed to get container status \"c22dcab9d429f9da9d313f07a06eac95b830113297986312a3f7c3791376ce23\": rpc error: code = NotFound desc = could not find container \"c22dcab9d429f9da9d313f07a06eac95b830113297986312a3f7c3791376ce23\": container with ID starting with c22dcab9d429f9da9d313f07a06eac95b830113297986312a3f7c3791376ce23 not found: ID does not exist"
Sep 10 22:20:01.576094 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:01.576074    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-1c252b/test-model-1c252b-kserve-9b5ffcc55-j8qpx"]
Sep 10 22:20:01.579949 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:01.579927    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-1c252b/test-model-1c252b-kserve-9b5ffcc55-j8qpx"]
Sep 10 22:20:03.500192 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:03.500158    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ba783dae-1685-4cb9-b045-dc838e0a596c" path="/var/lib/kubelet/pods/ba783dae-1685-4cb9-b045-dc838e0a596c/volumes"
Sep 10 22:20:15.196872 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:15.196810    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-1c252b-76ffcd984c-vqmgv"]
Sep 10 22:20:15.197386 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:15.197080    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1c252b-76ffcd984c-vqmgv" podUID="cf38af6c-be32-4502-bcb7-05c47ffdce1f" containerName="maas-api" containerID="cri-o://478c18d79a85f52f1dde3dfaea199dee00494db34cb1d5aa0713f08e2042fa3d" gracePeriod=30
Sep 10 22:20:15.437556 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:15.437532    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1c252b-76ffcd984c-vqmgv"
Sep 10 22:20:15.521491 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:15.521418    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf38af6c-be32-4502-bcb7-05c47ffdce1f-openshift-service-ca\") pod \"cf38af6c-be32-4502-bcb7-05c47ffdce1f\" (UID: \"cf38af6c-be32-4502-bcb7-05c47ffdce1f\") "
Sep 10 22:20:15.521491 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:15.521451    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf38af6c-be32-4502-bcb7-05c47ffdce1f-metrics-certs\") pod \"cf38af6c-be32-4502-bcb7-05c47ffdce1f\" (UID: \"cf38af6c-be32-4502-bcb7-05c47ffdce1f\") "
Sep 10 22:20:15.521731 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:15.521505    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf38af6c-be32-4502-bcb7-05c47ffdce1f-maas-api-tls\") pod \"cf38af6c-be32-4502-bcb7-05c47ffdce1f\" (UID: \"cf38af6c-be32-4502-bcb7-05c47ffdce1f\") "
Sep 10 22:20:15.521731 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:15.521616    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9qz6n\" (UniqueName: \"kubernetes.io/projected/cf38af6c-be32-4502-bcb7-05c47ffdce1f-kube-api-access-9qz6n\") pod \"cf38af6c-be32-4502-bcb7-05c47ffdce1f\" (UID: \"cf38af6c-be32-4502-bcb7-05c47ffdce1f\") "
Sep 10 22:20:15.521833 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:15.521758    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf38af6c-be32-4502-bcb7-05c47ffdce1f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "cf38af6c-be32-4502-bcb7-05c47ffdce1f" (UID: "cf38af6c-be32-4502-bcb7-05c47ffdce1f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 22:20:15.521983 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:15.521954    2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf38af6c-be32-4502-bcb7-05c47ffdce1f-openshift-service-ca\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:20:15.523647 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:15.523617    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf38af6c-be32-4502-bcb7-05c47ffdce1f-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "cf38af6c-be32-4502-bcb7-05c47ffdce1f" (UID: "cf38af6c-be32-4502-bcb7-05c47ffdce1f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 22:20:15.523737 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:15.523662    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf38af6c-be32-4502-bcb7-05c47ffdce1f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "cf38af6c-be32-4502-bcb7-05c47ffdce1f" (UID: "cf38af6c-be32-4502-bcb7-05c47ffdce1f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 22:20:15.523737 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:15.523666    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf38af6c-be32-4502-bcb7-05c47ffdce1f-kube-api-access-9qz6n" (OuterVolumeSpecName: "kube-api-access-9qz6n") pod "cf38af6c-be32-4502-bcb7-05c47ffdce1f" (UID: "cf38af6c-be32-4502-bcb7-05c47ffdce1f"). InnerVolumeSpecName "kube-api-access-9qz6n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 22:20:15.622368 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:15.622339    2582 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf38af6c-be32-4502-bcb7-05c47ffdce1f-metrics-certs\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:20:15.622368 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:15.622364    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf38af6c-be32-4502-bcb7-05c47ffdce1f-maas-api-tls\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:20:15.622368 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:15.622374    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9qz6n\" (UniqueName: \"kubernetes.io/projected/cf38af6c-be32-4502-bcb7-05c47ffdce1f-kube-api-access-9qz6n\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:20:15.633388 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:15.633354    2582 generic.go:356] "Generic (PLEG): container finished" podID="cf38af6c-be32-4502-bcb7-05c47ffdce1f" containerID="478c18d79a85f52f1dde3dfaea199dee00494db34cb1d5aa0713f08e2042fa3d" exitCode=0
Sep 10 22:20:15.633549 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:15.633406    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1c252b-76ffcd984c-vqmgv"
Sep 10 22:20:15.633549 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:15.633441    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1c252b-76ffcd984c-vqmgv" event={"ID":"cf38af6c-be32-4502-bcb7-05c47ffdce1f","Type":"ContainerDied","Data":"478c18d79a85f52f1dde3dfaea199dee00494db34cb1d5aa0713f08e2042fa3d"}
Sep 10 22:20:15.633549 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:15.633509    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1c252b-76ffcd984c-vqmgv" event={"ID":"cf38af6c-be32-4502-bcb7-05c47ffdce1f","Type":"ContainerDied","Data":"09b643abf56796f8f3ea2e2953faa3b6a2c86af6d49626422741290cd33f8f8c"}
Sep 10 22:20:15.633702 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:15.633545    2582 scope.go:117] "RemoveContainer" containerID="478c18d79a85f52f1dde3dfaea199dee00494db34cb1d5aa0713f08e2042fa3d"
Sep 10 22:20:15.642590 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:15.642570    2582 scope.go:117] "RemoveContainer" containerID="478c18d79a85f52f1dde3dfaea199dee00494db34cb1d5aa0713f08e2042fa3d"
Sep 10 22:20:15.642840 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:20:15.642820    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"478c18d79a85f52f1dde3dfaea199dee00494db34cb1d5aa0713f08e2042fa3d\": container with ID starting with 478c18d79a85f52f1dde3dfaea199dee00494db34cb1d5aa0713f08e2042fa3d not found: ID does not exist" containerID="478c18d79a85f52f1dde3dfaea199dee00494db34cb1d5aa0713f08e2042fa3d"
Sep 10 22:20:15.642906 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:15.642860    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"478c18d79a85f52f1dde3dfaea199dee00494db34cb1d5aa0713f08e2042fa3d"} err="failed to get container status \"478c18d79a85f52f1dde3dfaea199dee00494db34cb1d5aa0713f08e2042fa3d\": rpc error: code = NotFound desc = could not find container \"478c18d79a85f52f1dde3dfaea199dee00494db34cb1d5aa0713f08e2042fa3d\": container with ID starting with 478c18d79a85f52f1dde3dfaea199dee00494db34cb1d5aa0713f08e2042fa3d not found: ID does not exist"
Sep 10 22:20:15.655190 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:15.655169    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-1c252b-76ffcd984c-vqmgv"]
Sep 10 22:20:15.660338 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:15.660306    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-1c252b-76ffcd984c-vqmgv"]
Sep 10 22:20:17.498209 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:17.498171    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cf38af6c-be32-4502-bcb7-05c47ffdce1f" path="/var/lib/kubelet/pods/cf38af6c-be32-4502-bcb7-05c47ffdce1f/volumes"
Sep 10 22:20:18.362677 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:18.362643    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-a9b6f6-openshift-default-558c5f6ff4-dkx44"]
Sep 10 22:20:18.362985 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:18.362972    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d97d86ed-a521-4792-a760-0d95aa5ee6e7" containerName="maas-api"
Sep 10 22:20:18.363045 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:18.362986    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="d97d86ed-a521-4792-a760-0d95aa5ee6e7" containerName="maas-api"
Sep 10 22:20:18.363045 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:18.362996    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae629c41-66d4-4754-b4f6-df23857b31de" containerName="main"
Sep 10 22:20:18.363045 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:18.363002    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae629c41-66d4-4754-b4f6-df23857b31de" containerName="main"
Sep 10 22:20:18.363045 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:18.363009    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf38af6c-be32-4502-bcb7-05c47ffdce1f" containerName="maas-api"
Sep 10 22:20:18.363045 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:18.363014    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf38af6c-be32-4502-bcb7-05c47ffdce1f" containerName="maas-api"
Sep 10 22:20:18.363045 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:18.363028    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba783dae-1685-4cb9-b045-dc838e0a596c" containerName="main"
Sep 10 22:20:18.363045 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:18.363033    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba783dae-1685-4cb9-b045-dc838e0a596c" containerName="main"
Sep 10 22:20:18.363045 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:18.363045    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d91987a5-bc23-4976-a1c7-c10448df0e0e" containerName="main"
Sep 10 22:20:18.363301 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:18.363050    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="d91987a5-bc23-4976-a1c7-c10448df0e0e" containerName="main"
Sep 10 22:20:18.363301 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:18.363056    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23fb1677-9564-480a-a2bd-2aec7f9b81bc" containerName="istio-proxy"
Sep 10 22:20:18.363301 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:18.363061    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="23fb1677-9564-480a-a2bd-2aec7f9b81bc" containerName="istio-proxy"
Sep 10 22:20:18.363301 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:18.363069    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be6caaba-5c60-4d62-b0db-512db79382a8" containerName="payload-pre-processing"
Sep 10 22:20:18.363301 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:18.363074    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="be6caaba-5c60-4d62-b0db-512db79382a8" containerName="payload-pre-processing"
Sep 10 22:20:18.363301 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:18.363124    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="23fb1677-9564-480a-a2bd-2aec7f9b81bc" containerName="istio-proxy"
Sep 10 22:20:18.363301 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:18.363130    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="d97d86ed-a521-4792-a760-0d95aa5ee6e7" containerName="maas-api"
Sep 10 22:20:18.363301 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:18.363136    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba783dae-1685-4cb9-b045-dc838e0a596c" containerName="main"
Sep 10 22:20:18.363301 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:18.363141    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="be6caaba-5c60-4d62-b0db-512db79382a8" containerName="payload-pre-processing"
Sep 10 22:20:18.363301 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:18.363147    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="cf38af6c-be32-4502-bcb7-05c47ffdce1f" containerName="maas-api"
Sep 10 22:20:18.363301 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:18.363152    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae629c41-66d4-4754-b4f6-df23857b31de" containerName="main"
Sep 10 22:20:18.363301 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:18.363161    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="d91987a5-bc23-4976-a1c7-c10448df0e0e" containerName="main"
Sep 10 22:20:18.366141 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:18.366126    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-a9b6f6-openshift-default-558c5f6ff4-dkx44"
Sep 10 22:20:18.371185 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:18.371164    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-a9b6f6-openshift-default-dockercfg-8lzm2\""
Sep 10 22:20:18.388224 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:18.388201    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-a9b6f6-openshift-default-558c5f6ff4-dkx44"]
Sep 10 22:20:18.445705 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:18.445677    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rsmll\" (UniqueName: \"kubernetes.io/projected/8b0dd305-1f45-45dc-8e31-01c6c225c275-kube-api-access-rsmll\") pod \"e2e-shared-a-w2-a9b6f6-openshift-default-558c5f6ff4-dkx44\" (UID: \"8b0dd305-1f45-45dc-8e31-01c6c225c275\") " pod="openshift-ingress/e2e-shared-a-w2-a9b6f6-openshift-default-558c5f6ff4-dkx44"
Sep 10 22:20:18.445840 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:18.445711    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b0dd305-1f45-45dc-8e31-01c6c225c275-istiod-ca-cert\") pod \"e2e-shared-a-w2-a9b6f6-openshift-default-558c5f6ff4-dkx44\" (UID: \"8b0dd305-1f45-45dc-8e31-01c6c225c275\") " pod="openshift-ingress/e2e-shared-a-w2-a9b6f6-openshift-default-558c5f6ff4-dkx44"
Sep 10 22:20:18.445840 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:18.445736    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b0dd305-1f45-45dc-8e31-01c6c225c275-credential-socket\") pod \"e2e-shared-a-w2-a9b6f6-openshift-default-558c5f6ff4-dkx44\" (UID: \"8b0dd305-1f45-45dc-8e31-01c6c225c275\") " pod="openshift-ingress/e2e-shared-a-w2-a9b6f6-openshift-default-558c5f6ff4-dkx44"
Sep 10 22:20:18.445840 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:18.445790    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b0dd305-1f45-45dc-8e31-01c6c225c275-istio-data\") pod \"e2e-shared-a-w2-a9b6f6-openshift-default-558c5f6ff4-dkx44\" (UID: \"8b0dd305-1f45-45dc-8e31-01c6c225c275\") " pod="openshift-ingress/e2e-shared-a-w2-a9b6f6-openshift-default-558c5f6ff4-dkx44"
Sep 10 22:20:18.445840 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:18.445811    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b0dd305-1f45-45dc-8e31-01c6c225c275-workload-certs\") pod \"e2e-shared-a-w2-a9b6f6-openshift-default-558c5f6ff4-dkx44\" (UID: \"8b0dd305-1f45-45dc-8e31-01c6c225c275\") " pod="openshift-ingress/e2e-shared-a-w2-a9b6f6-openshift-default-558c5f6ff4-dkx44"
Sep 10 22:20:18.445840 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:18.445829    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b0dd305-1f45-45dc-8e31-01c6c225c275-istio-podinfo\") pod \"e2e-shared-a-w2-a9b6f6-openshift-default-558c5f6ff4-dkx44\" (UID: \"8b0dd305-1f45-45dc-8e31-01c6c225c275\") " pod="openshift-ingress/e2e-shared-a-w2-a9b6f6-openshift-default-558c5f6ff4-dkx44"
Sep 10 22:20:18.446040 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:18.445922    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b0dd305-1f45-45dc-8e31-01c6c225c275-workload-socket\") pod \"e2e-shared-a-w2-a9b6f6-openshift-default-558c5f6ff4-dkx44\" (UID: \"8b0dd305-1f45-45dc-8e31-01c6c225c275\") " pod="openshift-ingress/e2e-shared-a-w2-a9b6f6-openshift-default-558c5f6ff4-dkx44"
Sep 10 22:20:18.446040 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:18.445963    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b0dd305-1f45-45dc-8e31-01c6c225c275-istio-token\") pod \"e2e-shared-a-w2-a9b6f6-openshift-default-558c5f6ff4-dkx44\" (UID: \"8b0dd305-1f45-45dc-8e31-01c6c225c275\") " pod="openshift-ingress/e2e-shared-a-w2-a9b6f6-openshift-default-558c5f6ff4-dkx44"
Sep 10 22:20:18.446040 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:18.446018    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b0dd305-1f45-45dc-8e31-01c6c225c275-istio-envoy\") pod \"e2e-shared-a-w2-a9b6f6-openshift-default-558c5f6ff4-dkx44\" (UID: \"8b0dd305-1f45-45dc-8e31-01c6c225c275\") " pod="openshift-ingress/e2e-shared-a-w2-a9b6f6-openshift-default-558c5f6ff4-dkx44"
Sep 10 22:20:18.547162 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:18.547129    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsmll\" (UniqueName: \"kubernetes.io/projected/8b0dd305-1f45-45dc-8e31-01c6c225c275-kube-api-access-rsmll\") pod \"e2e-shared-a-w2-a9b6f6-openshift-default-558c5f6ff4-dkx44\" (UID: \"8b0dd305-1f45-45dc-8e31-01c6c225c275\") " pod="openshift-ingress/e2e-shared-a-w2-a9b6f6-openshift-default-558c5f6ff4-dkx44"
Sep 10 22:20:18.547162 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:18.547166    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b0dd305-1f45-45dc-8e31-01c6c225c275-istiod-ca-cert\") pod \"e2e-shared-a-w2-a9b6f6-openshift-default-558c5f6ff4-dkx44\" (UID: \"8b0dd305-1f45-45dc-8e31-01c6c225c275\") " pod="openshift-ingress/e2e-shared-a-w2-a9b6f6-openshift-default-558c5f6ff4-dkx44"
Sep 10 22:20:18.547674 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:18.547204    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b0dd305-1f45-45dc-8e31-01c6c225c275-credential-socket\") pod \"e2e-shared-a-w2-a9b6f6-openshift-default-558c5f6ff4-dkx44\" (UID: \"8b0dd305-1f45-45dc-8e31-01c6c225c275\") " pod="openshift-ingress/e2e-shared-a-w2-a9b6f6-openshift-default-558c5f6ff4-dkx44"
Sep 10 22:20:18.547674 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:18.547230    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b0dd305-1f45-45dc-8e31-01c6c225c275-istio-data\") pod \"e2e-shared-a-w2-a9b6f6-openshift-default-558c5f6ff4-dkx44\" (UID: \"8b0dd305-1f45-45dc-8e31-01c6c225c275\") " pod="openshift-ingress/e2e-shared-a-w2-a9b6f6-openshift-default-558c5f6ff4-dkx44"
Sep 10 22:20:18.547674 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:18.547250    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b0dd305-1f45-45dc-8e31-01c6c225c275-workload-certs\") pod \"e2e-shared-a-w2-a9b6f6-openshift-default-558c5f6ff4-dkx44\" (UID: \"8b0dd305-1f45-45dc-8e31-01c6c225c275\") " pod="openshift-ingress/e2e-shared-a-w2-a9b6f6-openshift-default-558c5f6ff4-dkx44"
Sep 10 22:20:18.547674 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:18.547264    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b0dd305-1f45-45dc-8e31-01c6c225c275-istio-podinfo\") pod \"e2e-shared-a-w2-a9b6f6-openshift-default-558c5f6ff4-dkx44\" (UID: \"8b0dd305-1f45-45dc-8e31-01c6c225c275\") " pod="openshift-ingress/e2e-shared-a-w2-a9b6f6-openshift-default-558c5f6ff4-dkx44"
Sep 10 22:20:18.547674 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:18.547281    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b0dd305-1f45-45dc-8e31-01c6c225c275-workload-socket\") pod \"e2e-shared-a-w2-a9b6f6-openshift-default-558c5f6ff4-dkx44\" (UID: \"8b0dd305-1f45-45dc-8e31-01c6c225c275\") " pod="openshift-ingress/e2e-shared-a-w2-a9b6f6-openshift-default-558c5f6ff4-dkx44"
Sep 10 22:20:18.547674 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:18.547320    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b0dd305-1f45-45dc-8e31-01c6c225c275-istio-token\") pod \"e2e-shared-a-w2-a9b6f6-openshift-default-558c5f6ff4-dkx44\" (UID: \"8b0dd305-1f45-45dc-8e31-01c6c225c275\") " pod="openshift-ingress/e2e-shared-a-w2-a9b6f6-openshift-default-558c5f6ff4-dkx44"
Sep 10 22:20:18.547674 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:18.547378    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b0dd305-1f45-45dc-8e31-01c6c225c275-istio-envoy\") pod \"e2e-shared-a-w2-a9b6f6-openshift-default-558c5f6ff4-dkx44\" (UID: \"8b0dd305-1f45-45dc-8e31-01c6c225c275\") " pod="openshift-ingress/e2e-shared-a-w2-a9b6f6-openshift-default-558c5f6ff4-dkx44"
Sep 10 22:20:18.548060 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:18.547689    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b0dd305-1f45-45dc-8e31-01c6c225c275-workload-certs\") pod \"e2e-shared-a-w2-a9b6f6-openshift-default-558c5f6ff4-dkx44\" (UID: \"8b0dd305-1f45-45dc-8e31-01c6c225c275\") " pod="openshift-ingress/e2e-shared-a-w2-a9b6f6-openshift-default-558c5f6ff4-dkx44"
Sep 10 22:20:18.548060 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:18.547714    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b0dd305-1f45-45dc-8e31-01c6c225c275-istio-data\") pod \"e2e-shared-a-w2-a9b6f6-openshift-default-558c5f6ff4-dkx44\" (UID: \"8b0dd305-1f45-45dc-8e31-01c6c225c275\") " pod="openshift-ingress/e2e-shared-a-w2-a9b6f6-openshift-default-558c5f6ff4-dkx44"
Sep 10 22:20:18.548060 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:18.547897    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b0dd305-1f45-45dc-8e31-01c6c225c275-workload-socket\") pod \"e2e-shared-a-w2-a9b6f6-openshift-default-558c5f6ff4-dkx44\" (UID: \"8b0dd305-1f45-45dc-8e31-01c6c225c275\") " pod="openshift-ingress/e2e-shared-a-w2-a9b6f6-openshift-default-558c5f6ff4-dkx44"
Sep 10 22:20:18.548060 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:18.547988    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b0dd305-1f45-45dc-8e31-01c6c225c275-credential-socket\") pod \"e2e-shared-a-w2-a9b6f6-openshift-default-558c5f6ff4-dkx44\" (UID: \"8b0dd305-1f45-45dc-8e31-01c6c225c275\") " pod="openshift-ingress/e2e-shared-a-w2-a9b6f6-openshift-default-558c5f6ff4-dkx44"
Sep 10 22:20:18.548203 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:18.548060    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b0dd305-1f45-45dc-8e31-01c6c225c275-istiod-ca-cert\") pod \"e2e-shared-a-w2-a9b6f6-openshift-default-558c5f6ff4-dkx44\" (UID: \"8b0dd305-1f45-45dc-8e31-01c6c225c275\") " pod="openshift-ingress/e2e-shared-a-w2-a9b6f6-openshift-default-558c5f6ff4-dkx44"
Sep 10 22:20:18.549650 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:18.549631    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b0dd305-1f45-45dc-8e31-01c6c225c275-istio-envoy\") pod \"e2e-shared-a-w2-a9b6f6-openshift-default-558c5f6ff4-dkx44\" (UID: \"8b0dd305-1f45-45dc-8e31-01c6c225c275\") " pod="openshift-ingress/e2e-shared-a-w2-a9b6f6-openshift-default-558c5f6ff4-dkx44"
Sep 10 22:20:18.549794 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:18.549777    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b0dd305-1f45-45dc-8e31-01c6c225c275-istio-podinfo\") pod \"e2e-shared-a-w2-a9b6f6-openshift-default-558c5f6ff4-dkx44\" (UID: \"8b0dd305-1f45-45dc-8e31-01c6c225c275\") " pod="openshift-ingress/e2e-shared-a-w2-a9b6f6-openshift-default-558c5f6ff4-dkx44"
Sep 10 22:20:18.559601 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:18.559579    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b0dd305-1f45-45dc-8e31-01c6c225c275-istio-token\") pod \"e2e-shared-a-w2-a9b6f6-openshift-default-558c5f6ff4-dkx44\" (UID: \"8b0dd305-1f45-45dc-8e31-01c6c225c275\") " pod="openshift-ingress/e2e-shared-a-w2-a9b6f6-openshift-default-558c5f6ff4-dkx44"
Sep 10 22:20:18.559712 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:18.559622    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rsmll\" (UniqueName: \"kubernetes.io/projected/8b0dd305-1f45-45dc-8e31-01c6c225c275-kube-api-access-rsmll\") pod \"e2e-shared-a-w2-a9b6f6-openshift-default-558c5f6ff4-dkx44\" (UID: \"8b0dd305-1f45-45dc-8e31-01c6c225c275\") " pod="openshift-ingress/e2e-shared-a-w2-a9b6f6-openshift-default-558c5f6ff4-dkx44"
Sep 10 22:20:18.675752 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:18.675727    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-a9b6f6-openshift-default-558c5f6ff4-dkx44"
Sep 10 22:20:18.973882 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:18.973783    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-a9b6f6-openshift-default-558c5f6ff4-dkx44"]
Sep 10 22:20:18.976723 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:20:18.976696    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b0dd305_1f45_45dc_8e31_01c6c225c275.slice/crio-47ffca5af9f8c8287763b615182671d4371aaba794ab3161c6a9703b6081fb59 WatchSource:0}: Error finding container 47ffca5af9f8c8287763b615182671d4371aaba794ab3161c6a9703b6081fb59: Status 404 returned error can't find the container with id 47ffca5af9f8c8287763b615182671d4371aaba794ab3161c6a9703b6081fb59
Sep 10 22:20:18.978907 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:18.978875    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 22:20:18.978964 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:18.978944    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 22:20:18.979009 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:18.978991    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 22:20:19.653115 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:19.653086    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-a9b6f6-openshift-default-558c5f6ff4-dkx44" event={"ID":"8b0dd305-1f45-45dc-8e31-01c6c225c275","Type":"ContainerStarted","Data":"f71afe4c9e4c8e7ddce7c1f212014436da4a8bb3bf1415294fa4272e51438ed0"}
Sep 10 22:20:19.653491 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:19.653140    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-a9b6f6-openshift-default-558c5f6ff4-dkx44" event={"ID":"8b0dd305-1f45-45dc-8e31-01c6c225c275","Type":"ContainerStarted","Data":"47ffca5af9f8c8287763b615182671d4371aaba794ab3161c6a9703b6081fb59"}
Sep 10 22:20:19.676053 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:19.676004    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-a9b6f6-openshift-default-558c5f6ff4-dkx44" podStartSLOduration=1.675988075 podStartE2EDuration="1.675988075s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 22:20:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:20:19.673966927 +0000 UTC m=+1140.630034633" watchObservedRunningTime="2026-09-10 22:20:19.675988075 +0000 UTC m=+1140.632055780"
Sep 10 22:20:19.676678 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:19.676660    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-a9b6f6-openshift-default-558c5f6ff4-dkx44"
Sep 10 22:20:20.681493 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:20.681456    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-a9b6f6-openshift-default-558c5f6ff4-dkx44"
Sep 10 22:20:21.662654 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:21.662622    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-a9b6f6-openshift-default-558c5f6ff4-dkx44"
Sep 10 22:20:21.663532 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:21.663513    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-a9b6f6-openshift-default-558c5f6ff4-dkx44"
Sep 10 22:20:23.450390 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:23.450350    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-d07dfb-openshift-default-6b77565746-fcc6m"]
Sep 10 22:20:23.455861 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:23.455824    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-d07dfb-openshift-default-6b77565746-fcc6m"
Sep 10 22:20:23.458927 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:23.458904    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-d07dfb-openshift-default-dockercfg-cb22w\""
Sep 10 22:20:23.480492 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:23.480468    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-d07dfb-openshift-default-6b77565746-fcc6m"]
Sep 10 22:20:23.595674 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:23.595637    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de372118-61dc-45ee-b7db-ae68c945f4bc-istio-envoy\") pod \"e2e-shared-a-w3-d07dfb-openshift-default-6b77565746-fcc6m\" (UID: \"de372118-61dc-45ee-b7db-ae68c945f4bc\") " pod="openshift-ingress/e2e-shared-a-w3-d07dfb-openshift-default-6b77565746-fcc6m"
Sep 10 22:20:23.595674 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:23.595679    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de372118-61dc-45ee-b7db-ae68c945f4bc-istio-data\") pod \"e2e-shared-a-w3-d07dfb-openshift-default-6b77565746-fcc6m\" (UID: \"de372118-61dc-45ee-b7db-ae68c945f4bc\") " pod="openshift-ingress/e2e-shared-a-w3-d07dfb-openshift-default-6b77565746-fcc6m"
Sep 10 22:20:23.595988 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:23.595757    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de372118-61dc-45ee-b7db-ae68c945f4bc-workload-certs\") pod \"e2e-shared-a-w3-d07dfb-openshift-default-6b77565746-fcc6m\" (UID: \"de372118-61dc-45ee-b7db-ae68c945f4bc\") " pod="openshift-ingress/e2e-shared-a-w3-d07dfb-openshift-default-6b77565746-fcc6m"
Sep 10 22:20:23.595988 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:23.595818    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de372118-61dc-45ee-b7db-ae68c945f4bc-workload-socket\") pod \"e2e-shared-a-w3-d07dfb-openshift-default-6b77565746-fcc6m\" (UID: \"de372118-61dc-45ee-b7db-ae68c945f4bc\") " pod="openshift-ingress/e2e-shared-a-w3-d07dfb-openshift-default-6b77565746-fcc6m"
Sep 10 22:20:23.595988 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:23.595881    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mf4r6\" (UniqueName: \"kubernetes.io/projected/de372118-61dc-45ee-b7db-ae68c945f4bc-kube-api-access-mf4r6\") pod \"e2e-shared-a-w3-d07dfb-openshift-default-6b77565746-fcc6m\" (UID: \"de372118-61dc-45ee-b7db-ae68c945f4bc\") " pod="openshift-ingress/e2e-shared-a-w3-d07dfb-openshift-default-6b77565746-fcc6m"
Sep 10 22:20:23.595988 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:23.595906    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de372118-61dc-45ee-b7db-ae68c945f4bc-istiod-ca-cert\") pod \"e2e-shared-a-w3-d07dfb-openshift-default-6b77565746-fcc6m\" (UID: \"de372118-61dc-45ee-b7db-ae68c945f4bc\") " pod="openshift-ingress/e2e-shared-a-w3-d07dfb-openshift-default-6b77565746-fcc6m"
Sep 10 22:20:23.595988 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:23.595966    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de372118-61dc-45ee-b7db-ae68c945f4bc-credential-socket\") pod \"e2e-shared-a-w3-d07dfb-openshift-default-6b77565746-fcc6m\" (UID: \"de372118-61dc-45ee-b7db-ae68c945f4bc\") " pod="openshift-ingress/e2e-shared-a-w3-d07dfb-openshift-default-6b77565746-fcc6m"
Sep 10 22:20:23.596190 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:23.595989    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de372118-61dc-45ee-b7db-ae68c945f4bc-istio-podinfo\") pod \"e2e-shared-a-w3-d07dfb-openshift-default-6b77565746-fcc6m\" (UID: \"de372118-61dc-45ee-b7db-ae68c945f4bc\") " pod="openshift-ingress/e2e-shared-a-w3-d07dfb-openshift-default-6b77565746-fcc6m"
Sep 10 22:20:23.596190 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:23.596015    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de372118-61dc-45ee-b7db-ae68c945f4bc-istio-token\") pod \"e2e-shared-a-w3-d07dfb-openshift-default-6b77565746-fcc6m\" (UID: \"de372118-61dc-45ee-b7db-ae68c945f4bc\") " pod="openshift-ingress/e2e-shared-a-w3-d07dfb-openshift-default-6b77565746-fcc6m"
Sep 10 22:20:23.697339 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:23.697310    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de372118-61dc-45ee-b7db-ae68c945f4bc-credential-socket\") pod \"e2e-shared-a-w3-d07dfb-openshift-default-6b77565746-fcc6m\" (UID: \"de372118-61dc-45ee-b7db-ae68c945f4bc\") " pod="openshift-ingress/e2e-shared-a-w3-d07dfb-openshift-default-6b77565746-fcc6m"
Sep 10 22:20:23.697339 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:23.697341    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de372118-61dc-45ee-b7db-ae68c945f4bc-istio-podinfo\") pod \"e2e-shared-a-w3-d07dfb-openshift-default-6b77565746-fcc6m\" (UID: \"de372118-61dc-45ee-b7db-ae68c945f4bc\") " pod="openshift-ingress/e2e-shared-a-w3-d07dfb-openshift-default-6b77565746-fcc6m"
Sep 10 22:20:23.697783 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:23.697360    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de372118-61dc-45ee-b7db-ae68c945f4bc-istio-token\") pod \"e2e-shared-a-w3-d07dfb-openshift-default-6b77565746-fcc6m\" (UID: \"de372118-61dc-45ee-b7db-ae68c945f4bc\") " pod="openshift-ingress/e2e-shared-a-w3-d07dfb-openshift-default-6b77565746-fcc6m"
Sep 10 22:20:23.697783 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:23.697395    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de372118-61dc-45ee-b7db-ae68c945f4bc-istio-envoy\") pod \"e2e-shared-a-w3-d07dfb-openshift-default-6b77565746-fcc6m\" (UID: \"de372118-61dc-45ee-b7db-ae68c945f4bc\") " pod="openshift-ingress/e2e-shared-a-w3-d07dfb-openshift-default-6b77565746-fcc6m"
Sep 10 22:20:23.697783 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:23.697414    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de372118-61dc-45ee-b7db-ae68c945f4bc-istio-data\") pod \"e2e-shared-a-w3-d07dfb-openshift-default-6b77565746-fcc6m\" (UID: \"de372118-61dc-45ee-b7db-ae68c945f4bc\") " pod="openshift-ingress/e2e-shared-a-w3-d07dfb-openshift-default-6b77565746-fcc6m"
Sep 10 22:20:23.697783 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:23.697446    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de372118-61dc-45ee-b7db-ae68c945f4bc-workload-certs\") pod \"e2e-shared-a-w3-d07dfb-openshift-default-6b77565746-fcc6m\" (UID: \"de372118-61dc-45ee-b7db-ae68c945f4bc\") " pod="openshift-ingress/e2e-shared-a-w3-d07dfb-openshift-default-6b77565746-fcc6m"
Sep 10 22:20:23.697783 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:23.697495    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de372118-61dc-45ee-b7db-ae68c945f4bc-workload-socket\") pod \"e2e-shared-a-w3-d07dfb-openshift-default-6b77565746-fcc6m\" (UID: \"de372118-61dc-45ee-b7db-ae68c945f4bc\") " pod="openshift-ingress/e2e-shared-a-w3-d07dfb-openshift-default-6b77565746-fcc6m"
Sep 10 22:20:23.697783 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:23.697750    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de372118-61dc-45ee-b7db-ae68c945f4bc-credential-socket\") pod \"e2e-shared-a-w3-d07dfb-openshift-default-6b77565746-fcc6m\" (UID: \"de372118-61dc-45ee-b7db-ae68c945f4bc\") " pod="openshift-ingress/e2e-shared-a-w3-d07dfb-openshift-default-6b77565746-fcc6m"
Sep 10 22:20:23.697783 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:23.697751    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mf4r6\" (UniqueName: \"kubernetes.io/projected/de372118-61dc-45ee-b7db-ae68c945f4bc-kube-api-access-mf4r6\") pod \"e2e-shared-a-w3-d07dfb-openshift-default-6b77565746-fcc6m\" (UID: \"de372118-61dc-45ee-b7db-ae68c945f4bc\") " pod="openshift-ingress/e2e-shared-a-w3-d07dfb-openshift-default-6b77565746-fcc6m"
Sep 10 22:20:23.698156 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:23.697805    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de372118-61dc-45ee-b7db-ae68c945f4bc-istio-data\") pod \"e2e-shared-a-w3-d07dfb-openshift-default-6b77565746-fcc6m\" (UID: \"de372118-61dc-45ee-b7db-ae68c945f4bc\") " pod="openshift-ingress/e2e-shared-a-w3-d07dfb-openshift-default-6b77565746-fcc6m"
Sep 10 22:20:23.698156 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:23.697823    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de372118-61dc-45ee-b7db-ae68c945f4bc-istiod-ca-cert\") pod \"e2e-shared-a-w3-d07dfb-openshift-default-6b77565746-fcc6m\" (UID: \"de372118-61dc-45ee-b7db-ae68c945f4bc\") " pod="openshift-ingress/e2e-shared-a-w3-d07dfb-openshift-default-6b77565746-fcc6m"
Sep 10 22:20:23.698156 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:23.697915    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de372118-61dc-45ee-b7db-ae68c945f4bc-workload-certs\") pod \"e2e-shared-a-w3-d07dfb-openshift-default-6b77565746-fcc6m\" (UID: \"de372118-61dc-45ee-b7db-ae68c945f4bc\") " pod="openshift-ingress/e2e-shared-a-w3-d07dfb-openshift-default-6b77565746-fcc6m"
Sep 10 22:20:23.698156 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:23.697933    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de372118-61dc-45ee-b7db-ae68c945f4bc-workload-socket\") pod \"e2e-shared-a-w3-d07dfb-openshift-default-6b77565746-fcc6m\" (UID: \"de372118-61dc-45ee-b7db-ae68c945f4bc\") " pod="openshift-ingress/e2e-shared-a-w3-d07dfb-openshift-default-6b77565746-fcc6m"
Sep 10 22:20:23.698446 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:23.698427    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de372118-61dc-45ee-b7db-ae68c945f4bc-istiod-ca-cert\") pod \"e2e-shared-a-w3-d07dfb-openshift-default-6b77565746-fcc6m\" (UID: \"de372118-61dc-45ee-b7db-ae68c945f4bc\") " pod="openshift-ingress/e2e-shared-a-w3-d07dfb-openshift-default-6b77565746-fcc6m"
Sep 10 22:20:23.699874 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:23.699833    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de372118-61dc-45ee-b7db-ae68c945f4bc-istio-envoy\") pod \"e2e-shared-a-w3-d07dfb-openshift-default-6b77565746-fcc6m\" (UID: \"de372118-61dc-45ee-b7db-ae68c945f4bc\") " pod="openshift-ingress/e2e-shared-a-w3-d07dfb-openshift-default-6b77565746-fcc6m"
Sep 10 22:20:23.700017 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:23.700000    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de372118-61dc-45ee-b7db-ae68c945f4bc-istio-podinfo\") pod \"e2e-shared-a-w3-d07dfb-openshift-default-6b77565746-fcc6m\" (UID: \"de372118-61dc-45ee-b7db-ae68c945f4bc\") " pod="openshift-ingress/e2e-shared-a-w3-d07dfb-openshift-default-6b77565746-fcc6m"
Sep 10 22:20:23.710865 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:23.710792    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mf4r6\" (UniqueName: \"kubernetes.io/projected/de372118-61dc-45ee-b7db-ae68c945f4bc-kube-api-access-mf4r6\") pod \"e2e-shared-a-w3-d07dfb-openshift-default-6b77565746-fcc6m\" (UID: \"de372118-61dc-45ee-b7db-ae68c945f4bc\") " pod="openshift-ingress/e2e-shared-a-w3-d07dfb-openshift-default-6b77565746-fcc6m"
Sep 10 22:20:23.711768 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:23.711748    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de372118-61dc-45ee-b7db-ae68c945f4bc-istio-token\") pod \"e2e-shared-a-w3-d07dfb-openshift-default-6b77565746-fcc6m\" (UID: \"de372118-61dc-45ee-b7db-ae68c945f4bc\") " pod="openshift-ingress/e2e-shared-a-w3-d07dfb-openshift-default-6b77565746-fcc6m"
Sep 10 22:20:23.767266 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:23.767227    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-d07dfb-openshift-default-6b77565746-fcc6m"
Sep 10 22:20:24.044875 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:24.044741    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-d07dfb-openshift-default-6b77565746-fcc6m"]
Sep 10 22:20:24.047497 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:20:24.047470    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde372118_61dc_45ee_b7db_ae68c945f4bc.slice/crio-ec21f9dce953617a37f1ed4e8783bee23374da4e1113d892d950e58474dfa66f WatchSource:0}: Error finding container ec21f9dce953617a37f1ed4e8783bee23374da4e1113d892d950e58474dfa66f: Status 404 returned error can't find the container with id ec21f9dce953617a37f1ed4e8783bee23374da4e1113d892d950e58474dfa66f
Sep 10 22:20:24.049706 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:24.049672    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 22:20:24.049802 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:24.049740    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 22:20:24.049802 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:24.049768    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 22:20:24.679065 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:24.679028    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-d07dfb-openshift-default-6b77565746-fcc6m" event={"ID":"de372118-61dc-45ee-b7db-ae68c945f4bc","Type":"ContainerStarted","Data":"3c34f92956b034b46b4c0cf7fa6d3987dbb3a0524d809205dce916ca197fd271"}
Sep 10 22:20:24.679557 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:24.679076    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-d07dfb-openshift-default-6b77565746-fcc6m" event={"ID":"de372118-61dc-45ee-b7db-ae68c945f4bc","Type":"ContainerStarted","Data":"ec21f9dce953617a37f1ed4e8783bee23374da4e1113d892d950e58474dfa66f"}
Sep 10 22:20:24.705561 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:24.705515    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-d07dfb-openshift-default-6b77565746-fcc6m" podStartSLOduration=1.705499925 podStartE2EDuration="1.705499925s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 22:20:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:20:24.703292036 +0000 UTC m=+1145.659359742" watchObservedRunningTime="2026-09-10 22:20:24.705499925 +0000 UTC m=+1145.661567630"
Sep 10 22:20:24.768352 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:24.768322    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-d07dfb-openshift-default-6b77565746-fcc6m"
Sep 10 22:20:25.772376 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:25.772348    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-d07dfb-openshift-default-6b77565746-fcc6m"
Sep 10 22:20:26.688780 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:26.688743    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-d07dfb-openshift-default-6b77565746-fcc6m"
Sep 10 22:20:26.689846 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:26.689825    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-d07dfb-openshift-default-6b77565746-fcc6m"
Sep 10 22:20:28.595425 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:28.595369    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-d07dfb-5bf7b6b89c-8psmq"]
Sep 10 22:20:28.600270 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:28.600236    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-d07dfb-5bf7b6b89c-8psmq"
Sep 10 22:20:28.604778 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:28.604752    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-d07dfb\""
Sep 10 22:20:28.619729 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:28.619698    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-d07dfb-5bf7b6b89c-8psmq"]
Sep 10 22:20:28.744477 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:28.744442    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcz8p\" (UniqueName: \"kubernetes.io/projected/0ae76787-4199-472e-8858-ab3c1d35724d-kube-api-access-lcz8p\") pod \"payload-processing-e2e-shared-a-w3-d07dfb-5bf7b6b89c-8psmq\" (UID: \"0ae76787-4199-472e-8858-ab3c1d35724d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-d07dfb-5bf7b6b89c-8psmq"
Sep 10 22:20:28.744477 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:28.744481    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ae76787-4199-472e-8858-ab3c1d35724d-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-d07dfb-5bf7b6b89c-8psmq\" (UID: \"0ae76787-4199-472e-8858-ab3c1d35724d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-d07dfb-5bf7b6b89c-8psmq"
Sep 10 22:20:28.845344 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:28.845298    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcz8p\" (UniqueName: \"kubernetes.io/projected/0ae76787-4199-472e-8858-ab3c1d35724d-kube-api-access-lcz8p\") pod \"payload-processing-e2e-shared-a-w3-d07dfb-5bf7b6b89c-8psmq\" (UID: \"0ae76787-4199-472e-8858-ab3c1d35724d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-d07dfb-5bf7b6b89c-8psmq"
Sep 10 22:20:28.845344 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:28.845345    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ae76787-4199-472e-8858-ab3c1d35724d-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-d07dfb-5bf7b6b89c-8psmq\" (UID: \"0ae76787-4199-472e-8858-ab3c1d35724d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-d07dfb-5bf7b6b89c-8psmq"
Sep 10 22:20:28.846149 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:28.846086    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ae76787-4199-472e-8858-ab3c1d35724d-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-d07dfb-5bf7b6b89c-8psmq\" (UID: \"0ae76787-4199-472e-8858-ab3c1d35724d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-d07dfb-5bf7b6b89c-8psmq"
Sep 10 22:20:28.857014 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:28.856989    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcz8p\" (UniqueName: \"kubernetes.io/projected/0ae76787-4199-472e-8858-ab3c1d35724d-kube-api-access-lcz8p\") pod \"payload-processing-e2e-shared-a-w3-d07dfb-5bf7b6b89c-8psmq\" (UID: \"0ae76787-4199-472e-8858-ab3c1d35724d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-d07dfb-5bf7b6b89c-8psmq"
Sep 10 22:20:28.910962 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:28.910900    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-d07dfb-5bf7b6b89c-8psmq"
Sep 10 22:20:29.184695 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:29.184650    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-d07dfb-5bf7b6b89c-8psmq"]
Sep 10 22:20:29.707446 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:29.707408    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-d07dfb-5bf7b6b89c-8psmq" event={"ID":"0ae76787-4199-472e-8858-ab3c1d35724d","Type":"ContainerStarted","Data":"0718e2c82b0185c2db907febc8d092f7e46f61a6d34e4271983a9c5aa75dd4a8"}
Sep 10 22:20:29.707866 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:29.707464    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-d07dfb-5bf7b6b89c-8psmq" event={"ID":"0ae76787-4199-472e-8858-ab3c1d35724d","Type":"ContainerStarted","Data":"5c9d6f8f3f00a2ae88d7e7f94512ffc44d59480ab1808a61da904a3dcbb7eae0"}
Sep 10 22:20:29.707866 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:29.707492    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-d07dfb-5bf7b6b89c-8psmq"
Sep 10 22:20:29.752545 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:29.752488    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-d07dfb-5bf7b6b89c-8psmq" podStartSLOduration=1.75246727 podStartE2EDuration="1.75246727s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 22:20:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:20:29.750495644 +0000 UTC m=+1150.706563349" watchObservedRunningTime="2026-09-10 22:20:29.75246727 +0000 UTC m=+1150.708534977"
Sep 10 22:20:37.616516 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:37.616465    2582 patch_prober.go:28] interesting pod/e2e-shared-a-w2-a9b6f6-openshift-default-558c5f6ff4-dkx44 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": read tcp 10.133.0.2:56186->10.133.0.61:15021: read: connection reset by peer" start-of-body=
Sep 10 22:20:37.617030 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:37.616545    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-a9b6f6-openshift-default-558c5f6ff4-dkx44" podUID="8b0dd305-1f45-45dc-8e31-01c6c225c275" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": read tcp 10.133.0.2:56186->10.133.0.61:15021: read: connection reset by peer"
Sep 10 22:20:37.747967 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:37.747884    2582 generic.go:356] "Generic (PLEG): container finished" podID="8b0dd305-1f45-45dc-8e31-01c6c225c275" containerID="f71afe4c9e4c8e7ddce7c1f212014436da4a8bb3bf1415294fa4272e51438ed0" exitCode=0
Sep 10 22:20:37.747967 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:37.747956    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-a9b6f6-openshift-default-558c5f6ff4-dkx44" event={"ID":"8b0dd305-1f45-45dc-8e31-01c6c225c275","Type":"ContainerDied","Data":"f71afe4c9e4c8e7ddce7c1f212014436da4a8bb3bf1415294fa4272e51438ed0"}
Sep 10 22:20:37.748342 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:37.748329    2582 scope.go:117] "RemoveContainer" containerID="f71afe4c9e4c8e7ddce7c1f212014436da4a8bb3bf1415294fa4272e51438ed0"
Sep 10 22:20:37.749388 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:37.749356    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 22:20:37.749499 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:37.749448    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 22:20:37.749562 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:37.749496    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 22:20:38.755227 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:38.755189    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-a9b6f6-openshift-default-558c5f6ff4-dkx44" event={"ID":"8b0dd305-1f45-45dc-8e31-01c6c225c275","Type":"ContainerStarted","Data":"bdf91189a87183186ae56f003c215affd6a59f1b8c32e65b23f17cc41d55ff09"}
Sep 10 22:20:39.676614 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:39.676145    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-a9b6f6-openshift-default-558c5f6ff4-dkx44"
Sep 10 22:20:39.676614 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:39.676276    2582 patch_prober.go:28] interesting pod/e2e-shared-a-w2-a9b6f6-openshift-default-558c5f6ff4-dkx44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Sep 10 22:20:39.676614 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:39.676320    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-a9b6f6-openshift-default-558c5f6ff4-dkx44" podUID="8b0dd305-1f45-45dc-8e31-01c6c225c275" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Sep 10 22:20:40.676261 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:40.676221    2582 patch_prober.go:28] interesting pod/e2e-shared-a-w2-a9b6f6-openshift-default-558c5f6ff4-dkx44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Sep 10 22:20:40.677313 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:40.676836    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-a9b6f6-openshift-default-558c5f6ff4-dkx44" podUID="8b0dd305-1f45-45dc-8e31-01c6c225c275" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Sep 10 22:20:40.714598 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:40.714564    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-d07dfb-5bf7b6b89c-8psmq"
Sep 10 22:20:41.676874 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:41.676801    2582 patch_prober.go:28] interesting pod/e2e-shared-a-w2-a9b6f6-openshift-default-558c5f6ff4-dkx44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Sep 10 22:20:41.677379 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:41.676902    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-a9b6f6-openshift-default-558c5f6ff4-dkx44" podUID="8b0dd305-1f45-45dc-8e31-01c6c225c275" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Sep 10 22:20:42.676436 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:42.676388    2582 patch_prober.go:28] interesting pod/e2e-shared-a-w2-a9b6f6-openshift-default-558c5f6ff4-dkx44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Sep 10 22:20:42.678664 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:42.676471    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-a9b6f6-openshift-default-558c5f6ff4-dkx44" podUID="8b0dd305-1f45-45dc-8e31-01c6c225c275" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Sep 10 22:20:42.730035 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:42.729995    2582 patch_prober.go:28] interesting pod/e2e-shared-a-w3-d07dfb-openshift-default-6b77565746-fcc6m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 10 22:20:42.730222 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:42.730090    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-d07dfb-openshift-default-6b77565746-fcc6m" podUID="de372118-61dc-45ee-b7db-ae68c945f4bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 10 22:20:43.676416 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:43.676376    2582 patch_prober.go:28] interesting pod/e2e-shared-a-w2-a9b6f6-openshift-default-558c5f6ff4-dkx44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Sep 10 22:20:43.676727 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:43.676695    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-a9b6f6-openshift-default-558c5f6ff4-dkx44" podUID="8b0dd305-1f45-45dc-8e31-01c6c225c275" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Sep 10 22:20:43.785285 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:43.785249    2582 generic.go:356] "Generic (PLEG): container finished" podID="de372118-61dc-45ee-b7db-ae68c945f4bc" containerID="3c34f92956b034b46b4c0cf7fa6d3987dbb3a0524d809205dce916ca197fd271" exitCode=0
Sep 10 22:20:43.785693 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:43.785322    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-d07dfb-openshift-default-6b77565746-fcc6m" event={"ID":"de372118-61dc-45ee-b7db-ae68c945f4bc","Type":"ContainerDied","Data":"3c34f92956b034b46b4c0cf7fa6d3987dbb3a0524d809205dce916ca197fd271"}
Sep 10 22:20:43.785760 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:43.785742    2582 scope.go:117] "RemoveContainer" containerID="3c34f92956b034b46b4c0cf7fa6d3987dbb3a0524d809205dce916ca197fd271"
Sep 10 22:20:43.787049 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:43.787017    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 22:20:43.787166 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:43.787108    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 22:20:43.787166 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:43.787156    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 22:20:44.685442 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:44.685406    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-a9b6f6-openshift-default-558c5f6ff4-dkx44"
Sep 10 22:20:44.686426 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:44.686358    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-a9b6f6-openshift-default-558c5f6ff4-dkx44"
Sep 10 22:20:44.693326 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:44.693296    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-a9b6f6-openshift-default-558c5f6ff4-dkx44"
Sep 10 22:20:44.803514 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:44.803470    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-d07dfb-openshift-default-6b77565746-fcc6m" event={"ID":"de372118-61dc-45ee-b7db-ae68c945f4bc","Type":"ContainerStarted","Data":"5e7d852998864b7edc92d018366c6b7ad51789086bf8d41ee71aadc41dff14fd"}
Sep 10 22:20:45.767828 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:45.767802    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-d07dfb-openshift-default-6b77565746-fcc6m"
Sep 10 22:20:45.768030 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:45.768006    2582 patch_prober.go:28] interesting pod/e2e-shared-a-w3-d07dfb-openshift-default-6b77565746-fcc6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Sep 10 22:20:45.768074 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:45.768049    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-d07dfb-openshift-default-6b77565746-fcc6m" podUID="de372118-61dc-45ee-b7db-ae68c945f4bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Sep 10 22:20:46.768618 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:46.768582    2582 patch_prober.go:28] interesting pod/e2e-shared-a-w3-d07dfb-openshift-default-6b77565746-fcc6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Sep 10 22:20:46.769113 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:46.768649    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-d07dfb-openshift-default-6b77565746-fcc6m" podUID="de372118-61dc-45ee-b7db-ae68c945f4bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Sep 10 22:20:47.768344 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:47.768301    2582 patch_prober.go:28] interesting pod/e2e-shared-a-w3-d07dfb-openshift-default-6b77565746-fcc6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Sep 10 22:20:47.768556 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:47.768365    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-d07dfb-openshift-default-6b77565746-fcc6m" podUID="de372118-61dc-45ee-b7db-ae68c945f4bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Sep 10 22:20:48.768449 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:48.768418    2582 patch_prober.go:28] interesting pod/e2e-shared-a-w3-d07dfb-openshift-default-6b77565746-fcc6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Sep 10 22:20:48.768921 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:48.768894    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-d07dfb-openshift-default-6b77565746-fcc6m" podUID="de372118-61dc-45ee-b7db-ae68c945f4bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Sep 10 22:20:49.767647 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:49.767610    2582 patch_prober.go:28] interesting pod/e2e-shared-a-w3-d07dfb-openshift-default-6b77565746-fcc6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Sep 10 22:20:49.767841 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:49.767668    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-d07dfb-openshift-default-6b77565746-fcc6m" podUID="de372118-61dc-45ee-b7db-ae68c945f4bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Sep 10 22:20:50.771441 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:50.771408    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-d07dfb-openshift-default-6b77565746-fcc6m"
Sep 10 22:20:50.771836 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:50.771672    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-d07dfb-openshift-default-6b77565746-fcc6m"
Sep 10 22:20:50.772463 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:50.772447    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-d07dfb-openshift-default-6b77565746-fcc6m"
Sep 10 22:20:52.262750 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:52.262721    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-ec6d47-openshift-default-c4db8956d-h7bnw"]
Sep 10 22:20:52.265993 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:52.265974    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-ec6d47-openshift-default-c4db8956d-h7bnw"
Sep 10 22:20:52.268386 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:52.268368    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-ec6d47-openshift-default-dockercfg-ddrcf\""
Sep 10 22:20:52.278154 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:52.278132    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-ec6d47-openshift-default-c4db8956d-h7bnw"]
Sep 10 22:20:52.361499 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:52.361469    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aeeca89b-577d-4dd5-990f-cf0706a478e7-workload-socket\") pod \"e2e-shared-b-w3-ec6d47-openshift-default-c4db8956d-h7bnw\" (UID: \"aeeca89b-577d-4dd5-990f-cf0706a478e7\") " pod="openshift-ingress/e2e-shared-b-w3-ec6d47-openshift-default-c4db8956d-h7bnw"
Sep 10 22:20:52.361627 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:52.361507    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aeeca89b-577d-4dd5-990f-cf0706a478e7-workload-certs\") pod \"e2e-shared-b-w3-ec6d47-openshift-default-c4db8956d-h7bnw\" (UID: \"aeeca89b-577d-4dd5-990f-cf0706a478e7\") " pod="openshift-ingress/e2e-shared-b-w3-ec6d47-openshift-default-c4db8956d-h7bnw"
Sep 10 22:20:52.361627 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:52.361526    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aeeca89b-577d-4dd5-990f-cf0706a478e7-istio-data\") pod \"e2e-shared-b-w3-ec6d47-openshift-default-c4db8956d-h7bnw\" (UID: \"aeeca89b-577d-4dd5-990f-cf0706a478e7\") " pod="openshift-ingress/e2e-shared-b-w3-ec6d47-openshift-default-c4db8956d-h7bnw"
Sep 10 22:20:52.361627 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:52.361569    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aeeca89b-577d-4dd5-990f-cf0706a478e7-istio-envoy\") pod \"e2e-shared-b-w3-ec6d47-openshift-default-c4db8956d-h7bnw\" (UID: \"aeeca89b-577d-4dd5-990f-cf0706a478e7\") " pod="openshift-ingress/e2e-shared-b-w3-ec6d47-openshift-default-c4db8956d-h7bnw"
Sep 10 22:20:52.361627 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:52.361607    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aeeca89b-577d-4dd5-990f-cf0706a478e7-istio-podinfo\") pod \"e2e-shared-b-w3-ec6d47-openshift-default-c4db8956d-h7bnw\" (UID: \"aeeca89b-577d-4dd5-990f-cf0706a478e7\") " pod="openshift-ingress/e2e-shared-b-w3-ec6d47-openshift-default-c4db8956d-h7bnw"
Sep 10 22:20:52.361869 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:52.361663    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aeeca89b-577d-4dd5-990f-cf0706a478e7-credential-socket\") pod \"e2e-shared-b-w3-ec6d47-openshift-default-c4db8956d-h7bnw\" (UID: \"aeeca89b-577d-4dd5-990f-cf0706a478e7\") " pod="openshift-ingress/e2e-shared-b-w3-ec6d47-openshift-default-c4db8956d-h7bnw"
Sep 10 22:20:52.361869 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:52.361726    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7ftf\" (UniqueName: \"kubernetes.io/projected/aeeca89b-577d-4dd5-990f-cf0706a478e7-kube-api-access-m7ftf\") pod \"e2e-shared-b-w3-ec6d47-openshift-default-c4db8956d-h7bnw\" (UID: \"aeeca89b-577d-4dd5-990f-cf0706a478e7\") " pod="openshift-ingress/e2e-shared-b-w3-ec6d47-openshift-default-c4db8956d-h7bnw"
Sep 10 22:20:52.361869 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:52.361766    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aeeca89b-577d-4dd5-990f-cf0706a478e7-istio-token\") pod \"e2e-shared-b-w3-ec6d47-openshift-default-c4db8956d-h7bnw\" (UID: \"aeeca89b-577d-4dd5-990f-cf0706a478e7\") " pod="openshift-ingress/e2e-shared-b-w3-ec6d47-openshift-default-c4db8956d-h7bnw"
Sep 10 22:20:52.361869 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:52.361859    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aeeca89b-577d-4dd5-990f-cf0706a478e7-istiod-ca-cert\") pod \"e2e-shared-b-w3-ec6d47-openshift-default-c4db8956d-h7bnw\" (UID: \"aeeca89b-577d-4dd5-990f-cf0706a478e7\") " pod="openshift-ingress/e2e-shared-b-w3-ec6d47-openshift-default-c4db8956d-h7bnw"
Sep 10 22:20:52.463037 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:52.462991    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aeeca89b-577d-4dd5-990f-cf0706a478e7-workload-certs\") pod \"e2e-shared-b-w3-ec6d47-openshift-default-c4db8956d-h7bnw\" (UID: \"aeeca89b-577d-4dd5-990f-cf0706a478e7\") " pod="openshift-ingress/e2e-shared-b-w3-ec6d47-openshift-default-c4db8956d-h7bnw"
Sep 10 22:20:52.463037 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:52.463036    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aeeca89b-577d-4dd5-990f-cf0706a478e7-istio-data\") pod \"e2e-shared-b-w3-ec6d47-openshift-default-c4db8956d-h7bnw\" (UID: \"aeeca89b-577d-4dd5-990f-cf0706a478e7\") " pod="openshift-ingress/e2e-shared-b-w3-ec6d47-openshift-default-c4db8956d-h7bnw"
Sep 10 22:20:52.463285 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:52.463146    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aeeca89b-577d-4dd5-990f-cf0706a478e7-istio-envoy\") pod \"e2e-shared-b-w3-ec6d47-openshift-default-c4db8956d-h7bnw\" (UID: \"aeeca89b-577d-4dd5-990f-cf0706a478e7\") " pod="openshift-ingress/e2e-shared-b-w3-ec6d47-openshift-default-c4db8956d-h7bnw"
Sep 10 22:20:52.463285 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:52.463207    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aeeca89b-577d-4dd5-990f-cf0706a478e7-istio-podinfo\") pod \"e2e-shared-b-w3-ec6d47-openshift-default-c4db8956d-h7bnw\" (UID: \"aeeca89b-577d-4dd5-990f-cf0706a478e7\") " pod="openshift-ingress/e2e-shared-b-w3-ec6d47-openshift-default-c4db8956d-h7bnw"
Sep 10 22:20:52.463285 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:52.463251    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aeeca89b-577d-4dd5-990f-cf0706a478e7-credential-socket\") pod \"e2e-shared-b-w3-ec6d47-openshift-default-c4db8956d-h7bnw\" (UID: \"aeeca89b-577d-4dd5-990f-cf0706a478e7\") " pod="openshift-ingress/e2e-shared-b-w3-ec6d47-openshift-default-c4db8956d-h7bnw"
Sep 10 22:20:52.463456 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:52.463307    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7ftf\" (UniqueName: \"kubernetes.io/projected/aeeca89b-577d-4dd5-990f-cf0706a478e7-kube-api-access-m7ftf\") pod \"e2e-shared-b-w3-ec6d47-openshift-default-c4db8956d-h7bnw\" (UID: \"aeeca89b-577d-4dd5-990f-cf0706a478e7\") " pod="openshift-ingress/e2e-shared-b-w3-ec6d47-openshift-default-c4db8956d-h7bnw"
Sep 10 22:20:52.463456 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:52.463443    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aeeca89b-577d-4dd5-990f-cf0706a478e7-workload-certs\") pod \"e2e-shared-b-w3-ec6d47-openshift-default-c4db8956d-h7bnw\" (UID: \"aeeca89b-577d-4dd5-990f-cf0706a478e7\") " pod="openshift-ingress/e2e-shared-b-w3-ec6d47-openshift-default-c4db8956d-h7bnw"
Sep 10 22:20:52.463564 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:52.463503    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aeeca89b-577d-4dd5-990f-cf0706a478e7-istio-data\") pod \"e2e-shared-b-w3-ec6d47-openshift-default-c4db8956d-h7bnw\" (UID: \"aeeca89b-577d-4dd5-990f-cf0706a478e7\") " pod="openshift-ingress/e2e-shared-b-w3-ec6d47-openshift-default-c4db8956d-h7bnw"
Sep 10 22:20:52.463618 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:52.463573    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aeeca89b-577d-4dd5-990f-cf0706a478e7-credential-socket\") pod \"e2e-shared-b-w3-ec6d47-openshift-default-c4db8956d-h7bnw\" (UID: \"aeeca89b-577d-4dd5-990f-cf0706a478e7\") " pod="openshift-ingress/e2e-shared-b-w3-ec6d47-openshift-default-c4db8956d-h7bnw"
Sep 10 22:20:52.463757 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:52.463735    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aeeca89b-577d-4dd5-990f-cf0706a478e7-istio-token\") pod \"e2e-shared-b-w3-ec6d47-openshift-default-c4db8956d-h7bnw\" (UID: \"aeeca89b-577d-4dd5-990f-cf0706a478e7\") " pod="openshift-ingress/e2e-shared-b-w3-ec6d47-openshift-default-c4db8956d-h7bnw"
Sep 10 22:20:52.463875 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:52.463838    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aeeca89b-577d-4dd5-990f-cf0706a478e7-istiod-ca-cert\") pod \"e2e-shared-b-w3-ec6d47-openshift-default-c4db8956d-h7bnw\" (UID: \"aeeca89b-577d-4dd5-990f-cf0706a478e7\") " pod="openshift-ingress/e2e-shared-b-w3-ec6d47-openshift-default-c4db8956d-h7bnw"
Sep 10 22:20:52.463964 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:52.463950    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aeeca89b-577d-4dd5-990f-cf0706a478e7-workload-socket\") pod \"e2e-shared-b-w3-ec6d47-openshift-default-c4db8956d-h7bnw\" (UID: \"aeeca89b-577d-4dd5-990f-cf0706a478e7\") " pod="openshift-ingress/e2e-shared-b-w3-ec6d47-openshift-default-c4db8956d-h7bnw"
Sep 10 22:20:52.464309 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:52.464285    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aeeca89b-577d-4dd5-990f-cf0706a478e7-workload-socket\") pod \"e2e-shared-b-w3-ec6d47-openshift-default-c4db8956d-h7bnw\" (UID: \"aeeca89b-577d-4dd5-990f-cf0706a478e7\") " pod="openshift-ingress/e2e-shared-b-w3-ec6d47-openshift-default-c4db8956d-h7bnw"
Sep 10 22:20:52.464507 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:52.464483    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aeeca89b-577d-4dd5-990f-cf0706a478e7-istiod-ca-cert\") pod \"e2e-shared-b-w3-ec6d47-openshift-default-c4db8956d-h7bnw\" (UID: \"aeeca89b-577d-4dd5-990f-cf0706a478e7\") " pod="openshift-ingress/e2e-shared-b-w3-ec6d47-openshift-default-c4db8956d-h7bnw"
Sep 10 22:20:52.465538 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:52.465503    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aeeca89b-577d-4dd5-990f-cf0706a478e7-istio-envoy\") pod \"e2e-shared-b-w3-ec6d47-openshift-default-c4db8956d-h7bnw\" (UID: \"aeeca89b-577d-4dd5-990f-cf0706a478e7\") " pod="openshift-ingress/e2e-shared-b-w3-ec6d47-openshift-default-c4db8956d-h7bnw"
Sep 10 22:20:52.465918 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:52.465900    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aeeca89b-577d-4dd5-990f-cf0706a478e7-istio-podinfo\") pod \"e2e-shared-b-w3-ec6d47-openshift-default-c4db8956d-h7bnw\" (UID: \"aeeca89b-577d-4dd5-990f-cf0706a478e7\") " pod="openshift-ingress/e2e-shared-b-w3-ec6d47-openshift-default-c4db8956d-h7bnw"
Sep 10 22:20:52.478712 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:52.478684    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aeeca89b-577d-4dd5-990f-cf0706a478e7-istio-token\") pod \"e2e-shared-b-w3-ec6d47-openshift-default-c4db8956d-h7bnw\" (UID: \"aeeca89b-577d-4dd5-990f-cf0706a478e7\") " pod="openshift-ingress/e2e-shared-b-w3-ec6d47-openshift-default-c4db8956d-h7bnw"
Sep 10 22:20:52.478872 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:52.478819    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7ftf\" (UniqueName: \"kubernetes.io/projected/aeeca89b-577d-4dd5-990f-cf0706a478e7-kube-api-access-m7ftf\") pod \"e2e-shared-b-w3-ec6d47-openshift-default-c4db8956d-h7bnw\" (UID: \"aeeca89b-577d-4dd5-990f-cf0706a478e7\") " pod="openshift-ingress/e2e-shared-b-w3-ec6d47-openshift-default-c4db8956d-h7bnw"
Sep 10 22:20:52.576347 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:52.576267    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-ec6d47-openshift-default-c4db8956d-h7bnw"
Sep 10 22:20:52.606543 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:52.606516    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a9b3ec-565f78dc58-nrsf6"]
Sep 10 22:20:52.609939 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:52.609912    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a9b3ec-565f78dc58-nrsf6"
Sep 10 22:20:52.612436 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:52.612408    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-a9b3ec\""
Sep 10 22:20:52.612913 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:52.612894    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-a9b3ec-dockercfg-hl85s\""
Sep 10 22:20:52.620641 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:52.620616    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a9b3ec-565f78dc58-nrsf6"]
Sep 10 22:20:52.666644 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:52.666514    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qp2x9\" (UniqueName: \"kubernetes.io/projected/94040552-d14f-4d5a-bf04-b1d5adad903b-kube-api-access-qp2x9\") pod \"payload-pre-processing-e2e-shared-b-w2-a9b3ec-565f78dc58-nrsf6\" (UID: \"94040552-d14f-4d5a-bf04-b1d5adad903b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a9b3ec-565f78dc58-nrsf6"
Sep 10 22:20:52.666874 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:52.666712    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94040552-d14f-4d5a-bf04-b1d5adad903b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-a9b3ec-565f78dc58-nrsf6\" (UID: \"94040552-d14f-4d5a-bf04-b1d5adad903b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a9b3ec-565f78dc58-nrsf6"
Sep 10 22:20:52.767761 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:52.767726    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94040552-d14f-4d5a-bf04-b1d5adad903b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-a9b3ec-565f78dc58-nrsf6\" (UID: \"94040552-d14f-4d5a-bf04-b1d5adad903b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a9b3ec-565f78dc58-nrsf6"
Sep 10 22:20:52.767958 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:52.767783    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qp2x9\" (UniqueName: \"kubernetes.io/projected/94040552-d14f-4d5a-bf04-b1d5adad903b-kube-api-access-qp2x9\") pod \"payload-pre-processing-e2e-shared-b-w2-a9b3ec-565f78dc58-nrsf6\" (UID: \"94040552-d14f-4d5a-bf04-b1d5adad903b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a9b3ec-565f78dc58-nrsf6"
Sep 10 22:20:52.768492 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:52.768471    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94040552-d14f-4d5a-bf04-b1d5adad903b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-a9b3ec-565f78dc58-nrsf6\" (UID: \"94040552-d14f-4d5a-bf04-b1d5adad903b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a9b3ec-565f78dc58-nrsf6"
Sep 10 22:20:52.776131 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:52.776104    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qp2x9\" (UniqueName: \"kubernetes.io/projected/94040552-d14f-4d5a-bf04-b1d5adad903b-kube-api-access-qp2x9\") pod \"payload-pre-processing-e2e-shared-b-w2-a9b3ec-565f78dc58-nrsf6\" (UID: \"94040552-d14f-4d5a-bf04-b1d5adad903b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a9b3ec-565f78dc58-nrsf6"
Sep 10 22:20:52.854741 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:52.854707    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-ec6d47-openshift-default-c4db8956d-h7bnw"]
Sep 10 22:20:52.856634 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:20:52.856608    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaeeca89b_577d_4dd5_990f_cf0706a478e7.slice/crio-2be0514d432c3a5fdadda359c3c91464fcd89d8d481e17b02c9c25787a4b284d WatchSource:0}: Error finding container 2be0514d432c3a5fdadda359c3c91464fcd89d8d481e17b02c9c25787a4b284d: Status 404 returned error can't find the container with id 2be0514d432c3a5fdadda359c3c91464fcd89d8d481e17b02c9c25787a4b284d
Sep 10 22:20:52.858904 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:52.858844    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 22:20:52.858980 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:52.858935    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 22:20:52.858980 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:52.858962    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 22:20:52.922003 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:52.921982    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a9b3ec-565f78dc58-nrsf6"
Sep 10 22:20:53.195723 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:53.195686    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a9b3ec-565f78dc58-nrsf6"]
Sep 10 22:20:53.198351 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:20:53.198313    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94040552_d14f_4d5a_bf04_b1d5adad903b.slice/crio-4d3295914af0d5ccfa7603f9114ca1e238085c0bb8e1b3ead7b9d39f9502b024 WatchSource:0}: Error finding container 4d3295914af0d5ccfa7603f9114ca1e238085c0bb8e1b3ead7b9d39f9502b024: Status 404 returned error can't find the container with id 4d3295914af0d5ccfa7603f9114ca1e238085c0bb8e1b3ead7b9d39f9502b024
Sep 10 22:20:53.851260 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:53.851225    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a9b3ec-565f78dc58-nrsf6" event={"ID":"94040552-d14f-4d5a-bf04-b1d5adad903b","Type":"ContainerStarted","Data":"08d20b630543f2312aee7221fb480c39d4036d201e6d42ef7be45c5a9f5defba"}
Sep 10 22:20:53.851712 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:53.851283    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a9b3ec-565f78dc58-nrsf6" event={"ID":"94040552-d14f-4d5a-bf04-b1d5adad903b","Type":"ContainerStarted","Data":"4d3295914af0d5ccfa7603f9114ca1e238085c0bb8e1b3ead7b9d39f9502b024"}
Sep 10 22:20:53.851712 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:53.851430    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a9b3ec-565f78dc58-nrsf6"
Sep 10 22:20:53.852986 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:53.852962    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-ec6d47-openshift-default-c4db8956d-h7bnw" event={"ID":"aeeca89b-577d-4dd5-990f-cf0706a478e7","Type":"ContainerStarted","Data":"226a5112c7d11f88754a06fa72a7e3f8d259a7a9a675f80157904c106b40a68b"}
Sep 10 22:20:53.853097 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:53.852999    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-ec6d47-openshift-default-c4db8956d-h7bnw" event={"ID":"aeeca89b-577d-4dd5-990f-cf0706a478e7","Type":"ContainerStarted","Data":"2be0514d432c3a5fdadda359c3c91464fcd89d8d481e17b02c9c25787a4b284d"}
Sep 10 22:20:53.868317 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:53.868273    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a9b3ec-565f78dc58-nrsf6" podStartSLOduration=1.868260538 podStartE2EDuration="1.868260538s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 22:20:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:20:53.86690846 +0000 UTC m=+1174.822976166" watchObservedRunningTime="2026-09-10 22:20:53.868260538 +0000 UTC m=+1174.824328245"
Sep 10 22:20:53.885672 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:53.885631    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-ec6d47-openshift-default-c4db8956d-h7bnw" podStartSLOduration=1.885618866 podStartE2EDuration="1.885618866s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 22:20:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:20:53.884415307 +0000 UTC m=+1174.840483012" watchObservedRunningTime="2026-09-10 22:20:53.885618866 +0000 UTC m=+1174.841686572"
Sep 10 22:20:54.576893 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:54.576842    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-ec6d47-openshift-default-c4db8956d-h7bnw"
Sep 10 22:20:54.581513 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:54.581493    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-ec6d47-openshift-default-c4db8956d-h7bnw"
Sep 10 22:20:54.858957 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:54.858881    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-ec6d47-openshift-default-c4db8956d-h7bnw"
Sep 10 22:20:54.859646 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:54.859629    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-ec6d47-openshift-default-c4db8956d-h7bnw"
Sep 10 22:20:57.891217 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:57.891184    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ec6d47-684fb785b6-llwjf"]
Sep 10 22:20:57.895570 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:57.895548    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ec6d47-684fb785b6-llwjf"
Sep 10 22:20:57.902786 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:57.902760    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-ec6d47\""
Sep 10 22:20:57.902786 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:57.902782    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-ec6d47-dockercfg-cgd4m\""
Sep 10 22:20:57.914150 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:57.914124    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ec6d47-684fb785b6-llwjf"]
Sep 10 22:20:58.014007 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:58.013972    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7kb6\" (UniqueName: \"kubernetes.io/projected/8269d673-185f-421c-b5bb-ca7d77226be6-kube-api-access-c7kb6\") pod \"payload-pre-processing-e2e-shared-b-w3-ec6d47-684fb785b6-llwjf\" (UID: \"8269d673-185f-421c-b5bb-ca7d77226be6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ec6d47-684fb785b6-llwjf"
Sep 10 22:20:58.014194 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:58.014077    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8269d673-185f-421c-b5bb-ca7d77226be6-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-ec6d47-684fb785b6-llwjf\" (UID: \"8269d673-185f-421c-b5bb-ca7d77226be6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ec6d47-684fb785b6-llwjf"
Sep 10 22:20:58.114924 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:58.114881    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8269d673-185f-421c-b5bb-ca7d77226be6-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-ec6d47-684fb785b6-llwjf\" (UID: \"8269d673-185f-421c-b5bb-ca7d77226be6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ec6d47-684fb785b6-llwjf"
Sep 10 22:20:58.115120 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:58.114959    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7kb6\" (UniqueName: \"kubernetes.io/projected/8269d673-185f-421c-b5bb-ca7d77226be6-kube-api-access-c7kb6\") pod \"payload-pre-processing-e2e-shared-b-w3-ec6d47-684fb785b6-llwjf\" (UID: \"8269d673-185f-421c-b5bb-ca7d77226be6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ec6d47-684fb785b6-llwjf"
Sep 10 22:20:58.115576 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:58.115554    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8269d673-185f-421c-b5bb-ca7d77226be6-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-ec6d47-684fb785b6-llwjf\" (UID: \"8269d673-185f-421c-b5bb-ca7d77226be6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ec6d47-684fb785b6-llwjf"
Sep 10 22:20:58.122885 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:58.122840    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7kb6\" (UniqueName: \"kubernetes.io/projected/8269d673-185f-421c-b5bb-ca7d77226be6-kube-api-access-c7kb6\") pod \"payload-pre-processing-e2e-shared-b-w3-ec6d47-684fb785b6-llwjf\" (UID: \"8269d673-185f-421c-b5bb-ca7d77226be6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ec6d47-684fb785b6-llwjf"
Sep 10 22:20:58.206158 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:58.206131    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ec6d47-684fb785b6-llwjf"
Sep 10 22:20:58.478430 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:58.478354    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ec6d47-684fb785b6-llwjf"]
Sep 10 22:20:58.481521 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:20:58.481494    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8269d673_185f_421c_b5bb_ca7d77226be6.slice/crio-82a750da4a2fbb05d2fa7868f745382aceb8709247617841e635c9c01bc10a4e WatchSource:0}: Error finding container 82a750da4a2fbb05d2fa7868f745382aceb8709247617841e635c9c01bc10a4e: Status 404 returned error can't find the container with id 82a750da4a2fbb05d2fa7868f745382aceb8709247617841e635c9c01bc10a4e
Sep 10 22:20:58.880233 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:58.880154    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ec6d47-684fb785b6-llwjf" event={"ID":"8269d673-185f-421c-b5bb-ca7d77226be6","Type":"ContainerStarted","Data":"552930ed95112188648b5d43832e95f56e61e37e4a0c051b2561464ff77211fe"}
Sep 10 22:20:58.880233 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:58.880205    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ec6d47-684fb785b6-llwjf" event={"ID":"8269d673-185f-421c-b5bb-ca7d77226be6","Type":"ContainerStarted","Data":"82a750da4a2fbb05d2fa7868f745382aceb8709247617841e635c9c01bc10a4e"}
Sep 10 22:20:58.880435 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:58.880285    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ec6d47-684fb785b6-llwjf"
Sep 10 22:20:58.898894 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:20:58.898834    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ec6d47-684fb785b6-llwjf" podStartSLOduration=1.898821678 podStartE2EDuration="1.898821678s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 22:20:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:20:58.897089625 +0000 UTC m=+1179.853157331" watchObservedRunningTime="2026-09-10 22:20:58.898821678 +0000 UTC m=+1179.854889383"
Sep 10 22:21:04.861573 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:21:04.861541    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a9b3ec-565f78dc58-nrsf6"
Sep 10 22:21:09.887633 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:21:09.887597    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ec6d47-684fb785b6-llwjf"
Sep 10 22:21:10.900047 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:21:10.899998    2582 patch_prober.go:28] interesting pod/e2e-shared-b-w3-ec6d47-openshift-default-c4db8956d-h7bnw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 10 22:21:10.900583 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:21:10.900064    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-ec6d47-openshift-default-c4db8956d-h7bnw" podUID="aeeca89b-577d-4dd5-990f-cf0706a478e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 10 22:21:11.948274 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:21:11.948237    2582 generic.go:356] "Generic (PLEG): container finished" podID="aeeca89b-577d-4dd5-990f-cf0706a478e7" containerID="226a5112c7d11f88754a06fa72a7e3f8d259a7a9a675f80157904c106b40a68b" exitCode=0
Sep 10 22:21:11.948760 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:21:11.948307    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-ec6d47-openshift-default-c4db8956d-h7bnw" event={"ID":"aeeca89b-577d-4dd5-990f-cf0706a478e7","Type":"ContainerDied","Data":"226a5112c7d11f88754a06fa72a7e3f8d259a7a9a675f80157904c106b40a68b"}
Sep 10 22:21:11.948760 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:21:11.948747    2582 scope.go:117] "RemoveContainer" containerID="226a5112c7d11f88754a06fa72a7e3f8d259a7a9a675f80157904c106b40a68b"
Sep 10 22:21:11.949912 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:21:11.949863    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 22:21:11.950036 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:21:11.949960    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 22:21:11.950036 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:21:11.950010    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 22:21:12.955838 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:21:12.955799    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-ec6d47-openshift-default-c4db8956d-h7bnw" event={"ID":"aeeca89b-577d-4dd5-990f-cf0706a478e7","Type":"ContainerStarted","Data":"b0f64faaa8af94d43a965943c4e76e9eb074bea1454151c00249c8ec4eac8bd2"}
Sep 10 22:21:13.577383 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:21:13.577343    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-ec6d47-openshift-default-c4db8956d-h7bnw"
Sep 10 22:21:13.577596 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:21:13.577460    2582 patch_prober.go:28] interesting pod/e2e-shared-b-w3-ec6d47-openshift-default-c4db8956d-h7bnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Sep 10 22:21:13.577596 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:21:13.577505    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-ec6d47-openshift-default-c4db8956d-h7bnw" podUID="aeeca89b-577d-4dd5-990f-cf0706a478e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Sep 10 22:21:14.576970 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:21:14.576925    2582 patch_prober.go:28] interesting pod/e2e-shared-b-w3-ec6d47-openshift-default-c4db8956d-h7bnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Sep 10 22:21:14.577469 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:21:14.577006    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-ec6d47-openshift-default-c4db8956d-h7bnw" podUID="aeeca89b-577d-4dd5-990f-cf0706a478e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Sep 10 22:21:15.577785 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:21:15.577746    2582 patch_prober.go:28] interesting pod/e2e-shared-b-w3-ec6d47-openshift-default-c4db8956d-h7bnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Sep 10 22:21:15.578339 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:21:15.577817    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-ec6d47-openshift-default-c4db8956d-h7bnw" podUID="aeeca89b-577d-4dd5-990f-cf0706a478e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Sep 10 22:21:16.577440 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:21:16.577391    2582 patch_prober.go:28] interesting pod/e2e-shared-b-w3-ec6d47-openshift-default-c4db8956d-h7bnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Sep 10 22:21:16.577687 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:21:16.577464    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-ec6d47-openshift-default-c4db8956d-h7bnw" podUID="aeeca89b-577d-4dd5-990f-cf0706a478e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Sep 10 22:21:17.577097 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:21:17.577049    2582 patch_prober.go:28] interesting pod/e2e-shared-b-w3-ec6d47-openshift-default-c4db8956d-h7bnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Sep 10 22:21:17.577617 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:21:17.577124    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-ec6d47-openshift-default-c4db8956d-h7bnw" podUID="aeeca89b-577d-4dd5-990f-cf0706a478e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Sep 10 22:21:18.580932 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:21:18.580895    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-ec6d47-openshift-default-c4db8956d-h7bnw"
Sep 10 22:21:18.581328 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:21:18.581167    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-ec6d47-openshift-default-c4db8956d-h7bnw"
Sep 10 22:21:18.581865 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:21:18.581827    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-ec6d47-openshift-default-c4db8956d-h7bnw"
Sep 10 22:21:19.506020 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:21:19.505994    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-w6vlb_8517c9a7-382a-40fd-bc52-e6da90ebc584/console-operator/2.log"
Sep 10 22:21:19.508403 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:21:19.508382    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-w6vlb_8517c9a7-382a-40fd-bc52-e6da90ebc584/console-operator/2.log"
Sep 10 22:21:19.630719 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:21:19.630693    2582 scope.go:117] "RemoveContainer" containerID="d3e6de3c985d1fdcdc27a17f6adb46474c85e760f9e749a00e1e688decaaa015"
Sep 10 22:21:19.638791 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:21:19.638775    2582 scope.go:117] "RemoveContainer" containerID="9ed94665562ab3cb5bce48d3e1e4b718ed48595990dbf2cf8be56f3fa6e4ade4"
Sep 10 22:21:32.754070 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:21:32.754032    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-676c26ab-57b7db5c49-fskg2"]
Sep 10 22:21:32.758108 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:21:32.758086    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-676c26ab-57b7db5c49-fskg2"
Sep 10 22:21:32.760244 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:21:32.760220    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-676c26ab\""
Sep 10 22:21:32.776607 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:21:32.776586    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-676c26ab-57b7db5c49-fskg2"]
Sep 10 22:21:32.840188 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:21:32.840159    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ce4bed6-f1db-42dc-a2a7-86d398926c24-plugins-config-volume\") pod \"payload-processing-e2e-mt-676c26ab-57b7db5c49-fskg2\" (UID: \"4ce4bed6-f1db-42dc-a2a7-86d398926c24\") " pod="openshift-ingress/payload-processing-e2e-mt-676c26ab-57b7db5c49-fskg2"
Sep 10 22:21:32.840334 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:21:32.840195    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvw9v\" (UniqueName: \"kubernetes.io/projected/4ce4bed6-f1db-42dc-a2a7-86d398926c24-kube-api-access-jvw9v\") pod \"payload-processing-e2e-mt-676c26ab-57b7db5c49-fskg2\" (UID: \"4ce4bed6-f1db-42dc-a2a7-86d398926c24\") " pod="openshift-ingress/payload-processing-e2e-mt-676c26ab-57b7db5c49-fskg2"
Sep 10 22:21:32.941250 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:21:32.941218    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ce4bed6-f1db-42dc-a2a7-86d398926c24-plugins-config-volume\") pod \"payload-processing-e2e-mt-676c26ab-57b7db5c49-fskg2\" (UID: \"4ce4bed6-f1db-42dc-a2a7-86d398926c24\") " pod="openshift-ingress/payload-processing-e2e-mt-676c26ab-57b7db5c49-fskg2"
Sep 10 22:21:32.941250 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:21:32.941255    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvw9v\" (UniqueName: \"kubernetes.io/projected/4ce4bed6-f1db-42dc-a2a7-86d398926c24-kube-api-access-jvw9v\") pod \"payload-processing-e2e-mt-676c26ab-57b7db5c49-fskg2\" (UID: \"4ce4bed6-f1db-42dc-a2a7-86d398926c24\") " pod="openshift-ingress/payload-processing-e2e-mt-676c26ab-57b7db5c49-fskg2"
Sep 10 22:21:32.942599 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:21:32.942570    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ce4bed6-f1db-42dc-a2a7-86d398926c24-plugins-config-volume\") pod \"payload-processing-e2e-mt-676c26ab-57b7db5c49-fskg2\" (UID: \"4ce4bed6-f1db-42dc-a2a7-86d398926c24\") " pod="openshift-ingress/payload-processing-e2e-mt-676c26ab-57b7db5c49-fskg2"
Sep 10 22:21:32.950693 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:21:32.950657    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvw9v\" (UniqueName: \"kubernetes.io/projected/4ce4bed6-f1db-42dc-a2a7-86d398926c24-kube-api-access-jvw9v\") pod \"payload-processing-e2e-mt-676c26ab-57b7db5c49-fskg2\" (UID: \"4ce4bed6-f1db-42dc-a2a7-86d398926c24\") " pod="openshift-ingress/payload-processing-e2e-mt-676c26ab-57b7db5c49-fskg2"
Sep 10 22:21:33.068015 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:21:33.067938    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-676c26ab-57b7db5c49-fskg2"
Sep 10 22:21:33.337454 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:21:33.337426    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-676c26ab-57b7db5c49-fskg2"]
Sep 10 22:21:33.339464 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:21:33.339434    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ce4bed6_f1db_42dc_a2a7_86d398926c24.slice/crio-17b6718263c2fcc13e73a4e09e3d3f936ba80cac9cb7b7d95c99c1d122df2a47 WatchSource:0}: Error finding container 17b6718263c2fcc13e73a4e09e3d3f936ba80cac9cb7b7d95c99c1d122df2a47: Status 404 returned error can't find the container with id 17b6718263c2fcc13e73a4e09e3d3f936ba80cac9cb7b7d95c99c1d122df2a47
Sep 10 22:21:34.062283 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:21:34.062242    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-676c26ab-57b7db5c49-fskg2" event={"ID":"4ce4bed6-f1db-42dc-a2a7-86d398926c24","Type":"ContainerStarted","Data":"8642e0f2b641cdcd46edb8f62bc02e58700dca0fb5c7bdffab9546eaf38aa70c"}
Sep 10 22:21:34.062720 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:21:34.062297    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-676c26ab-57b7db5c49-fskg2" event={"ID":"4ce4bed6-f1db-42dc-a2a7-86d398926c24","Type":"ContainerStarted","Data":"17b6718263c2fcc13e73a4e09e3d3f936ba80cac9cb7b7d95c99c1d122df2a47"}
Sep 10 22:21:34.062720 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:21:34.062363    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-676c26ab-57b7db5c49-fskg2"
Sep 10 22:21:34.079578 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:21:34.079534    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-676c26ab-57b7db5c49-fskg2" podStartSLOduration=2.079521937 podStartE2EDuration="2.079521937s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 22:21:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:21:34.077890405 +0000 UTC m=+1215.033958113" watchObservedRunningTime="2026-09-10 22:21:34.079521937 +0000 UTC m=+1215.035589692"
Sep 10 22:21:40.301950 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:21:40.301911    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-d07dfb/auth-test-model-d07dfb-kserve-6764ddf5c9-vzq4s"]
Sep 10 22:21:40.305912 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:21:40.305890    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-d07dfb/auth-test-model-d07dfb-kserve-6764ddf5c9-vzq4s"
Sep 10 22:21:40.308794 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:21:40.308767    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-d07dfb\"/\"kube-root-ca.crt\""
Sep 10 22:21:40.308954 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:21:40.308804    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-d07dfb\"/\"default-dockercfg-nhwgr\""
Sep 10 22:21:40.309627 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:21:40.309607    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-d07dfb\"/\"openshift-service-ca.crt\""
Sep 10 22:21:40.309729 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:21:40.309644    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-d07dfb\"/\"auth-test-model-d07dfb-kserve-self-signed-certs\""
Sep 10 22:21:40.320674 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:21:40.320646    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-d07dfb/auth-test-model-d07dfb-kserve-6764ddf5c9-vzq4s"]
Sep 10 22:21:40.408265 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:21:40.408231    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32794fee-1c78-4799-b55d-a6dfd8086493-tmp-dir\") pod \"auth-test-model-d07dfb-kserve-6764ddf5c9-vzq4s\" (UID: \"32794fee-1c78-4799-b55d-a6dfd8086493\") " pod="ai-tenant-e2e-shared-a-w3-d07dfb/auth-test-model-d07dfb-kserve-6764ddf5c9-vzq4s"
Sep 10 22:21:40.408446 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:21:40.408281    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32794fee-1c78-4799-b55d-a6dfd8086493-model-cache\") pod \"auth-test-model-d07dfb-kserve-6764ddf5c9-vzq4s\" (UID: \"32794fee-1c78-4799-b55d-a6dfd8086493\") " pod="ai-tenant-e2e-shared-a-w3-d07dfb/auth-test-model-d07dfb-kserve-6764ddf5c9-vzq4s"
Sep 10 22:21:40.408446 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:21:40.408303    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32794fee-1c78-4799-b55d-a6dfd8086493-home\") pod \"auth-test-model-d07dfb-kserve-6764ddf5c9-vzq4s\" (UID: \"32794fee-1c78-4799-b55d-a6dfd8086493\") " pod="ai-tenant-e2e-shared-a-w3-d07dfb/auth-test-model-d07dfb-kserve-6764ddf5c9-vzq4s"
Sep 10 22:21:40.408446 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:21:40.408392    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4f8nr\" (UniqueName: \"kubernetes.io/projected/32794fee-1c78-4799-b55d-a6dfd8086493-kube-api-access-4f8nr\") pod \"auth-test-model-d07dfb-kserve-6764ddf5c9-vzq4s\" (UID: \"32794fee-1c78-4799-b55d-a6dfd8086493\") " pod="ai-tenant-e2e-shared-a-w3-d07dfb/auth-test-model-d07dfb-kserve-6764ddf5c9-vzq4s"
Sep 10 22:21:40.408576 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:21:40.408480    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32794fee-1c78-4799-b55d-a6dfd8086493-dshm\") pod \"auth-test-model-d07dfb-kserve-6764ddf5c9-vzq4s\" (UID: \"32794fee-1c78-4799-b55d-a6dfd8086493\") " pod="ai-tenant-e2e-shared-a-w3-d07dfb/auth-test-model-d07dfb-kserve-6764ddf5c9-vzq4s"
Sep 10 22:21:40.408576 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:21:40.408517    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32794fee-1c78-4799-b55d-a6dfd8086493-tls-certs\") pod \"auth-test-model-d07dfb-kserve-6764ddf5c9-vzq4s\" (UID: \"32794fee-1c78-4799-b55d-a6dfd8086493\") " pod="ai-tenant-e2e-shared-a-w3-d07dfb/auth-test-model-d07dfb-kserve-6764ddf5c9-vzq4s"
Sep 10 22:21:40.509601 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:21:40.509567    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32794fee-1c78-4799-b55d-a6dfd8086493-dshm\") pod \"auth-test-model-d07dfb-kserve-6764ddf5c9-vzq4s\" (UID: \"32794fee-1c78-4799-b55d-a6dfd8086493\") " pod="ai-tenant-e2e-shared-a-w3-d07dfb/auth-test-model-d07dfb-kserve-6764ddf5c9-vzq4s"
Sep 10 22:21:40.509800 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:21:40.509606    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32794fee-1c78-4799-b55d-a6dfd8086493-tls-certs\") pod \"auth-test-model-d07dfb-kserve-6764ddf5c9-vzq4s\" (UID: \"32794fee-1c78-4799-b55d-a6dfd8086493\") " pod="ai-tenant-e2e-shared-a-w3-d07dfb/auth-test-model-d07dfb-kserve-6764ddf5c9-vzq4s"
Sep 10 22:21:40.509800 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:21:40.509650    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32794fee-1c78-4799-b55d-a6dfd8086493-tmp-dir\") pod \"auth-test-model-d07dfb-kserve-6764ddf5c9-vzq4s\" (UID: \"32794fee-1c78-4799-b55d-a6dfd8086493\") " pod="ai-tenant-e2e-shared-a-w3-d07dfb/auth-test-model-d07dfb-kserve-6764ddf5c9-vzq4s"
Sep 10 22:21:40.509800 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:21:40.509700    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32794fee-1c78-4799-b55d-a6dfd8086493-model-cache\") pod \"auth-test-model-d07dfb-kserve-6764ddf5c9-vzq4s\" (UID: \"32794fee-1c78-4799-b55d-a6dfd8086493\") " pod="ai-tenant-e2e-shared-a-w3-d07dfb/auth-test-model-d07dfb-kserve-6764ddf5c9-vzq4s"
Sep 10 22:21:40.509800 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:21:40.509730    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32794fee-1c78-4799-b55d-a6dfd8086493-home\") pod \"auth-test-model-d07dfb-kserve-6764ddf5c9-vzq4s\" (UID: \"32794fee-1c78-4799-b55d-a6dfd8086493\") " pod="ai-tenant-e2e-shared-a-w3-d07dfb/auth-test-model-d07dfb-kserve-6764ddf5c9-vzq4s"
Sep 10 22:21:40.509800 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:21:40.509778    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4f8nr\" (UniqueName: \"kubernetes.io/projected/32794fee-1c78-4799-b55d-a6dfd8086493-kube-api-access-4f8nr\") pod \"auth-test-model-d07dfb-kserve-6764ddf5c9-vzq4s\" (UID: \"32794fee-1c78-4799-b55d-a6dfd8086493\") " pod="ai-tenant-e2e-shared-a-w3-d07dfb/auth-test-model-d07dfb-kserve-6764ddf5c9-vzq4s"
Sep 10 22:21:40.510191 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:21:40.510153    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32794fee-1c78-4799-b55d-a6dfd8086493-tmp-dir\") pod \"auth-test-model-d07dfb-kserve-6764ddf5c9-vzq4s\" (UID: \"32794fee-1c78-4799-b55d-a6dfd8086493\") " pod="ai-tenant-e2e-shared-a-w3-d07dfb/auth-test-model-d07dfb-kserve-6764ddf5c9-vzq4s"
Sep 10 22:21:40.510191 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:21:40.510182    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32794fee-1c78-4799-b55d-a6dfd8086493-home\") pod \"auth-test-model-d07dfb-kserve-6764ddf5c9-vzq4s\" (UID: \"32794fee-1c78-4799-b55d-a6dfd8086493\") " pod="ai-tenant-e2e-shared-a-w3-d07dfb/auth-test-model-d07dfb-kserve-6764ddf5c9-vzq4s"
Sep 10 22:21:40.510391 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:21:40.510350    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32794fee-1c78-4799-b55d-a6dfd8086493-model-cache\") pod \"auth-test-model-d07dfb-kserve-6764ddf5c9-vzq4s\" (UID: \"32794fee-1c78-4799-b55d-a6dfd8086493\") " pod="ai-tenant-e2e-shared-a-w3-d07dfb/auth-test-model-d07dfb-kserve-6764ddf5c9-vzq4s"
Sep 10 22:21:40.512100 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:21:40.512076    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32794fee-1c78-4799-b55d-a6dfd8086493-dshm\") pod \"auth-test-model-d07dfb-kserve-6764ddf5c9-vzq4s\" (UID: \"32794fee-1c78-4799-b55d-a6dfd8086493\") " pod="ai-tenant-e2e-shared-a-w3-d07dfb/auth-test-model-d07dfb-kserve-6764ddf5c9-vzq4s"
Sep 10 22:21:40.512227 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:21:40.512210    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32794fee-1c78-4799-b55d-a6dfd8086493-tls-certs\") pod \"auth-test-model-d07dfb-kserve-6764ddf5c9-vzq4s\" (UID: \"32794fee-1c78-4799-b55d-a6dfd8086493\") " pod="ai-tenant-e2e-shared-a-w3-d07dfb/auth-test-model-d07dfb-kserve-6764ddf5c9-vzq4s"
Sep 10 22:21:40.519184 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:21:40.519160    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4f8nr\" (UniqueName: \"kubernetes.io/projected/32794fee-1c78-4799-b55d-a6dfd8086493-kube-api-access-4f8nr\") pod \"auth-test-model-d07dfb-kserve-6764ddf5c9-vzq4s\" (UID: \"32794fee-1c78-4799-b55d-a6dfd8086493\") " pod="ai-tenant-e2e-shared-a-w3-d07dfb/auth-test-model-d07dfb-kserve-6764ddf5c9-vzq4s"
Sep 10 22:21:40.615670 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:21:40.615576    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-d07dfb/auth-test-model-d07dfb-kserve-6764ddf5c9-vzq4s"
Sep 10 22:21:40.898872 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:21:40.898819    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-d07dfb/auth-test-model-d07dfb-kserve-6764ddf5c9-vzq4s"]
Sep 10 22:21:40.902773 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:21:40.902738    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod32794fee_1c78_4799_b55d_a6dfd8086493.slice/crio-cda6c37579032b08dc1b07ff842f261fe624a4f3e003d262c01f050adf7ff1e4 WatchSource:0}: Error finding container cda6c37579032b08dc1b07ff842f261fe624a4f3e003d262c01f050adf7ff1e4: Status 404 returned error can't find the container with id cda6c37579032b08dc1b07ff842f261fe624a4f3e003d262c01f050adf7ff1e4
Sep 10 22:21:41.098429 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:21:41.098395    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-d07dfb/auth-test-model-d07dfb-kserve-6764ddf5c9-vzq4s" event={"ID":"32794fee-1c78-4799-b55d-a6dfd8086493","Type":"ContainerStarted","Data":"b621172f8e3feb00d73bfea48c3e006de323dca40a4449939b1e0daf70cceb49"}
Sep 10 22:21:41.098593 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:21:41.098436    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-d07dfb/auth-test-model-d07dfb-kserve-6764ddf5c9-vzq4s" event={"ID":"32794fee-1c78-4799-b55d-a6dfd8086493","Type":"ContainerStarted","Data":"cda6c37579032b08dc1b07ff842f261fe624a4f3e003d262c01f050adf7ff1e4"}
Sep 10 22:21:41.121070 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:21:41.121027    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-d07dfb/auth-test-model-d07dfb-kserve-6764ddf5c9-vzq4s" podStartSLOduration=1.121010586 podStartE2EDuration="1.121010586s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 22:21:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:21:41.119839267 +0000 UTC m=+1222.075906974" watchObservedRunningTime="2026-09-10 22:21:41.121010586 +0000 UTC m=+1222.077078292"
Sep 10 22:21:45.070403 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:21:45.070368    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-676c26ab-57b7db5c49-fskg2"
Sep 10 22:21:50.616255 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:21:50.616215    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-d07dfb/auth-test-model-d07dfb-kserve-6764ddf5c9-vzq4s"
Sep 10 22:21:50.616788 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:21:50.616357    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-d07dfb/auth-test-model-d07dfb-kserve-6764ddf5c9-vzq4s"
Sep 10 22:21:50.628980 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:21:50.628955    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-d07dfb/auth-test-model-d07dfb-kserve-6764ddf5c9-vzq4s"
Sep 10 22:21:51.159521 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:21:51.159487    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-d07dfb/auth-test-model-d07dfb-kserve-6764ddf5c9-vzq4s"
Sep 10 22:22:01.820211 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:22:01.820170    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-ec6d47/auth-test-model-ec6d47-kserve-6df9c55887-hckbr"]
Sep 10 22:22:01.823728 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:22:01.823704    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-ec6d47/auth-test-model-ec6d47-kserve-6df9c55887-hckbr"
Sep 10 22:22:01.827262 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:22:01.827241    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-ec6d47\"/\"kube-root-ca.crt\""
Sep 10 22:22:01.827939 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:22:01.827919    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-ec6d47\"/\"auth-test-model-ec6d47-kserve-self-signed-certs\""
Sep 10 22:22:01.828053 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:22:01.827923    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-ec6d47\"/\"openshift-service-ca.crt\""
Sep 10 22:22:01.828053 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:22:01.827923    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-ec6d47\"/\"default-dockercfg-vvdmr\""
Sep 10 22:22:01.866930 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:22:01.866900    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-ec6d47/auth-test-model-ec6d47-kserve-6df9c55887-hckbr"]
Sep 10 22:22:01.999505 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:22:01.999465    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b45375ed-a680-4d89-bde2-758ed6c24222-tmp-dir\") pod \"auth-test-model-ec6d47-kserve-6df9c55887-hckbr\" (UID: \"b45375ed-a680-4d89-bde2-758ed6c24222\") " pod="ai-tenant-e2e-shared-b-w3-ec6d47/auth-test-model-ec6d47-kserve-6df9c55887-hckbr"
Sep 10 22:22:01.999713 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:22:01.999518    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b45375ed-a680-4d89-bde2-758ed6c24222-tls-certs\") pod \"auth-test-model-ec6d47-kserve-6df9c55887-hckbr\" (UID: \"b45375ed-a680-4d89-bde2-758ed6c24222\") " pod="ai-tenant-e2e-shared-b-w3-ec6d47/auth-test-model-ec6d47-kserve-6df9c55887-hckbr"
Sep 10 22:22:01.999713 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:22:01.999547    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b45375ed-a680-4d89-bde2-758ed6c24222-home\") pod \"auth-test-model-ec6d47-kserve-6df9c55887-hckbr\" (UID: \"b45375ed-a680-4d89-bde2-758ed6c24222\") " pod="ai-tenant-e2e-shared-b-w3-ec6d47/auth-test-model-ec6d47-kserve-6df9c55887-hckbr"
Sep 10 22:22:01.999713 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:22:01.999647    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdtl8\" (UniqueName: \"kubernetes.io/projected/b45375ed-a680-4d89-bde2-758ed6c24222-kube-api-access-fdtl8\") pod \"auth-test-model-ec6d47-kserve-6df9c55887-hckbr\" (UID: \"b45375ed-a680-4d89-bde2-758ed6c24222\") " pod="ai-tenant-e2e-shared-b-w3-ec6d47/auth-test-model-ec6d47-kserve-6df9c55887-hckbr"
Sep 10 22:22:01.999713 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:22:01.999705    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b45375ed-a680-4d89-bde2-758ed6c24222-dshm\") pod \"auth-test-model-ec6d47-kserve-6df9c55887-hckbr\" (UID: \"b45375ed-a680-4d89-bde2-758ed6c24222\") " pod="ai-tenant-e2e-shared-b-w3-ec6d47/auth-test-model-ec6d47-kserve-6df9c55887-hckbr"
Sep 10 22:22:01.999918 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:22:01.999765    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b45375ed-a680-4d89-bde2-758ed6c24222-model-cache\") pod \"auth-test-model-ec6d47-kserve-6df9c55887-hckbr\" (UID: \"b45375ed-a680-4d89-bde2-758ed6c24222\") " pod="ai-tenant-e2e-shared-b-w3-ec6d47/auth-test-model-ec6d47-kserve-6df9c55887-hckbr"
Sep 10 22:22:02.100648 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:22:02.100545    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b45375ed-a680-4d89-bde2-758ed6c24222-tmp-dir\") pod \"auth-test-model-ec6d47-kserve-6df9c55887-hckbr\" (UID: \"b45375ed-a680-4d89-bde2-758ed6c24222\") " pod="ai-tenant-e2e-shared-b-w3-ec6d47/auth-test-model-ec6d47-kserve-6df9c55887-hckbr"
Sep 10 22:22:02.100648 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:22:02.100602    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b45375ed-a680-4d89-bde2-758ed6c24222-tls-certs\") pod \"auth-test-model-ec6d47-kserve-6df9c55887-hckbr\" (UID: \"b45375ed-a680-4d89-bde2-758ed6c24222\") " pod="ai-tenant-e2e-shared-b-w3-ec6d47/auth-test-model-ec6d47-kserve-6df9c55887-hckbr"
Sep 10 22:22:02.100844 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:22:02.100726    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b45375ed-a680-4d89-bde2-758ed6c24222-home\") pod \"auth-test-model-ec6d47-kserve-6df9c55887-hckbr\" (UID: \"b45375ed-a680-4d89-bde2-758ed6c24222\") " pod="ai-tenant-e2e-shared-b-w3-ec6d47/auth-test-model-ec6d47-kserve-6df9c55887-hckbr"
Sep 10 22:22:02.100844 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:22:02.100787    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdtl8\" (UniqueName: \"kubernetes.io/projected/b45375ed-a680-4d89-bde2-758ed6c24222-kube-api-access-fdtl8\") pod \"auth-test-model-ec6d47-kserve-6df9c55887-hckbr\" (UID: \"b45375ed-a680-4d89-bde2-758ed6c24222\") " pod="ai-tenant-e2e-shared-b-w3-ec6d47/auth-test-model-ec6d47-kserve-6df9c55887-hckbr"
Sep 10 22:22:02.100958 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:22:02.100840    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b45375ed-a680-4d89-bde2-758ed6c24222-dshm\") pod \"auth-test-model-ec6d47-kserve-6df9c55887-hckbr\" (UID: \"b45375ed-a680-4d89-bde2-758ed6c24222\") " pod="ai-tenant-e2e-shared-b-w3-ec6d47/auth-test-model-ec6d47-kserve-6df9c55887-hckbr"
Sep 10 22:22:02.100958 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:22:02.100945    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b45375ed-a680-4d89-bde2-758ed6c24222-model-cache\") pod \"auth-test-model-ec6d47-kserve-6df9c55887-hckbr\" (UID: \"b45375ed-a680-4d89-bde2-758ed6c24222\") " pod="ai-tenant-e2e-shared-b-w3-ec6d47/auth-test-model-ec6d47-kserve-6df9c55887-hckbr"
Sep 10 22:22:02.101050 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:22:02.100990    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b45375ed-a680-4d89-bde2-758ed6c24222-tmp-dir\") pod \"auth-test-model-ec6d47-kserve-6df9c55887-hckbr\" (UID: \"b45375ed-a680-4d89-bde2-758ed6c24222\") " pod="ai-tenant-e2e-shared-b-w3-ec6d47/auth-test-model-ec6d47-kserve-6df9c55887-hckbr"
Sep 10 22:22:02.101134 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:22:02.101115    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b45375ed-a680-4d89-bde2-758ed6c24222-home\") pod \"auth-test-model-ec6d47-kserve-6df9c55887-hckbr\" (UID: \"b45375ed-a680-4d89-bde2-758ed6c24222\") " pod="ai-tenant-e2e-shared-b-w3-ec6d47/auth-test-model-ec6d47-kserve-6df9c55887-hckbr"
Sep 10 22:22:02.101256 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:22:02.101239    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b45375ed-a680-4d89-bde2-758ed6c24222-model-cache\") pod \"auth-test-model-ec6d47-kserve-6df9c55887-hckbr\" (UID: \"b45375ed-a680-4d89-bde2-758ed6c24222\") " pod="ai-tenant-e2e-shared-b-w3-ec6d47/auth-test-model-ec6d47-kserve-6df9c55887-hckbr"
Sep 10 22:22:02.103012 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:22:02.102982    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b45375ed-a680-4d89-bde2-758ed6c24222-dshm\") pod \"auth-test-model-ec6d47-kserve-6df9c55887-hckbr\" (UID: \"b45375ed-a680-4d89-bde2-758ed6c24222\") " pod="ai-tenant-e2e-shared-b-w3-ec6d47/auth-test-model-ec6d47-kserve-6df9c55887-hckbr"
Sep 10 22:22:02.103191 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:22:02.103174    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b45375ed-a680-4d89-bde2-758ed6c24222-tls-certs\") pod \"auth-test-model-ec6d47-kserve-6df9c55887-hckbr\" (UID: \"b45375ed-a680-4d89-bde2-758ed6c24222\") " pod="ai-tenant-e2e-shared-b-w3-ec6d47/auth-test-model-ec6d47-kserve-6df9c55887-hckbr"
Sep 10 22:22:02.110054 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:22:02.110030    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdtl8\" (UniqueName: \"kubernetes.io/projected/b45375ed-a680-4d89-bde2-758ed6c24222-kube-api-access-fdtl8\") pod \"auth-test-model-ec6d47-kserve-6df9c55887-hckbr\" (UID: \"b45375ed-a680-4d89-bde2-758ed6c24222\") " pod="ai-tenant-e2e-shared-b-w3-ec6d47/auth-test-model-ec6d47-kserve-6df9c55887-hckbr"
Sep 10 22:22:02.133863 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:22:02.133823    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-ec6d47/auth-test-model-ec6d47-kserve-6df9c55887-hckbr"
Sep 10 22:22:02.406196 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:22:02.406171    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-ec6d47/auth-test-model-ec6d47-kserve-6df9c55887-hckbr"]
Sep 10 22:22:02.408316 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:22:02.408279    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb45375ed_a680_4d89_bde2_758ed6c24222.slice/crio-95a66e6e0a01cfb6779cd8cfeafb31f4721936eb75530631c59fcfc9bf671c12 WatchSource:0}: Error finding container 95a66e6e0a01cfb6779cd8cfeafb31f4721936eb75530631c59fcfc9bf671c12: Status 404 returned error can't find the container with id 95a66e6e0a01cfb6779cd8cfeafb31f4721936eb75530631c59fcfc9bf671c12
Sep 10 22:22:03.210455 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:22:03.210411    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-ec6d47/auth-test-model-ec6d47-kserve-6df9c55887-hckbr" event={"ID":"b45375ed-a680-4d89-bde2-758ed6c24222","Type":"ContainerStarted","Data":"fb591d508ca7ee5093a1686fbb43168d6b30eb7b4f66b32f39081a7eb76b5947"}
Sep 10 22:22:03.210455 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:22:03.210458    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-ec6d47/auth-test-model-ec6d47-kserve-6df9c55887-hckbr" event={"ID":"b45375ed-a680-4d89-bde2-758ed6c24222","Type":"ContainerStarted","Data":"95a66e6e0a01cfb6779cd8cfeafb31f4721936eb75530631c59fcfc9bf671c12"}
Sep 10 22:22:03.229085 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:22:03.229032    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-ec6d47/auth-test-model-ec6d47-kserve-6df9c55887-hckbr" podStartSLOduration=2.229016188 podStartE2EDuration="2.229016188s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 22:22:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:22:03.227442537 +0000 UTC m=+1244.183510244" watchObservedRunningTime="2026-09-10 22:22:03.229016188 +0000 UTC m=+1244.185083893"
Sep 10 22:22:12.134335 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:22:12.134247    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-ec6d47/auth-test-model-ec6d47-kserve-6df9c55887-hckbr"
Sep 10 22:22:12.134335 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:22:12.134309    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-ec6d47/auth-test-model-ec6d47-kserve-6df9c55887-hckbr"
Sep 10 22:22:12.146979 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:22:12.146943    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-ec6d47/auth-test-model-ec6d47-kserve-6df9c55887-hckbr"
Sep 10 22:22:12.264752 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:22:12.264720    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-ec6d47/auth-test-model-ec6d47-kserve-6df9c55887-hckbr"
Sep 10 22:22:15.608395 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:22:15.608359    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-676c26ab/e2e-lifecycle-model-676c26ab-kserve-777db9ddbc-w9vbm"]
Sep 10 22:22:15.612973 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:22:15.612954    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-676c26ab/e2e-lifecycle-model-676c26ab-kserve-777db9ddbc-w9vbm"
Sep 10 22:22:15.616741 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:22:15.616721    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-676c26ab\"/\"default-dockercfg-7lklx\""
Sep 10 22:22:15.616875 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:22:15.616765    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-676c26ab\"/\"e2e-lifecycle-model-676c26ab-kserve-self-signed-certs\""
Sep 10 22:22:15.616875 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:22:15.616765    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-676c26ab\"/\"openshift-service-ca.crt\""
Sep 10 22:22:15.616875 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:22:15.616794    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-676c26ab\"/\"kube-root-ca.crt\""
Sep 10 22:22:15.624069 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:22:15.624042    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-676c26ab/e2e-lifecycle-model-676c26ab-kserve-777db9ddbc-w9vbm"]
Sep 10 22:22:15.714401 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:22:15.714351    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2aeb5cc-525f-4a64-919d-ec15adc61bbc-model-cache\") pod \"e2e-lifecycle-model-676c26ab-kserve-777db9ddbc-w9vbm\" (UID: \"a2aeb5cc-525f-4a64-919d-ec15adc61bbc\") " pod="ai-tenant-e2e-mt-676c26ab/e2e-lifecycle-model-676c26ab-kserve-777db9ddbc-w9vbm"
Sep 10 22:22:15.714401 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:22:15.714406    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2aeb5cc-525f-4a64-919d-ec15adc61bbc-tmp-dir\") pod \"e2e-lifecycle-model-676c26ab-kserve-777db9ddbc-w9vbm\" (UID: \"a2aeb5cc-525f-4a64-919d-ec15adc61bbc\") " pod="ai-tenant-e2e-mt-676c26ab/e2e-lifecycle-model-676c26ab-kserve-777db9ddbc-w9vbm"
Sep 10 22:22:15.714625 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:22:15.714471    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2aeb5cc-525f-4a64-919d-ec15adc61bbc-dshm\") pod \"e2e-lifecycle-model-676c26ab-kserve-777db9ddbc-w9vbm\" (UID: \"a2aeb5cc-525f-4a64-919d-ec15adc61bbc\") " pod="ai-tenant-e2e-mt-676c26ab/e2e-lifecycle-model-676c26ab-kserve-777db9ddbc-w9vbm"
Sep 10 22:22:15.714625 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:22:15.714533    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9fz5\" (UniqueName: \"kubernetes.io/projected/a2aeb5cc-525f-4a64-919d-ec15adc61bbc-kube-api-access-p9fz5\") pod \"e2e-lifecycle-model-676c26ab-kserve-777db9ddbc-w9vbm\" (UID: \"a2aeb5cc-525f-4a64-919d-ec15adc61bbc\") " pod="ai-tenant-e2e-mt-676c26ab/e2e-lifecycle-model-676c26ab-kserve-777db9ddbc-w9vbm"
Sep 10 22:22:15.714625 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:22:15.714591    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2aeb5cc-525f-4a64-919d-ec15adc61bbc-home\") pod \"e2e-lifecycle-model-676c26ab-kserve-777db9ddbc-w9vbm\" (UID: \"a2aeb5cc-525f-4a64-919d-ec15adc61bbc\") " pod="ai-tenant-e2e-mt-676c26ab/e2e-lifecycle-model-676c26ab-kserve-777db9ddbc-w9vbm"
Sep 10 22:22:15.714753 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:22:15.714631    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2aeb5cc-525f-4a64-919d-ec15adc61bbc-tls-certs\") pod \"e2e-lifecycle-model-676c26ab-kserve-777db9ddbc-w9vbm\" (UID: \"a2aeb5cc-525f-4a64-919d-ec15adc61bbc\") " pod="ai-tenant-e2e-mt-676c26ab/e2e-lifecycle-model-676c26ab-kserve-777db9ddbc-w9vbm"
Sep 10 22:22:15.815923 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:22:15.815876    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9fz5\" (UniqueName: \"kubernetes.io/projected/a2aeb5cc-525f-4a64-919d-ec15adc61bbc-kube-api-access-p9fz5\") pod \"e2e-lifecycle-model-676c26ab-kserve-777db9ddbc-w9vbm\" (UID: \"a2aeb5cc-525f-4a64-919d-ec15adc61bbc\") " pod="ai-tenant-e2e-mt-676c26ab/e2e-lifecycle-model-676c26ab-kserve-777db9ddbc-w9vbm"
Sep 10 22:22:15.816158 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:22:15.815962    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2aeb5cc-525f-4a64-919d-ec15adc61bbc-home\") pod \"e2e-lifecycle-model-676c26ab-kserve-777db9ddbc-w9vbm\" (UID: \"a2aeb5cc-525f-4a64-919d-ec15adc61bbc\") " pod="ai-tenant-e2e-mt-676c26ab/e2e-lifecycle-model-676c26ab-kserve-777db9ddbc-w9vbm"
Sep 10 22:22:15.816158 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:22:15.816015    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2aeb5cc-525f-4a64-919d-ec15adc61bbc-tls-certs\") pod \"e2e-lifecycle-model-676c26ab-kserve-777db9ddbc-w9vbm\" (UID: \"a2aeb5cc-525f-4a64-919d-ec15adc61bbc\") " pod="ai-tenant-e2e-mt-676c26ab/e2e-lifecycle-model-676c26ab-kserve-777db9ddbc-w9vbm"
Sep 10 22:22:15.816158 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:22:15.816089    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2aeb5cc-525f-4a64-919d-ec15adc61bbc-model-cache\") pod \"e2e-lifecycle-model-676c26ab-kserve-777db9ddbc-w9vbm\" (UID: \"a2aeb5cc-525f-4a64-919d-ec15adc61bbc\") " pod="ai-tenant-e2e-mt-676c26ab/e2e-lifecycle-model-676c26ab-kserve-777db9ddbc-w9vbm"
Sep 10 22:22:15.816158 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:22:15.816120    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2aeb5cc-525f-4a64-919d-ec15adc61bbc-tmp-dir\") pod \"e2e-lifecycle-model-676c26ab-kserve-777db9ddbc-w9vbm\" (UID: \"a2aeb5cc-525f-4a64-919d-ec15adc61bbc\") " pod="ai-tenant-e2e-mt-676c26ab/e2e-lifecycle-model-676c26ab-kserve-777db9ddbc-w9vbm"
Sep 10 22:22:15.816158 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:22:15.816144    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2aeb5cc-525f-4a64-919d-ec15adc61bbc-dshm\") pod \"e2e-lifecycle-model-676c26ab-kserve-777db9ddbc-w9vbm\" (UID: \"a2aeb5cc-525f-4a64-919d-ec15adc61bbc\") " pod="ai-tenant-e2e-mt-676c26ab/e2e-lifecycle-model-676c26ab-kserve-777db9ddbc-w9vbm"
Sep 10 22:22:15.816510 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:22:15.816434    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2aeb5cc-525f-4a64-919d-ec15adc61bbc-home\") pod \"e2e-lifecycle-model-676c26ab-kserve-777db9ddbc-w9vbm\" (UID: \"a2aeb5cc-525f-4a64-919d-ec15adc61bbc\") " pod="ai-tenant-e2e-mt-676c26ab/e2e-lifecycle-model-676c26ab-kserve-777db9ddbc-w9vbm"
Sep 10 22:22:15.816510 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:22:15.816479    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2aeb5cc-525f-4a64-919d-ec15adc61bbc-tmp-dir\") pod \"e2e-lifecycle-model-676c26ab-kserve-777db9ddbc-w9vbm\" (UID: \"a2aeb5cc-525f-4a64-919d-ec15adc61bbc\") " pod="ai-tenant-e2e-mt-676c26ab/e2e-lifecycle-model-676c26ab-kserve-777db9ddbc-w9vbm"
Sep 10 22:22:15.816510 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:22:15.816505    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2aeb5cc-525f-4a64-919d-ec15adc61bbc-model-cache\") pod \"e2e-lifecycle-model-676c26ab-kserve-777db9ddbc-w9vbm\" (UID: \"a2aeb5cc-525f-4a64-919d-ec15adc61bbc\") " pod="ai-tenant-e2e-mt-676c26ab/e2e-lifecycle-model-676c26ab-kserve-777db9ddbc-w9vbm"
Sep 10 22:22:15.818400 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:22:15.818372    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2aeb5cc-525f-4a64-919d-ec15adc61bbc-dshm\") pod \"e2e-lifecycle-model-676c26ab-kserve-777db9ddbc-w9vbm\" (UID: \"a2aeb5cc-525f-4a64-919d-ec15adc61bbc\") " pod="ai-tenant-e2e-mt-676c26ab/e2e-lifecycle-model-676c26ab-kserve-777db9ddbc-w9vbm"
Sep 10 22:22:15.818654 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:22:15.818639    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2aeb5cc-525f-4a64-919d-ec15adc61bbc-tls-certs\") pod \"e2e-lifecycle-model-676c26ab-kserve-777db9ddbc-w9vbm\" (UID: \"a2aeb5cc-525f-4a64-919d-ec15adc61bbc\") " pod="ai-tenant-e2e-mt-676c26ab/e2e-lifecycle-model-676c26ab-kserve-777db9ddbc-w9vbm"
Sep 10 22:22:15.826106 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:22:15.826081    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9fz5\" (UniqueName: \"kubernetes.io/projected/a2aeb5cc-525f-4a64-919d-ec15adc61bbc-kube-api-access-p9fz5\") pod \"e2e-lifecycle-model-676c26ab-kserve-777db9ddbc-w9vbm\" (UID: \"a2aeb5cc-525f-4a64-919d-ec15adc61bbc\") " pod="ai-tenant-e2e-mt-676c26ab/e2e-lifecycle-model-676c26ab-kserve-777db9ddbc-w9vbm"
Sep 10 22:22:15.922756 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:22:15.922705    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-676c26ab/e2e-lifecycle-model-676c26ab-kserve-777db9ddbc-w9vbm"
Sep 10 22:22:16.199117 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:22:16.199044    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-676c26ab/e2e-lifecycle-model-676c26ab-kserve-777db9ddbc-w9vbm"]
Sep 10 22:22:16.202375 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:22:16.202347    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda2aeb5cc_525f_4a64_919d_ec15adc61bbc.slice/crio-2f0f668628dbcd94dcc7096c605425c56942f2e0f7d8c97ac6bec1f30efe888b WatchSource:0}: Error finding container 2f0f668628dbcd94dcc7096c605425c56942f2e0f7d8c97ac6bec1f30efe888b: Status 404 returned error can't find the container with id 2f0f668628dbcd94dcc7096c605425c56942f2e0f7d8c97ac6bec1f30efe888b
Sep 10 22:22:16.275003 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:22:16.274968    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-676c26ab/e2e-lifecycle-model-676c26ab-kserve-777db9ddbc-w9vbm" event={"ID":"a2aeb5cc-525f-4a64-919d-ec15adc61bbc","Type":"ContainerStarted","Data":"077741c9e4f015aa0ad409dfb7b2d97402b93d628916be5250883f597c0c10ef"}
Sep 10 22:22:16.275160 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:22:16.275023    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-676c26ab/e2e-lifecycle-model-676c26ab-kserve-777db9ddbc-w9vbm" event={"ID":"a2aeb5cc-525f-4a64-919d-ec15adc61bbc","Type":"ContainerStarted","Data":"2f0f668628dbcd94dcc7096c605425c56942f2e0f7d8c97ac6bec1f30efe888b"}
Sep 10 22:22:16.292075 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:22:16.292024    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-676c26ab/e2e-lifecycle-model-676c26ab-kserve-777db9ddbc-w9vbm" podStartSLOduration=1.292005385 podStartE2EDuration="1.292005385s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 22:22:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:22:16.291189866 +0000 UTC m=+1257.247257596" watchObservedRunningTime="2026-09-10 22:22:16.292005385 +0000 UTC m=+1257.248073091"
Sep 10 22:22:19.659901 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:22:19.659872    2582 scope.go:117] "RemoveContainer" containerID="004ce21bd2aa5ccd1e8f029f17d8d66c43bae2265325ede95af68b1b07d0a1e9"
Sep 10 22:22:25.922944 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:22:25.922908    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-676c26ab/e2e-lifecycle-model-676c26ab-kserve-777db9ddbc-w9vbm"
Sep 10 22:22:25.923344 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:22:25.922956    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-676c26ab/e2e-lifecycle-model-676c26ab-kserve-777db9ddbc-w9vbm"
Sep 10 22:22:25.935270 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:22:25.935246    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-676c26ab/e2e-lifecycle-model-676c26ab-kserve-777db9ddbc-w9vbm"
Sep 10 22:22:26.339614 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:22:26.339531    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-676c26ab/e2e-lifecycle-model-676c26ab-kserve-777db9ddbc-w9vbm"
Sep 10 22:22:50.778099 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:22:50.778057    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-676c26ab-4tmjv"]
Sep 10 22:22:50.781669 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:22:50.781644    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-676c26ab-4tmjv"
Sep 10 22:22:50.784817 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:22:50.784798    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 22:22:50.784948 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:22:50.784930    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8hjrv\""
Sep 10 22:22:50.794742 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:22:50.794718    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-676c26ab-4tmjv"]
Sep 10 22:22:50.928785 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:22:50.928757    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/441b2f44-f770-41cf-87ce-0927bee26491-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-676c26ab-4tmjv\" (UID: \"441b2f44-f770-41cf-87ce-0927bee26491\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-676c26ab-4tmjv"
Sep 10 22:22:51.030077 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:22:51.029999    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/441b2f44-f770-41cf-87ce-0927bee26491-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-676c26ab-4tmjv\" (UID: \"441b2f44-f770-41cf-87ce-0927bee26491\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-676c26ab-4tmjv"
Sep 10 22:22:51.030659 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:22:51.030626    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/441b2f44-f770-41cf-87ce-0927bee26491-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-676c26ab-4tmjv\" (UID: \"441b2f44-f770-41cf-87ce-0927bee26491\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-676c26ab-4tmjv"
Sep 10 22:22:51.092268 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:22:51.092245    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-676c26ab-4tmjv"
Sep 10 22:22:51.375375 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:22:51.375350    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-676c26ab-4tmjv"]
Sep 10 22:22:51.377945 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:22:51.377919    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod441b2f44_f770_41cf_87ce_0927bee26491.slice/crio-2e40ddd6737c4e9c24127e52ac2b7b0439a7b5e6d11c286b58f8b1c8c9e91319 WatchSource:0}: Error finding container 2e40ddd6737c4e9c24127e52ac2b7b0439a7b5e6d11c286b58f8b1c8c9e91319: Status 404 returned error can't find the container with id 2e40ddd6737c4e9c24127e52ac2b7b0439a7b5e6d11c286b58f8b1c8c9e91319
Sep 10 22:22:51.449545 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:22:51.449518    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-676c26ab-4tmjv" event={"ID":"441b2f44-f770-41cf-87ce-0927bee26491","Type":"ContainerStarted","Data":"2e40ddd6737c4e9c24127e52ac2b7b0439a7b5e6d11c286b58f8b1c8c9e91319"}
Sep 10 22:22:52.462458 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:22:52.462412    2582 generic.go:356] "Generic (PLEG): container finished" podID="441b2f44-f770-41cf-87ce-0927bee26491" containerID="b411ee29f9789d9d9799fb8c1a81c1eb6b084dd7ac7547cf4f90c221f5251653" exitCode=0
Sep 10 22:22:52.462941 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:22:52.462464    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-676c26ab-4tmjv" event={"ID":"441b2f44-f770-41cf-87ce-0927bee26491","Type":"ContainerDied","Data":"b411ee29f9789d9d9799fb8c1a81c1eb6b084dd7ac7547cf4f90c221f5251653"}
Sep 10 22:22:53.593518 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:22:53.593496    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-676c26ab-4tmjv"
Sep 10 22:22:53.753542 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:22:53.753472    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/441b2f44-f770-41cf-87ce-0927bee26491-maas-api-service-ca\") pod \"441b2f44-f770-41cf-87ce-0927bee26491\" (UID: \"441b2f44-f770-41cf-87ce-0927bee26491\") "
Sep 10 22:22:53.753840 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:22:53.753819    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/441b2f44-f770-41cf-87ce-0927bee26491-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "441b2f44-f770-41cf-87ce-0927bee26491" (UID: "441b2f44-f770-41cf-87ce-0927bee26491"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 22:22:53.855028 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:22:53.855004    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/441b2f44-f770-41cf-87ce-0927bee26491-maas-api-service-ca\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:22:54.474326 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:22:54.474291    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-676c26ab-4tmjv" event={"ID":"441b2f44-f770-41cf-87ce-0927bee26491","Type":"ContainerDied","Data":"2e40ddd6737c4e9c24127e52ac2b7b0439a7b5e6d11c286b58f8b1c8c9e91319"}
Sep 10 22:22:54.474326 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:22:54.474327    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-676c26ab-4tmjv"
Sep 10 22:22:54.474533 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:22:54.474341    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2e40ddd6737c4e9c24127e52ac2b7b0439a7b5e6d11c286b58f8b1c8c9e91319"
Sep 10 22:23:06.750118 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:23:06.750077    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-676c26ab-57b7db5c49-fskg2"]
Sep 10 22:23:06.750619 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:23:06.750295    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-676c26ab-57b7db5c49-fskg2" podUID="4ce4bed6-f1db-42dc-a2a7-86d398926c24" containerName="payload-processing" containerID="cri-o://8642e0f2b641cdcd46edb8f62bc02e58700dca0fb5c7bdffab9546eaf38aa70c" gracePeriod=30
Sep 10 22:23:06.954935 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:23:06.954905    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-676c26ab-4tmjv"]
Sep 10 22:23:06.957549 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:23:06.957523    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-676c26ab-4tmjv"]
Sep 10 22:23:07.006129 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:23:07.006083    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-676c26ab-57b7db5c49-fskg2"
Sep 10 22:23:07.069910 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:23:07.069884    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jvw9v\" (UniqueName: \"kubernetes.io/projected/4ce4bed6-f1db-42dc-a2a7-86d398926c24-kube-api-access-jvw9v\") pod \"4ce4bed6-f1db-42dc-a2a7-86d398926c24\" (UID: \"4ce4bed6-f1db-42dc-a2a7-86d398926c24\") "
Sep 10 22:23:07.070040 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:23:07.069996    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ce4bed6-f1db-42dc-a2a7-86d398926c24-plugins-config-volume\") pod \"4ce4bed6-f1db-42dc-a2a7-86d398926c24\" (UID: \"4ce4bed6-f1db-42dc-a2a7-86d398926c24\") "
Sep 10 22:23:07.070335 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:23:07.070306    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ce4bed6-f1db-42dc-a2a7-86d398926c24-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4ce4bed6-f1db-42dc-a2a7-86d398926c24" (UID: "4ce4bed6-f1db-42dc-a2a7-86d398926c24"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 22:23:07.071805 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:23:07.071786    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ce4bed6-f1db-42dc-a2a7-86d398926c24-kube-api-access-jvw9v" (OuterVolumeSpecName: "kube-api-access-jvw9v") pod "4ce4bed6-f1db-42dc-a2a7-86d398926c24" (UID: "4ce4bed6-f1db-42dc-a2a7-86d398926c24"). InnerVolumeSpecName "kube-api-access-jvw9v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 22:23:07.170898 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:23:07.170844    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ce4bed6-f1db-42dc-a2a7-86d398926c24-plugins-config-volume\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:23:07.170898 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:23:07.170895    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvw9v\" (UniqueName: \"kubernetes.io/projected/4ce4bed6-f1db-42dc-a2a7-86d398926c24-kube-api-access-jvw9v\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:23:07.498259 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:23:07.498220    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="441b2f44-f770-41cf-87ce-0927bee26491" path="/var/lib/kubelet/pods/441b2f44-f770-41cf-87ce-0927bee26491/volumes"
Sep 10 22:23:07.537479 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:23:07.537451    2582 generic.go:356] "Generic (PLEG): container finished" podID="4ce4bed6-f1db-42dc-a2a7-86d398926c24" containerID="8642e0f2b641cdcd46edb8f62bc02e58700dca0fb5c7bdffab9546eaf38aa70c" exitCode=0
Sep 10 22:23:07.537617 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:23:07.537493    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-676c26ab-57b7db5c49-fskg2"
Sep 10 22:23:07.537617 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:23:07.537541    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-676c26ab-57b7db5c49-fskg2" event={"ID":"4ce4bed6-f1db-42dc-a2a7-86d398926c24","Type":"ContainerDied","Data":"8642e0f2b641cdcd46edb8f62bc02e58700dca0fb5c7bdffab9546eaf38aa70c"}
Sep 10 22:23:07.537617 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:23:07.537592    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-676c26ab-57b7db5c49-fskg2" event={"ID":"4ce4bed6-f1db-42dc-a2a7-86d398926c24","Type":"ContainerDied","Data":"17b6718263c2fcc13e73a4e09e3d3f936ba80cac9cb7b7d95c99c1d122df2a47"}
Sep 10 22:23:07.537746 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:23:07.537620    2582 scope.go:117] "RemoveContainer" containerID="8642e0f2b641cdcd46edb8f62bc02e58700dca0fb5c7bdffab9546eaf38aa70c"
Sep 10 22:23:07.546363 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:23:07.546339    2582 scope.go:117] "RemoveContainer" containerID="8642e0f2b641cdcd46edb8f62bc02e58700dca0fb5c7bdffab9546eaf38aa70c"
Sep 10 22:23:07.546632 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:23:07.546613    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8642e0f2b641cdcd46edb8f62bc02e58700dca0fb5c7bdffab9546eaf38aa70c\": container with ID starting with 8642e0f2b641cdcd46edb8f62bc02e58700dca0fb5c7bdffab9546eaf38aa70c not found: ID does not exist" containerID="8642e0f2b641cdcd46edb8f62bc02e58700dca0fb5c7bdffab9546eaf38aa70c"
Sep 10 22:23:07.546681 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:23:07.546640    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8642e0f2b641cdcd46edb8f62bc02e58700dca0fb5c7bdffab9546eaf38aa70c"} err="failed to get container status \"8642e0f2b641cdcd46edb8f62bc02e58700dca0fb5c7bdffab9546eaf38aa70c\": rpc error: code = NotFound desc = could not find container \"8642e0f2b641cdcd46edb8f62bc02e58700dca0fb5c7bdffab9546eaf38aa70c\": container with ID starting with 8642e0f2b641cdcd46edb8f62bc02e58700dca0fb5c7bdffab9546eaf38aa70c not found: ID does not exist"
Sep 10 22:23:07.555025 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:23:07.555002    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-676c26ab-57b7db5c49-fskg2"]
Sep 10 22:23:07.558683 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:23:07.558659    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-676c26ab-57b7db5c49-fskg2"]
Sep 10 22:23:09.500182 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:23:09.500146    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4ce4bed6-f1db-42dc-a2a7-86d398926c24" path="/var/lib/kubelet/pods/4ce4bed6-f1db-42dc-a2a7-86d398926c24/volumes"
Sep 10 22:23:15.498823 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:23:15.498783    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-676c26ab/e2e-lifecycle-model-676c26ab-kserve-777db9ddbc-w9vbm"]
Sep 10 22:23:30.518207 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:23:30.518142    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-676c26ab/e2e-lifecycle-model-676c26ab-kserve-777db9ddbc-w9vbm" podUID="a2aeb5cc-525f-4a64-919d-ec15adc61bbc" containerName="main" containerID="cri-o://077741c9e4f015aa0ad409dfb7b2d97402b93d628916be5250883f597c0c10ef" gracePeriod=45
Sep 10 22:23:30.650173 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:23:30.650142    2582 generic.go:356] "Generic (PLEG): container finished" podID="a2aeb5cc-525f-4a64-919d-ec15adc61bbc" containerID="077741c9e4f015aa0ad409dfb7b2d97402b93d628916be5250883f597c0c10ef" exitCode=0
Sep 10 22:23:30.650332 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:23:30.650201    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-676c26ab/e2e-lifecycle-model-676c26ab-kserve-777db9ddbc-w9vbm" event={"ID":"a2aeb5cc-525f-4a64-919d-ec15adc61bbc","Type":"ContainerDied","Data":"077741c9e4f015aa0ad409dfb7b2d97402b93d628916be5250883f597c0c10ef"}
Sep 10 22:23:30.783757 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:23:30.783683    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-676c26ab/e2e-lifecycle-model-676c26ab-kserve-777db9ddbc-w9vbm"
Sep 10 22:23:30.877053 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:23:30.877003    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2aeb5cc-525f-4a64-919d-ec15adc61bbc-tls-certs\") pod \"a2aeb5cc-525f-4a64-919d-ec15adc61bbc\" (UID: \"a2aeb5cc-525f-4a64-919d-ec15adc61bbc\") "
Sep 10 22:23:30.877256 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:23:30.877080    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2aeb5cc-525f-4a64-919d-ec15adc61bbc-model-cache\") pod \"a2aeb5cc-525f-4a64-919d-ec15adc61bbc\" (UID: \"a2aeb5cc-525f-4a64-919d-ec15adc61bbc\") "
Sep 10 22:23:30.877256 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:23:30.877191    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2aeb5cc-525f-4a64-919d-ec15adc61bbc-tmp-dir\") pod \"a2aeb5cc-525f-4a64-919d-ec15adc61bbc\" (UID: \"a2aeb5cc-525f-4a64-919d-ec15adc61bbc\") "
Sep 10 22:23:30.877380 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:23:30.877309    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2aeb5cc-525f-4a64-919d-ec15adc61bbc-home\") pod \"a2aeb5cc-525f-4a64-919d-ec15adc61bbc\" (UID: \"a2aeb5cc-525f-4a64-919d-ec15adc61bbc\") "
Sep 10 22:23:30.877380 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:23:30.877358    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2aeb5cc-525f-4a64-919d-ec15adc61bbc-model-cache" (OuterVolumeSpecName: "model-cache") pod "a2aeb5cc-525f-4a64-919d-ec15adc61bbc" (UID: "a2aeb5cc-525f-4a64-919d-ec15adc61bbc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 22:23:30.877380 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:23:30.877373    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2aeb5cc-525f-4a64-919d-ec15adc61bbc-dshm\") pod \"a2aeb5cc-525f-4a64-919d-ec15adc61bbc\" (UID: \"a2aeb5cc-525f-4a64-919d-ec15adc61bbc\") "
Sep 10 22:23:30.877548 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:23:30.877453    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p9fz5\" (UniqueName: \"kubernetes.io/projected/a2aeb5cc-525f-4a64-919d-ec15adc61bbc-kube-api-access-p9fz5\") pod \"a2aeb5cc-525f-4a64-919d-ec15adc61bbc\" (UID: \"a2aeb5cc-525f-4a64-919d-ec15adc61bbc\") "
Sep 10 22:23:30.877686 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:23:30.877623    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2aeb5cc-525f-4a64-919d-ec15adc61bbc-home" (OuterVolumeSpecName: "home") pod "a2aeb5cc-525f-4a64-919d-ec15adc61bbc" (UID: "a2aeb5cc-525f-4a64-919d-ec15adc61bbc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 22:23:30.877964 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:23:30.877937    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2aeb5cc-525f-4a64-919d-ec15adc61bbc-home\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:23:30.878080 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:23:30.877963    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2aeb5cc-525f-4a64-919d-ec15adc61bbc-model-cache\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:23:30.878080 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:23:30.878049    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2aeb5cc-525f-4a64-919d-ec15adc61bbc-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a2aeb5cc-525f-4a64-919d-ec15adc61bbc" (UID: "a2aeb5cc-525f-4a64-919d-ec15adc61bbc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 22:23:30.880147 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:23:30.880110    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2aeb5cc-525f-4a64-919d-ec15adc61bbc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a2aeb5cc-525f-4a64-919d-ec15adc61bbc" (UID: "a2aeb5cc-525f-4a64-919d-ec15adc61bbc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 22:23:30.880147 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:23:30.880125    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2aeb5cc-525f-4a64-919d-ec15adc61bbc-kube-api-access-p9fz5" (OuterVolumeSpecName: "kube-api-access-p9fz5") pod "a2aeb5cc-525f-4a64-919d-ec15adc61bbc" (UID: "a2aeb5cc-525f-4a64-919d-ec15adc61bbc"). InnerVolumeSpecName "kube-api-access-p9fz5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 22:23:30.880321 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:23:30.880175    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2aeb5cc-525f-4a64-919d-ec15adc61bbc-dshm" (OuterVolumeSpecName: "dshm") pod "a2aeb5cc-525f-4a64-919d-ec15adc61bbc" (UID: "a2aeb5cc-525f-4a64-919d-ec15adc61bbc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 22:23:30.979116 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:23:30.979084    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2aeb5cc-525f-4a64-919d-ec15adc61bbc-dshm\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:23:30.979116 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:23:30.979111    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9fz5\" (UniqueName: \"kubernetes.io/projected/a2aeb5cc-525f-4a64-919d-ec15adc61bbc-kube-api-access-p9fz5\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:23:30.979116 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:23:30.979120    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2aeb5cc-525f-4a64-919d-ec15adc61bbc-tls-certs\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:23:30.979116 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:23:30.979129    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2aeb5cc-525f-4a64-919d-ec15adc61bbc-tmp-dir\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:23:31.656924 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:23:31.656892    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-676c26ab/e2e-lifecycle-model-676c26ab-kserve-777db9ddbc-w9vbm"
Sep 10 22:23:31.656924 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:23:31.656902    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-676c26ab/e2e-lifecycle-model-676c26ab-kserve-777db9ddbc-w9vbm" event={"ID":"a2aeb5cc-525f-4a64-919d-ec15adc61bbc","Type":"ContainerDied","Data":"2f0f668628dbcd94dcc7096c605425c56942f2e0f7d8c97ac6bec1f30efe888b"}
Sep 10 22:23:31.657398 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:23:31.656960    2582 scope.go:117] "RemoveContainer" containerID="077741c9e4f015aa0ad409dfb7b2d97402b93d628916be5250883f597c0c10ef"
Sep 10 22:23:31.689247 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:23:31.688426    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-676c26ab/e2e-lifecycle-model-676c26ab-kserve-777db9ddbc-w9vbm"]
Sep 10 22:23:31.692056 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:23:31.692034    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-676c26ab/e2e-lifecycle-model-676c26ab-kserve-777db9ddbc-w9vbm"]
Sep 10 22:23:33.498730 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:23:33.498695    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a2aeb5cc-525f-4a64-919d-ec15adc61bbc" path="/var/lib/kubelet/pods/a2aeb5cc-525f-4a64-919d-ec15adc61bbc/volumes"
Sep 10 22:24:24.835546 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:24:24.835513    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-d07dfb/auth-test-model-d07dfb-kserve-6764ddf5c9-vzq4s"]
Sep 10 22:24:25.900865 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:24:25.900812    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-ec6d47/auth-test-model-ec6d47-kserve-6df9c55887-hckbr"]
Sep 10 22:24:35.609177 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:24:35.609130    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-d07dfb/sub-test-model-d07dfb-kserve-78c8db4f9c-hp4z6"]
Sep 10 22:24:35.609678 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:24:35.609661    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ce4bed6-f1db-42dc-a2a7-86d398926c24" containerName="payload-processing"
Sep 10 22:24:35.609765 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:24:35.609680    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ce4bed6-f1db-42dc-a2a7-86d398926c24" containerName="payload-processing"
Sep 10 22:24:35.609765 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:24:35.609693    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="441b2f44-f770-41cf-87ce-0927bee26491" containerName="revoke-keys"
Sep 10 22:24:35.609765 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:24:35.609702    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="441b2f44-f770-41cf-87ce-0927bee26491" containerName="revoke-keys"
Sep 10 22:24:35.609765 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:24:35.609738    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2aeb5cc-525f-4a64-919d-ec15adc61bbc" containerName="main"
Sep 10 22:24:35.609765 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:24:35.609747    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2aeb5cc-525f-4a64-919d-ec15adc61bbc" containerName="main"
Sep 10 22:24:35.610087 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:24:35.609829    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2aeb5cc-525f-4a64-919d-ec15adc61bbc" containerName="main"
Sep 10 22:24:35.610087 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:24:35.609842    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ce4bed6-f1db-42dc-a2a7-86d398926c24" containerName="payload-processing"
Sep 10 22:24:35.610087 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:24:35.609879    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="441b2f44-f770-41cf-87ce-0927bee26491" containerName="revoke-keys"
Sep 10 22:24:35.614379 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:24:35.614355    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-d07dfb/sub-test-model-d07dfb-kserve-78c8db4f9c-hp4z6"
Sep 10 22:24:35.616763 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:24:35.616744    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-d07dfb\"/\"sub-test-model-d07dfb-kserve-self-signed-certs\""
Sep 10 22:24:35.623004 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:24:35.622978    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-d07dfb/sub-test-model-d07dfb-kserve-78c8db4f9c-hp4z6"]
Sep 10 22:24:35.744993 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:24:35.744947    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab05d721-124d-46c5-948f-6e83c5aa2854-home\") pod \"sub-test-model-d07dfb-kserve-78c8db4f9c-hp4z6\" (UID: \"ab05d721-124d-46c5-948f-6e83c5aa2854\") " pod="ai-tenant-e2e-shared-a-w3-d07dfb/sub-test-model-d07dfb-kserve-78c8db4f9c-hp4z6"
Sep 10 22:24:35.744993 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:24:35.744998    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab05d721-124d-46c5-948f-6e83c5aa2854-model-cache\") pod \"sub-test-model-d07dfb-kserve-78c8db4f9c-hp4z6\" (UID: \"ab05d721-124d-46c5-948f-6e83c5aa2854\") " pod="ai-tenant-e2e-shared-a-w3-d07dfb/sub-test-model-d07dfb-kserve-78c8db4f9c-hp4z6"
Sep 10 22:24:35.745220 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:24:35.745069    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab05d721-124d-46c5-948f-6e83c5aa2854-tls-certs\") pod \"sub-test-model-d07dfb-kserve-78c8db4f9c-hp4z6\" (UID: \"ab05d721-124d-46c5-948f-6e83c5aa2854\") " pod="ai-tenant-e2e-shared-a-w3-d07dfb/sub-test-model-d07dfb-kserve-78c8db4f9c-hp4z6"
Sep 10 22:24:35.745220 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:24:35.745189    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab05d721-124d-46c5-948f-6e83c5aa2854-tmp-dir\") pod \"sub-test-model-d07dfb-kserve-78c8db4f9c-hp4z6\" (UID: \"ab05d721-124d-46c5-948f-6e83c5aa2854\") " pod="ai-tenant-e2e-shared-a-w3-d07dfb/sub-test-model-d07dfb-kserve-78c8db4f9c-hp4z6"
Sep 10 22:24:35.745290 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:24:35.745224    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab05d721-124d-46c5-948f-6e83c5aa2854-dshm\") pod \"sub-test-model-d07dfb-kserve-78c8db4f9c-hp4z6\" (UID: \"ab05d721-124d-46c5-948f-6e83c5aa2854\") " pod="ai-tenant-e2e-shared-a-w3-d07dfb/sub-test-model-d07dfb-kserve-78c8db4f9c-hp4z6"
Sep 10 22:24:35.745290 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:24:35.745247    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ndm2\" (UniqueName: \"kubernetes.io/projected/ab05d721-124d-46c5-948f-6e83c5aa2854-kube-api-access-8ndm2\") pod \"sub-test-model-d07dfb-kserve-78c8db4f9c-hp4z6\" (UID: \"ab05d721-124d-46c5-948f-6e83c5aa2854\") " pod="ai-tenant-e2e-shared-a-w3-d07dfb/sub-test-model-d07dfb-kserve-78c8db4f9c-hp4z6"
Sep 10 22:24:35.846171 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:24:35.846131    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab05d721-124d-46c5-948f-6e83c5aa2854-tmp-dir\") pod \"sub-test-model-d07dfb-kserve-78c8db4f9c-hp4z6\" (UID: \"ab05d721-124d-46c5-948f-6e83c5aa2854\") " pod="ai-tenant-e2e-shared-a-w3-d07dfb/sub-test-model-d07dfb-kserve-78c8db4f9c-hp4z6"
Sep 10 22:24:35.846171 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:24:35.846175    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab05d721-124d-46c5-948f-6e83c5aa2854-dshm\") pod \"sub-test-model-d07dfb-kserve-78c8db4f9c-hp4z6\" (UID: \"ab05d721-124d-46c5-948f-6e83c5aa2854\") " pod="ai-tenant-e2e-shared-a-w3-d07dfb/sub-test-model-d07dfb-kserve-78c8db4f9c-hp4z6"
Sep 10 22:24:35.846434 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:24:35.846203    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ndm2\" (UniqueName: \"kubernetes.io/projected/ab05d721-124d-46c5-948f-6e83c5aa2854-kube-api-access-8ndm2\") pod \"sub-test-model-d07dfb-kserve-78c8db4f9c-hp4z6\" (UID: \"ab05d721-124d-46c5-948f-6e83c5aa2854\") " pod="ai-tenant-e2e-shared-a-w3-d07dfb/sub-test-model-d07dfb-kserve-78c8db4f9c-hp4z6"
Sep 10 22:24:35.846434 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:24:35.846243    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab05d721-124d-46c5-948f-6e83c5aa2854-home\") pod \"sub-test-model-d07dfb-kserve-78c8db4f9c-hp4z6\" (UID: \"ab05d721-124d-46c5-948f-6e83c5aa2854\") " pod="ai-tenant-e2e-shared-a-w3-d07dfb/sub-test-model-d07dfb-kserve-78c8db4f9c-hp4z6"
Sep 10 22:24:35.846434 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:24:35.846281    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab05d721-124d-46c5-948f-6e83c5aa2854-model-cache\") pod \"sub-test-model-d07dfb-kserve-78c8db4f9c-hp4z6\" (UID: \"ab05d721-124d-46c5-948f-6e83c5aa2854\") " pod="ai-tenant-e2e-shared-a-w3-d07dfb/sub-test-model-d07dfb-kserve-78c8db4f9c-hp4z6"
Sep 10 22:24:35.846434 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:24:35.846337    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab05d721-124d-46c5-948f-6e83c5aa2854-tls-certs\") pod \"sub-test-model-d07dfb-kserve-78c8db4f9c-hp4z6\" (UID: \"ab05d721-124d-46c5-948f-6e83c5aa2854\") " pod="ai-tenant-e2e-shared-a-w3-d07dfb/sub-test-model-d07dfb-kserve-78c8db4f9c-hp4z6"
Sep 10 22:24:35.846715 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:24:35.846683    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab05d721-124d-46c5-948f-6e83c5aa2854-tmp-dir\") pod \"sub-test-model-d07dfb-kserve-78c8db4f9c-hp4z6\" (UID: \"ab05d721-124d-46c5-948f-6e83c5aa2854\") " pod="ai-tenant-e2e-shared-a-w3-d07dfb/sub-test-model-d07dfb-kserve-78c8db4f9c-hp4z6"
Sep 10 22:24:35.846789 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:24:35.846720    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab05d721-124d-46c5-948f-6e83c5aa2854-home\") pod \"sub-test-model-d07dfb-kserve-78c8db4f9c-hp4z6\" (UID: \"ab05d721-124d-46c5-948f-6e83c5aa2854\") " pod="ai-tenant-e2e-shared-a-w3-d07dfb/sub-test-model-d07dfb-kserve-78c8db4f9c-hp4z6"
Sep 10 22:24:35.846789 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:24:35.846738    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab05d721-124d-46c5-948f-6e83c5aa2854-model-cache\") pod \"sub-test-model-d07dfb-kserve-78c8db4f9c-hp4z6\" (UID: \"ab05d721-124d-46c5-948f-6e83c5aa2854\") " pod="ai-tenant-e2e-shared-a-w3-d07dfb/sub-test-model-d07dfb-kserve-78c8db4f9c-hp4z6"
Sep 10 22:24:35.848507 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:24:35.848483    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab05d721-124d-46c5-948f-6e83c5aa2854-dshm\") pod \"sub-test-model-d07dfb-kserve-78c8db4f9c-hp4z6\" (UID: \"ab05d721-124d-46c5-948f-6e83c5aa2854\") " pod="ai-tenant-e2e-shared-a-w3-d07dfb/sub-test-model-d07dfb-kserve-78c8db4f9c-hp4z6"
Sep 10 22:24:35.848758 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:24:35.848741    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab05d721-124d-46c5-948f-6e83c5aa2854-tls-certs\") pod \"sub-test-model-d07dfb-kserve-78c8db4f9c-hp4z6\" (UID: \"ab05d721-124d-46c5-948f-6e83c5aa2854\") " pod="ai-tenant-e2e-shared-a-w3-d07dfb/sub-test-model-d07dfb-kserve-78c8db4f9c-hp4z6"
Sep 10 22:24:35.854624 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:24:35.854604    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ndm2\" (UniqueName: \"kubernetes.io/projected/ab05d721-124d-46c5-948f-6e83c5aa2854-kube-api-access-8ndm2\") pod \"sub-test-model-d07dfb-kserve-78c8db4f9c-hp4z6\" (UID: \"ab05d721-124d-46c5-948f-6e83c5aa2854\") " pod="ai-tenant-e2e-shared-a-w3-d07dfb/sub-test-model-d07dfb-kserve-78c8db4f9c-hp4z6"
Sep 10 22:24:35.924405 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:24:35.924369    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-d07dfb/sub-test-model-d07dfb-kserve-78c8db4f9c-hp4z6"
Sep 10 22:24:36.200317 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:24:36.200287    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-d07dfb/sub-test-model-d07dfb-kserve-78c8db4f9c-hp4z6"]
Sep 10 22:24:36.202279 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:24:36.202251    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podab05d721_124d_46c5_948f_6e83c5aa2854.slice/crio-4f67896a7705cf4653d7a22f2ad8590a4ffabc27ea54f6be259b77c35a4032b5 WatchSource:0}: Error finding container 4f67896a7705cf4653d7a22f2ad8590a4ffabc27ea54f6be259b77c35a4032b5: Status 404 returned error can't find the container with id 4f67896a7705cf4653d7a22f2ad8590a4ffabc27ea54f6be259b77c35a4032b5
Sep 10 22:24:36.966816 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:24:36.966773    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-d07dfb/sub-test-model-d07dfb-kserve-78c8db4f9c-hp4z6" event={"ID":"ab05d721-124d-46c5-948f-6e83c5aa2854","Type":"ContainerStarted","Data":"01dbeb2c2496cb14aff3068d160ba04e118a31d232e90444b51400756f6c0c1b"}
Sep 10 22:24:36.966816 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:24:36.966818    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-d07dfb/sub-test-model-d07dfb-kserve-78c8db4f9c-hp4z6" event={"ID":"ab05d721-124d-46c5-948f-6e83c5aa2854","Type":"ContainerStarted","Data":"4f67896a7705cf4653d7a22f2ad8590a4ffabc27ea54f6be259b77c35a4032b5"}
Sep 10 22:24:36.982128 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:24:36.982077    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-d07dfb/sub-test-model-d07dfb-kserve-78c8db4f9c-hp4z6" podStartSLOduration=1.982062597 podStartE2EDuration="1.982062597s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 22:24:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:24:36.982011376 +0000 UTC m=+1397.938079085" watchObservedRunningTime="2026-09-10 22:24:36.982062597 +0000 UTC m=+1397.938130304"
Sep 10 22:24:39.854828 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:24:39.854789    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-d07dfb/auth-test-model-d07dfb-kserve-6764ddf5c9-vzq4s" podUID="32794fee-1c78-4799-b55d-a6dfd8086493" containerName="main" containerID="cri-o://b621172f8e3feb00d73bfea48c3e006de323dca40a4449939b1e0daf70cceb49" gracePeriod=45
Sep 10 22:24:39.981518 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:24:39.981490    2582 generic.go:356] "Generic (PLEG): container finished" podID="32794fee-1c78-4799-b55d-a6dfd8086493" containerID="b621172f8e3feb00d73bfea48c3e006de323dca40a4449939b1e0daf70cceb49" exitCode=0
Sep 10 22:24:39.981637 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:24:39.981538    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-d07dfb/auth-test-model-d07dfb-kserve-6764ddf5c9-vzq4s" event={"ID":"32794fee-1c78-4799-b55d-a6dfd8086493","Type":"ContainerDied","Data":"b621172f8e3feb00d73bfea48c3e006de323dca40a4449939b1e0daf70cceb49"}
Sep 10 22:24:40.106653 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:24:40.106592    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-d07dfb/auth-test-model-d07dfb-kserve-6764ddf5c9-vzq4s"
Sep 10 22:24:40.289694 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:24:40.289662    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32794fee-1c78-4799-b55d-a6dfd8086493-home\") pod \"32794fee-1c78-4799-b55d-a6dfd8086493\" (UID: \"32794fee-1c78-4799-b55d-a6dfd8086493\") "
Sep 10 22:24:40.289902 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:24:40.289702    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4f8nr\" (UniqueName: \"kubernetes.io/projected/32794fee-1c78-4799-b55d-a6dfd8086493-kube-api-access-4f8nr\") pod \"32794fee-1c78-4799-b55d-a6dfd8086493\" (UID: \"32794fee-1c78-4799-b55d-a6dfd8086493\") "
Sep 10 22:24:40.289902 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:24:40.289729    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32794fee-1c78-4799-b55d-a6dfd8086493-tls-certs\") pod \"32794fee-1c78-4799-b55d-a6dfd8086493\" (UID: \"32794fee-1c78-4799-b55d-a6dfd8086493\") "
Sep 10 22:24:40.289902 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:24:40.289767    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32794fee-1c78-4799-b55d-a6dfd8086493-model-cache\") pod \"32794fee-1c78-4799-b55d-a6dfd8086493\" (UID: \"32794fee-1c78-4799-b55d-a6dfd8086493\") "
Sep 10 22:24:40.289902 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:24:40.289837    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32794fee-1c78-4799-b55d-a6dfd8086493-dshm\") pod \"32794fee-1c78-4799-b55d-a6dfd8086493\" (UID: \"32794fee-1c78-4799-b55d-a6dfd8086493\") "
Sep 10 22:24:40.289902 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:24:40.289879    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32794fee-1c78-4799-b55d-a6dfd8086493-tmp-dir\") pod \"32794fee-1c78-4799-b55d-a6dfd8086493\" (UID: \"32794fee-1c78-4799-b55d-a6dfd8086493\") "
Sep 10 22:24:40.290195 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:24:40.289957    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32794fee-1c78-4799-b55d-a6dfd8086493-home" (OuterVolumeSpecName: "home") pod "32794fee-1c78-4799-b55d-a6dfd8086493" (UID: "32794fee-1c78-4799-b55d-a6dfd8086493"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 22:24:40.290195 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:24:40.290128    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32794fee-1c78-4799-b55d-a6dfd8086493-model-cache" (OuterVolumeSpecName: "model-cache") pod "32794fee-1c78-4799-b55d-a6dfd8086493" (UID: "32794fee-1c78-4799-b55d-a6dfd8086493"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 22:24:40.290195 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:24:40.290149    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32794fee-1c78-4799-b55d-a6dfd8086493-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "32794fee-1c78-4799-b55d-a6dfd8086493" (UID: "32794fee-1c78-4799-b55d-a6dfd8086493"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 22:24:40.290357 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:24:40.290251    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32794fee-1c78-4799-b55d-a6dfd8086493-home\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:24:40.290357 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:24:40.290268    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32794fee-1c78-4799-b55d-a6dfd8086493-model-cache\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:24:40.290357 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:24:40.290296    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32794fee-1c78-4799-b55d-a6dfd8086493-tmp-dir\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:24:40.292060 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:24:40.291976    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32794fee-1c78-4799-b55d-a6dfd8086493-dshm" (OuterVolumeSpecName: "dshm") pod "32794fee-1c78-4799-b55d-a6dfd8086493" (UID: "32794fee-1c78-4799-b55d-a6dfd8086493"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 22:24:40.292178 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:24:40.292081    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32794fee-1c78-4799-b55d-a6dfd8086493-kube-api-access-4f8nr" (OuterVolumeSpecName: "kube-api-access-4f8nr") pod "32794fee-1c78-4799-b55d-a6dfd8086493" (UID: "32794fee-1c78-4799-b55d-a6dfd8086493"). InnerVolumeSpecName "kube-api-access-4f8nr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 22:24:40.292178 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:24:40.292092    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32794fee-1c78-4799-b55d-a6dfd8086493-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "32794fee-1c78-4799-b55d-a6dfd8086493" (UID: "32794fee-1c78-4799-b55d-a6dfd8086493"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 22:24:40.391341 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:24:40.391278    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4f8nr\" (UniqueName: \"kubernetes.io/projected/32794fee-1c78-4799-b55d-a6dfd8086493-kube-api-access-4f8nr\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:24:40.391341 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:24:40.391302    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32794fee-1c78-4799-b55d-a6dfd8086493-tls-certs\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:24:40.391341 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:24:40.391311    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32794fee-1c78-4799-b55d-a6dfd8086493-dshm\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:24:40.920367 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:24:40.920318    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-ec6d47/auth-test-model-ec6d47-kserve-6df9c55887-hckbr" podUID="b45375ed-a680-4d89-bde2-758ed6c24222" containerName="main" containerID="cri-o://fb591d508ca7ee5093a1686fbb43168d6b30eb7b4f66b32f39081a7eb76b5947" gracePeriod=45
Sep 10 22:24:40.988609 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:24:40.988576    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-d07dfb/auth-test-model-d07dfb-kserve-6764ddf5c9-vzq4s"
Sep 10 22:24:40.988779 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:24:40.988609    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-d07dfb/auth-test-model-d07dfb-kserve-6764ddf5c9-vzq4s" event={"ID":"32794fee-1c78-4799-b55d-a6dfd8086493","Type":"ContainerDied","Data":"cda6c37579032b08dc1b07ff842f261fe624a4f3e003d262c01f050adf7ff1e4"}
Sep 10 22:24:40.988779 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:24:40.988669    2582 scope.go:117] "RemoveContainer" containerID="b621172f8e3feb00d73bfea48c3e006de323dca40a4449939b1e0daf70cceb49"
Sep 10 22:24:41.008404 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:24:41.008379    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-d07dfb/auth-test-model-d07dfb-kserve-6764ddf5c9-vzq4s"]
Sep 10 22:24:41.010813 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:24:41.010788    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-d07dfb/auth-test-model-d07dfb-kserve-6764ddf5c9-vzq4s"]
Sep 10 22:24:41.171234 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:24:41.171168    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-ec6d47/auth-test-model-ec6d47-kserve-6df9c55887-hckbr"
Sep 10 22:24:41.296732 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:24:41.296702    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b45375ed-a680-4d89-bde2-758ed6c24222-tls-certs\") pod \"b45375ed-a680-4d89-bde2-758ed6c24222\" (UID: \"b45375ed-a680-4d89-bde2-758ed6c24222\") "
Sep 10 22:24:41.296732 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:24:41.296739    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b45375ed-a680-4d89-bde2-758ed6c24222-dshm\") pod \"b45375ed-a680-4d89-bde2-758ed6c24222\" (UID: \"b45375ed-a680-4d89-bde2-758ed6c24222\") "
Sep 10 22:24:41.296964 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:24:41.296763    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b45375ed-a680-4d89-bde2-758ed6c24222-model-cache\") pod \"b45375ed-a680-4d89-bde2-758ed6c24222\" (UID: \"b45375ed-a680-4d89-bde2-758ed6c24222\") "
Sep 10 22:24:41.296964 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:24:41.296866    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b45375ed-a680-4d89-bde2-758ed6c24222-tmp-dir\") pod \"b45375ed-a680-4d89-bde2-758ed6c24222\" (UID: \"b45375ed-a680-4d89-bde2-758ed6c24222\") "
Sep 10 22:24:41.296964 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:24:41.296892    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fdtl8\" (UniqueName: \"kubernetes.io/projected/b45375ed-a680-4d89-bde2-758ed6c24222-kube-api-access-fdtl8\") pod \"b45375ed-a680-4d89-bde2-758ed6c24222\" (UID: \"b45375ed-a680-4d89-bde2-758ed6c24222\") "
Sep 10 22:24:41.296964 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:24:41.296937    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b45375ed-a680-4d89-bde2-758ed6c24222-home\") pod \"b45375ed-a680-4d89-bde2-758ed6c24222\" (UID: \"b45375ed-a680-4d89-bde2-758ed6c24222\") "
Sep 10 22:24:41.297188 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:24:41.297085    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b45375ed-a680-4d89-bde2-758ed6c24222-model-cache" (OuterVolumeSpecName: "model-cache") pod "b45375ed-a680-4d89-bde2-758ed6c24222" (UID: "b45375ed-a680-4d89-bde2-758ed6c24222"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 22:24:41.297188 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:24:41.297090    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b45375ed-a680-4d89-bde2-758ed6c24222-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b45375ed-a680-4d89-bde2-758ed6c24222" (UID: "b45375ed-a680-4d89-bde2-758ed6c24222"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 22:24:41.297310 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:24:41.297213    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b45375ed-a680-4d89-bde2-758ed6c24222-tmp-dir\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:24:41.297310 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:24:41.297223    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b45375ed-a680-4d89-bde2-758ed6c24222-home" (OuterVolumeSpecName: "home") pod "b45375ed-a680-4d89-bde2-758ed6c24222" (UID: "b45375ed-a680-4d89-bde2-758ed6c24222"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 22:24:41.297310 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:24:41.297232    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b45375ed-a680-4d89-bde2-758ed6c24222-model-cache\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:24:41.298993 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:24:41.298964    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b45375ed-a680-4d89-bde2-758ed6c24222-dshm" (OuterVolumeSpecName: "dshm") pod "b45375ed-a680-4d89-bde2-758ed6c24222" (UID: "b45375ed-a680-4d89-bde2-758ed6c24222"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 22:24:41.299103 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:24:41.299058    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b45375ed-a680-4d89-bde2-758ed6c24222-kube-api-access-fdtl8" (OuterVolumeSpecName: "kube-api-access-fdtl8") pod "b45375ed-a680-4d89-bde2-758ed6c24222" (UID: "b45375ed-a680-4d89-bde2-758ed6c24222"). InnerVolumeSpecName "kube-api-access-fdtl8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 22:24:41.299149 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:24:41.299099    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b45375ed-a680-4d89-bde2-758ed6c24222-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b45375ed-a680-4d89-bde2-758ed6c24222" (UID: "b45375ed-a680-4d89-bde2-758ed6c24222"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 22:24:41.398207 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:24:41.398172    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fdtl8\" (UniqueName: \"kubernetes.io/projected/b45375ed-a680-4d89-bde2-758ed6c24222-kube-api-access-fdtl8\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:24:41.398207 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:24:41.398203    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b45375ed-a680-4d89-bde2-758ed6c24222-home\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:24:41.398207 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:24:41.398213    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b45375ed-a680-4d89-bde2-758ed6c24222-tls-certs\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:24:41.398523 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:24:41.398221    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b45375ed-a680-4d89-bde2-758ed6c24222-dshm\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:24:41.501033 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:24:41.500942    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="32794fee-1c78-4799-b55d-a6dfd8086493" path="/var/lib/kubelet/pods/32794fee-1c78-4799-b55d-a6dfd8086493/volumes"
Sep 10 22:24:41.995786 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:24:41.995749    2582 generic.go:356] "Generic (PLEG): container finished" podID="b45375ed-a680-4d89-bde2-758ed6c24222" containerID="fb591d508ca7ee5093a1686fbb43168d6b30eb7b4f66b32f39081a7eb76b5947" exitCode=0
Sep 10 22:24:41.996289 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:24:41.995789    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-ec6d47/auth-test-model-ec6d47-kserve-6df9c55887-hckbr"
Sep 10 22:24:41.996289 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:24:41.995810    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-ec6d47/auth-test-model-ec6d47-kserve-6df9c55887-hckbr" event={"ID":"b45375ed-a680-4d89-bde2-758ed6c24222","Type":"ContainerDied","Data":"fb591d508ca7ee5093a1686fbb43168d6b30eb7b4f66b32f39081a7eb76b5947"}
Sep 10 22:24:41.996289 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:24:41.995866    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-ec6d47/auth-test-model-ec6d47-kserve-6df9c55887-hckbr" event={"ID":"b45375ed-a680-4d89-bde2-758ed6c24222","Type":"ContainerDied","Data":"95a66e6e0a01cfb6779cd8cfeafb31f4721936eb75530631c59fcfc9bf671c12"}
Sep 10 22:24:41.996289 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:24:41.995887    2582 scope.go:117] "RemoveContainer" containerID="fb591d508ca7ee5093a1686fbb43168d6b30eb7b4f66b32f39081a7eb76b5947"
Sep 10 22:24:42.004245 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:24:42.004225    2582 scope.go:117] "RemoveContainer" containerID="fb591d508ca7ee5093a1686fbb43168d6b30eb7b4f66b32f39081a7eb76b5947"
Sep 10 22:24:42.004546 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:24:42.004522    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb591d508ca7ee5093a1686fbb43168d6b30eb7b4f66b32f39081a7eb76b5947\": container with ID starting with fb591d508ca7ee5093a1686fbb43168d6b30eb7b4f66b32f39081a7eb76b5947 not found: ID does not exist" containerID="fb591d508ca7ee5093a1686fbb43168d6b30eb7b4f66b32f39081a7eb76b5947"
Sep 10 22:24:42.004589 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:24:42.004555    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb591d508ca7ee5093a1686fbb43168d6b30eb7b4f66b32f39081a7eb76b5947"} err="failed to get container status \"fb591d508ca7ee5093a1686fbb43168d6b30eb7b4f66b32f39081a7eb76b5947\": rpc error: code = NotFound desc = could not find container \"fb591d508ca7ee5093a1686fbb43168d6b30eb7b4f66b32f39081a7eb76b5947\": container with ID starting with fb591d508ca7ee5093a1686fbb43168d6b30eb7b4f66b32f39081a7eb76b5947 not found: ID does not exist"
Sep 10 22:24:42.013663 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:24:42.013637    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-ec6d47/auth-test-model-ec6d47-kserve-6df9c55887-hckbr"]
Sep 10 22:24:42.020211 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:24:42.020190    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-ec6d47/auth-test-model-ec6d47-kserve-6df9c55887-hckbr"]
Sep 10 22:24:43.498521 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:24:43.498486    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b45375ed-a680-4d89-bde2-758ed6c24222" path="/var/lib/kubelet/pods/b45375ed-a680-4d89-bde2-758ed6c24222/volumes"
Sep 10 22:24:45.925381 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:24:45.925341    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-d07dfb/sub-test-model-d07dfb-kserve-78c8db4f9c-hp4z6"
Sep 10 22:24:45.925770 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:24:45.925405    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-d07dfb/sub-test-model-d07dfb-kserve-78c8db4f9c-hp4z6"
Sep 10 22:24:45.937802 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:24:45.937775    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-d07dfb/sub-test-model-d07dfb-kserve-78c8db4f9c-hp4z6"
Sep 10 22:24:46.027507 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:24:46.027479    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-d07dfb/sub-test-model-d07dfb-kserve-78c8db4f9c-hp4z6"
Sep 10 22:25:04.037374 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:25:04.037339    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-9c0e33-75ddc78994-fn98c"]
Sep 10 22:25:04.037815 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:25:04.037693    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b45375ed-a680-4d89-bde2-758ed6c24222" containerName="main"
Sep 10 22:25:04.037815 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:25:04.037705    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="b45375ed-a680-4d89-bde2-758ed6c24222" containerName="main"
Sep 10 22:25:04.037815 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:25:04.037727    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32794fee-1c78-4799-b55d-a6dfd8086493" containerName="main"
Sep 10 22:25:04.037815 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:25:04.037732    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="32794fee-1c78-4799-b55d-a6dfd8086493" containerName="main"
Sep 10 22:25:04.037815 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:25:04.037810    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="b45375ed-a680-4d89-bde2-758ed6c24222" containerName="main"
Sep 10 22:25:04.037815 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:25:04.037818    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="32794fee-1c78-4799-b55d-a6dfd8086493" containerName="main"
Sep 10 22:25:04.040787 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:25:04.040768    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9c0e33-75ddc78994-fn98c"
Sep 10 22:25:04.046006 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:25:04.045983    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-9c0e33-dockercfg-5rd8z\""
Sep 10 22:25:04.046300 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:25:04.046282    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-9c0e33\""
Sep 10 22:25:04.061578 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:25:04.061553    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-9c0e33-75ddc78994-fn98c"]
Sep 10 22:25:04.074687 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:25:04.074663    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cph4\" (UniqueName: \"kubernetes.io/projected/cb9c1682-4a88-4f66-8007-39e4c508d576-kube-api-access-6cph4\") pod \"payload-pre-processing-e2e-api-a-9c0e33-75ddc78994-fn98c\" (UID: \"cb9c1682-4a88-4f66-8007-39e4c508d576\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-9c0e33-75ddc78994-fn98c"
Sep 10 22:25:04.074795 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:25:04.074701    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb9c1682-4a88-4f66-8007-39e4c508d576-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-9c0e33-75ddc78994-fn98c\" (UID: \"cb9c1682-4a88-4f66-8007-39e4c508d576\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-9c0e33-75ddc78994-fn98c"
Sep 10 22:25:04.176328 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:25:04.176277    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cph4\" (UniqueName: \"kubernetes.io/projected/cb9c1682-4a88-4f66-8007-39e4c508d576-kube-api-access-6cph4\") pod \"payload-pre-processing-e2e-api-a-9c0e33-75ddc78994-fn98c\" (UID: \"cb9c1682-4a88-4f66-8007-39e4c508d576\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-9c0e33-75ddc78994-fn98c"
Sep 10 22:25:04.176649 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:25:04.176358    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb9c1682-4a88-4f66-8007-39e4c508d576-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-9c0e33-75ddc78994-fn98c\" (UID: \"cb9c1682-4a88-4f66-8007-39e4c508d576\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-9c0e33-75ddc78994-fn98c"
Sep 10 22:25:04.177420 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:25:04.177392    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb9c1682-4a88-4f66-8007-39e4c508d576-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-9c0e33-75ddc78994-fn98c\" (UID: \"cb9c1682-4a88-4f66-8007-39e4c508d576\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-9c0e33-75ddc78994-fn98c"
Sep 10 22:25:04.190579 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:25:04.190554    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cph4\" (UniqueName: \"kubernetes.io/projected/cb9c1682-4a88-4f66-8007-39e4c508d576-kube-api-access-6cph4\") pod \"payload-pre-processing-e2e-api-a-9c0e33-75ddc78994-fn98c\" (UID: \"cb9c1682-4a88-4f66-8007-39e4c508d576\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-9c0e33-75ddc78994-fn98c"
Sep 10 22:25:04.350479 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:25:04.350416    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9c0e33-75ddc78994-fn98c"
Sep 10 22:25:04.633569 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:25:04.633535    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-9c0e33-75ddc78994-fn98c"]
Sep 10 22:25:04.635037 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:25:04.635005    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb9c1682_4a88_4f66_8007_39e4c508d576.slice/crio-e7a7b21b2a998eabe49d49b9deeb96afc9fd06350769e220929cdd499cff6978 WatchSource:0}: Error finding container e7a7b21b2a998eabe49d49b9deeb96afc9fd06350769e220929cdd499cff6978: Status 404 returned error can't find the container with id e7a7b21b2a998eabe49d49b9deeb96afc9fd06350769e220929cdd499cff6978
Sep 10 22:25:05.105117 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:25:05.105076    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9c0e33-75ddc78994-fn98c" event={"ID":"cb9c1682-4a88-4f66-8007-39e4c508d576","Type":"ContainerStarted","Data":"a1fd684562fd355e6651edc76427a55304e210f483b1ffaa4e32c925b0ba7069"}
Sep 10 22:25:05.105547 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:25:05.105142    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9c0e33-75ddc78994-fn98c"
Sep 10 22:25:05.105547 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:25:05.105161    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9c0e33-75ddc78994-fn98c" event={"ID":"cb9c1682-4a88-4f66-8007-39e4c508d576","Type":"ContainerStarted","Data":"e7a7b21b2a998eabe49d49b9deeb96afc9fd06350769e220929cdd499cff6978"}
Sep 10 22:25:05.122516 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:25:05.122475    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9c0e33-75ddc78994-fn98c" podStartSLOduration=1.122460037 podStartE2EDuration="1.122460037s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 22:25:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:25:05.12167827 +0000 UTC m=+1426.077745979" watchObservedRunningTime="2026-09-10 22:25:05.122460037 +0000 UTC m=+1426.078527792"
Sep 10 22:25:16.111949 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:25:16.111870    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9c0e33-75ddc78994-fn98c"
Sep 10 22:25:19.700471 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:25:19.700440    2582 scope.go:117] "RemoveContainer" containerID="5aa75f0d4719bd6f78754819960a0fe33230b680242e54528118f6950e663959"
Sep 10 22:25:25.733361 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:25:25.733329    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-ec6d47/sub-test-model-ec6d47-kserve-64647bffb5-sprnx"]
Sep 10 22:25:25.737918 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:25:25.737896    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-ec6d47/sub-test-model-ec6d47-kserve-64647bffb5-sprnx"
Sep 10 22:25:25.740007 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:25:25.739987    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-ec6d47\"/\"kube-root-ca.crt\""
Sep 10 22:25:25.740138 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:25:25.739987    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-ec6d47\"/\"openshift-service-ca.crt\""
Sep 10 22:25:25.740680 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:25:25.740659    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-ec6d47\"/\"sub-test-model-ec6d47-kserve-self-signed-certs\""
Sep 10 22:25:25.740680 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:25:25.740678    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-ec6d47\"/\"default-dockercfg-vvdmr\""
Sep 10 22:25:25.744720 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:25:25.744673    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-ec6d47/sub-test-model-ec6d47-kserve-64647bffb5-sprnx"]
Sep 10 22:25:25.863501 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:25:25.863471    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c5cc847-16f0-44a9-b062-dc7a7c8a4364-dshm\") pod \"sub-test-model-ec6d47-kserve-64647bffb5-sprnx\" (UID: \"7c5cc847-16f0-44a9-b062-dc7a7c8a4364\") " pod="ai-tenant-e2e-shared-b-w3-ec6d47/sub-test-model-ec6d47-kserve-64647bffb5-sprnx"
Sep 10 22:25:25.863645 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:25:25.863526    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c5cc847-16f0-44a9-b062-dc7a7c8a4364-tmp-dir\") pod \"sub-test-model-ec6d47-kserve-64647bffb5-sprnx\" (UID: \"7c5cc847-16f0-44a9-b062-dc7a7c8a4364\") " pod="ai-tenant-e2e-shared-b-w3-ec6d47/sub-test-model-ec6d47-kserve-64647bffb5-sprnx"
Sep 10 22:25:25.863645 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:25:25.863562    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c5cc847-16f0-44a9-b062-dc7a7c8a4364-tls-certs\") pod \"sub-test-model-ec6d47-kserve-64647bffb5-sprnx\" (UID: \"7c5cc847-16f0-44a9-b062-dc7a7c8a4364\") " pod="ai-tenant-e2e-shared-b-w3-ec6d47/sub-test-model-ec6d47-kserve-64647bffb5-sprnx"
Sep 10 22:25:25.863645 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:25:25.863584    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnprg\" (UniqueName: \"kubernetes.io/projected/7c5cc847-16f0-44a9-b062-dc7a7c8a4364-kube-api-access-hnprg\") pod \"sub-test-model-ec6d47-kserve-64647bffb5-sprnx\" (UID: \"7c5cc847-16f0-44a9-b062-dc7a7c8a4364\") " pod="ai-tenant-e2e-shared-b-w3-ec6d47/sub-test-model-ec6d47-kserve-64647bffb5-sprnx"
Sep 10 22:25:25.863645 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:25:25.863629    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c5cc847-16f0-44a9-b062-dc7a7c8a4364-model-cache\") pod \"sub-test-model-ec6d47-kserve-64647bffb5-sprnx\" (UID: \"7c5cc847-16f0-44a9-b062-dc7a7c8a4364\") " pod="ai-tenant-e2e-shared-b-w3-ec6d47/sub-test-model-ec6d47-kserve-64647bffb5-sprnx"
Sep 10 22:25:25.863786 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:25:25.863666    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c5cc847-16f0-44a9-b062-dc7a7c8a4364-home\") pod \"sub-test-model-ec6d47-kserve-64647bffb5-sprnx\" (UID: \"7c5cc847-16f0-44a9-b062-dc7a7c8a4364\") " pod="ai-tenant-e2e-shared-b-w3-ec6d47/sub-test-model-ec6d47-kserve-64647bffb5-sprnx"
Sep 10 22:25:25.964973 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:25:25.964941    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c5cc847-16f0-44a9-b062-dc7a7c8a4364-home\") pod \"sub-test-model-ec6d47-kserve-64647bffb5-sprnx\" (UID: \"7c5cc847-16f0-44a9-b062-dc7a7c8a4364\") " pod="ai-tenant-e2e-shared-b-w3-ec6d47/sub-test-model-ec6d47-kserve-64647bffb5-sprnx"
Sep 10 22:25:25.965250 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:25:25.965229    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c5cc847-16f0-44a9-b062-dc7a7c8a4364-dshm\") pod \"sub-test-model-ec6d47-kserve-64647bffb5-sprnx\" (UID: \"7c5cc847-16f0-44a9-b062-dc7a7c8a4364\") " pod="ai-tenant-e2e-shared-b-w3-ec6d47/sub-test-model-ec6d47-kserve-64647bffb5-sprnx"
Sep 10 22:25:25.965398 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:25:25.965368    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c5cc847-16f0-44a9-b062-dc7a7c8a4364-home\") pod \"sub-test-model-ec6d47-kserve-64647bffb5-sprnx\" (UID: \"7c5cc847-16f0-44a9-b062-dc7a7c8a4364\") " pod="ai-tenant-e2e-shared-b-w3-ec6d47/sub-test-model-ec6d47-kserve-64647bffb5-sprnx"
Sep 10 22:25:25.965532 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:25:25.965399    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c5cc847-16f0-44a9-b062-dc7a7c8a4364-tmp-dir\") pod \"sub-test-model-ec6d47-kserve-64647bffb5-sprnx\" (UID: \"7c5cc847-16f0-44a9-b062-dc7a7c8a4364\") " pod="ai-tenant-e2e-shared-b-w3-ec6d47/sub-test-model-ec6d47-kserve-64647bffb5-sprnx"
Sep 10 22:25:25.965532 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:25:25.965428    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c5cc847-16f0-44a9-b062-dc7a7c8a4364-tls-certs\") pod \"sub-test-model-ec6d47-kserve-64647bffb5-sprnx\" (UID: \"7c5cc847-16f0-44a9-b062-dc7a7c8a4364\") " pod="ai-tenant-e2e-shared-b-w3-ec6d47/sub-test-model-ec6d47-kserve-64647bffb5-sprnx"
Sep 10 22:25:25.965532 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:25:25.965454    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnprg\" (UniqueName: \"kubernetes.io/projected/7c5cc847-16f0-44a9-b062-dc7a7c8a4364-kube-api-access-hnprg\") pod \"sub-test-model-ec6d47-kserve-64647bffb5-sprnx\" (UID: \"7c5cc847-16f0-44a9-b062-dc7a7c8a4364\") " pod="ai-tenant-e2e-shared-b-w3-ec6d47/sub-test-model-ec6d47-kserve-64647bffb5-sprnx"
Sep 10 22:25:25.965532 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:25:25.965520    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c5cc847-16f0-44a9-b062-dc7a7c8a4364-model-cache\") pod \"sub-test-model-ec6d47-kserve-64647bffb5-sprnx\" (UID: \"7c5cc847-16f0-44a9-b062-dc7a7c8a4364\") " pod="ai-tenant-e2e-shared-b-w3-ec6d47/sub-test-model-ec6d47-kserve-64647bffb5-sprnx"
Sep 10 22:25:25.965774 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:25:25.965749    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c5cc847-16f0-44a9-b062-dc7a7c8a4364-tmp-dir\") pod \"sub-test-model-ec6d47-kserve-64647bffb5-sprnx\" (UID: \"7c5cc847-16f0-44a9-b062-dc7a7c8a4364\") " pod="ai-tenant-e2e-shared-b-w3-ec6d47/sub-test-model-ec6d47-kserve-64647bffb5-sprnx"
Sep 10 22:25:25.965904 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:25:25.965881    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c5cc847-16f0-44a9-b062-dc7a7c8a4364-model-cache\") pod \"sub-test-model-ec6d47-kserve-64647bffb5-sprnx\" (UID: \"7c5cc847-16f0-44a9-b062-dc7a7c8a4364\") " pod="ai-tenant-e2e-shared-b-w3-ec6d47/sub-test-model-ec6d47-kserve-64647bffb5-sprnx"
Sep 10 22:25:25.967664 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:25:25.967639    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c5cc847-16f0-44a9-b062-dc7a7c8a4364-dshm\") pod \"sub-test-model-ec6d47-kserve-64647bffb5-sprnx\" (UID: \"7c5cc847-16f0-44a9-b062-dc7a7c8a4364\") " pod="ai-tenant-e2e-shared-b-w3-ec6d47/sub-test-model-ec6d47-kserve-64647bffb5-sprnx"
Sep 10 22:25:25.969092 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:25:25.969070    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c5cc847-16f0-44a9-b062-dc7a7c8a4364-tls-certs\") pod \"sub-test-model-ec6d47-kserve-64647bffb5-sprnx\" (UID: \"7c5cc847-16f0-44a9-b062-dc7a7c8a4364\") " pod="ai-tenant-e2e-shared-b-w3-ec6d47/sub-test-model-ec6d47-kserve-64647bffb5-sprnx"
Sep 10 22:25:25.974280 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:25:25.974258    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnprg\" (UniqueName: \"kubernetes.io/projected/7c5cc847-16f0-44a9-b062-dc7a7c8a4364-kube-api-access-hnprg\") pod \"sub-test-model-ec6d47-kserve-64647bffb5-sprnx\" (UID: \"7c5cc847-16f0-44a9-b062-dc7a7c8a4364\") " pod="ai-tenant-e2e-shared-b-w3-ec6d47/sub-test-model-ec6d47-kserve-64647bffb5-sprnx"
Sep 10 22:25:26.048227 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:25:26.048163    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-ec6d47/sub-test-model-ec6d47-kserve-64647bffb5-sprnx"
Sep 10 22:25:26.320729 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:25:26.320628    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-ec6d47/sub-test-model-ec6d47-kserve-64647bffb5-sprnx"]
Sep 10 22:25:26.323831 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:25:26.323804    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7c5cc847_16f0_44a9_b062_dc7a7c8a4364.slice/crio-e535acee79b27576889266b76a339faa31aba4c4567c01af7238678ee0e15ce4 WatchSource:0}: Error finding container e535acee79b27576889266b76a339faa31aba4c4567c01af7238678ee0e15ce4: Status 404 returned error can't find the container with id e535acee79b27576889266b76a339faa31aba4c4567c01af7238678ee0e15ce4
Sep 10 22:25:27.210308 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:25:27.210275    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-ec6d47/sub-test-model-ec6d47-kserve-64647bffb5-sprnx" event={"ID":"7c5cc847-16f0-44a9-b062-dc7a7c8a4364","Type":"ContainerStarted","Data":"206822043f231426329fddaf04ffbf97a6f462df9f47e40f22feac3da1331784"}
Sep 10 22:25:27.210724 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:25:27.210320    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-ec6d47/sub-test-model-ec6d47-kserve-64647bffb5-sprnx" event={"ID":"7c5cc847-16f0-44a9-b062-dc7a7c8a4364","Type":"ContainerStarted","Data":"e535acee79b27576889266b76a339faa31aba4c4567c01af7238678ee0e15ce4"}
Sep 10 22:25:27.244702 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:25:27.244651    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-ec6d47/sub-test-model-ec6d47-kserve-64647bffb5-sprnx" podStartSLOduration=2.244636033 podStartE2EDuration="2.244636033s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 22:25:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:25:27.243751444 +0000 UTC m=+1448.199819150" watchObservedRunningTime="2026-09-10 22:25:27.244636033 +0000 UTC m=+1448.200703795"
Sep 10 22:25:34.275782 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:25:34.275746    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c142c5-6f45675c5d-gb4vs"]
Sep 10 22:25:34.279490 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:25:34.279469    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c142c5-6f45675c5d-gb4vs"
Sep 10 22:25:34.286676 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:25:34.286649    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-c142c5-dockercfg-f4rn7\""
Sep 10 22:25:34.286783 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:25:34.286697    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-c142c5\""
Sep 10 22:25:34.304627 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:25:34.304606    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c142c5-6f45675c5d-gb4vs"]
Sep 10 22:25:34.444972 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:25:34.444935    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8004ab8-dd2d-4a64-91bb-3a2da4acd1a6-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-c142c5-6f45675c5d-gb4vs\" (UID: \"d8004ab8-dd2d-4a64-91bb-3a2da4acd1a6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c142c5-6f45675c5d-gb4vs"
Sep 10 22:25:34.445151 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:25:34.444985    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g86l4\" (UniqueName: \"kubernetes.io/projected/d8004ab8-dd2d-4a64-91bb-3a2da4acd1a6-kube-api-access-g86l4\") pod \"payload-pre-processing-e2e-api-b-c142c5-6f45675c5d-gb4vs\" (UID: \"d8004ab8-dd2d-4a64-91bb-3a2da4acd1a6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c142c5-6f45675c5d-gb4vs"
Sep 10 22:25:34.546479 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:25:34.546402    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8004ab8-dd2d-4a64-91bb-3a2da4acd1a6-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-c142c5-6f45675c5d-gb4vs\" (UID: \"d8004ab8-dd2d-4a64-91bb-3a2da4acd1a6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c142c5-6f45675c5d-gb4vs"
Sep 10 22:25:34.546479 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:25:34.546451    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g86l4\" (UniqueName: \"kubernetes.io/projected/d8004ab8-dd2d-4a64-91bb-3a2da4acd1a6-kube-api-access-g86l4\") pod \"payload-pre-processing-e2e-api-b-c142c5-6f45675c5d-gb4vs\" (UID: \"d8004ab8-dd2d-4a64-91bb-3a2da4acd1a6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c142c5-6f45675c5d-gb4vs"
Sep 10 22:25:34.547250 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:25:34.547229    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8004ab8-dd2d-4a64-91bb-3a2da4acd1a6-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-c142c5-6f45675c5d-gb4vs\" (UID: \"d8004ab8-dd2d-4a64-91bb-3a2da4acd1a6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c142c5-6f45675c5d-gb4vs"
Sep 10 22:25:34.565810 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:25:34.565780    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g86l4\" (UniqueName: \"kubernetes.io/projected/d8004ab8-dd2d-4a64-91bb-3a2da4acd1a6-kube-api-access-g86l4\") pod \"payload-pre-processing-e2e-api-b-c142c5-6f45675c5d-gb4vs\" (UID: \"d8004ab8-dd2d-4a64-91bb-3a2da4acd1a6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c142c5-6f45675c5d-gb4vs"
Sep 10 22:25:34.589717 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:25:34.589696    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c142c5-6f45675c5d-gb4vs"
Sep 10 22:25:34.889612 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:25:34.889587    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c142c5-6f45675c5d-gb4vs"]
Sep 10 22:25:34.892248 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:25:34.892213    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8004ab8_dd2d_4a64_91bb_3a2da4acd1a6.slice/crio-841bda7ae491d52c50d410cf77f78b863bea91330574c5180e38b7d4e65b2f6f WatchSource:0}: Error finding container 841bda7ae491d52c50d410cf77f78b863bea91330574c5180e38b7d4e65b2f6f: Status 404 returned error can't find the container with id 841bda7ae491d52c50d410cf77f78b863bea91330574c5180e38b7d4e65b2f6f
Sep 10 22:25:35.253011 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:25:35.252974    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c142c5-6f45675c5d-gb4vs" event={"ID":"d8004ab8-dd2d-4a64-91bb-3a2da4acd1a6","Type":"ContainerStarted","Data":"b80d1c6b1106c201163e4127db4d0a423a0b476bdbc81ccbe5430c9bae135da3"}
Sep 10 22:25:35.253188 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:25:35.253027    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c142c5-6f45675c5d-gb4vs" event={"ID":"d8004ab8-dd2d-4a64-91bb-3a2da4acd1a6","Type":"ContainerStarted","Data":"841bda7ae491d52c50d410cf77f78b863bea91330574c5180e38b7d4e65b2f6f"}
Sep 10 22:25:35.253188 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:25:35.253121    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c142c5-6f45675c5d-gb4vs"
Sep 10 22:25:35.283244 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:25:35.283197    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c142c5-6f45675c5d-gb4vs" podStartSLOduration=1.2831825860000001 podStartE2EDuration="1.283182586s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 22:25:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:25:35.279972456 +0000 UTC m=+1456.236040163" watchObservedRunningTime="2026-09-10 22:25:35.283182586 +0000 UTC m=+1456.239250335"
Sep 10 22:25:36.048960 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:25:36.048925    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-ec6d47/sub-test-model-ec6d47-kserve-64647bffb5-sprnx"
Sep 10 22:25:36.049138 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:25:36.049045    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-ec6d47/sub-test-model-ec6d47-kserve-64647bffb5-sprnx"
Sep 10 22:25:36.061202 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:25:36.061171    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-ec6d47/sub-test-model-ec6d47-kserve-64647bffb5-sprnx"
Sep 10 22:25:36.269384 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:25:36.269359    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-ec6d47/sub-test-model-ec6d47-kserve-64647bffb5-sprnx"
Sep 10 22:25:46.260651 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:25:46.260620    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c142c5-6f45675c5d-gb4vs"
Sep 10 22:26:04.244881 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:04.244824    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9c0e33-7rcnj"]
Sep 10 22:26:04.248527 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:04.248510    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9c0e33-7rcnj"
Sep 10 22:26:04.250640 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:04.250617    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 22:26:04.250723 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:04.250677    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8hjrv\""
Sep 10 22:26:04.261151 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:04.261128    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9c0e33-7rcnj"]
Sep 10 22:26:04.314721 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:04.314695    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b546c17-c9f7-41be-831e-b43579c1409b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-9c0e33-7rcnj\" (UID: \"2b546c17-c9f7-41be-831e-b43579c1409b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9c0e33-7rcnj"
Sep 10 22:26:04.415259 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:04.415232    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b546c17-c9f7-41be-831e-b43579c1409b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-9c0e33-7rcnj\" (UID: \"2b546c17-c9f7-41be-831e-b43579c1409b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9c0e33-7rcnj"
Sep 10 22:26:04.415822 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:04.415804    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b546c17-c9f7-41be-831e-b43579c1409b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-9c0e33-7rcnj\" (UID: \"2b546c17-c9f7-41be-831e-b43579c1409b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9c0e33-7rcnj"
Sep 10 22:26:04.558488 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:04.558391    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9c0e33-7rcnj"
Sep 10 22:26:04.831963 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:04.831938    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9c0e33-7rcnj"]
Sep 10 22:26:04.833769 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:26:04.833741    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b546c17_c9f7_41be_831e_b43579c1409b.slice/crio-7e8bc182bb821e6a9d3e17ff287a4141ea4391901229d864701fa731b3916bbc WatchSource:0}: Error finding container 7e8bc182bb821e6a9d3e17ff287a4141ea4391901229d864701fa731b3916bbc: Status 404 returned error can't find the container with id 7e8bc182bb821e6a9d3e17ff287a4141ea4391901229d864701fa731b3916bbc
Sep 10 22:26:05.399774 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:05.399691    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9c0e33-7rcnj" event={"ID":"2b546c17-c9f7-41be-831e-b43579c1409b","Type":"ContainerStarted","Data":"65846ba9a26559973e19f432a7542f4bf9f6ba76d062de0b23a296997c6588b4"}
Sep 10 22:26:05.399774 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:05.399743    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9c0e33-7rcnj" event={"ID":"2b546c17-c9f7-41be-831e-b43579c1409b","Type":"ContainerStarted","Data":"7e8bc182bb821e6a9d3e17ff287a4141ea4391901229d864701fa731b3916bbc"}
Sep 10 22:26:05.440788 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:05.440738    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9c0e33-7rcnj" podStartSLOduration=1.440722657 podStartE2EDuration="1.440722657s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 22:26:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:26:05.439461743 +0000 UTC m=+1486.395529448" watchObservedRunningTime="2026-09-10 22:26:05.440722657 +0000 UTC m=+1486.396790392"
Sep 10 22:26:06.407254 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:06.407221    2582 generic.go:356] "Generic (PLEG): container finished" podID="2b546c17-c9f7-41be-831e-b43579c1409b" containerID="65846ba9a26559973e19f432a7542f4bf9f6ba76d062de0b23a296997c6588b4" exitCode=0
Sep 10 22:26:06.407726 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:06.407297    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9c0e33-7rcnj" event={"ID":"2b546c17-c9f7-41be-831e-b43579c1409b","Type":"ContainerDied","Data":"65846ba9a26559973e19f432a7542f4bf9f6ba76d062de0b23a296997c6588b4"}
Sep 10 22:26:07.538358 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:07.538333    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9c0e33-7rcnj"
Sep 10 22:26:07.650232 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:07.650193    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b546c17-c9f7-41be-831e-b43579c1409b-maas-api-service-ca\") pod \"2b546c17-c9f7-41be-831e-b43579c1409b\" (UID: \"2b546c17-c9f7-41be-831e-b43579c1409b\") "
Sep 10 22:26:07.650590 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:07.650564    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b546c17-c9f7-41be-831e-b43579c1409b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2b546c17-c9f7-41be-831e-b43579c1409b" (UID: "2b546c17-c9f7-41be-831e-b43579c1409b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 22:26:07.751550 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:07.751460    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b546c17-c9f7-41be-831e-b43579c1409b-maas-api-service-ca\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:26:08.418107 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:08.418074    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9c0e33-7rcnj"
Sep 10 22:26:08.418289 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:08.418082    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9c0e33-7rcnj" event={"ID":"2b546c17-c9f7-41be-831e-b43579c1409b","Type":"ContainerDied","Data":"7e8bc182bb821e6a9d3e17ff287a4141ea4391901229d864701fa731b3916bbc"}
Sep 10 22:26:08.418289 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:08.418197    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7e8bc182bb821e6a9d3e17ff287a4141ea4391901229d864701fa731b3916bbc"
Sep 10 22:26:19.535233 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:19.535192    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-w6vlb_8517c9a7-382a-40fd-bc52-e6da90ebc584/console-operator/2.log"
Sep 10 22:26:19.539062 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:19.539031    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-w6vlb_8517c9a7-382a-40fd-bc52-e6da90ebc584/console-operator/2.log"
Sep 10 22:26:19.834251 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:19.834167    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-9c0e33-75ddc78994-fn98c"]
Sep 10 22:26:19.834437 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:19.834394    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9c0e33-75ddc78994-fn98c" podUID="cb9c1682-4a88-4f66-8007-39e4c508d576" containerName="payload-pre-processing" containerID="cri-o://a1fd684562fd355e6651edc76427a55304e210f483b1ffaa4e32c925b0ba7069" gracePeriod=30
Sep 10 22:26:20.025495 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:20.025453    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9c0e33-7rcnj"]
Sep 10 22:26:20.027643 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:20.027615    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9c0e33-7rcnj"]
Sep 10 22:26:20.086492 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:20.086415    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9c0e33-75ddc78994-fn98c"
Sep 10 22:26:20.282150 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:20.282119    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb9c1682-4a88-4f66-8007-39e4c508d576-plugins-config-volume\") pod \"cb9c1682-4a88-4f66-8007-39e4c508d576\" (UID: \"cb9c1682-4a88-4f66-8007-39e4c508d576\") "
Sep 10 22:26:20.282316 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:20.282176    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6cph4\" (UniqueName: \"kubernetes.io/projected/cb9c1682-4a88-4f66-8007-39e4c508d576-kube-api-access-6cph4\") pod \"cb9c1682-4a88-4f66-8007-39e4c508d576\" (UID: \"cb9c1682-4a88-4f66-8007-39e4c508d576\") "
Sep 10 22:26:20.282572 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:20.282543    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb9c1682-4a88-4f66-8007-39e4c508d576-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "cb9c1682-4a88-4f66-8007-39e4c508d576" (UID: "cb9c1682-4a88-4f66-8007-39e4c508d576"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 22:26:20.284262 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:20.284235    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb9c1682-4a88-4f66-8007-39e4c508d576-kube-api-access-6cph4" (OuterVolumeSpecName: "kube-api-access-6cph4") pod "cb9c1682-4a88-4f66-8007-39e4c508d576" (UID: "cb9c1682-4a88-4f66-8007-39e4c508d576"). InnerVolumeSpecName "kube-api-access-6cph4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 22:26:20.382958 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:20.382903    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb9c1682-4a88-4f66-8007-39e4c508d576-plugins-config-volume\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:26:20.382958 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:20.382923    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6cph4\" (UniqueName: \"kubernetes.io/projected/cb9c1682-4a88-4f66-8007-39e4c508d576-kube-api-access-6cph4\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:26:20.480776 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:20.480744    2582 generic.go:356] "Generic (PLEG): container finished" podID="cb9c1682-4a88-4f66-8007-39e4c508d576" containerID="a1fd684562fd355e6651edc76427a55304e210f483b1ffaa4e32c925b0ba7069" exitCode=0
Sep 10 22:26:20.480904 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:20.480793    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9c0e33-75ddc78994-fn98c"
Sep 10 22:26:20.480904 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:20.480823    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9c0e33-75ddc78994-fn98c" event={"ID":"cb9c1682-4a88-4f66-8007-39e4c508d576","Type":"ContainerDied","Data":"a1fd684562fd355e6651edc76427a55304e210f483b1ffaa4e32c925b0ba7069"}
Sep 10 22:26:20.480998 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:20.480917    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9c0e33-75ddc78994-fn98c" event={"ID":"cb9c1682-4a88-4f66-8007-39e4c508d576","Type":"ContainerDied","Data":"e7a7b21b2a998eabe49d49b9deeb96afc9fd06350769e220929cdd499cff6978"}
Sep 10 22:26:20.480998 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:20.480939    2582 scope.go:117] "RemoveContainer" containerID="a1fd684562fd355e6651edc76427a55304e210f483b1ffaa4e32c925b0ba7069"
Sep 10 22:26:20.489600 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:20.489580    2582 scope.go:117] "RemoveContainer" containerID="a1fd684562fd355e6651edc76427a55304e210f483b1ffaa4e32c925b0ba7069"
Sep 10 22:26:20.489892 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:26:20.489871    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1fd684562fd355e6651edc76427a55304e210f483b1ffaa4e32c925b0ba7069\": container with ID starting with a1fd684562fd355e6651edc76427a55304e210f483b1ffaa4e32c925b0ba7069 not found: ID does not exist" containerID="a1fd684562fd355e6651edc76427a55304e210f483b1ffaa4e32c925b0ba7069"
Sep 10 22:26:20.489950 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:20.489902    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1fd684562fd355e6651edc76427a55304e210f483b1ffaa4e32c925b0ba7069"} err="failed to get container status \"a1fd684562fd355e6651edc76427a55304e210f483b1ffaa4e32c925b0ba7069\": rpc error: code = NotFound desc = could not find container \"a1fd684562fd355e6651edc76427a55304e210f483b1ffaa4e32c925b0ba7069\": container with ID starting with a1fd684562fd355e6651edc76427a55304e210f483b1ffaa4e32c925b0ba7069 not found: ID does not exist"
Sep 10 22:26:20.500077 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:20.500052    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-9c0e33-75ddc78994-fn98c"]
Sep 10 22:26:20.503030 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:20.503009    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-9c0e33-75ddc78994-fn98c"]
Sep 10 22:26:21.498380 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:21.498349    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2b546c17-c9f7-41be-831e-b43579c1409b" path="/var/lib/kubelet/pods/2b546c17-c9f7-41be-831e-b43579c1409b/volumes"
Sep 10 22:26:21.498748 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:21.498692    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cb9c1682-4a88-4f66-8007-39e4c508d576" path="/var/lib/kubelet/pods/cb9c1682-4a88-4f66-8007-39e4c508d576/volumes"
Sep 10 22:26:23.454106 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:23.454072    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7511-ef7ce8"]
Sep 10 22:26:23.454475 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:23.454410    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b546c17-c9f7-41be-831e-b43579c1409b" containerName="revoke-keys"
Sep 10 22:26:23.454475 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:23.454421    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b546c17-c9f7-41be-831e-b43579c1409b" containerName="revoke-keys"
Sep 10 22:26:23.454475 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:23.454436    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb9c1682-4a88-4f66-8007-39e4c508d576" containerName="payload-pre-processing"
Sep 10 22:26:23.454475 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:23.454442    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb9c1682-4a88-4f66-8007-39e4c508d576" containerName="payload-pre-processing"
Sep 10 22:26:23.454604 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:23.454505    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b546c17-c9f7-41be-831e-b43579c1409b" containerName="revoke-keys"
Sep 10 22:26:23.454604 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:23.454514    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb9c1682-4a88-4f66-8007-39e4c508d576" containerName="payload-pre-processing"
Sep 10 22:26:23.458797 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:23.458778    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7511-ef7ce8"
Sep 10 22:26:23.460883 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:23.460836    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-znq4v\""
Sep 10 22:26:23.463443 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:23.463416    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7511-ef7ce8"]
Sep 10 22:26:23.509200 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:23.509173    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bs4tw\" (UniqueName: \"kubernetes.io/projected/99c309d2-4ca6-40eb-9863-b21dd6808c8b-kube-api-access-bs4tw\") pod \"test-curl-7511-ef7ce8\" (UID: \"99c309d2-4ca6-40eb-9863-b21dd6808c8b\") " pod="opendatahub/test-curl-7511-ef7ce8"
Sep 10 22:26:23.610392 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:23.610367    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bs4tw\" (UniqueName: \"kubernetes.io/projected/99c309d2-4ca6-40eb-9863-b21dd6808c8b-kube-api-access-bs4tw\") pod \"test-curl-7511-ef7ce8\" (UID: \"99c309d2-4ca6-40eb-9863-b21dd6808c8b\") " pod="opendatahub/test-curl-7511-ef7ce8"
Sep 10 22:26:23.617733 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:23.617715    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bs4tw\" (UniqueName: \"kubernetes.io/projected/99c309d2-4ca6-40eb-9863-b21dd6808c8b-kube-api-access-bs4tw\") pod \"test-curl-7511-ef7ce8\" (UID: \"99c309d2-4ca6-40eb-9863-b21dd6808c8b\") " pod="opendatahub/test-curl-7511-ef7ce8"
Sep 10 22:26:23.769917 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:23.769862    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7511-ef7ce8"
Sep 10 22:26:24.044417 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:24.044380    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7511-ef7ce8"]
Sep 10 22:26:24.503439 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:24.503399    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7511-ef7ce8" event={"ID":"99c309d2-4ca6-40eb-9863-b21dd6808c8b","Type":"ContainerStarted","Data":"f799f0d6362dc1f7f0997260ccf28fefc80bed4371afef920adf5c6ba43ff95f"}
Sep 10 22:26:24.503439 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:24.503442    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7511-ef7ce8" event={"ID":"99c309d2-4ca6-40eb-9863-b21dd6808c8b","Type":"ContainerStarted","Data":"662b9779c410b1e2c4810a8dbb32c8dba703751a536ea8ff4398343ebf4cf4d4"}
Sep 10 22:26:24.526105 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:24.526047    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7511-ef7ce8" podStartSLOduration=1.526032643 podStartE2EDuration="1.526032643s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 22:26:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:26:24.523433179 +0000 UTC m=+1505.479500885" watchObservedRunningTime="2026-09-10 22:26:24.526032643 +0000 UTC m=+1505.482100348"
Sep 10 22:26:25.130841 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:25.130805    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7511-ef7ce8"]
Sep 10 22:26:25.135457 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:25.135431    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7511-ef7ce8"]
Sep 10 22:26:25.508140 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:26:25.508112    2582 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7511-ef7ce8" container="test-curl-7511-ef7ce8"
Sep 10 22:26:26.513318 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:26.513257    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7511-ef7ce8" podUID="99c309d2-4ca6-40eb-9863-b21dd6808c8b" containerName="test-curl-7511-ef7ce8" containerID="cri-o://f799f0d6362dc1f7f0997260ccf28fefc80bed4371afef920adf5c6ba43ff95f" gracePeriod=2
Sep 10 22:26:26.839823 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:26.839746    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c142c5-sxlzv"]
Sep 10 22:26:26.840142 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:26.840129    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99c309d2-4ca6-40eb-9863-b21dd6808c8b" containerName="test-curl-7511-ef7ce8"
Sep 10 22:26:26.840190 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:26.840144    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="99c309d2-4ca6-40eb-9863-b21dd6808c8b" containerName="test-curl-7511-ef7ce8"
Sep 10 22:26:26.840233 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:26.840207    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="99c309d2-4ca6-40eb-9863-b21dd6808c8b" containerName="test-curl-7511-ef7ce8"
Sep 10 22:26:26.843291 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:26.843274    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c142c5-sxlzv"
Sep 10 22:26:26.849963 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:26.849939    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 22:26:26.850086 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:26.850032    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8hjrv\""
Sep 10 22:26:26.868465 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:26.868439    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c142c5-sxlzv"]
Sep 10 22:26:26.939272 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:26.939246    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/76e6b1b8-ff59-4b88-9dcd-b6743f9adfa9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-c142c5-sxlzv\" (UID: \"76e6b1b8-ff59-4b88-9dcd-b6743f9adfa9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c142c5-sxlzv"
Sep 10 22:26:27.040395 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:27.040368    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/76e6b1b8-ff59-4b88-9dcd-b6743f9adfa9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-c142c5-sxlzv\" (UID: \"76e6b1b8-ff59-4b88-9dcd-b6743f9adfa9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c142c5-sxlzv"
Sep 10 22:26:27.040972 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:27.040952    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/76e6b1b8-ff59-4b88-9dcd-b6743f9adfa9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-c142c5-sxlzv\" (UID: \"76e6b1b8-ff59-4b88-9dcd-b6743f9adfa9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c142c5-sxlzv"
Sep 10 22:26:27.150515 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:27.150441    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-d07dfb/sub-test-model-d07dfb-kserve-78c8db4f9c-hp4z6"]
Sep 10 22:26:27.153134 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:27.153109    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c142c5-sxlzv"
Sep 10 22:26:27.423977 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:27.423954    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c142c5-sxlzv"]
Sep 10 22:26:27.426175 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:26:27.426150    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76e6b1b8_ff59_4b88_9dcd_b6743f9adfa9.slice/crio-112eb4ba4b373b2bdf1b45c2a317cee70131d526fca14bce1f23b8aa45099d5d WatchSource:0}: Error finding container 112eb4ba4b373b2bdf1b45c2a317cee70131d526fca14bce1f23b8aa45099d5d: Status 404 returned error can't find the container with id 112eb4ba4b373b2bdf1b45c2a317cee70131d526fca14bce1f23b8aa45099d5d
Sep 10 22:26:27.518841 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:27.518817    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c142c5-sxlzv" event={"ID":"76e6b1b8-ff59-4b88-9dcd-b6743f9adfa9","Type":"ContainerStarted","Data":"112eb4ba4b373b2bdf1b45c2a317cee70131d526fca14bce1f23b8aa45099d5d"}
Sep 10 22:26:28.167233 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:28.167195    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-ec6d47/sub-test-model-ec6d47-kserve-64647bffb5-sprnx"]
Sep 10 22:26:28.525573 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:28.525544    2582 generic.go:356] "Generic (PLEG): container finished" podID="76e6b1b8-ff59-4b88-9dcd-b6743f9adfa9" containerID="3b73caebf662d7a931f61c9503957ba1f3bde86d7f8400fa94c9e5ba23c22bd7" exitCode=0
Sep 10 22:26:28.525900 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:28.525625    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c142c5-sxlzv" event={"ID":"76e6b1b8-ff59-4b88-9dcd-b6743f9adfa9","Type":"ContainerDied","Data":"3b73caebf662d7a931f61c9503957ba1f3bde86d7f8400fa94c9e5ba23c22bd7"}
Sep 10 22:26:28.691936 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:28.691908    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7511-ef7ce8"
Sep 10 22:26:28.756299 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:28.756272    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bs4tw\" (UniqueName: \"kubernetes.io/projected/99c309d2-4ca6-40eb-9863-b21dd6808c8b-kube-api-access-bs4tw\") pod \"99c309d2-4ca6-40eb-9863-b21dd6808c8b\" (UID: \"99c309d2-4ca6-40eb-9863-b21dd6808c8b\") "
Sep 10 22:26:28.758300 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:28.758279    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99c309d2-4ca6-40eb-9863-b21dd6808c8b-kube-api-access-bs4tw" (OuterVolumeSpecName: "kube-api-access-bs4tw") pod "99c309d2-4ca6-40eb-9863-b21dd6808c8b" (UID: "99c309d2-4ca6-40eb-9863-b21dd6808c8b"). InnerVolumeSpecName "kube-api-access-bs4tw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 22:26:28.857803 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:28.857728    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bs4tw\" (UniqueName: \"kubernetes.io/projected/99c309d2-4ca6-40eb-9863-b21dd6808c8b-kube-api-access-bs4tw\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:26:29.499337 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:29.499298    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="99c309d2-4ca6-40eb-9863-b21dd6808c8b" path="/var/lib/kubelet/pods/99c309d2-4ca6-40eb-9863-b21dd6808c8b/volumes"
Sep 10 22:26:29.531441 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:29.531413    2582 generic.go:356] "Generic (PLEG): container finished" podID="99c309d2-4ca6-40eb-9863-b21dd6808c8b" containerID="f799f0d6362dc1f7f0997260ccf28fefc80bed4371afef920adf5c6ba43ff95f" exitCode=137
Sep 10 22:26:29.531784 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:29.531454    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7511-ef7ce8"
Sep 10 22:26:29.531784 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:29.531522    2582 scope.go:117] "RemoveContainer" containerID="f799f0d6362dc1f7f0997260ccf28fefc80bed4371afef920adf5c6ba43ff95f"
Sep 10 22:26:29.556782 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:29.556761    2582 scope.go:117] "RemoveContainer" containerID="f799f0d6362dc1f7f0997260ccf28fefc80bed4371afef920adf5c6ba43ff95f"
Sep 10 22:26:29.557163 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:26:29.557138    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f799f0d6362dc1f7f0997260ccf28fefc80bed4371afef920adf5c6ba43ff95f\": container with ID starting with f799f0d6362dc1f7f0997260ccf28fefc80bed4371afef920adf5c6ba43ff95f not found: ID does not exist" containerID="f799f0d6362dc1f7f0997260ccf28fefc80bed4371afef920adf5c6ba43ff95f"
Sep 10 22:26:29.557262 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:29.557174    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f799f0d6362dc1f7f0997260ccf28fefc80bed4371afef920adf5c6ba43ff95f"} err="failed to get container status \"f799f0d6362dc1f7f0997260ccf28fefc80bed4371afef920adf5c6ba43ff95f\": rpc error: code = NotFound desc = could not find container \"f799f0d6362dc1f7f0997260ccf28fefc80bed4371afef920adf5c6ba43ff95f\": container with ID starting with f799f0d6362dc1f7f0997260ccf28fefc80bed4371afef920adf5c6ba43ff95f not found: ID does not exist"
Sep 10 22:26:29.663431 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:29.663405    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c142c5-sxlzv"
Sep 10 22:26:29.768049 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:29.767950    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/76e6b1b8-ff59-4b88-9dcd-b6743f9adfa9-maas-api-service-ca\") pod \"76e6b1b8-ff59-4b88-9dcd-b6743f9adfa9\" (UID: \"76e6b1b8-ff59-4b88-9dcd-b6743f9adfa9\") "
Sep 10 22:26:29.768398 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:29.768372    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76e6b1b8-ff59-4b88-9dcd-b6743f9adfa9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "76e6b1b8-ff59-4b88-9dcd-b6743f9adfa9" (UID: "76e6b1b8-ff59-4b88-9dcd-b6743f9adfa9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 22:26:29.868703 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:29.868664    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/76e6b1b8-ff59-4b88-9dcd-b6743f9adfa9-maas-api-service-ca\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:26:30.538202 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:30.538168    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c142c5-sxlzv"
Sep 10 22:26:30.538202 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:30.538183    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c142c5-sxlzv" event={"ID":"76e6b1b8-ff59-4b88-9dcd-b6743f9adfa9","Type":"ContainerDied","Data":"112eb4ba4b373b2bdf1b45c2a317cee70131d526fca14bce1f23b8aa45099d5d"}
Sep 10 22:26:30.538682 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:30.538226    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="112eb4ba4b373b2bdf1b45c2a317cee70131d526fca14bce1f23b8aa45099d5d"
Sep 10 22:26:33.487434 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:33.487398    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-d07dfb/rate-test-model-d07dfb-kserve-5dbf77f559-fttfk"]
Sep 10 22:26:33.487925 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:33.487904    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76e6b1b8-ff59-4b88-9dcd-b6743f9adfa9" containerName="revoke-keys"
Sep 10 22:26:33.487925 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:33.487923    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="76e6b1b8-ff59-4b88-9dcd-b6743f9adfa9" containerName="revoke-keys"
Sep 10 22:26:33.488075 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:33.487987    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="76e6b1b8-ff59-4b88-9dcd-b6743f9adfa9" containerName="revoke-keys"
Sep 10 22:26:33.491017 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:33.491000    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-d07dfb/rate-test-model-d07dfb-kserve-5dbf77f559-fttfk"
Sep 10 22:26:33.493074 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:33.493054    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-d07dfb\"/\"rate-test-model-d07dfb-kserve-self-signed-certs\""
Sep 10 22:26:33.501416 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:33.501393    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-d07dfb/rate-test-model-d07dfb-kserve-5dbf77f559-fttfk"]
Sep 10 22:26:33.602280 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:33.602250    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jr56\" (UniqueName: \"kubernetes.io/projected/8046dbac-5187-4501-aad4-682083e61e14-kube-api-access-7jr56\") pod \"rate-test-model-d07dfb-kserve-5dbf77f559-fttfk\" (UID: \"8046dbac-5187-4501-aad4-682083e61e14\") " pod="ai-tenant-e2e-shared-a-w3-d07dfb/rate-test-model-d07dfb-kserve-5dbf77f559-fttfk"
Sep 10 22:26:33.602428 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:33.602329    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8046dbac-5187-4501-aad4-682083e61e14-tls-certs\") pod \"rate-test-model-d07dfb-kserve-5dbf77f559-fttfk\" (UID: \"8046dbac-5187-4501-aad4-682083e61e14\") " pod="ai-tenant-e2e-shared-a-w3-d07dfb/rate-test-model-d07dfb-kserve-5dbf77f559-fttfk"
Sep 10 22:26:33.602428 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:33.602390    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8046dbac-5187-4501-aad4-682083e61e14-model-cache\") pod \"rate-test-model-d07dfb-kserve-5dbf77f559-fttfk\" (UID: \"8046dbac-5187-4501-aad4-682083e61e14\") " pod="ai-tenant-e2e-shared-a-w3-d07dfb/rate-test-model-d07dfb-kserve-5dbf77f559-fttfk"
Sep 10 22:26:33.602567 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:33.602449    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8046dbac-5187-4501-aad4-682083e61e14-dshm\") pod \"rate-test-model-d07dfb-kserve-5dbf77f559-fttfk\" (UID: \"8046dbac-5187-4501-aad4-682083e61e14\") " pod="ai-tenant-e2e-shared-a-w3-d07dfb/rate-test-model-d07dfb-kserve-5dbf77f559-fttfk"
Sep 10 22:26:33.602567 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:33.602492    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8046dbac-5187-4501-aad4-682083e61e14-home\") pod \"rate-test-model-d07dfb-kserve-5dbf77f559-fttfk\" (UID: \"8046dbac-5187-4501-aad4-682083e61e14\") " pod="ai-tenant-e2e-shared-a-w3-d07dfb/rate-test-model-d07dfb-kserve-5dbf77f559-fttfk"
Sep 10 22:26:33.602567 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:33.602534    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8046dbac-5187-4501-aad4-682083e61e14-tmp-dir\") pod \"rate-test-model-d07dfb-kserve-5dbf77f559-fttfk\" (UID: \"8046dbac-5187-4501-aad4-682083e61e14\") " pod="ai-tenant-e2e-shared-a-w3-d07dfb/rate-test-model-d07dfb-kserve-5dbf77f559-fttfk"
Sep 10 22:26:33.703811 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:33.703775    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8046dbac-5187-4501-aad4-682083e61e14-dshm\") pod \"rate-test-model-d07dfb-kserve-5dbf77f559-fttfk\" (UID: \"8046dbac-5187-4501-aad4-682083e61e14\") " pod="ai-tenant-e2e-shared-a-w3-d07dfb/rate-test-model-d07dfb-kserve-5dbf77f559-fttfk"
Sep 10 22:26:33.704029 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:33.703815    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8046dbac-5187-4501-aad4-682083e61e14-home\") pod \"rate-test-model-d07dfb-kserve-5dbf77f559-fttfk\" (UID: \"8046dbac-5187-4501-aad4-682083e61e14\") " pod="ai-tenant-e2e-shared-a-w3-d07dfb/rate-test-model-d07dfb-kserve-5dbf77f559-fttfk"
Sep 10 22:26:33.704029 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:33.703844    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8046dbac-5187-4501-aad4-682083e61e14-tmp-dir\") pod \"rate-test-model-d07dfb-kserve-5dbf77f559-fttfk\" (UID: \"8046dbac-5187-4501-aad4-682083e61e14\") " pod="ai-tenant-e2e-shared-a-w3-d07dfb/rate-test-model-d07dfb-kserve-5dbf77f559-fttfk"
Sep 10 22:26:33.704029 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:33.703902    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jr56\" (UniqueName: \"kubernetes.io/projected/8046dbac-5187-4501-aad4-682083e61e14-kube-api-access-7jr56\") pod \"rate-test-model-d07dfb-kserve-5dbf77f559-fttfk\" (UID: \"8046dbac-5187-4501-aad4-682083e61e14\") " pod="ai-tenant-e2e-shared-a-w3-d07dfb/rate-test-model-d07dfb-kserve-5dbf77f559-fttfk"
Sep 10 22:26:33.704029 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:33.703974    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8046dbac-5187-4501-aad4-682083e61e14-tls-certs\") pod \"rate-test-model-d07dfb-kserve-5dbf77f559-fttfk\" (UID: \"8046dbac-5187-4501-aad4-682083e61e14\") " pod="ai-tenant-e2e-shared-a-w3-d07dfb/rate-test-model-d07dfb-kserve-5dbf77f559-fttfk"
Sep 10 22:26:33.704029 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:33.704016    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8046dbac-5187-4501-aad4-682083e61e14-model-cache\") pod \"rate-test-model-d07dfb-kserve-5dbf77f559-fttfk\" (UID: \"8046dbac-5187-4501-aad4-682083e61e14\") " pod="ai-tenant-e2e-shared-a-w3-d07dfb/rate-test-model-d07dfb-kserve-5dbf77f559-fttfk"
Sep 10 22:26:33.704287 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:33.704217    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8046dbac-5187-4501-aad4-682083e61e14-home\") pod \"rate-test-model-d07dfb-kserve-5dbf77f559-fttfk\" (UID: \"8046dbac-5187-4501-aad4-682083e61e14\") " pod="ai-tenant-e2e-shared-a-w3-d07dfb/rate-test-model-d07dfb-kserve-5dbf77f559-fttfk"
Sep 10 22:26:33.704341 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:33.704284    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8046dbac-5187-4501-aad4-682083e61e14-tmp-dir\") pod \"rate-test-model-d07dfb-kserve-5dbf77f559-fttfk\" (UID: \"8046dbac-5187-4501-aad4-682083e61e14\") " pod="ai-tenant-e2e-shared-a-w3-d07dfb/rate-test-model-d07dfb-kserve-5dbf77f559-fttfk"
Sep 10 22:26:33.704427 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:33.704406    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8046dbac-5187-4501-aad4-682083e61e14-model-cache\") pod \"rate-test-model-d07dfb-kserve-5dbf77f559-fttfk\" (UID: \"8046dbac-5187-4501-aad4-682083e61e14\") " pod="ai-tenant-e2e-shared-a-w3-d07dfb/rate-test-model-d07dfb-kserve-5dbf77f559-fttfk"
Sep 10 22:26:33.706086 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:33.706059    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8046dbac-5187-4501-aad4-682083e61e14-dshm\") pod \"rate-test-model-d07dfb-kserve-5dbf77f559-fttfk\" (UID: \"8046dbac-5187-4501-aad4-682083e61e14\") " pod="ai-tenant-e2e-shared-a-w3-d07dfb/rate-test-model-d07dfb-kserve-5dbf77f559-fttfk"
Sep 10 22:26:33.706313 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:33.706295    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8046dbac-5187-4501-aad4-682083e61e14-tls-certs\") pod \"rate-test-model-d07dfb-kserve-5dbf77f559-fttfk\" (UID: \"8046dbac-5187-4501-aad4-682083e61e14\") " pod="ai-tenant-e2e-shared-a-w3-d07dfb/rate-test-model-d07dfb-kserve-5dbf77f559-fttfk"
Sep 10 22:26:33.712181 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:33.712160    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jr56\" (UniqueName: \"kubernetes.io/projected/8046dbac-5187-4501-aad4-682083e61e14-kube-api-access-7jr56\") pod \"rate-test-model-d07dfb-kserve-5dbf77f559-fttfk\" (UID: \"8046dbac-5187-4501-aad4-682083e61e14\") " pod="ai-tenant-e2e-shared-a-w3-d07dfb/rate-test-model-d07dfb-kserve-5dbf77f559-fttfk"
Sep 10 22:26:33.801877 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:33.801801    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-d07dfb/rate-test-model-d07dfb-kserve-5dbf77f559-fttfk"
Sep 10 22:26:34.098785 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:34.098753    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-d07dfb/rate-test-model-d07dfb-kserve-5dbf77f559-fttfk"]
Sep 10 22:26:34.100512 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:26:34.100487    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8046dbac_5187_4501_aad4_682083e61e14.slice/crio-2443539f38cd1cd282d38962cb1a3917cd1e2f9f92850526dd4e800f141ffa0a WatchSource:0}: Error finding container 2443539f38cd1cd282d38962cb1a3917cd1e2f9f92850526dd4e800f141ffa0a: Status 404 returned error can't find the container with id 2443539f38cd1cd282d38962cb1a3917cd1e2f9f92850526dd4e800f141ffa0a
Sep 10 22:26:34.560215 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:34.560177    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-d07dfb/rate-test-model-d07dfb-kserve-5dbf77f559-fttfk" event={"ID":"8046dbac-5187-4501-aad4-682083e61e14","Type":"ContainerStarted","Data":"2f12e460f952b9ef3298f0ed04c23cb979fc20cfc0a13e6666d7cd62c5558ae6"}
Sep 10 22:26:34.560215 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:34.560219    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-d07dfb/rate-test-model-d07dfb-kserve-5dbf77f559-fttfk" event={"ID":"8046dbac-5187-4501-aad4-682083e61e14","Type":"ContainerStarted","Data":"2443539f38cd1cd282d38962cb1a3917cd1e2f9f92850526dd4e800f141ffa0a"}
Sep 10 22:26:34.579476 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:34.579429    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-d07dfb/rate-test-model-d07dfb-kserve-5dbf77f559-fttfk" podStartSLOduration=1.579414464 podStartE2EDuration="1.579414464s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 22:26:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:26:34.577743619 +0000 UTC m=+1515.533811328" watchObservedRunningTime="2026-09-10 22:26:34.579414464 +0000 UTC m=+1515.535482230"
Sep 10 22:26:42.170447 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:42.170350    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-d07dfb/sub-test-model-d07dfb-kserve-78c8db4f9c-hp4z6" podUID="ab05d721-124d-46c5-948f-6e83c5aa2854" containerName="main" containerID="cri-o://01dbeb2c2496cb14aff3068d160ba04e118a31d232e90444b51400756f6c0c1b" gracePeriod=45
Sep 10 22:26:42.411728 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:42.411704    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-d07dfb/sub-test-model-d07dfb-kserve-78c8db4f9c-hp4z6"
Sep 10 22:26:42.504245 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:42.504164    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8ndm2\" (UniqueName: \"kubernetes.io/projected/ab05d721-124d-46c5-948f-6e83c5aa2854-kube-api-access-8ndm2\") pod \"ab05d721-124d-46c5-948f-6e83c5aa2854\" (UID: \"ab05d721-124d-46c5-948f-6e83c5aa2854\") "
Sep 10 22:26:42.504245 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:42.504219    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab05d721-124d-46c5-948f-6e83c5aa2854-model-cache\") pod \"ab05d721-124d-46c5-948f-6e83c5aa2854\" (UID: \"ab05d721-124d-46c5-948f-6e83c5aa2854\") "
Sep 10 22:26:42.504411 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:42.504327    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab05d721-124d-46c5-948f-6e83c5aa2854-tmp-dir\") pod \"ab05d721-124d-46c5-948f-6e83c5aa2854\" (UID: \"ab05d721-124d-46c5-948f-6e83c5aa2854\") "
Sep 10 22:26:42.504411 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:42.504362    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab05d721-124d-46c5-948f-6e83c5aa2854-home\") pod \"ab05d721-124d-46c5-948f-6e83c5aa2854\" (UID: \"ab05d721-124d-46c5-948f-6e83c5aa2854\") "
Sep 10 22:26:42.504411 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:42.504400    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab05d721-124d-46c5-948f-6e83c5aa2854-tls-certs\") pod \"ab05d721-124d-46c5-948f-6e83c5aa2854\" (UID: \"ab05d721-124d-46c5-948f-6e83c5aa2854\") "
Sep 10 22:26:42.504546 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:42.504436    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab05d721-124d-46c5-948f-6e83c5aa2854-dshm\") pod \"ab05d721-124d-46c5-948f-6e83c5aa2854\" (UID: \"ab05d721-124d-46c5-948f-6e83c5aa2854\") "
Sep 10 22:26:42.504546 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:42.504522    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab05d721-124d-46c5-948f-6e83c5aa2854-model-cache" (OuterVolumeSpecName: "model-cache") pod "ab05d721-124d-46c5-948f-6e83c5aa2854" (UID: "ab05d721-124d-46c5-948f-6e83c5aa2854"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 22:26:42.504655 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:42.504573    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab05d721-124d-46c5-948f-6e83c5aa2854-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ab05d721-124d-46c5-948f-6e83c5aa2854" (UID: "ab05d721-124d-46c5-948f-6e83c5aa2854"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 22:26:42.504655 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:42.504618    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab05d721-124d-46c5-948f-6e83c5aa2854-home" (OuterVolumeSpecName: "home") pod "ab05d721-124d-46c5-948f-6e83c5aa2854" (UID: "ab05d721-124d-46c5-948f-6e83c5aa2854"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 22:26:42.504818 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:42.504799    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab05d721-124d-46c5-948f-6e83c5aa2854-model-cache\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:26:42.504899 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:42.504824    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab05d721-124d-46c5-948f-6e83c5aa2854-tmp-dir\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:26:42.504899 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:42.504838    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab05d721-124d-46c5-948f-6e83c5aa2854-home\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:26:42.506472 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:42.506440    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab05d721-124d-46c5-948f-6e83c5aa2854-kube-api-access-8ndm2" (OuterVolumeSpecName: "kube-api-access-8ndm2") pod "ab05d721-124d-46c5-948f-6e83c5aa2854" (UID: "ab05d721-124d-46c5-948f-6e83c5aa2854"). InnerVolumeSpecName "kube-api-access-8ndm2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 22:26:42.506573 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:42.506471    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab05d721-124d-46c5-948f-6e83c5aa2854-dshm" (OuterVolumeSpecName: "dshm") pod "ab05d721-124d-46c5-948f-6e83c5aa2854" (UID: "ab05d721-124d-46c5-948f-6e83c5aa2854"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 22:26:42.506767 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:42.506748    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab05d721-124d-46c5-948f-6e83c5aa2854-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ab05d721-124d-46c5-948f-6e83c5aa2854" (UID: "ab05d721-124d-46c5-948f-6e83c5aa2854"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 22:26:42.554249 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:42.554220    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c142c5-6f45675c5d-gb4vs"]
Sep 10 22:26:42.554451 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:42.554415    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c142c5-6f45675c5d-gb4vs" podUID="d8004ab8-dd2d-4a64-91bb-3a2da4acd1a6" containerName="payload-pre-processing" containerID="cri-o://b80d1c6b1106c201163e4127db4d0a423a0b476bdbc81ccbe5430c9bae135da3" gracePeriod=30
Sep 10 22:26:42.607407 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:42.607370    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ndm2\" (UniqueName: \"kubernetes.io/projected/ab05d721-124d-46c5-948f-6e83c5aa2854-kube-api-access-8ndm2\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:26:42.607407 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:42.607407    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab05d721-124d-46c5-948f-6e83c5aa2854-tls-certs\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:26:42.607610 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:42.607421    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab05d721-124d-46c5-948f-6e83c5aa2854-dshm\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:26:42.609375 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:42.609329    2582 generic.go:356] "Generic (PLEG): container finished" podID="ab05d721-124d-46c5-948f-6e83c5aa2854" containerID="01dbeb2c2496cb14aff3068d160ba04e118a31d232e90444b51400756f6c0c1b" exitCode=0
Sep 10 22:26:42.609506 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:42.609467    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-d07dfb/sub-test-model-d07dfb-kserve-78c8db4f9c-hp4z6" event={"ID":"ab05d721-124d-46c5-948f-6e83c5aa2854","Type":"ContainerDied","Data":"01dbeb2c2496cb14aff3068d160ba04e118a31d232e90444b51400756f6c0c1b"}
Sep 10 22:26:42.609571 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:42.609499    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-d07dfb/sub-test-model-d07dfb-kserve-78c8db4f9c-hp4z6"
Sep 10 22:26:42.609571 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:42.609560    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-d07dfb/sub-test-model-d07dfb-kserve-78c8db4f9c-hp4z6" event={"ID":"ab05d721-124d-46c5-948f-6e83c5aa2854","Type":"ContainerDied","Data":"4f67896a7705cf4653d7a22f2ad8590a4ffabc27ea54f6be259b77c35a4032b5"}
Sep 10 22:26:42.609674 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:42.609603    2582 scope.go:117] "RemoveContainer" containerID="01dbeb2c2496cb14aff3068d160ba04e118a31d232e90444b51400756f6c0c1b"
Sep 10 22:26:42.639252 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:42.639226    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-d07dfb/sub-test-model-d07dfb-kserve-78c8db4f9c-hp4z6"]
Sep 10 22:26:42.643534 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:42.643514    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-d07dfb/sub-test-model-d07dfb-kserve-78c8db4f9c-hp4z6"]
Sep 10 22:26:42.665634 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:42.665613    2582 scope.go:117] "RemoveContainer" containerID="01dbeb2c2496cb14aff3068d160ba04e118a31d232e90444b51400756f6c0c1b"
Sep 10 22:26:42.666031 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:26:42.666012    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01dbeb2c2496cb14aff3068d160ba04e118a31d232e90444b51400756f6c0c1b\": container with ID starting with 01dbeb2c2496cb14aff3068d160ba04e118a31d232e90444b51400756f6c0c1b not found: ID does not exist" containerID="01dbeb2c2496cb14aff3068d160ba04e118a31d232e90444b51400756f6c0c1b"
Sep 10 22:26:42.666102 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:42.666039    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01dbeb2c2496cb14aff3068d160ba04e118a31d232e90444b51400756f6c0c1b"} err="failed to get container status \"01dbeb2c2496cb14aff3068d160ba04e118a31d232e90444b51400756f6c0c1b\": rpc error: code = NotFound desc = could not find container \"01dbeb2c2496cb14aff3068d160ba04e118a31d232e90444b51400756f6c0c1b\": container with ID starting with 01dbeb2c2496cb14aff3068d160ba04e118a31d232e90444b51400756f6c0c1b not found: ID does not exist"
Sep 10 22:26:42.737001 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:42.736966    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c142c5-sxlzv"]
Sep 10 22:26:42.740001 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:42.739974    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c142c5-sxlzv"]
Sep 10 22:26:42.787822 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:42.787800    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c142c5-6f45675c5d-gb4vs"
Sep 10 22:26:42.910758 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:42.910722    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8004ab8-dd2d-4a64-91bb-3a2da4acd1a6-plugins-config-volume\") pod \"d8004ab8-dd2d-4a64-91bb-3a2da4acd1a6\" (UID: \"d8004ab8-dd2d-4a64-91bb-3a2da4acd1a6\") "
Sep 10 22:26:42.910974 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:42.910808    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g86l4\" (UniqueName: \"kubernetes.io/projected/d8004ab8-dd2d-4a64-91bb-3a2da4acd1a6-kube-api-access-g86l4\") pod \"d8004ab8-dd2d-4a64-91bb-3a2da4acd1a6\" (UID: \"d8004ab8-dd2d-4a64-91bb-3a2da4acd1a6\") "
Sep 10 22:26:42.911126 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:42.911104    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8004ab8-dd2d-4a64-91bb-3a2da4acd1a6-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d8004ab8-dd2d-4a64-91bb-3a2da4acd1a6" (UID: "d8004ab8-dd2d-4a64-91bb-3a2da4acd1a6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 22:26:42.912825 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:42.912802    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8004ab8-dd2d-4a64-91bb-3a2da4acd1a6-kube-api-access-g86l4" (OuterVolumeSpecName: "kube-api-access-g86l4") pod "d8004ab8-dd2d-4a64-91bb-3a2da4acd1a6" (UID: "d8004ab8-dd2d-4a64-91bb-3a2da4acd1a6"). InnerVolumeSpecName "kube-api-access-g86l4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 22:26:43.012334 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:43.012298    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8004ab8-dd2d-4a64-91bb-3a2da4acd1a6-plugins-config-volume\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:26:43.012334 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:43.012330    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g86l4\" (UniqueName: \"kubernetes.io/projected/d8004ab8-dd2d-4a64-91bb-3a2da4acd1a6-kube-api-access-g86l4\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:26:43.190161 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:43.190123    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-ec6d47/sub-test-model-ec6d47-kserve-64647bffb5-sprnx" podUID="7c5cc847-16f0-44a9-b062-dc7a7c8a4364" containerName="main" containerID="cri-o://206822043f231426329fddaf04ffbf97a6f462df9f47e40f22feac3da1331784" gracePeriod=45
Sep 10 22:26:43.430317 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:43.430292    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-ec6d47/sub-test-model-ec6d47-kserve-64647bffb5-sprnx"
Sep 10 22:26:43.498514 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:43.498440    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="76e6b1b8-ff59-4b88-9dcd-b6743f9adfa9" path="/var/lib/kubelet/pods/76e6b1b8-ff59-4b88-9dcd-b6743f9adfa9/volumes"
Sep 10 22:26:43.498898 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:43.498883    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ab05d721-124d-46c5-948f-6e83c5aa2854" path="/var/lib/kubelet/pods/ab05d721-124d-46c5-948f-6e83c5aa2854/volumes"
Sep 10 22:26:43.516756 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:43.516727    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c5cc847-16f0-44a9-b062-dc7a7c8a4364-dshm\") pod \"7c5cc847-16f0-44a9-b062-dc7a7c8a4364\" (UID: \"7c5cc847-16f0-44a9-b062-dc7a7c8a4364\") "
Sep 10 22:26:43.516897 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:43.516778    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c5cc847-16f0-44a9-b062-dc7a7c8a4364-tls-certs\") pod \"7c5cc847-16f0-44a9-b062-dc7a7c8a4364\" (UID: \"7c5cc847-16f0-44a9-b062-dc7a7c8a4364\") "
Sep 10 22:26:43.516897 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:43.516819    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c5cc847-16f0-44a9-b062-dc7a7c8a4364-model-cache\") pod \"7c5cc847-16f0-44a9-b062-dc7a7c8a4364\" (UID: \"7c5cc847-16f0-44a9-b062-dc7a7c8a4364\") "
Sep 10 22:26:43.516897 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:43.516886    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c5cc847-16f0-44a9-b062-dc7a7c8a4364-home\") pod \"7c5cc847-16f0-44a9-b062-dc7a7c8a4364\" (UID: \"7c5cc847-16f0-44a9-b062-dc7a7c8a4364\") "
Sep 10 22:26:43.517035 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:43.516931    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c5cc847-16f0-44a9-b062-dc7a7c8a4364-tmp-dir\") pod \"7c5cc847-16f0-44a9-b062-dc7a7c8a4364\" (UID: \"7c5cc847-16f0-44a9-b062-dc7a7c8a4364\") "
Sep 10 22:26:43.517035 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:43.516958    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hnprg\" (UniqueName: \"kubernetes.io/projected/7c5cc847-16f0-44a9-b062-dc7a7c8a4364-kube-api-access-hnprg\") pod \"7c5cc847-16f0-44a9-b062-dc7a7c8a4364\" (UID: \"7c5cc847-16f0-44a9-b062-dc7a7c8a4364\") "
Sep 10 22:26:43.517196 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:43.517167    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c5cc847-16f0-44a9-b062-dc7a7c8a4364-model-cache" (OuterVolumeSpecName: "model-cache") pod "7c5cc847-16f0-44a9-b062-dc7a7c8a4364" (UID: "7c5cc847-16f0-44a9-b062-dc7a7c8a4364"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 22:26:43.517249 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:43.517185    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c5cc847-16f0-44a9-b062-dc7a7c8a4364-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7c5cc847-16f0-44a9-b062-dc7a7c8a4364" (UID: "7c5cc847-16f0-44a9-b062-dc7a7c8a4364"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 22:26:43.517249 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:43.517192    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c5cc847-16f0-44a9-b062-dc7a7c8a4364-home" (OuterVolumeSpecName: "home") pod "7c5cc847-16f0-44a9-b062-dc7a7c8a4364" (UID: "7c5cc847-16f0-44a9-b062-dc7a7c8a4364"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 22:26:43.519044 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:43.519013    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c5cc847-16f0-44a9-b062-dc7a7c8a4364-kube-api-access-hnprg" (OuterVolumeSpecName: "kube-api-access-hnprg") pod "7c5cc847-16f0-44a9-b062-dc7a7c8a4364" (UID: "7c5cc847-16f0-44a9-b062-dc7a7c8a4364"). InnerVolumeSpecName "kube-api-access-hnprg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 22:26:43.519262 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:43.519242    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c5cc847-16f0-44a9-b062-dc7a7c8a4364-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7c5cc847-16f0-44a9-b062-dc7a7c8a4364" (UID: "7c5cc847-16f0-44a9-b062-dc7a7c8a4364"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 22:26:43.519327 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:43.519291    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c5cc847-16f0-44a9-b062-dc7a7c8a4364-dshm" (OuterVolumeSpecName: "dshm") pod "7c5cc847-16f0-44a9-b062-dc7a7c8a4364" (UID: "7c5cc847-16f0-44a9-b062-dc7a7c8a4364"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 22:26:43.615908 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:43.615880    2582 generic.go:356] "Generic (PLEG): container finished" podID="7c5cc847-16f0-44a9-b062-dc7a7c8a4364" containerID="206822043f231426329fddaf04ffbf97a6f462df9f47e40f22feac3da1331784" exitCode=0
Sep 10 22:26:43.616007 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:43.615908    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-ec6d47/sub-test-model-ec6d47-kserve-64647bffb5-sprnx" event={"ID":"7c5cc847-16f0-44a9-b062-dc7a7c8a4364","Type":"ContainerDied","Data":"206822043f231426329fddaf04ffbf97a6f462df9f47e40f22feac3da1331784"}
Sep 10 22:26:43.616007 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:43.615919    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-ec6d47/sub-test-model-ec6d47-kserve-64647bffb5-sprnx"
Sep 10 22:26:43.616007 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:43.615943    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-ec6d47/sub-test-model-ec6d47-kserve-64647bffb5-sprnx" event={"ID":"7c5cc847-16f0-44a9-b062-dc7a7c8a4364","Type":"ContainerDied","Data":"e535acee79b27576889266b76a339faa31aba4c4567c01af7238678ee0e15ce4"}
Sep 10 22:26:43.616007 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:43.615962    2582 scope.go:117] "RemoveContainer" containerID="206822043f231426329fddaf04ffbf97a6f462df9f47e40f22feac3da1331784"
Sep 10 22:26:43.617778 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:43.617757    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c5cc847-16f0-44a9-b062-dc7a7c8a4364-model-cache\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:26:43.617898 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:43.617784    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c5cc847-16f0-44a9-b062-dc7a7c8a4364-home\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:26:43.617898 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:43.617797    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c5cc847-16f0-44a9-b062-dc7a7c8a4364-tmp-dir\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:26:43.617898 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:43.617811    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hnprg\" (UniqueName: \"kubernetes.io/projected/7c5cc847-16f0-44a9-b062-dc7a7c8a4364-kube-api-access-hnprg\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:26:43.617898 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:43.617826    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c5cc847-16f0-44a9-b062-dc7a7c8a4364-dshm\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:26:43.617898 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:43.617839    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c5cc847-16f0-44a9-b062-dc7a7c8a4364-tls-certs\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:26:43.618433 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:43.618412    2582 generic.go:356] "Generic (PLEG): container finished" podID="d8004ab8-dd2d-4a64-91bb-3a2da4acd1a6" containerID="b80d1c6b1106c201163e4127db4d0a423a0b476bdbc81ccbe5430c9bae135da3" exitCode=0
Sep 10 22:26:43.618515 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:43.618454    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c142c5-6f45675c5d-gb4vs"
Sep 10 22:26:43.618515 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:43.618495    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c142c5-6f45675c5d-gb4vs" event={"ID":"d8004ab8-dd2d-4a64-91bb-3a2da4acd1a6","Type":"ContainerDied","Data":"b80d1c6b1106c201163e4127db4d0a423a0b476bdbc81ccbe5430c9bae135da3"}
Sep 10 22:26:43.618598 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:43.618541    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c142c5-6f45675c5d-gb4vs" event={"ID":"d8004ab8-dd2d-4a64-91bb-3a2da4acd1a6","Type":"ContainerDied","Data":"841bda7ae491d52c50d410cf77f78b863bea91330574c5180e38b7d4e65b2f6f"}
Sep 10 22:26:43.625062 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:43.625047    2582 scope.go:117] "RemoveContainer" containerID="206822043f231426329fddaf04ffbf97a6f462df9f47e40f22feac3da1331784"
Sep 10 22:26:43.625333 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:26:43.625315    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"206822043f231426329fddaf04ffbf97a6f462df9f47e40f22feac3da1331784\": container with ID starting with 206822043f231426329fddaf04ffbf97a6f462df9f47e40f22feac3da1331784 not found: ID does not exist" containerID="206822043f231426329fddaf04ffbf97a6f462df9f47e40f22feac3da1331784"
Sep 10 22:26:43.625381 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:43.625340    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"206822043f231426329fddaf04ffbf97a6f462df9f47e40f22feac3da1331784"} err="failed to get container status \"206822043f231426329fddaf04ffbf97a6f462df9f47e40f22feac3da1331784\": rpc error: code = NotFound desc = could not find container \"206822043f231426329fddaf04ffbf97a6f462df9f47e40f22feac3da1331784\": container with ID starting with 206822043f231426329fddaf04ffbf97a6f462df9f47e40f22feac3da1331784 not found: ID does not exist"
Sep 10 22:26:43.625381 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:43.625368    2582 scope.go:117] "RemoveContainer" containerID="b80d1c6b1106c201163e4127db4d0a423a0b476bdbc81ccbe5430c9bae135da3"
Sep 10 22:26:43.632670 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:43.632654    2582 scope.go:117] "RemoveContainer" containerID="b80d1c6b1106c201163e4127db4d0a423a0b476bdbc81ccbe5430c9bae135da3"
Sep 10 22:26:43.632982 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:26:43.632964    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b80d1c6b1106c201163e4127db4d0a423a0b476bdbc81ccbe5430c9bae135da3\": container with ID starting with b80d1c6b1106c201163e4127db4d0a423a0b476bdbc81ccbe5430c9bae135da3 not found: ID does not exist" containerID="b80d1c6b1106c201163e4127db4d0a423a0b476bdbc81ccbe5430c9bae135da3"
Sep 10 22:26:43.633046 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:43.632988    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b80d1c6b1106c201163e4127db4d0a423a0b476bdbc81ccbe5430c9bae135da3"} err="failed to get container status \"b80d1c6b1106c201163e4127db4d0a423a0b476bdbc81ccbe5430c9bae135da3\": rpc error: code = NotFound desc = could not find container \"b80d1c6b1106c201163e4127db4d0a423a0b476bdbc81ccbe5430c9bae135da3\": container with ID starting with b80d1c6b1106c201163e4127db4d0a423a0b476bdbc81ccbe5430c9bae135da3 not found: ID does not exist"
Sep 10 22:26:43.637455 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:43.637434    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c142c5-6f45675c5d-gb4vs"]
Sep 10 22:26:43.641794 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:43.641772    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c142c5-6f45675c5d-gb4vs"]
Sep 10 22:26:43.652215 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:43.652191    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-ec6d47/sub-test-model-ec6d47-kserve-64647bffb5-sprnx"]
Sep 10 22:26:43.654355 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:43.654336    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-ec6d47/sub-test-model-ec6d47-kserve-64647bffb5-sprnx"]
Sep 10 22:26:43.801972 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:43.801894    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-d07dfb/rate-test-model-d07dfb-kserve-5dbf77f559-fttfk"
Sep 10 22:26:43.801972 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:43.801943    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-d07dfb/rate-test-model-d07dfb-kserve-5dbf77f559-fttfk"
Sep 10 22:26:43.814387 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:43.814364    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-d07dfb/rate-test-model-d07dfb-kserve-5dbf77f559-fttfk"
Sep 10 22:26:44.636444 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:44.636412    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-d07dfb/rate-test-model-d07dfb-kserve-5dbf77f559-fttfk"
Sep 10 22:26:45.499389 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:45.499356    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7c5cc847-16f0-44a9-b062-dc7a7c8a4364" path="/var/lib/kubelet/pods/7c5cc847-16f0-44a9-b062-dc7a7c8a4364/volumes"
Sep 10 22:26:45.499723 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:45.499710    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d8004ab8-dd2d-4a64-91bb-3a2da4acd1a6" path="/var/lib/kubelet/pods/d8004ab8-dd2d-4a64-91bb-3a2da4acd1a6/volumes"
Sep 10 22:26:49.498677 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:49.498648    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a9b6f6-8h5zq"]
Sep 10 22:26:49.499073 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:49.498975    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab05d721-124d-46c5-948f-6e83c5aa2854" containerName="main"
Sep 10 22:26:49.499073 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:49.498986    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab05d721-124d-46c5-948f-6e83c5aa2854" containerName="main"
Sep 10 22:26:49.499073 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:49.498995    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8004ab8-dd2d-4a64-91bb-3a2da4acd1a6" containerName="payload-pre-processing"
Sep 10 22:26:49.499073 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:49.499001    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8004ab8-dd2d-4a64-91bb-3a2da4acd1a6" containerName="payload-pre-processing"
Sep 10 22:26:49.499073 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:49.499015    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c5cc847-16f0-44a9-b062-dc7a7c8a4364" containerName="main"
Sep 10 22:26:49.499073 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:49.499020    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c5cc847-16f0-44a9-b062-dc7a7c8a4364" containerName="main"
Sep 10 22:26:49.499073 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:49.499074    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="d8004ab8-dd2d-4a64-91bb-3a2da4acd1a6" containerName="payload-pre-processing"
Sep 10 22:26:49.499292 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:49.499083    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c5cc847-16f0-44a9-b062-dc7a7c8a4364" containerName="main"
Sep 10 22:26:49.499292 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:49.499090    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab05d721-124d-46c5-948f-6e83c5aa2854" containerName="main"
Sep 10 22:26:49.503390 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:49.503364    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a9b6f6-8h5zq"]
Sep 10 22:26:49.503545 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:49.503450    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a9b6f6-8h5zq"
Sep 10 22:26:49.505636 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:49.505615    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 22:26:49.505769 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:49.505615    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8hjrv\""
Sep 10 22:26:49.670263 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:49.670218    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ce1446b-606e-46ba-a19e-98c0f9645170-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-a9b6f6-8h5zq\" (UID: \"7ce1446b-606e-46ba-a19e-98c0f9645170\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a9b6f6-8h5zq"
Sep 10 22:26:49.771499 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:49.771406    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ce1446b-606e-46ba-a19e-98c0f9645170-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-a9b6f6-8h5zq\" (UID: \"7ce1446b-606e-46ba-a19e-98c0f9645170\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a9b6f6-8h5zq"
Sep 10 22:26:49.772126 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:49.772097    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ce1446b-606e-46ba-a19e-98c0f9645170-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-a9b6f6-8h5zq\" (UID: \"7ce1446b-606e-46ba-a19e-98c0f9645170\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a9b6f6-8h5zq"
Sep 10 22:26:49.813924 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:49.813897    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a9b6f6-8h5zq"
Sep 10 22:26:50.083809 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:50.083781    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a9b6f6-8h5zq"]
Sep 10 22:26:50.085897 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:26:50.085867    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ce1446b_606e_46ba_a19e_98c0f9645170.slice/crio-46c52ff3f17791853cca664c5837689bd0543d3c2090999d3a8c5c1a61f7cbc3 WatchSource:0}: Error finding container 46c52ff3f17791853cca664c5837689bd0543d3c2090999d3a8c5c1a61f7cbc3: Status 404 returned error can't find the container with id 46c52ff3f17791853cca664c5837689bd0543d3c2090999d3a8c5c1a61f7cbc3
Sep 10 22:26:50.657113 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:50.657080    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a9b6f6-8h5zq" event={"ID":"7ce1446b-606e-46ba-a19e-98c0f9645170","Type":"ContainerStarted","Data":"e9e139b6ac11ddd62a7d177c09ad2800930cc69deb79aff54c367075629aff0f"}
Sep 10 22:26:50.657505 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:50.657144    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a9b6f6-8h5zq" event={"ID":"7ce1446b-606e-46ba-a19e-98c0f9645170","Type":"ContainerStarted","Data":"46c52ff3f17791853cca664c5837689bd0543d3c2090999d3a8c5c1a61f7cbc3"}
Sep 10 22:26:50.671630 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:50.671583    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a9b6f6-8h5zq" podStartSLOduration=1.6715697 podStartE2EDuration="1.6715697s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 22:26:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:26:50.669697696 +0000 UTC m=+1531.625765401" watchObservedRunningTime="2026-09-10 22:26:50.6715697 +0000 UTC m=+1531.627637406"
Sep 10 22:26:51.663290 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:51.663249    2582 generic.go:356] "Generic (PLEG): container finished" podID="7ce1446b-606e-46ba-a19e-98c0f9645170" containerID="e9e139b6ac11ddd62a7d177c09ad2800930cc69deb79aff54c367075629aff0f" exitCode=0
Sep 10 22:26:51.663661 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:51.663331    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a9b6f6-8h5zq" event={"ID":"7ce1446b-606e-46ba-a19e-98c0f9645170","Type":"ContainerDied","Data":"e9e139b6ac11ddd62a7d177c09ad2800930cc69deb79aff54c367075629aff0f"}
Sep 10 22:26:52.791991 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:52.791960    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a9b6f6-8h5zq"
Sep 10 22:26:52.797981 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:52.797957    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ce1446b-606e-46ba-a19e-98c0f9645170-maas-api-service-ca\") pod \"7ce1446b-606e-46ba-a19e-98c0f9645170\" (UID: \"7ce1446b-606e-46ba-a19e-98c0f9645170\") "
Sep 10 22:26:52.798278 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:52.798254    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ce1446b-606e-46ba-a19e-98c0f9645170-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7ce1446b-606e-46ba-a19e-98c0f9645170" (UID: "7ce1446b-606e-46ba-a19e-98c0f9645170"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 22:26:52.898964 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:52.898930    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ce1446b-606e-46ba-a19e-98c0f9645170-maas-api-service-ca\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:26:53.673745 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:53.673706    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a9b6f6-8h5zq" event={"ID":"7ce1446b-606e-46ba-a19e-98c0f9645170","Type":"ContainerDied","Data":"46c52ff3f17791853cca664c5837689bd0543d3c2090999d3a8c5c1a61f7cbc3"}
Sep 10 22:26:53.673745 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:53.673744    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a9b6f6-8h5zq"
Sep 10 22:26:53.674010 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:26:53.673752    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="46c52ff3f17791853cca664c5837689bd0543d3c2090999d3a8c5c1a61f7cbc3"
Sep 10 22:27:00.272560 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:27:00.272523    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a9b6f6-8h5zq"]
Sep 10 22:27:00.275819 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:27:00.275795    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a9b6f6-8h5zq"]
Sep 10 22:27:01.497780 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:27:01.497747    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7ce1446b-606e-46ba-a19e-98c0f9645170" path="/var/lib/kubelet/pods/7ce1446b-606e-46ba-a19e-98c0f9645170/volumes"
Sep 10 22:27:06.729630 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:27:06.729592    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-a9b6f6-openshift-default-558c5f6ff4-dkx44"]
Sep 10 22:27:06.730172 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:27:06.729814    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-a9b6f6-openshift-default-558c5f6ff4-dkx44" podUID="8b0dd305-1f45-45dc-8e31-01c6c225c275" containerName="istio-proxy" containerID="cri-o://bdf91189a87183186ae56f003c215affd6a59f1b8c32e65b23f17cc41d55ff09" gracePeriod=30
Sep 10 22:27:07.116351 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:27:07.116272    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-a9b3ec-w5pqf"]
Sep 10 22:27:07.116633 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:27:07.116622    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ce1446b-606e-46ba-a19e-98c0f9645170" containerName="revoke-keys"
Sep 10 22:27:07.116681 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:27:07.116635    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ce1446b-606e-46ba-a19e-98c0f9645170" containerName="revoke-keys"
Sep 10 22:27:07.116720 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:27:07.116687    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ce1446b-606e-46ba-a19e-98c0f9645170" containerName="revoke-keys"
Sep 10 22:27:07.120693 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:27:07.120677    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-a9b3ec-w5pqf"
Sep 10 22:27:07.122805 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:27:07.122783    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8hjrv\""
Sep 10 22:27:07.122954 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:27:07.122788    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 22:27:07.126908 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:27:07.126882    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-a9b3ec-w5pqf"]
Sep 10 22:27:07.221101 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:27:07.221071    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1176f1c-9683-4eaf-933e-dd8d576551df-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-a9b3ec-w5pqf\" (UID: \"d1176f1c-9683-4eaf-933e-dd8d576551df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-a9b3ec-w5pqf"
Sep 10 22:27:07.321920 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:27:07.321891    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1176f1c-9683-4eaf-933e-dd8d576551df-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-a9b3ec-w5pqf\" (UID: \"d1176f1c-9683-4eaf-933e-dd8d576551df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-a9b3ec-w5pqf"
Sep 10 22:27:07.322480 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:27:07.322460    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1176f1c-9683-4eaf-933e-dd8d576551df-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-a9b3ec-w5pqf\" (UID: \"d1176f1c-9683-4eaf-933e-dd8d576551df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-a9b3ec-w5pqf"
Sep 10 22:27:07.432219 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:27:07.432193    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-a9b3ec-w5pqf"
Sep 10 22:27:07.703808 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:27:07.703783    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-a9b3ec-w5pqf"]
Sep 10 22:27:07.706640 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:27:07.706603    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1176f1c_9683_4eaf_933e_dd8d576551df.slice/crio-71187a52e1c9776b8c36a3e5e517909f663252e0eea2dab76940b66e87cf3960 WatchSource:0}: Error finding container 71187a52e1c9776b8c36a3e5e517909f663252e0eea2dab76940b66e87cf3960: Status 404 returned error can't find the container with id 71187a52e1c9776b8c36a3e5e517909f663252e0eea2dab76940b66e87cf3960
Sep 10 22:27:07.743445 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:27:07.743422    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-a9b3ec-w5pqf" event={"ID":"d1176f1c-9683-4eaf-933e-dd8d576551df","Type":"ContainerStarted","Data":"71187a52e1c9776b8c36a3e5e517909f663252e0eea2dab76940b66e87cf3960"}
Sep 10 22:27:08.749395 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:27:08.749363    2582 generic.go:356] "Generic (PLEG): container finished" podID="d1176f1c-9683-4eaf-933e-dd8d576551df" containerID="dbb3869315d03be99af693f8b9d55b12221bcaafc52b4972cebf9045f7c47def" exitCode=0
Sep 10 22:27:08.749795 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:27:08.749447    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-a9b3ec-w5pqf" event={"ID":"d1176f1c-9683-4eaf-933e-dd8d576551df","Type":"ContainerDied","Data":"dbb3869315d03be99af693f8b9d55b12221bcaafc52b4972cebf9045f7c47def"}
Sep 10 22:27:09.884315 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:27:09.884288    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-a9b3ec-w5pqf"
Sep 10 22:27:09.942911 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:27:09.942886    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1176f1c-9683-4eaf-933e-dd8d576551df-maas-api-service-ca\") pod \"d1176f1c-9683-4eaf-933e-dd8d576551df\" (UID: \"d1176f1c-9683-4eaf-933e-dd8d576551df\") "
Sep 10 22:27:09.943212 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:27:09.943190    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1176f1c-9683-4eaf-933e-dd8d576551df-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d1176f1c-9683-4eaf-933e-dd8d576551df" (UID: "d1176f1c-9683-4eaf-933e-dd8d576551df"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 22:27:09.998075 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:27:09.995435    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-ec6d47/rate-test-model-ec6d47-kserve-74f74599f5-jnfg2"]
Sep 10 22:27:09.998075 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:27:09.996311    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1176f1c-9683-4eaf-933e-dd8d576551df" containerName="revoke-keys"
Sep 10 22:27:09.998075 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:27:09.996332    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1176f1c-9683-4eaf-933e-dd8d576551df" containerName="revoke-keys"
Sep 10 22:27:09.998075 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:27:09.996490    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="d1176f1c-9683-4eaf-933e-dd8d576551df" containerName="revoke-keys"
Sep 10 22:27:10.000414 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:27:10.000179    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-ec6d47/rate-test-model-ec6d47-kserve-74f74599f5-jnfg2"
Sep 10 22:27:10.002380 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:27:10.002358    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-ec6d47\"/\"default-dockercfg-vvdmr\""
Sep 10 22:27:10.002495 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:27:10.002415    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-ec6d47\"/\"rate-test-model-ec6d47-kserve-self-signed-certs\""
Sep 10 22:27:10.002640 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:27:10.002627    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-ec6d47\"/\"openshift-service-ca.crt\""
Sep 10 22:27:10.002684 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:27:10.002649    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-ec6d47\"/\"kube-root-ca.crt\""
Sep 10 22:27:10.008496 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:27:10.008473    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-ec6d47/rate-test-model-ec6d47-kserve-74f74599f5-jnfg2"]
Sep 10 22:27:10.043949 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:27:10.043926    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/234ec161-4aff-4b6c-aee6-d9294074ee33-home\") pod \"rate-test-model-ec6d47-kserve-74f74599f5-jnfg2\" (UID: \"234ec161-4aff-4b6c-aee6-d9294074ee33\") " pod="ai-tenant-e2e-shared-b-w3-ec6d47/rate-test-model-ec6d47-kserve-74f74599f5-jnfg2"
Sep 10 22:27:10.044079 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:27:10.043963    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/234ec161-4aff-4b6c-aee6-d9294074ee33-dshm\") pod \"rate-test-model-ec6d47-kserve-74f74599f5-jnfg2\" (UID: \"234ec161-4aff-4b6c-aee6-d9294074ee33\") " pod="ai-tenant-e2e-shared-b-w3-ec6d47/rate-test-model-ec6d47-kserve-74f74599f5-jnfg2"
Sep 10 22:27:10.044079 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:27:10.044058    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/234ec161-4aff-4b6c-aee6-d9294074ee33-model-cache\") pod \"rate-test-model-ec6d47-kserve-74f74599f5-jnfg2\" (UID: \"234ec161-4aff-4b6c-aee6-d9294074ee33\") " pod="ai-tenant-e2e-shared-b-w3-ec6d47/rate-test-model-ec6d47-kserve-74f74599f5-jnfg2"
Sep 10 22:27:10.044182 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:27:10.044109    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/234ec161-4aff-4b6c-aee6-d9294074ee33-tmp-dir\") pod \"rate-test-model-ec6d47-kserve-74f74599f5-jnfg2\" (UID: \"234ec161-4aff-4b6c-aee6-d9294074ee33\") " pod="ai-tenant-e2e-shared-b-w3-ec6d47/rate-test-model-ec6d47-kserve-74f74599f5-jnfg2"
Sep 10 22:27:10.044241 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:27:10.044176    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkhk7\" (UniqueName: \"kubernetes.io/projected/234ec161-4aff-4b6c-aee6-d9294074ee33-kube-api-access-wkhk7\") pod \"rate-test-model-ec6d47-kserve-74f74599f5-jnfg2\" (UID: \"234ec161-4aff-4b6c-aee6-d9294074ee33\") " pod="ai-tenant-e2e-shared-b-w3-ec6d47/rate-test-model-ec6d47-kserve-74f74599f5-jnfg2"
Sep 10 22:27:10.044241 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:27:10.044221    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/234ec161-4aff-4b6c-aee6-d9294074ee33-tls-certs\") pod \"rate-test-model-ec6d47-kserve-74f74599f5-jnfg2\" (UID: \"234ec161-4aff-4b6c-aee6-d9294074ee33\") " pod="ai-tenant-e2e-shared-b-w3-ec6d47/rate-test-model-ec6d47-kserve-74f74599f5-jnfg2"
Sep 10 22:27:10.044341 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:27:10.044284    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1176f1c-9683-4eaf-933e-dd8d576551df-maas-api-service-ca\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:27:10.145295 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:27:10.145267    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/234ec161-4aff-4b6c-aee6-d9294074ee33-tls-certs\") pod \"rate-test-model-ec6d47-kserve-74f74599f5-jnfg2\" (UID: \"234ec161-4aff-4b6c-aee6-d9294074ee33\") " pod="ai-tenant-e2e-shared-b-w3-ec6d47/rate-test-model-ec6d47-kserve-74f74599f5-jnfg2"
Sep 10 22:27:10.145413 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:27:10.145311    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/234ec161-4aff-4b6c-aee6-d9294074ee33-home\") pod \"rate-test-model-ec6d47-kserve-74f74599f5-jnfg2\" (UID: \"234ec161-4aff-4b6c-aee6-d9294074ee33\") " pod="ai-tenant-e2e-shared-b-w3-ec6d47/rate-test-model-ec6d47-kserve-74f74599f5-jnfg2"
Sep 10 22:27:10.145413 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:27:10.145348    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/234ec161-4aff-4b6c-aee6-d9294074ee33-dshm\") pod \"rate-test-model-ec6d47-kserve-74f74599f5-jnfg2\" (UID: \"234ec161-4aff-4b6c-aee6-d9294074ee33\") " pod="ai-tenant-e2e-shared-b-w3-ec6d47/rate-test-model-ec6d47-kserve-74f74599f5-jnfg2"
Sep 10 22:27:10.145413 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:27:10.145378    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/234ec161-4aff-4b6c-aee6-d9294074ee33-model-cache\") pod \"rate-test-model-ec6d47-kserve-74f74599f5-jnfg2\" (UID: \"234ec161-4aff-4b6c-aee6-d9294074ee33\") " pod="ai-tenant-e2e-shared-b-w3-ec6d47/rate-test-model-ec6d47-kserve-74f74599f5-jnfg2"
Sep 10 22:27:10.145413 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:27:10.145398    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/234ec161-4aff-4b6c-aee6-d9294074ee33-tmp-dir\") pod \"rate-test-model-ec6d47-kserve-74f74599f5-jnfg2\" (UID: \"234ec161-4aff-4b6c-aee6-d9294074ee33\") " pod="ai-tenant-e2e-shared-b-w3-ec6d47/rate-test-model-ec6d47-kserve-74f74599f5-jnfg2"
Sep 10 22:27:10.145616 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:27:10.145435    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkhk7\" (UniqueName: \"kubernetes.io/projected/234ec161-4aff-4b6c-aee6-d9294074ee33-kube-api-access-wkhk7\") pod \"rate-test-model-ec6d47-kserve-74f74599f5-jnfg2\" (UID: \"234ec161-4aff-4b6c-aee6-d9294074ee33\") " pod="ai-tenant-e2e-shared-b-w3-ec6d47/rate-test-model-ec6d47-kserve-74f74599f5-jnfg2"
Sep 10 22:27:10.145790 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:27:10.145767    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/234ec161-4aff-4b6c-aee6-d9294074ee33-model-cache\") pod \"rate-test-model-ec6d47-kserve-74f74599f5-jnfg2\" (UID: \"234ec161-4aff-4b6c-aee6-d9294074ee33\") " pod="ai-tenant-e2e-shared-b-w3-ec6d47/rate-test-model-ec6d47-kserve-74f74599f5-jnfg2"
Sep 10 22:27:10.145790 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:27:10.145786    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/234ec161-4aff-4b6c-aee6-d9294074ee33-home\") pod \"rate-test-model-ec6d47-kserve-74f74599f5-jnfg2\" (UID: \"234ec161-4aff-4b6c-aee6-d9294074ee33\") " pod="ai-tenant-e2e-shared-b-w3-ec6d47/rate-test-model-ec6d47-kserve-74f74599f5-jnfg2"
Sep 10 22:27:10.145965 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:27:10.145843    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/234ec161-4aff-4b6c-aee6-d9294074ee33-tmp-dir\") pod \"rate-test-model-ec6d47-kserve-74f74599f5-jnfg2\" (UID: \"234ec161-4aff-4b6c-aee6-d9294074ee33\") " pod="ai-tenant-e2e-shared-b-w3-ec6d47/rate-test-model-ec6d47-kserve-74f74599f5-jnfg2"
Sep 10 22:27:10.147549 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:27:10.147530    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/234ec161-4aff-4b6c-aee6-d9294074ee33-dshm\") pod \"rate-test-model-ec6d47-kserve-74f74599f5-jnfg2\" (UID: \"234ec161-4aff-4b6c-aee6-d9294074ee33\") " pod="ai-tenant-e2e-shared-b-w3-ec6d47/rate-test-model-ec6d47-kserve-74f74599f5-jnfg2"
Sep 10 22:27:10.147808 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:27:10.147790    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/234ec161-4aff-4b6c-aee6-d9294074ee33-tls-certs\") pod \"rate-test-model-ec6d47-kserve-74f74599f5-jnfg2\" (UID: \"234ec161-4aff-4b6c-aee6-d9294074ee33\") " pod="ai-tenant-e2e-shared-b-w3-ec6d47/rate-test-model-ec6d47-kserve-74f74599f5-jnfg2"
Sep 10 22:27:10.152767 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:27:10.152741    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkhk7\" (UniqueName: \"kubernetes.io/projected/234ec161-4aff-4b6c-aee6-d9294074ee33-kube-api-access-wkhk7\") pod \"rate-test-model-ec6d47-kserve-74f74599f5-jnfg2\" (UID: \"234ec161-4aff-4b6c-aee6-d9294074ee33\") " pod="ai-tenant-e2e-shared-b-w3-ec6d47/rate-test-model-ec6d47-kserve-74f74599f5-jnfg2"
Sep 10 22:27:10.310788 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:27:10.310700    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-ec6d47/rate-test-model-ec6d47-kserve-74f74599f5-jnfg2"
Sep 10 22:27:10.582907 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:27:10.582885    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-ec6d47/rate-test-model-ec6d47-kserve-74f74599f5-jnfg2"]
Sep 10 22:27:10.584811 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:27:10.584787    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod234ec161_4aff_4b6c_aee6_d9294074ee33.slice/crio-8344aa4382631ef7eb5287cd77b9a65f91878bfb3e45c8ce650255a58e98e0c8 WatchSource:0}: Error finding container 8344aa4382631ef7eb5287cd77b9a65f91878bfb3e45c8ce650255a58e98e0c8: Status 404 returned error can't find the container with id 8344aa4382631ef7eb5287cd77b9a65f91878bfb3e45c8ce650255a58e98e0c8
Sep 10 22:27:10.761233 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:27:10.761203    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-a9b3ec-w5pqf"
Sep 10 22:27:10.761400 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:27:10.761202    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-a9b3ec-w5pqf" event={"ID":"d1176f1c-9683-4eaf-933e-dd8d576551df","Type":"ContainerDied","Data":"71187a52e1c9776b8c36a3e5e517909f663252e0eea2dab76940b66e87cf3960"}
Sep 10 22:27:10.761400 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:27:10.761325    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="71187a52e1c9776b8c36a3e5e517909f663252e0eea2dab76940b66e87cf3960"
Sep 10 22:27:10.762917 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:27:10.762893    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-ec6d47/rate-test-model-ec6d47-kserve-74f74599f5-jnfg2" event={"ID":"234ec161-4aff-4b6c-aee6-d9294074ee33","Type":"ContainerStarted","Data":"4d2281781f17e3ed1ef1845906783312afd04605be6bdf559a71c70cca96353a"}
Sep 10 22:27:10.763040 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:27:10.762929    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-ec6d47/rate-test-model-ec6d47-kserve-74f74599f5-jnfg2" event={"ID":"234ec161-4aff-4b6c-aee6-d9294074ee33","Type":"ContainerStarted","Data":"8344aa4382631ef7eb5287cd77b9a65f91878bfb3e45c8ce650255a58e98e0c8"}
Sep 10 22:27:10.778704 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:27:10.778653    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-ec6d47/rate-test-model-ec6d47-kserve-74f74599f5-jnfg2" podStartSLOduration=1.7786360509999999 podStartE2EDuration="1.778636051s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 22:27:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:27:10.778076835 +0000 UTC m=+1551.734144541" watchObservedRunningTime="2026-09-10 22:27:10.778636051 +0000 UTC m=+1551.734703759"
Sep 10 22:27:11.972840 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:27:11.972815    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-a9b6f6-openshift-default-558c5f6ff4-dkx44"
Sep 10 22:27:12.061570 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:27:12.061506    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b0dd305-1f45-45dc-8e31-01c6c225c275-istio-data\") pod \"8b0dd305-1f45-45dc-8e31-01c6c225c275\" (UID: \"8b0dd305-1f45-45dc-8e31-01c6c225c275\") "
Sep 10 22:27:12.061570 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:27:12.061540    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b0dd305-1f45-45dc-8e31-01c6c225c275-istio-envoy\") pod \"8b0dd305-1f45-45dc-8e31-01c6c225c275\" (UID: \"8b0dd305-1f45-45dc-8e31-01c6c225c275\") "
Sep 10 22:27:12.061733 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:27:12.061585    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b0dd305-1f45-45dc-8e31-01c6c225c275-workload-certs\") pod \"8b0dd305-1f45-45dc-8e31-01c6c225c275\" (UID: \"8b0dd305-1f45-45dc-8e31-01c6c225c275\") "
Sep 10 22:27:12.061733 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:27:12.061606    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b0dd305-1f45-45dc-8e31-01c6c225c275-credential-socket\") pod \"8b0dd305-1f45-45dc-8e31-01c6c225c275\" (UID: \"8b0dd305-1f45-45dc-8e31-01c6c225c275\") "
Sep 10 22:27:12.061733 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:27:12.061631    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b0dd305-1f45-45dc-8e31-01c6c225c275-istio-token\") pod \"8b0dd305-1f45-45dc-8e31-01c6c225c275\" (UID: \"8b0dd305-1f45-45dc-8e31-01c6c225c275\") "
Sep 10 22:27:12.061733 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:27:12.061672    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rsmll\" (UniqueName: \"kubernetes.io/projected/8b0dd305-1f45-45dc-8e31-01c6c225c275-kube-api-access-rsmll\") pod \"8b0dd305-1f45-45dc-8e31-01c6c225c275\" (UID: \"8b0dd305-1f45-45dc-8e31-01c6c225c275\") "
Sep 10 22:27:12.061733 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:27:12.061711    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b0dd305-1f45-45dc-8e31-01c6c225c275-istiod-ca-cert\") pod \"8b0dd305-1f45-45dc-8e31-01c6c225c275\" (UID: \"8b0dd305-1f45-45dc-8e31-01c6c225c275\") "
Sep 10 22:27:12.062008 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:27:12.061735    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b0dd305-1f45-45dc-8e31-01c6c225c275-workload-socket\") pod \"8b0dd305-1f45-45dc-8e31-01c6c225c275\" (UID: \"8b0dd305-1f45-45dc-8e31-01c6c225c275\") "
Sep 10 22:27:12.062008 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:27:12.061767    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b0dd305-1f45-45dc-8e31-01c6c225c275-istio-data" (OuterVolumeSpecName: "istio-data") pod "8b0dd305-1f45-45dc-8e31-01c6c225c275" (UID: "8b0dd305-1f45-45dc-8e31-01c6c225c275"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 22:27:12.062008 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:27:12.061780    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b0dd305-1f45-45dc-8e31-01c6c225c275-istio-podinfo\") pod \"8b0dd305-1f45-45dc-8e31-01c6c225c275\" (UID: \"8b0dd305-1f45-45dc-8e31-01c6c225c275\") "
Sep 10 22:27:12.062008 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:27:12.061932    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b0dd305-1f45-45dc-8e31-01c6c225c275-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8b0dd305-1f45-45dc-8e31-01c6c225c275" (UID: "8b0dd305-1f45-45dc-8e31-01c6c225c275"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 22:27:12.062238 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:27:12.062005    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b0dd305-1f45-45dc-8e31-01c6c225c275-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8b0dd305-1f45-45dc-8e31-01c6c225c275" (UID: "8b0dd305-1f45-45dc-8e31-01c6c225c275"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 22:27:12.062238 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:27:12.062220    2582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b0dd305-1f45-45dc-8e31-01c6c225c275-credential-socket\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:27:12.062342 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:27:12.062245    2582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b0dd305-1f45-45dc-8e31-01c6c225c275-workload-socket\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:27:12.062342 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:27:12.062262    2582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b0dd305-1f45-45dc-8e31-01c6c225c275-istio-data\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:27:12.062342 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:27:12.062307    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b0dd305-1f45-45dc-8e31-01c6c225c275-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8b0dd305-1f45-45dc-8e31-01c6c225c275" (UID: "8b0dd305-1f45-45dc-8e31-01c6c225c275"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 22:27:12.062497 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:27:12.062374    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b0dd305-1f45-45dc-8e31-01c6c225c275-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8b0dd305-1f45-45dc-8e31-01c6c225c275" (UID: "8b0dd305-1f45-45dc-8e31-01c6c225c275"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 22:27:12.064056 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:27:12.064027    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8b0dd305-1f45-45dc-8e31-01c6c225c275-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8b0dd305-1f45-45dc-8e31-01c6c225c275" (UID: "8b0dd305-1f45-45dc-8e31-01c6c225c275"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 22:27:12.064173 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:27:12.064094    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b0dd305-1f45-45dc-8e31-01c6c225c275-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8b0dd305-1f45-45dc-8e31-01c6c225c275" (UID: "8b0dd305-1f45-45dc-8e31-01c6c225c275"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 22:27:12.064296 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:27:12.064273    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b0dd305-1f45-45dc-8e31-01c6c225c275-kube-api-access-rsmll" (OuterVolumeSpecName: "kube-api-access-rsmll") pod "8b0dd305-1f45-45dc-8e31-01c6c225c275" (UID: "8b0dd305-1f45-45dc-8e31-01c6c225c275"). InnerVolumeSpecName "kube-api-access-rsmll". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 22:27:12.064477 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:27:12.064459    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b0dd305-1f45-45dc-8e31-01c6c225c275-istio-token" (OuterVolumeSpecName: "istio-token") pod "8b0dd305-1f45-45dc-8e31-01c6c225c275" (UID: "8b0dd305-1f45-45dc-8e31-01c6c225c275"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 22:27:12.163632 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:27:12.163608    2582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b0dd305-1f45-45dc-8e31-01c6c225c275-istio-envoy\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:27:12.163632 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:27:12.163631    2582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b0dd305-1f45-45dc-8e31-01c6c225c275-workload-certs\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:27:12.163779 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:27:12.163640    2582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b0dd305-1f45-45dc-8e31-01c6c225c275-istio-token\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:27:12.163779 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:27:12.163648    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rsmll\" (UniqueName: \"kubernetes.io/projected/8b0dd305-1f45-45dc-8e31-01c6c225c275-kube-api-access-rsmll\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:27:12.163779 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:27:12.163657    2582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b0dd305-1f45-45dc-8e31-01c6c225c275-istiod-ca-cert\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:27:12.163779 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:27:12.163667    2582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b0dd305-1f45-45dc-8e31-01c6c225c275-istio-podinfo\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:27:12.773465 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:27:12.773431    2582 generic.go:356] "Generic (PLEG): container finished" podID="8b0dd305-1f45-45dc-8e31-01c6c225c275" containerID="bdf91189a87183186ae56f003c215affd6a59f1b8c32e65b23f17cc41d55ff09" exitCode=0
Sep 10 22:27:12.773650 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:27:12.773473    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-a9b6f6-openshift-default-558c5f6ff4-dkx44"
Sep 10 22:27:12.773650 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:27:12.773514    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-a9b6f6-openshift-default-558c5f6ff4-dkx44" event={"ID":"8b0dd305-1f45-45dc-8e31-01c6c225c275","Type":"ContainerDied","Data":"bdf91189a87183186ae56f003c215affd6a59f1b8c32e65b23f17cc41d55ff09"}
Sep 10 22:27:12.773650 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:27:12.773567    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-a9b6f6-openshift-default-558c5f6ff4-dkx44" event={"ID":"8b0dd305-1f45-45dc-8e31-01c6c225c275","Type":"ContainerDied","Data":"47ffca5af9f8c8287763b615182671d4371aaba794ab3161c6a9703b6081fb59"}
Sep 10 22:27:12.773650 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:27:12.773588    2582 scope.go:117] "RemoveContainer" containerID="bdf91189a87183186ae56f003c215affd6a59f1b8c32e65b23f17cc41d55ff09"
Sep 10 22:27:12.782832 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:27:12.782815    2582 scope.go:117] "RemoveContainer" containerID="f71afe4c9e4c8e7ddce7c1f212014436da4a8bb3bf1415294fa4272e51438ed0"
Sep 10 22:27:12.791068 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:27:12.791052    2582 scope.go:117] "RemoveContainer" containerID="bdf91189a87183186ae56f003c215affd6a59f1b8c32e65b23f17cc41d55ff09"
Sep 10 22:27:12.791320 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:27:12.791300    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bdf91189a87183186ae56f003c215affd6a59f1b8c32e65b23f17cc41d55ff09\": container with ID starting with bdf91189a87183186ae56f003c215affd6a59f1b8c32e65b23f17cc41d55ff09 not found: ID does not exist" containerID="bdf91189a87183186ae56f003c215affd6a59f1b8c32e65b23f17cc41d55ff09"
Sep 10 22:27:12.791367 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:27:12.791329    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bdf91189a87183186ae56f003c215affd6a59f1b8c32e65b23f17cc41d55ff09"} err="failed to get container status \"bdf91189a87183186ae56f003c215affd6a59f1b8c32e65b23f17cc41d55ff09\": rpc error: code = NotFound desc = could not find container \"bdf91189a87183186ae56f003c215affd6a59f1b8c32e65b23f17cc41d55ff09\": container with ID starting with bdf91189a87183186ae56f003c215affd6a59f1b8c32e65b23f17cc41d55ff09 not found: ID does not exist"
Sep 10 22:27:12.791367 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:27:12.791345    2582 scope.go:117] "RemoveContainer" containerID="f71afe4c9e4c8e7ddce7c1f212014436da4a8bb3bf1415294fa4272e51438ed0"
Sep 10 22:27:12.791588 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:27:12.791571    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f71afe4c9e4c8e7ddce7c1f212014436da4a8bb3bf1415294fa4272e51438ed0\": container with ID starting with f71afe4c9e4c8e7ddce7c1f212014436da4a8bb3bf1415294fa4272e51438ed0 not found: ID does not exist" containerID="f71afe4c9e4c8e7ddce7c1f212014436da4a8bb3bf1415294fa4272e51438ed0"
Sep 10 22:27:12.791626 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:27:12.791595    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f71afe4c9e4c8e7ddce7c1f212014436da4a8bb3bf1415294fa4272e51438ed0"} err="failed to get container status \"f71afe4c9e4c8e7ddce7c1f212014436da4a8bb3bf1415294fa4272e51438ed0\": rpc error: code = NotFound desc = could not find container \"f71afe4c9e4c8e7ddce7c1f212014436da4a8bb3bf1415294fa4272e51438ed0\": container with ID starting with f71afe4c9e4c8e7ddce7c1f212014436da4a8bb3bf1415294fa4272e51438ed0 not found: ID does not exist"
Sep 10 22:27:12.797097 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:27:12.797074    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-a9b6f6-openshift-default-558c5f6ff4-dkx44"]
Sep 10 22:27:12.803584 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:27:12.803563    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-a9b6f6-openshift-default-558c5f6ff4-dkx44"]
Sep 10 22:27:13.498945 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:27:13.498906    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8b0dd305-1f45-45dc-8e31-01c6c225c275" path="/var/lib/kubelet/pods/8b0dd305-1f45-45dc-8e31-01c6c225c275/volumes"
Sep 10 22:27:17.985160 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:27:17.985122    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a9b3ec-565f78dc58-nrsf6"]
Sep 10 22:27:17.985637 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:27:17.985394    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a9b3ec-565f78dc58-nrsf6" podUID="94040552-d14f-4d5a-bf04-b1d5adad903b" containerName="payload-pre-processing" containerID="cri-o://08d20b630543f2312aee7221fb480c39d4036d201e6d42ef7be45c5a9f5defba" gracePeriod=30
Sep 10 22:27:18.162251 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:27:18.162221    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-a9b3ec-w5pqf"]
Sep 10 22:27:18.166794 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:27:18.166769    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-a9b3ec-w5pqf"]
Sep 10 22:27:18.234397 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:27:18.234374    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a9b3ec-565f78dc58-nrsf6"
Sep 10 22:27:18.319894 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:27:18.319797    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94040552-d14f-4d5a-bf04-b1d5adad903b-plugins-config-volume\") pod \"94040552-d14f-4d5a-bf04-b1d5adad903b\" (UID: \"94040552-d14f-4d5a-bf04-b1d5adad903b\") "
Sep 10 22:27:18.319894 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:27:18.319840    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qp2x9\" (UniqueName: \"kubernetes.io/projected/94040552-d14f-4d5a-bf04-b1d5adad903b-kube-api-access-qp2x9\") pod \"94040552-d14f-4d5a-bf04-b1d5adad903b\" (UID: \"94040552-d14f-4d5a-bf04-b1d5adad903b\") "
Sep 10 22:27:18.320216 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:27:18.320184    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94040552-d14f-4d5a-bf04-b1d5adad903b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "94040552-d14f-4d5a-bf04-b1d5adad903b" (UID: "94040552-d14f-4d5a-bf04-b1d5adad903b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 22:27:18.321915 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:27:18.321890    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94040552-d14f-4d5a-bf04-b1d5adad903b-kube-api-access-qp2x9" (OuterVolumeSpecName: "kube-api-access-qp2x9") pod "94040552-d14f-4d5a-bf04-b1d5adad903b" (UID: "94040552-d14f-4d5a-bf04-b1d5adad903b"). InnerVolumeSpecName "kube-api-access-qp2x9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 22:27:18.421021 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:27:18.420995    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94040552-d14f-4d5a-bf04-b1d5adad903b-plugins-config-volume\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:27:18.421021 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:27:18.421019    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qp2x9\" (UniqueName: \"kubernetes.io/projected/94040552-d14f-4d5a-bf04-b1d5adad903b-kube-api-access-qp2x9\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:27:18.804500 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:27:18.804472    2582 generic.go:356] "Generic (PLEG): container finished" podID="94040552-d14f-4d5a-bf04-b1d5adad903b" containerID="08d20b630543f2312aee7221fb480c39d4036d201e6d42ef7be45c5a9f5defba" exitCode=0
Sep 10 22:27:18.804661 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:27:18.804512    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a9b3ec-565f78dc58-nrsf6"
Sep 10 22:27:18.804661 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:27:18.804549    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a9b3ec-565f78dc58-nrsf6" event={"ID":"94040552-d14f-4d5a-bf04-b1d5adad903b","Type":"ContainerDied","Data":"08d20b630543f2312aee7221fb480c39d4036d201e6d42ef7be45c5a9f5defba"}
Sep 10 22:27:18.804661 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:27:18.804603    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a9b3ec-565f78dc58-nrsf6" event={"ID":"94040552-d14f-4d5a-bf04-b1d5adad903b","Type":"ContainerDied","Data":"4d3295914af0d5ccfa7603f9114ca1e238085c0bb8e1b3ead7b9d39f9502b024"}
Sep 10 22:27:18.804661 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:27:18.804627    2582 scope.go:117] "RemoveContainer" containerID="08d20b630543f2312aee7221fb480c39d4036d201e6d42ef7be45c5a9f5defba"
Sep 10 22:27:18.812989 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:27:18.812971    2582 scope.go:117] "RemoveContainer" containerID="08d20b630543f2312aee7221fb480c39d4036d201e6d42ef7be45c5a9f5defba"
Sep 10 22:27:18.813260 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:27:18.813240    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08d20b630543f2312aee7221fb480c39d4036d201e6d42ef7be45c5a9f5defba\": container with ID starting with 08d20b630543f2312aee7221fb480c39d4036d201e6d42ef7be45c5a9f5defba not found: ID does not exist" containerID="08d20b630543f2312aee7221fb480c39d4036d201e6d42ef7be45c5a9f5defba"
Sep 10 22:27:18.813323 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:27:18.813267    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08d20b630543f2312aee7221fb480c39d4036d201e6d42ef7be45c5a9f5defba"} err="failed to get container status \"08d20b630543f2312aee7221fb480c39d4036d201e6d42ef7be45c5a9f5defba\": rpc error: code = NotFound desc = could not find container \"08d20b630543f2312aee7221fb480c39d4036d201e6d42ef7be45c5a9f5defba\": container with ID starting with 08d20b630543f2312aee7221fb480c39d4036d201e6d42ef7be45c5a9f5defba not found: ID does not exist"
Sep 10 22:27:18.824611 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:27:18.824585    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a9b3ec-565f78dc58-nrsf6"]
Sep 10 22:27:18.827879 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:27:18.827842    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a9b3ec-565f78dc58-nrsf6"]
Sep 10 22:27:19.498392 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:27:19.498358    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="94040552-d14f-4d5a-bf04-b1d5adad903b" path="/var/lib/kubelet/pods/94040552-d14f-4d5a-bf04-b1d5adad903b/volumes"
Sep 10 22:27:19.498737 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:27:19.498663    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d1176f1c-9683-4eaf-933e-dd8d576551df" path="/var/lib/kubelet/pods/d1176f1c-9683-4eaf-933e-dd8d576551df/volumes"
Sep 10 22:27:20.311722 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:27:20.311693    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-ec6d47/rate-test-model-ec6d47-kserve-74f74599f5-jnfg2"
Sep 10 22:27:20.311910 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:27:20.311740    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-ec6d47/rate-test-model-ec6d47-kserve-74f74599f5-jnfg2"
Sep 10 22:27:20.323911 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:27:20.323885    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-ec6d47/rate-test-model-ec6d47-kserve-74f74599f5-jnfg2"
Sep 10 22:27:20.825104 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:27:20.825075    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-ec6d47/rate-test-model-ec6d47-kserve-74f74599f5-jnfg2"
Sep 10 22:27:48.833791 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:27:48.833751    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-d07dfb/rate-test-model-d07dfb-kserve-5dbf77f559-fttfk"]
Sep 10 22:27:49.781174 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:27:49.781098    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-ec6d47/rate-test-model-ec6d47-kserve-74f74599f5-jnfg2"]
Sep 10 22:28:03.852116 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:28:03.852075    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-d07dfb/rate-test-model-d07dfb-kserve-5dbf77f559-fttfk" podUID="8046dbac-5187-4501-aad4-682083e61e14" containerName="main" containerID="cri-o://2f12e460f952b9ef3298f0ed04c23cb979fc20cfc0a13e6666d7cd62c5558ae6" gracePeriod=45
Sep 10 22:28:04.026498 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:28:04.026339    2582 generic.go:356] "Generic (PLEG): container finished" podID="8046dbac-5187-4501-aad4-682083e61e14" containerID="2f12e460f952b9ef3298f0ed04c23cb979fc20cfc0a13e6666d7cd62c5558ae6" exitCode=0
Sep 10 22:28:04.026498 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:28:04.026451    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-d07dfb/rate-test-model-d07dfb-kserve-5dbf77f559-fttfk" event={"ID":"8046dbac-5187-4501-aad4-682083e61e14","Type":"ContainerDied","Data":"2f12e460f952b9ef3298f0ed04c23cb979fc20cfc0a13e6666d7cd62c5558ae6"}
Sep 10 22:28:04.098338 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:28:04.098315    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-d07dfb/rate-test-model-d07dfb-kserve-5dbf77f559-fttfk"
Sep 10 22:28:04.194197 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:28:04.194164    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8046dbac-5187-4501-aad4-682083e61e14-tls-certs\") pod \"8046dbac-5187-4501-aad4-682083e61e14\" (UID: \"8046dbac-5187-4501-aad4-682083e61e14\") "
Sep 10 22:28:04.194370 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:28:04.194244    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7jr56\" (UniqueName: \"kubernetes.io/projected/8046dbac-5187-4501-aad4-682083e61e14-kube-api-access-7jr56\") pod \"8046dbac-5187-4501-aad4-682083e61e14\" (UID: \"8046dbac-5187-4501-aad4-682083e61e14\") "
Sep 10 22:28:04.194370 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:28:04.194328    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8046dbac-5187-4501-aad4-682083e61e14-tmp-dir\") pod \"8046dbac-5187-4501-aad4-682083e61e14\" (UID: \"8046dbac-5187-4501-aad4-682083e61e14\") "
Sep 10 22:28:04.194446 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:28:04.194410    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8046dbac-5187-4501-aad4-682083e61e14-model-cache\") pod \"8046dbac-5187-4501-aad4-682083e61e14\" (UID: \"8046dbac-5187-4501-aad4-682083e61e14\") "
Sep 10 22:28:04.194483 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:28:04.194454    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8046dbac-5187-4501-aad4-682083e61e14-dshm\") pod \"8046dbac-5187-4501-aad4-682083e61e14\" (UID: \"8046dbac-5187-4501-aad4-682083e61e14\") "
Sep 10 22:28:04.194483 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:28:04.194475    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8046dbac-5187-4501-aad4-682083e61e14-home\") pod \"8046dbac-5187-4501-aad4-682083e61e14\" (UID: \"8046dbac-5187-4501-aad4-682083e61e14\") "
Sep 10 22:28:04.194605 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:28:04.194577    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8046dbac-5187-4501-aad4-682083e61e14-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8046dbac-5187-4501-aad4-682083e61e14" (UID: "8046dbac-5187-4501-aad4-682083e61e14"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 22:28:04.194706 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:28:04.194685    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8046dbac-5187-4501-aad4-682083e61e14-model-cache" (OuterVolumeSpecName: "model-cache") pod "8046dbac-5187-4501-aad4-682083e61e14" (UID: "8046dbac-5187-4501-aad4-682083e61e14"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 22:28:04.194832 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:28:04.194794    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8046dbac-5187-4501-aad4-682083e61e14-home" (OuterVolumeSpecName: "home") pod "8046dbac-5187-4501-aad4-682083e61e14" (UID: "8046dbac-5187-4501-aad4-682083e61e14"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 22:28:04.194832 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:28:04.194815    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8046dbac-5187-4501-aad4-682083e61e14-model-cache\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:28:04.194998 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:28:04.194871    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8046dbac-5187-4501-aad4-682083e61e14-tmp-dir\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:28:04.196550 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:28:04.196527    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8046dbac-5187-4501-aad4-682083e61e14-dshm" (OuterVolumeSpecName: "dshm") pod "8046dbac-5187-4501-aad4-682083e61e14" (UID: "8046dbac-5187-4501-aad4-682083e61e14"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 22:28:04.196899 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:28:04.196864    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8046dbac-5187-4501-aad4-682083e61e14-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8046dbac-5187-4501-aad4-682083e61e14" (UID: "8046dbac-5187-4501-aad4-682083e61e14"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 22:28:04.196899 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:28:04.196886    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8046dbac-5187-4501-aad4-682083e61e14-kube-api-access-7jr56" (OuterVolumeSpecName: "kube-api-access-7jr56") pod "8046dbac-5187-4501-aad4-682083e61e14" (UID: "8046dbac-5187-4501-aad4-682083e61e14"). InnerVolumeSpecName "kube-api-access-7jr56". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 22:28:04.295509 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:28:04.295485    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8046dbac-5187-4501-aad4-682083e61e14-tls-certs\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:28:04.295509 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:28:04.295505    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7jr56\" (UniqueName: \"kubernetes.io/projected/8046dbac-5187-4501-aad4-682083e61e14-kube-api-access-7jr56\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:28:04.295642 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:28:04.295515    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8046dbac-5187-4501-aad4-682083e61e14-dshm\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:28:04.295642 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:28:04.295523    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8046dbac-5187-4501-aad4-682083e61e14-home\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:28:04.807617 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:28:04.807555    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-ec6d47/rate-test-model-ec6d47-kserve-74f74599f5-jnfg2" podUID="234ec161-4aff-4b6c-aee6-d9294074ee33" containerName="main" containerID="cri-o://4d2281781f17e3ed1ef1845906783312afd04605be6bdf559a71c70cca96353a" gracePeriod=45
Sep 10 22:28:05.033339 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:28:05.033280    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-d07dfb/rate-test-model-d07dfb-kserve-5dbf77f559-fttfk"
Sep 10 22:28:05.033339 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:28:05.033304    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-d07dfb/rate-test-model-d07dfb-kserve-5dbf77f559-fttfk" event={"ID":"8046dbac-5187-4501-aad4-682083e61e14","Type":"ContainerDied","Data":"2443539f38cd1cd282d38962cb1a3917cd1e2f9f92850526dd4e800f141ffa0a"}
Sep 10 22:28:05.033801 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:28:05.033356    2582 scope.go:117] "RemoveContainer" containerID="2f12e460f952b9ef3298f0ed04c23cb979fc20cfc0a13e6666d7cd62c5558ae6"
Sep 10 22:28:05.041324 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:28:05.041291    2582 generic.go:356] "Generic (PLEG): container finished" podID="234ec161-4aff-4b6c-aee6-d9294074ee33" containerID="4d2281781f17e3ed1ef1845906783312afd04605be6bdf559a71c70cca96353a" exitCode=0
Sep 10 22:28:05.041447 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:28:05.041354    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-ec6d47/rate-test-model-ec6d47-kserve-74f74599f5-jnfg2" event={"ID":"234ec161-4aff-4b6c-aee6-d9294074ee33","Type":"ContainerDied","Data":"4d2281781f17e3ed1ef1845906783312afd04605be6bdf559a71c70cca96353a"}
Sep 10 22:28:05.041447 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:28:05.041407    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-ec6d47/rate-test-model-ec6d47-kserve-74f74599f5-jnfg2" event={"ID":"234ec161-4aff-4b6c-aee6-d9294074ee33","Type":"ContainerDied","Data":"8344aa4382631ef7eb5287cd77b9a65f91878bfb3e45c8ce650255a58e98e0c8"}
Sep 10 22:28:05.041447 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:28:05.041443    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8344aa4382631ef7eb5287cd77b9a65f91878bfb3e45c8ce650255a58e98e0c8"
Sep 10 22:28:05.052449 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:28:05.052430    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-ec6d47/rate-test-model-ec6d47-kserve-74f74599f5-jnfg2"
Sep 10 22:28:05.067028 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:28:05.066976    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-d07dfb/rate-test-model-d07dfb-kserve-5dbf77f559-fttfk"]
Sep 10 22:28:05.069212 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:28:05.069188    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-d07dfb/rate-test-model-d07dfb-kserve-5dbf77f559-fttfk"]
Sep 10 22:28:05.103050 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:28:05.103027    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/234ec161-4aff-4b6c-aee6-d9294074ee33-home\") pod \"234ec161-4aff-4b6c-aee6-d9294074ee33\" (UID: \"234ec161-4aff-4b6c-aee6-d9294074ee33\") "
Sep 10 22:28:05.103138 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:28:05.103083    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wkhk7\" (UniqueName: \"kubernetes.io/projected/234ec161-4aff-4b6c-aee6-d9294074ee33-kube-api-access-wkhk7\") pod \"234ec161-4aff-4b6c-aee6-d9294074ee33\" (UID: \"234ec161-4aff-4b6c-aee6-d9294074ee33\") "
Sep 10 22:28:05.103138 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:28:05.103125    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/234ec161-4aff-4b6c-aee6-d9294074ee33-dshm\") pod \"234ec161-4aff-4b6c-aee6-d9294074ee33\" (UID: \"234ec161-4aff-4b6c-aee6-d9294074ee33\") "
Sep 10 22:28:05.103220 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:28:05.103165    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/234ec161-4aff-4b6c-aee6-d9294074ee33-tmp-dir\") pod \"234ec161-4aff-4b6c-aee6-d9294074ee33\" (UID: \"234ec161-4aff-4b6c-aee6-d9294074ee33\") "
Sep 10 22:28:05.103220 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:28:05.103201    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/234ec161-4aff-4b6c-aee6-d9294074ee33-model-cache\") pod \"234ec161-4aff-4b6c-aee6-d9294074ee33\" (UID: \"234ec161-4aff-4b6c-aee6-d9294074ee33\") "
Sep 10 22:28:05.103328 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:28:05.103230    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/234ec161-4aff-4b6c-aee6-d9294074ee33-tls-certs\") pod \"234ec161-4aff-4b6c-aee6-d9294074ee33\" (UID: \"234ec161-4aff-4b6c-aee6-d9294074ee33\") "
Sep 10 22:28:05.103328 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:28:05.103291    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/234ec161-4aff-4b6c-aee6-d9294074ee33-home" (OuterVolumeSpecName: "home") pod "234ec161-4aff-4b6c-aee6-d9294074ee33" (UID: "234ec161-4aff-4b6c-aee6-d9294074ee33"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 22:28:05.103536 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:28:05.103484    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/234ec161-4aff-4b6c-aee6-d9294074ee33-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "234ec161-4aff-4b6c-aee6-d9294074ee33" (UID: "234ec161-4aff-4b6c-aee6-d9294074ee33"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 22:28:05.103536 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:28:05.103523    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/234ec161-4aff-4b6c-aee6-d9294074ee33-model-cache" (OuterVolumeSpecName: "model-cache") pod "234ec161-4aff-4b6c-aee6-d9294074ee33" (UID: "234ec161-4aff-4b6c-aee6-d9294074ee33"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 22:28:05.103667 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:28:05.103572    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/234ec161-4aff-4b6c-aee6-d9294074ee33-home\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:28:05.103667 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:28:05.103589    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/234ec161-4aff-4b6c-aee6-d9294074ee33-tmp-dir\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:28:05.105290 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:28:05.105265    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/234ec161-4aff-4b6c-aee6-d9294074ee33-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "234ec161-4aff-4b6c-aee6-d9294074ee33" (UID: "234ec161-4aff-4b6c-aee6-d9294074ee33"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 22:28:05.105290 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:28:05.105282    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/234ec161-4aff-4b6c-aee6-d9294074ee33-dshm" (OuterVolumeSpecName: "dshm") pod "234ec161-4aff-4b6c-aee6-d9294074ee33" (UID: "234ec161-4aff-4b6c-aee6-d9294074ee33"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 22:28:05.105417 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:28:05.105283    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/234ec161-4aff-4b6c-aee6-d9294074ee33-kube-api-access-wkhk7" (OuterVolumeSpecName: "kube-api-access-wkhk7") pod "234ec161-4aff-4b6c-aee6-d9294074ee33" (UID: "234ec161-4aff-4b6c-aee6-d9294074ee33"). InnerVolumeSpecName "kube-api-access-wkhk7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 22:28:05.204747 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:28:05.204723    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wkhk7\" (UniqueName: \"kubernetes.io/projected/234ec161-4aff-4b6c-aee6-d9294074ee33-kube-api-access-wkhk7\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:28:05.204747 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:28:05.204746    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/234ec161-4aff-4b6c-aee6-d9294074ee33-dshm\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:28:05.204918 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:28:05.204756    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/234ec161-4aff-4b6c-aee6-d9294074ee33-model-cache\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:28:05.204918 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:28:05.204765    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/234ec161-4aff-4b6c-aee6-d9294074ee33-tls-certs\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:28:05.498710 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:28:05.498668    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8046dbac-5187-4501-aad4-682083e61e14" path="/var/lib/kubelet/pods/8046dbac-5187-4501-aad4-682083e61e14/volumes"
Sep 10 22:28:06.047532 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:28:06.047502    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-ec6d47/rate-test-model-ec6d47-kserve-74f74599f5-jnfg2"
Sep 10 22:28:06.064113 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:28:06.064087    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-ec6d47/rate-test-model-ec6d47-kserve-74f74599f5-jnfg2"]
Sep 10 22:28:06.070705 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:28:06.070681    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-ec6d47/rate-test-model-ec6d47-kserve-74f74599f5-jnfg2"]
Sep 10 22:28:07.499101 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:28:07.499071    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="234ec161-4aff-4b6c-aee6-d9294074ee33" path="/var/lib/kubelet/pods/234ec161-4aff-4b6c-aee6-d9294074ee33/volumes"
Sep 10 22:28:19.119840 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:28:19.119758    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-79434a-openshift-default-8f46dd48-r6zgk"]
Sep 10 22:28:19.120331 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:28:19.120318    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8046dbac-5187-4501-aad4-682083e61e14" containerName="main"
Sep 10 22:28:19.120399 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:28:19.120339    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="8046dbac-5187-4501-aad4-682083e61e14" containerName="main"
Sep 10 22:28:19.120399 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:28:19.120353    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="234ec161-4aff-4b6c-aee6-d9294074ee33" containerName="main"
Sep 10 22:28:19.120399 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:28:19.120361    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="234ec161-4aff-4b6c-aee6-d9294074ee33" containerName="main"
Sep 10 22:28:19.120399 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:28:19.120387    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b0dd305-1f45-45dc-8e31-01c6c225c275" containerName="istio-proxy"
Sep 10 22:28:19.120399 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:28:19.120395    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b0dd305-1f45-45dc-8e31-01c6c225c275" containerName="istio-proxy"
Sep 10 22:28:19.120644 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:28:19.120415    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b0dd305-1f45-45dc-8e31-01c6c225c275" containerName="istio-proxy"
Sep 10 22:28:19.120644 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:28:19.120424    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b0dd305-1f45-45dc-8e31-01c6c225c275" containerName="istio-proxy"
Sep 10 22:28:19.120644 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:28:19.120434    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94040552-d14f-4d5a-bf04-b1d5adad903b" containerName="payload-pre-processing"
Sep 10 22:28:19.120644 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:28:19.120443    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="94040552-d14f-4d5a-bf04-b1d5adad903b" containerName="payload-pre-processing"
Sep 10 22:28:19.120644 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:28:19.120527    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b0dd305-1f45-45dc-8e31-01c6c225c275" containerName="istio-proxy"
Sep 10 22:28:19.120644 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:28:19.120538    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="8046dbac-5187-4501-aad4-682083e61e14" containerName="main"
Sep 10 22:28:19.120644 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:28:19.120547    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="94040552-d14f-4d5a-bf04-b1d5adad903b" containerName="payload-pre-processing"
Sep 10 22:28:19.120644 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:28:19.120557    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b0dd305-1f45-45dc-8e31-01c6c225c275" containerName="istio-proxy"
Sep 10 22:28:19.120644 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:28:19.120568    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="234ec161-4aff-4b6c-aee6-d9294074ee33" containerName="main"
Sep 10 22:28:19.124995 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:28:19.124974    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-79434a-openshift-default-8f46dd48-r6zgk"
Sep 10 22:28:19.127310 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:28:19.127292    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-79434a-openshift-default-dockercfg-fcpw7\""
Sep 10 22:28:19.138001 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:28:19.137979    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-79434a-openshift-default-8f46dd48-r6zgk"]
Sep 10 22:28:19.221588 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:28:19.221555    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f9549c1-10dc-4700-804f-82d6b1241447-workload-certs\") pod \"e2e-ipp-b-79434a-openshift-default-8f46dd48-r6zgk\" (UID: \"0f9549c1-10dc-4700-804f-82d6b1241447\") " pod="openshift-ingress/e2e-ipp-b-79434a-openshift-default-8f46dd48-r6zgk"
Sep 10 22:28:19.221744 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:28:19.221603    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f9549c1-10dc-4700-804f-82d6b1241447-workload-socket\") pod \"e2e-ipp-b-79434a-openshift-default-8f46dd48-r6zgk\" (UID: \"0f9549c1-10dc-4700-804f-82d6b1241447\") " pod="openshift-ingress/e2e-ipp-b-79434a-openshift-default-8f46dd48-r6zgk"
Sep 10 22:28:19.221744 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:28:19.221707    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f9549c1-10dc-4700-804f-82d6b1241447-istio-envoy\") pod \"e2e-ipp-b-79434a-openshift-default-8f46dd48-r6zgk\" (UID: \"0f9549c1-10dc-4700-804f-82d6b1241447\") " pod="openshift-ingress/e2e-ipp-b-79434a-openshift-default-8f46dd48-r6zgk"
Sep 10 22:28:19.221905 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:28:19.221758    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f9549c1-10dc-4700-804f-82d6b1241447-credential-socket\") pod \"e2e-ipp-b-79434a-openshift-default-8f46dd48-r6zgk\" (UID: \"0f9549c1-10dc-4700-804f-82d6b1241447\") " pod="openshift-ingress/e2e-ipp-b-79434a-openshift-default-8f46dd48-r6zgk"
Sep 10 22:28:19.221905 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:28:19.221795    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f9549c1-10dc-4700-804f-82d6b1241447-istiod-ca-cert\") pod \"e2e-ipp-b-79434a-openshift-default-8f46dd48-r6zgk\" (UID: \"0f9549c1-10dc-4700-804f-82d6b1241447\") " pod="openshift-ingress/e2e-ipp-b-79434a-openshift-default-8f46dd48-r6zgk"
Sep 10 22:28:19.222047 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:28:19.221924    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f9549c1-10dc-4700-804f-82d6b1241447-istio-token\") pod \"e2e-ipp-b-79434a-openshift-default-8f46dd48-r6zgk\" (UID: \"0f9549c1-10dc-4700-804f-82d6b1241447\") " pod="openshift-ingress/e2e-ipp-b-79434a-openshift-default-8f46dd48-r6zgk"
Sep 10 22:28:19.222047 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:28:19.221981    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f9549c1-10dc-4700-804f-82d6b1241447-istio-podinfo\") pod \"e2e-ipp-b-79434a-openshift-default-8f46dd48-r6zgk\" (UID: \"0f9549c1-10dc-4700-804f-82d6b1241447\") " pod="openshift-ingress/e2e-ipp-b-79434a-openshift-default-8f46dd48-r6zgk"
Sep 10 22:28:19.222164 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:28:19.222049    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62p6p\" (UniqueName: \"kubernetes.io/projected/0f9549c1-10dc-4700-804f-82d6b1241447-kube-api-access-62p6p\") pod \"e2e-ipp-b-79434a-openshift-default-8f46dd48-r6zgk\" (UID: \"0f9549c1-10dc-4700-804f-82d6b1241447\") " pod="openshift-ingress/e2e-ipp-b-79434a-openshift-default-8f46dd48-r6zgk"
Sep 10 22:28:19.222164 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:28:19.222092    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f9549c1-10dc-4700-804f-82d6b1241447-istio-data\") pod \"e2e-ipp-b-79434a-openshift-default-8f46dd48-r6zgk\" (UID: \"0f9549c1-10dc-4700-804f-82d6b1241447\") " pod="openshift-ingress/e2e-ipp-b-79434a-openshift-default-8f46dd48-r6zgk"
Sep 10 22:28:19.323354 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:28:19.323322    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f9549c1-10dc-4700-804f-82d6b1241447-istio-podinfo\") pod \"e2e-ipp-b-79434a-openshift-default-8f46dd48-r6zgk\" (UID: \"0f9549c1-10dc-4700-804f-82d6b1241447\") " pod="openshift-ingress/e2e-ipp-b-79434a-openshift-default-8f46dd48-r6zgk"
Sep 10 22:28:19.323516 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:28:19.323361    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62p6p\" (UniqueName: \"kubernetes.io/projected/0f9549c1-10dc-4700-804f-82d6b1241447-kube-api-access-62p6p\") pod \"e2e-ipp-b-79434a-openshift-default-8f46dd48-r6zgk\" (UID: \"0f9549c1-10dc-4700-804f-82d6b1241447\") " pod="openshift-ingress/e2e-ipp-b-79434a-openshift-default-8f46dd48-r6zgk"
Sep 10 22:28:19.323516 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:28:19.323388    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f9549c1-10dc-4700-804f-82d6b1241447-istio-data\") pod \"e2e-ipp-b-79434a-openshift-default-8f46dd48-r6zgk\" (UID: \"0f9549c1-10dc-4700-804f-82d6b1241447\") " pod="openshift-ingress/e2e-ipp-b-79434a-openshift-default-8f46dd48-r6zgk"
Sep 10 22:28:19.323516 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:28:19.323422    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f9549c1-10dc-4700-804f-82d6b1241447-workload-certs\") pod \"e2e-ipp-b-79434a-openshift-default-8f46dd48-r6zgk\" (UID: \"0f9549c1-10dc-4700-804f-82d6b1241447\") " pod="openshift-ingress/e2e-ipp-b-79434a-openshift-default-8f46dd48-r6zgk"
Sep 10 22:28:19.323516 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:28:19.323441    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f9549c1-10dc-4700-804f-82d6b1241447-workload-socket\") pod \"e2e-ipp-b-79434a-openshift-default-8f46dd48-r6zgk\" (UID: \"0f9549c1-10dc-4700-804f-82d6b1241447\") " pod="openshift-ingress/e2e-ipp-b-79434a-openshift-default-8f46dd48-r6zgk"
Sep 10 22:28:19.323516 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:28:19.323495    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f9549c1-10dc-4700-804f-82d6b1241447-istio-envoy\") pod \"e2e-ipp-b-79434a-openshift-default-8f46dd48-r6zgk\" (UID: \"0f9549c1-10dc-4700-804f-82d6b1241447\") " pod="openshift-ingress/e2e-ipp-b-79434a-openshift-default-8f46dd48-r6zgk"
Sep 10 22:28:19.323770 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:28:19.323525    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f9549c1-10dc-4700-804f-82d6b1241447-credential-socket\") pod \"e2e-ipp-b-79434a-openshift-default-8f46dd48-r6zgk\" (UID: \"0f9549c1-10dc-4700-804f-82d6b1241447\") " pod="openshift-ingress/e2e-ipp-b-79434a-openshift-default-8f46dd48-r6zgk"
Sep 10 22:28:19.323770 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:28:19.323556    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f9549c1-10dc-4700-804f-82d6b1241447-istiod-ca-cert\") pod \"e2e-ipp-b-79434a-openshift-default-8f46dd48-r6zgk\" (UID: \"0f9549c1-10dc-4700-804f-82d6b1241447\") " pod="openshift-ingress/e2e-ipp-b-79434a-openshift-default-8f46dd48-r6zgk"
Sep 10 22:28:19.323770 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:28:19.323605    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f9549c1-10dc-4700-804f-82d6b1241447-istio-token\") pod \"e2e-ipp-b-79434a-openshift-default-8f46dd48-r6zgk\" (UID: \"0f9549c1-10dc-4700-804f-82d6b1241447\") " pod="openshift-ingress/e2e-ipp-b-79434a-openshift-default-8f46dd48-r6zgk"
Sep 10 22:28:19.323962 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:28:19.323926    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f9549c1-10dc-4700-804f-82d6b1241447-istio-data\") pod \"e2e-ipp-b-79434a-openshift-default-8f46dd48-r6zgk\" (UID: \"0f9549c1-10dc-4700-804f-82d6b1241447\") " pod="openshift-ingress/e2e-ipp-b-79434a-openshift-default-8f46dd48-r6zgk"
Sep 10 22:28:19.324108 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:28:19.324083    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f9549c1-10dc-4700-804f-82d6b1241447-workload-socket\") pod \"e2e-ipp-b-79434a-openshift-default-8f46dd48-r6zgk\" (UID: \"0f9549c1-10dc-4700-804f-82d6b1241447\") " pod="openshift-ingress/e2e-ipp-b-79434a-openshift-default-8f46dd48-r6zgk"
Sep 10 22:28:19.324195 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:28:19.324122    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f9549c1-10dc-4700-804f-82d6b1241447-workload-certs\") pod \"e2e-ipp-b-79434a-openshift-default-8f46dd48-r6zgk\" (UID: \"0f9549c1-10dc-4700-804f-82d6b1241447\") " pod="openshift-ingress/e2e-ipp-b-79434a-openshift-default-8f46dd48-r6zgk"
Sep 10 22:28:19.324518 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:28:19.324467    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f9549c1-10dc-4700-804f-82d6b1241447-istiod-ca-cert\") pod \"e2e-ipp-b-79434a-openshift-default-8f46dd48-r6zgk\" (UID: \"0f9549c1-10dc-4700-804f-82d6b1241447\") " pod="openshift-ingress/e2e-ipp-b-79434a-openshift-default-8f46dd48-r6zgk"
Sep 10 22:28:19.324639 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:28:19.324612    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f9549c1-10dc-4700-804f-82d6b1241447-credential-socket\") pod \"e2e-ipp-b-79434a-openshift-default-8f46dd48-r6zgk\" (UID: \"0f9549c1-10dc-4700-804f-82d6b1241447\") " pod="openshift-ingress/e2e-ipp-b-79434a-openshift-default-8f46dd48-r6zgk"
Sep 10 22:28:19.326782 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:28:19.326735    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f9549c1-10dc-4700-804f-82d6b1241447-istio-podinfo\") pod \"e2e-ipp-b-79434a-openshift-default-8f46dd48-r6zgk\" (UID: \"0f9549c1-10dc-4700-804f-82d6b1241447\") " pod="openshift-ingress/e2e-ipp-b-79434a-openshift-default-8f46dd48-r6zgk"
Sep 10 22:28:19.327145 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:28:19.327099    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f9549c1-10dc-4700-804f-82d6b1241447-istio-envoy\") pod \"e2e-ipp-b-79434a-openshift-default-8f46dd48-r6zgk\" (UID: \"0f9549c1-10dc-4700-804f-82d6b1241447\") " pod="openshift-ingress/e2e-ipp-b-79434a-openshift-default-8f46dd48-r6zgk"
Sep 10 22:28:19.331264 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:28:19.331239    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f9549c1-10dc-4700-804f-82d6b1241447-istio-token\") pod \"e2e-ipp-b-79434a-openshift-default-8f46dd48-r6zgk\" (UID: \"0f9549c1-10dc-4700-804f-82d6b1241447\") " pod="openshift-ingress/e2e-ipp-b-79434a-openshift-default-8f46dd48-r6zgk"
Sep 10 22:28:19.331443 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:28:19.331422    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-62p6p\" (UniqueName: \"kubernetes.io/projected/0f9549c1-10dc-4700-804f-82d6b1241447-kube-api-access-62p6p\") pod \"e2e-ipp-b-79434a-openshift-default-8f46dd48-r6zgk\" (UID: \"0f9549c1-10dc-4700-804f-82d6b1241447\") " pod="openshift-ingress/e2e-ipp-b-79434a-openshift-default-8f46dd48-r6zgk"
Sep 10 22:28:19.437303 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:28:19.437285    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-79434a-openshift-default-dockercfg-fcpw7\""
Sep 10 22:28:19.445784 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:28:19.445766    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-79434a-openshift-default-8f46dd48-r6zgk"
Sep 10 22:28:19.719793 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:28:19.719771    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-79434a-openshift-default-8f46dd48-r6zgk"]
Sep 10 22:28:19.723987 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:28:19.722542    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f9549c1_10dc_4700_804f_82d6b1241447.slice/crio-6eec2aac898bc4363734147008359a6e6e3892605529ae054f1f57a9fa9f87a3 WatchSource:0}: Error finding container 6eec2aac898bc4363734147008359a6e6e3892605529ae054f1f57a9fa9f87a3: Status 404 returned error can't find the container with id 6eec2aac898bc4363734147008359a6e6e3892605529ae054f1f57a9fa9f87a3
Sep 10 22:28:19.727102 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:28:19.727063    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 22:28:19.727203 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:28:19.727130    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 22:28:19.727203 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:28:19.727162    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 22:28:20.123769 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:28:20.123683    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-79434a-openshift-default-8f46dd48-r6zgk" event={"ID":"0f9549c1-10dc-4700-804f-82d6b1241447","Type":"ContainerStarted","Data":"17371bb521b421da173fb53058dd7f51bf314045f2d05b446d18f8ef9791ae72"}
Sep 10 22:28:20.123769 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:28:20.123747    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-79434a-openshift-default-8f46dd48-r6zgk" event={"ID":"0f9549c1-10dc-4700-804f-82d6b1241447","Type":"ContainerStarted","Data":"6eec2aac898bc4363734147008359a6e6e3892605529ae054f1f57a9fa9f87a3"}
Sep 10 22:28:20.143592 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:28:20.143541    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-79434a-openshift-default-8f46dd48-r6zgk" podStartSLOduration=1.143527742 podStartE2EDuration="1.143527742s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 22:28:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:28:20.141358594 +0000 UTC m=+1621.097426300" watchObservedRunningTime="2026-09-10 22:28:20.143527742 +0000 UTC m=+1621.099595448"
Sep 10 22:28:20.446764 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:28:20.446732    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-79434a-openshift-default-8f46dd48-r6zgk"
Sep 10 22:28:20.451599 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:28:20.451573    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-79434a-openshift-default-8f46dd48-r6zgk"
Sep 10 22:28:21.128184 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:28:21.128153    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-79434a-openshift-default-8f46dd48-r6zgk"
Sep 10 22:28:21.128953 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:28:21.128934    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-79434a-openshift-default-8f46dd48-r6zgk"
Sep 10 22:28:23.968772 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:28:23.968733    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-79434a-85db7444f4-bcj4j"]
Sep 10 22:28:23.972275 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:28:23.972252    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-79434a-85db7444f4-bcj4j"
Sep 10 22:28:23.976501 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:28:23.976481    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-79434a\""
Sep 10 22:28:23.976732 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:28:23.976715    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-79434a-dockercfg-prg7w\""
Sep 10 22:28:23.986686 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:28:23.986667    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-79434a-85db7444f4-bcj4j"]
Sep 10 22:28:24.060728 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:28:24.060705    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lc88f\" (UniqueName: \"kubernetes.io/projected/5ceabc6f-b6e9-4a9b-96b4-637182881a75-kube-api-access-lc88f\") pod \"payload-pre-processing-e2e-ipp-b-79434a-85db7444f4-bcj4j\" (UID: \"5ceabc6f-b6e9-4a9b-96b4-637182881a75\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-79434a-85db7444f4-bcj4j"
Sep 10 22:28:24.060834 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:28:24.060747    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ceabc6f-b6e9-4a9b-96b4-637182881a75-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-79434a-85db7444f4-bcj4j\" (UID: \"5ceabc6f-b6e9-4a9b-96b4-637182881a75\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-79434a-85db7444f4-bcj4j"
Sep 10 22:28:24.162243 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:28:24.162214    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lc88f\" (UniqueName: \"kubernetes.io/projected/5ceabc6f-b6e9-4a9b-96b4-637182881a75-kube-api-access-lc88f\") pod \"payload-pre-processing-e2e-ipp-b-79434a-85db7444f4-bcj4j\" (UID: \"5ceabc6f-b6e9-4a9b-96b4-637182881a75\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-79434a-85db7444f4-bcj4j"
Sep 10 22:28:24.162423 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:28:24.162261    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ceabc6f-b6e9-4a9b-96b4-637182881a75-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-79434a-85db7444f4-bcj4j\" (UID: \"5ceabc6f-b6e9-4a9b-96b4-637182881a75\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-79434a-85db7444f4-bcj4j"
Sep 10 22:28:24.162936 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:28:24.162918    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ceabc6f-b6e9-4a9b-96b4-637182881a75-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-79434a-85db7444f4-bcj4j\" (UID: \"5ceabc6f-b6e9-4a9b-96b4-637182881a75\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-79434a-85db7444f4-bcj4j"
Sep 10 22:28:24.170453 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:28:24.170427    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lc88f\" (UniqueName: \"kubernetes.io/projected/5ceabc6f-b6e9-4a9b-96b4-637182881a75-kube-api-access-lc88f\") pod \"payload-pre-processing-e2e-ipp-b-79434a-85db7444f4-bcj4j\" (UID: \"5ceabc6f-b6e9-4a9b-96b4-637182881a75\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-79434a-85db7444f4-bcj4j"
Sep 10 22:28:24.283028 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:28:24.282972    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-79434a-85db7444f4-bcj4j"
Sep 10 22:28:24.551842 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:28:24.551775    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-79434a-85db7444f4-bcj4j"]
Sep 10 22:28:24.553967 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:28:24.553944    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ceabc6f_b6e9_4a9b_96b4_637182881a75.slice/crio-4f30bcf1cafc0370987bd38ffa623deb3feb6f59678496366a515ea9edf132d2 WatchSource:0}: Error finding container 4f30bcf1cafc0370987bd38ffa623deb3feb6f59678496366a515ea9edf132d2: Status 404 returned error can't find the container with id 4f30bcf1cafc0370987bd38ffa623deb3feb6f59678496366a515ea9edf132d2
Sep 10 22:28:25.148729 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:28:25.148696    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-79434a-85db7444f4-bcj4j" event={"ID":"5ceabc6f-b6e9-4a9b-96b4-637182881a75","Type":"ContainerStarted","Data":"0142fe0cbc095c9f2fc18a548827932f3824f4cc2f2e41bbd93a51a6d0efa2f5"}
Sep 10 22:28:25.149228 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:28:25.148745    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-79434a-85db7444f4-bcj4j" event={"ID":"5ceabc6f-b6e9-4a9b-96b4-637182881a75","Type":"ContainerStarted","Data":"4f30bcf1cafc0370987bd38ffa623deb3feb6f59678496366a515ea9edf132d2"}
Sep 10 22:28:25.149228 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:28:25.148797    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-79434a-85db7444f4-bcj4j"
Sep 10 22:28:25.165031 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:28:25.164979    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-79434a-85db7444f4-bcj4j" podStartSLOduration=2.164959379 podStartE2EDuration="2.164959379s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 22:28:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:28:25.164681179 +0000 UTC m=+1626.120748876" watchObservedRunningTime="2026-09-10 22:28:25.164959379 +0000 UTC m=+1626.121027085"
Sep 10 22:28:36.155044 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:28:36.155004    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-79434a-85db7444f4-bcj4j"
Sep 10 22:28:40.232360 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:28:40.232328    2582 generic.go:356] "Generic (PLEG): container finished" podID="0f9549c1-10dc-4700-804f-82d6b1241447" containerID="17371bb521b421da173fb53058dd7f51bf314045f2d05b446d18f8ef9791ae72" exitCode=0
Sep 10 22:28:40.232711 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:28:40.232400    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-79434a-openshift-default-8f46dd48-r6zgk" event={"ID":"0f9549c1-10dc-4700-804f-82d6b1241447","Type":"ContainerDied","Data":"17371bb521b421da173fb53058dd7f51bf314045f2d05b446d18f8ef9791ae72"}
Sep 10 22:28:40.232869 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:28:40.232842    2582 scope.go:117] "RemoveContainer" containerID="17371bb521b421da173fb53058dd7f51bf314045f2d05b446d18f8ef9791ae72"
Sep 10 22:28:40.233941 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:28:40.233911    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 22:28:40.234029 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:28:40.233980    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 22:28:40.234029 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:28:40.234015    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 22:28:41.239867 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:28:41.239820    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-79434a-openshift-default-8f46dd48-r6zgk" event={"ID":"0f9549c1-10dc-4700-804f-82d6b1241447","Type":"ContainerStarted","Data":"541f181436b6e3889bfbee0bd2c0536345eabff04e57049039941ce0f774dbce"}
Sep 10 22:28:41.446759 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:28:41.446717    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-79434a-openshift-default-8f46dd48-r6zgk"
Sep 10 22:28:41.447037 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:28:41.447011    2582 patch_prober.go:28] interesting pod/e2e-ipp-b-79434a-openshift-default-8f46dd48-r6zgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Sep 10 22:28:41.447088 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:28:41.447067    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-79434a-openshift-default-8f46dd48-r6zgk" podUID="0f9549c1-10dc-4700-804f-82d6b1241447" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Sep 10 22:28:42.446543 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:28:42.446508    2582 patch_prober.go:28] interesting pod/e2e-ipp-b-79434a-openshift-default-8f46dd48-r6zgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Sep 10 22:28:42.446946 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:28:42.446562    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-79434a-openshift-default-8f46dd48-r6zgk" podUID="0f9549c1-10dc-4700-804f-82d6b1241447" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Sep 10 22:28:43.446805 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:28:43.446771    2582 patch_prober.go:28] interesting pod/e2e-ipp-b-79434a-openshift-default-8f46dd48-r6zgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Sep 10 22:28:43.447219 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:28:43.446831    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-79434a-openshift-default-8f46dd48-r6zgk" podUID="0f9549c1-10dc-4700-804f-82d6b1241447" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Sep 10 22:28:44.446400 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:28:44.446363    2582 patch_prober.go:28] interesting pod/e2e-ipp-b-79434a-openshift-default-8f46dd48-r6zgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Sep 10 22:28:44.446582 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:28:44.446417    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-79434a-openshift-default-8f46dd48-r6zgk" podUID="0f9549c1-10dc-4700-804f-82d6b1241447" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Sep 10 22:28:45.446235 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:28:45.446196    2582 patch_prober.go:28] interesting pod/e2e-ipp-b-79434a-openshift-default-8f46dd48-r6zgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Sep 10 22:28:45.446610 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:28:45.446264    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-79434a-openshift-default-8f46dd48-r6zgk" podUID="0f9549c1-10dc-4700-804f-82d6b1241447" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Sep 10 22:28:46.450632 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:28:46.450602    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-79434a-openshift-default-8f46dd48-r6zgk"
Sep 10 22:28:46.451112 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:28:46.450886    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-79434a-openshift-default-8f46dd48-r6zgk"
Sep 10 22:28:46.451527 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:28:46.451512    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-79434a-openshift-default-8f46dd48-r6zgk"
Sep 10 22:29:00.903215 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:29:00.903179    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-a12859/ipp-route-a12859-kserve-7dc58c5bc9-lqtq9"]
Sep 10 22:29:00.906577 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:29:00.906559    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-a12859/ipp-route-a12859-kserve-7dc58c5bc9-lqtq9"
Sep 10 22:29:00.909248 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:29:00.909221    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-a12859\"/\"default-dockercfg-cr9lm\""
Sep 10 22:29:00.909367 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:29:00.909251    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-a12859\"/\"kube-root-ca.crt\""
Sep 10 22:29:00.909367 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:29:00.909344    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-a12859\"/\"ipp-route-a12859-kserve-self-signed-certs\""
Sep 10 22:29:00.909367 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:29:00.909357    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-a12859\"/\"openshift-service-ca.crt\""
Sep 10 22:29:00.917387 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:29:00.917365    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-a12859/ipp-route-a12859-kserve-7dc58c5bc9-lqtq9"]
Sep 10 22:29:00.963130 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:29:00.963107    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0cb70f46-10a3-4c47-87a4-85cf4e69847d-dshm\") pod \"ipp-route-a12859-kserve-7dc58c5bc9-lqtq9\" (UID: \"0cb70f46-10a3-4c47-87a4-85cf4e69847d\") " pod="ai-tenant-e2e-ipp-a-a12859/ipp-route-a12859-kserve-7dc58c5bc9-lqtq9"
Sep 10 22:29:00.963243 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:29:00.963132    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0cb70f46-10a3-4c47-87a4-85cf4e69847d-home\") pod \"ipp-route-a12859-kserve-7dc58c5bc9-lqtq9\" (UID: \"0cb70f46-10a3-4c47-87a4-85cf4e69847d\") " pod="ai-tenant-e2e-ipp-a-a12859/ipp-route-a12859-kserve-7dc58c5bc9-lqtq9"
Sep 10 22:29:00.963243 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:29:00.963150    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0cb70f46-10a3-4c47-87a4-85cf4e69847d-tmp-dir\") pod \"ipp-route-a12859-kserve-7dc58c5bc9-lqtq9\" (UID: \"0cb70f46-10a3-4c47-87a4-85cf4e69847d\") " pod="ai-tenant-e2e-ipp-a-a12859/ipp-route-a12859-kserve-7dc58c5bc9-lqtq9"
Sep 10 22:29:00.963243 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:29:00.963180    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4vk6\" (UniqueName: \"kubernetes.io/projected/0cb70f46-10a3-4c47-87a4-85cf4e69847d-kube-api-access-z4vk6\") pod \"ipp-route-a12859-kserve-7dc58c5bc9-lqtq9\" (UID: \"0cb70f46-10a3-4c47-87a4-85cf4e69847d\") " pod="ai-tenant-e2e-ipp-a-a12859/ipp-route-a12859-kserve-7dc58c5bc9-lqtq9"
Sep 10 22:29:00.963416 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:29:00.963232    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0cb70f46-10a3-4c47-87a4-85cf4e69847d-model-cache\") pod \"ipp-route-a12859-kserve-7dc58c5bc9-lqtq9\" (UID: \"0cb70f46-10a3-4c47-87a4-85cf4e69847d\") " pod="ai-tenant-e2e-ipp-a-a12859/ipp-route-a12859-kserve-7dc58c5bc9-lqtq9"
Sep 10 22:29:00.963416 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:29:00.963319    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0cb70f46-10a3-4c47-87a4-85cf4e69847d-tls-certs\") pod \"ipp-route-a12859-kserve-7dc58c5bc9-lqtq9\" (UID: \"0cb70f46-10a3-4c47-87a4-85cf4e69847d\") " pod="ai-tenant-e2e-ipp-a-a12859/ipp-route-a12859-kserve-7dc58c5bc9-lqtq9"
Sep 10 22:29:01.063893 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:29:01.063841    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0cb70f46-10a3-4c47-87a4-85cf4e69847d-tls-certs\") pod \"ipp-route-a12859-kserve-7dc58c5bc9-lqtq9\" (UID: \"0cb70f46-10a3-4c47-87a4-85cf4e69847d\") " pod="ai-tenant-e2e-ipp-a-a12859/ipp-route-a12859-kserve-7dc58c5bc9-lqtq9"
Sep 10 22:29:01.064001 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:29:01.063935    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0cb70f46-10a3-4c47-87a4-85cf4e69847d-dshm\") pod \"ipp-route-a12859-kserve-7dc58c5bc9-lqtq9\" (UID: \"0cb70f46-10a3-4c47-87a4-85cf4e69847d\") " pod="ai-tenant-e2e-ipp-a-a12859/ipp-route-a12859-kserve-7dc58c5bc9-lqtq9"
Sep 10 22:29:01.064001 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:29:01.063952    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0cb70f46-10a3-4c47-87a4-85cf4e69847d-home\") pod \"ipp-route-a12859-kserve-7dc58c5bc9-lqtq9\" (UID: \"0cb70f46-10a3-4c47-87a4-85cf4e69847d\") " pod="ai-tenant-e2e-ipp-a-a12859/ipp-route-a12859-kserve-7dc58c5bc9-lqtq9"
Sep 10 22:29:01.064001 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:29:01.063969    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0cb70f46-10a3-4c47-87a4-85cf4e69847d-tmp-dir\") pod \"ipp-route-a12859-kserve-7dc58c5bc9-lqtq9\" (UID: \"0cb70f46-10a3-4c47-87a4-85cf4e69847d\") " pod="ai-tenant-e2e-ipp-a-a12859/ipp-route-a12859-kserve-7dc58c5bc9-lqtq9"
Sep 10 22:29:01.064001 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:29:01.063998    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4vk6\" (UniqueName: \"kubernetes.io/projected/0cb70f46-10a3-4c47-87a4-85cf4e69847d-kube-api-access-z4vk6\") pod \"ipp-route-a12859-kserve-7dc58c5bc9-lqtq9\" (UID: \"0cb70f46-10a3-4c47-87a4-85cf4e69847d\") " pod="ai-tenant-e2e-ipp-a-a12859/ipp-route-a12859-kserve-7dc58c5bc9-lqtq9"
Sep 10 22:29:01.064159 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:29:01.064020    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0cb70f46-10a3-4c47-87a4-85cf4e69847d-model-cache\") pod \"ipp-route-a12859-kserve-7dc58c5bc9-lqtq9\" (UID: \"0cb70f46-10a3-4c47-87a4-85cf4e69847d\") " pod="ai-tenant-e2e-ipp-a-a12859/ipp-route-a12859-kserve-7dc58c5bc9-lqtq9"
Sep 10 22:29:01.064406 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:29:01.064381    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0cb70f46-10a3-4c47-87a4-85cf4e69847d-tmp-dir\") pod \"ipp-route-a12859-kserve-7dc58c5bc9-lqtq9\" (UID: \"0cb70f46-10a3-4c47-87a4-85cf4e69847d\") " pod="ai-tenant-e2e-ipp-a-a12859/ipp-route-a12859-kserve-7dc58c5bc9-lqtq9"
Sep 10 22:29:01.064500 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:29:01.064404    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0cb70f46-10a3-4c47-87a4-85cf4e69847d-model-cache\") pod \"ipp-route-a12859-kserve-7dc58c5bc9-lqtq9\" (UID: \"0cb70f46-10a3-4c47-87a4-85cf4e69847d\") " pod="ai-tenant-e2e-ipp-a-a12859/ipp-route-a12859-kserve-7dc58c5bc9-lqtq9"
Sep 10 22:29:01.064500 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:29:01.064448    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0cb70f46-10a3-4c47-87a4-85cf4e69847d-home\") pod \"ipp-route-a12859-kserve-7dc58c5bc9-lqtq9\" (UID: \"0cb70f46-10a3-4c47-87a4-85cf4e69847d\") " pod="ai-tenant-e2e-ipp-a-a12859/ipp-route-a12859-kserve-7dc58c5bc9-lqtq9"
Sep 10 22:29:01.066280 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:29:01.066256    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0cb70f46-10a3-4c47-87a4-85cf4e69847d-dshm\") pod \"ipp-route-a12859-kserve-7dc58c5bc9-lqtq9\" (UID: \"0cb70f46-10a3-4c47-87a4-85cf4e69847d\") " pod="ai-tenant-e2e-ipp-a-a12859/ipp-route-a12859-kserve-7dc58c5bc9-lqtq9"
Sep 10 22:29:01.066374 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:29:01.066329    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0cb70f46-10a3-4c47-87a4-85cf4e69847d-tls-certs\") pod \"ipp-route-a12859-kserve-7dc58c5bc9-lqtq9\" (UID: \"0cb70f46-10a3-4c47-87a4-85cf4e69847d\") " pod="ai-tenant-e2e-ipp-a-a12859/ipp-route-a12859-kserve-7dc58c5bc9-lqtq9"
Sep 10 22:29:01.074380 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:29:01.074361    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4vk6\" (UniqueName: \"kubernetes.io/projected/0cb70f46-10a3-4c47-87a4-85cf4e69847d-kube-api-access-z4vk6\") pod \"ipp-route-a12859-kserve-7dc58c5bc9-lqtq9\" (UID: \"0cb70f46-10a3-4c47-87a4-85cf4e69847d\") " pod="ai-tenant-e2e-ipp-a-a12859/ipp-route-a12859-kserve-7dc58c5bc9-lqtq9"
Sep 10 22:29:01.215306 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:29:01.215271    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-a12859/ipp-route-a12859-kserve-7dc58c5bc9-lqtq9"
Sep 10 22:29:01.484435 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:29:01.484412    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-a12859/ipp-route-a12859-kserve-7dc58c5bc9-lqtq9"]
Sep 10 22:29:01.486695 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:29:01.486669    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0cb70f46_10a3_4c47_87a4_85cf4e69847d.slice/crio-1ffd52690972857025c9bdbbfa9c90880a6c37a908b9de9d01fe429f80e962f4 WatchSource:0}: Error finding container 1ffd52690972857025c9bdbbfa9c90880a6c37a908b9de9d01fe429f80e962f4: Status 404 returned error can't find the container with id 1ffd52690972857025c9bdbbfa9c90880a6c37a908b9de9d01fe429f80e962f4
Sep 10 22:29:02.344296 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:29:02.344255    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-a12859/ipp-route-a12859-kserve-7dc58c5bc9-lqtq9" event={"ID":"0cb70f46-10a3-4c47-87a4-85cf4e69847d","Type":"ContainerStarted","Data":"9e163ecec7b4597e12264d8d86a7887440352f62e489c21e6e4427dfd083b20c"}
Sep 10 22:29:02.344744 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:29:02.344306    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-a12859/ipp-route-a12859-kserve-7dc58c5bc9-lqtq9" event={"ID":"0cb70f46-10a3-4c47-87a4-85cf4e69847d","Type":"ContainerStarted","Data":"1ffd52690972857025c9bdbbfa9c90880a6c37a908b9de9d01fe429f80e962f4"}
Sep 10 22:29:02.365379 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:29:02.365332    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-a12859/ipp-route-a12859-kserve-7dc58c5bc9-lqtq9" podStartSLOduration=2.365318663 podStartE2EDuration="2.365318663s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 22:29:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:29:02.363782951 +0000 UTC m=+1663.319850657" watchObservedRunningTime="2026-09-10 22:29:02.365318663 +0000 UTC m=+1663.321386369"
Sep 10 22:29:11.215629 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:29:11.215594    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-a12859/ipp-route-a12859-kserve-7dc58c5bc9-lqtq9"
Sep 10 22:29:11.216122 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:29:11.215645    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-a12859/ipp-route-a12859-kserve-7dc58c5bc9-lqtq9"
Sep 10 22:29:11.228098 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:29:11.228065    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-a12859/ipp-route-a12859-kserve-7dc58c5bc9-lqtq9"
Sep 10 22:29:11.401906 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:29:11.401881    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-a12859/ipp-route-a12859-kserve-7dc58c5bc9-lqtq9"
Sep 10 22:29:19.781464 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:29:19.781432    2582 scope.go:117] "RemoveContainer" containerID="b411ee29f9789d9d9799fb8c1a81c1eb6b084dd7ac7547cf4f90c221f5251653"
Sep 10 22:29:33.820195 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:29:33.820159    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-996d51-85cd468b6-5mn2m"]
Sep 10 22:29:33.823613 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:29:33.823593    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-996d51-85cd468b6-5mn2m"
Sep 10 22:29:33.826327 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:29:33.826304    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-996d51-dockercfg-7kc27\""
Sep 10 22:29:33.827706 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:29:33.827689    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-996d51\""
Sep 10 22:29:33.841357 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:29:33.841318    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-996d51-85cd468b6-5mn2m"]
Sep 10 22:29:33.948069 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:29:33.948035    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e310f75-5591-4fc2-9bc7-6e6bd1ae7e81-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-996d51-85cd468b6-5mn2m\" (UID: \"4e310f75-5591-4fc2-9bc7-6e6bd1ae7e81\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-996d51-85cd468b6-5mn2m"
Sep 10 22:29:33.948230 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:29:33.948130    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbw6j\" (UniqueName: \"kubernetes.io/projected/4e310f75-5591-4fc2-9bc7-6e6bd1ae7e81-kube-api-access-fbw6j\") pod \"payload-pre-processing-e2e-ipp-cleanup-996d51-85cd468b6-5mn2m\" (UID: \"4e310f75-5591-4fc2-9bc7-6e6bd1ae7e81\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-996d51-85cd468b6-5mn2m"
Sep 10 22:29:34.049496 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:29:34.049461    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e310f75-5591-4fc2-9bc7-6e6bd1ae7e81-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-996d51-85cd468b6-5mn2m\" (UID: \"4e310f75-5591-4fc2-9bc7-6e6bd1ae7e81\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-996d51-85cd468b6-5mn2m"
Sep 10 22:29:34.049630 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:29:34.049514    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbw6j\" (UniqueName: \"kubernetes.io/projected/4e310f75-5591-4fc2-9bc7-6e6bd1ae7e81-kube-api-access-fbw6j\") pod \"payload-pre-processing-e2e-ipp-cleanup-996d51-85cd468b6-5mn2m\" (UID: \"4e310f75-5591-4fc2-9bc7-6e6bd1ae7e81\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-996d51-85cd468b6-5mn2m"
Sep 10 22:29:34.050267 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:29:34.050249    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e310f75-5591-4fc2-9bc7-6e6bd1ae7e81-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-996d51-85cd468b6-5mn2m\" (UID: \"4e310f75-5591-4fc2-9bc7-6e6bd1ae7e81\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-996d51-85cd468b6-5mn2m"
Sep 10 22:29:34.063806 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:29:34.063781    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbw6j\" (UniqueName: \"kubernetes.io/projected/4e310f75-5591-4fc2-9bc7-6e6bd1ae7e81-kube-api-access-fbw6j\") pod \"payload-pre-processing-e2e-ipp-cleanup-996d51-85cd468b6-5mn2m\" (UID: \"4e310f75-5591-4fc2-9bc7-6e6bd1ae7e81\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-996d51-85cd468b6-5mn2m"
Sep 10 22:29:34.134628 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:29:34.134572    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-996d51-85cd468b6-5mn2m"
Sep 10 22:29:34.439172 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:29:34.439143    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-996d51-85cd468b6-5mn2m"]
Sep 10 22:29:34.440515 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:29:34.440486    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e310f75_5591_4fc2_9bc7_6e6bd1ae7e81.slice/crio-f8ad8d6347f00a9971ab9461511ff08988760f878cf5095505a82323dad2314a WatchSource:0}: Error finding container f8ad8d6347f00a9971ab9461511ff08988760f878cf5095505a82323dad2314a: Status 404 returned error can't find the container with id f8ad8d6347f00a9971ab9461511ff08988760f878cf5095505a82323dad2314a
Sep 10 22:29:34.511919 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:29:34.511891    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-996d51-85cd468b6-5mn2m" event={"ID":"4e310f75-5591-4fc2-9bc7-6e6bd1ae7e81","Type":"ContainerStarted","Data":"8b1db90676b0cb931b835af0f1d7720d4a4dff23f4eab54eeb81300d9a6fc633"}
Sep 10 22:29:34.512039 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:29:34.511941    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-996d51-85cd468b6-5mn2m" event={"ID":"4e310f75-5591-4fc2-9bc7-6e6bd1ae7e81","Type":"ContainerStarted","Data":"f8ad8d6347f00a9971ab9461511ff08988760f878cf5095505a82323dad2314a"}
Sep 10 22:29:34.512104 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:29:34.512042    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-996d51-85cd468b6-5mn2m"
Sep 10 22:29:45.519562 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:29:45.519487    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-996d51-85cd468b6-5mn2m"
Sep 10 22:29:45.549194 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:29:45.549151    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-996d51-85cd468b6-5mn2m" podStartSLOduration=12.549136726 podStartE2EDuration="12.549136726s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 22:29:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:29:34.571374526 +0000 UTC m=+1695.527442233" watchObservedRunningTime="2026-09-10 22:29:45.549136726 +0000 UTC m=+1706.505204431"
Sep 10 22:29:57.498570 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:29:57.498538    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-996d51-dj5fq"]
Sep 10 22:29:57.501682 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:29:57.501662    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-996d51-dj5fq"
Sep 10 22:29:57.503919 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:29:57.503888    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 22:29:57.504259 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:29:57.504240    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8hjrv\""
Sep 10 22:29:57.506101 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:29:57.506077    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-996d51-dj5fq"]
Sep 10 22:29:57.540646 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:29:57.540622    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3895c295-4058-4354-a386-7f54e623d507-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-996d51-dj5fq\" (UID: \"3895c295-4058-4354-a386-7f54e623d507\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-996d51-dj5fq"
Sep 10 22:29:57.641893 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:29:57.641868    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3895c295-4058-4354-a386-7f54e623d507-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-996d51-dj5fq\" (UID: \"3895c295-4058-4354-a386-7f54e623d507\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-996d51-dj5fq"
Sep 10 22:29:57.642437 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:29:57.642419    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3895c295-4058-4354-a386-7f54e623d507-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-996d51-dj5fq\" (UID: \"3895c295-4058-4354-a386-7f54e623d507\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-996d51-dj5fq"
Sep 10 22:29:57.812980 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:29:57.812893    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-996d51-dj5fq"
Sep 10 22:29:58.082733 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:29:58.082662    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-996d51-dj5fq"]
Sep 10 22:29:58.631599 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:29:58.631556    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-996d51-dj5fq" event={"ID":"3895c295-4058-4354-a386-7f54e623d507","Type":"ContainerStarted","Data":"925d30d80ec7826367d46e18732b4e9f03dc969a731e144580b5e2eedb2569f9"}
Sep 10 22:29:58.632015 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:29:58.631655    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-996d51-dj5fq" event={"ID":"3895c295-4058-4354-a386-7f54e623d507","Type":"ContainerStarted","Data":"f63e13710638e2dc3028a6db1b148073b226567f6605bc0a0bd618560f48cb6b"}
Sep 10 22:29:58.647620 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:29:58.647576    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-996d51-dj5fq" podStartSLOduration=1.647560496 podStartE2EDuration="1.647560496s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 22:29:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:29:58.6458103 +0000 UTC m=+1719.601878007" watchObservedRunningTime="2026-09-10 22:29:58.647560496 +0000 UTC m=+1719.603628202"
Sep 10 22:29:59.638734 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:29:59.638699    2582 generic.go:356] "Generic (PLEG): container finished" podID="3895c295-4058-4354-a386-7f54e623d507" containerID="925d30d80ec7826367d46e18732b4e9f03dc969a731e144580b5e2eedb2569f9" exitCode=0
Sep 10 22:29:59.639167 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:29:59.638785    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-996d51-dj5fq" event={"ID":"3895c295-4058-4354-a386-7f54e623d507","Type":"ContainerDied","Data":"925d30d80ec7826367d46e18732b4e9f03dc969a731e144580b5e2eedb2569f9"}
Sep 10 22:30:00.134580 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:30:00.134539    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-79434a-29817990-rvbvd"]
Sep 10 22:30:00.138199 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:30:00.138178    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ec6d47-29817990-nrp45"]
Sep 10 22:30:00.138327 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:30:00.138292    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-79434a-29817990-rvbvd"
Sep 10 22:30:00.141486 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:30:00.141468    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a12859-29817990-fj4bh"]
Sep 10 22:30:00.141646 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:30:00.141626    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ec6d47-29817990-nrp45"
Sep 10 22:30:00.144613 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:30:00.144593    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a12859-29817990-fj4bh"
Sep 10 22:30:00.148222 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:30:00.148202    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ec6d47-29817990-nrp45"]
Sep 10 22:30:00.150560 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:30:00.150509    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-79434a-29817990-rvbvd"]
Sep 10 22:30:00.152171 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:30:00.152154    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-79434a-29817990-rvbvd"
Sep 10 22:30:00.153835 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:30:00.153815    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a12859-29817990-fj4bh"]
Sep 10 22:30:00.156463 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:30:00.156437    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-cleanup-996d51-29817990-k28h2"]
Sep 10 22:30:00.160718 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:30:00.160535    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-cleanup-996d51-29817990-k28h2"
Sep 10 22:30:00.163509 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:30:00.163490    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ec6d47-29817990-nrp45"
Sep 10 22:30:00.171080 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:30:00.170911    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-cleanup-996d51-29817990-k28h2"]
Sep 10 22:30:00.192963 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:30:00.192938    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a12859-29817990-fj4bh"
Sep 10 22:30:00.200432 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:30:00.200409    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-cleanup-996d51-29817990-k28h2"
Sep 10 22:30:00.440011 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:30:00.439875    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-79434a-29817990-rvbvd"]
Sep 10 22:30:00.442382 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:30:00.442357    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacd08608_79ba_4fd2_8dec_1409a5761af3.slice/crio-33d2b3aa80db2ffcd1d814dfd6c5f89a9e11395edde5c4a07fa749530b24232c WatchSource:0}: Error finding container 33d2b3aa80db2ffcd1d814dfd6c5f89a9e11395edde5c4a07fa749530b24232c: Status 404 returned error can't find the container with id 33d2b3aa80db2ffcd1d814dfd6c5f89a9e11395edde5c4a07fa749530b24232c
Sep 10 22:30:00.454948 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:30:00.454923    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ec6d47-29817990-nrp45"]
Sep 10 22:30:00.456892 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:30:00.456864    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5d85362_c5b9_4fca_84b4_78e231c8528d.slice/crio-ae7836a7b305f608dd026a462616985398c10d98d60c4542217983036b5ae18e WatchSource:0}: Error finding container ae7836a7b305f608dd026a462616985398c10d98d60c4542217983036b5ae18e: Status 404 returned error can't find the container with id ae7836a7b305f608dd026a462616985398c10d98d60c4542217983036b5ae18e
Sep 10 22:30:00.494016 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:30:00.493993    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a12859-29817990-fj4bh"]
Sep 10 22:30:00.495640 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:30:00.495616    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4268668_fa77_4642_a070_051e83e655f9.slice/crio-b0564824d88301eeb7992d9a55e5d088c1d03a770ffcef9e0a1e7d6eb8077aa1 WatchSource:0}: Error finding container b0564824d88301eeb7992d9a55e5d088c1d03a770ffcef9e0a1e7d6eb8077aa1: Status 404 returned error can't find the container with id b0564824d88301eeb7992d9a55e5d088c1d03a770ffcef9e0a1e7d6eb8077aa1
Sep 10 22:30:00.506735 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:30:00.506716    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-cleanup-996d51-29817990-k28h2"]
Sep 10 22:30:00.508914 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:30:00.508891    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70eb6cb4_d357_4c2c_b4ab_4eb978435598.slice/crio-697d3df7b58d4e9975a7f9b00fd68ba3f080b915bba37eb2da478c264669ad7b WatchSource:0}: Error finding container 697d3df7b58d4e9975a7f9b00fd68ba3f080b915bba37eb2da478c264669ad7b: Status 404 returned error can't find the container with id 697d3df7b58d4e9975a7f9b00fd68ba3f080b915bba37eb2da478c264669ad7b
Sep 10 22:30:00.652392 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:30:00.652361    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ec6d47-29817990-nrp45" event={"ID":"b5d85362-c5b9-4fca-84b4-78e231c8528d","Type":"ContainerStarted","Data":"ae7836a7b305f608dd026a462616985398c10d98d60c4542217983036b5ae18e"}
Sep 10 22:30:00.653700 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:30:00.653676    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a12859-29817990-fj4bh" event={"ID":"c4268668-fa77-4642-a070-051e83e655f9","Type":"ContainerStarted","Data":"b0564824d88301eeb7992d9a55e5d088c1d03a770ffcef9e0a1e7d6eb8077aa1"}
Sep 10 22:30:00.654897 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:30:00.654845    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-79434a-29817990-rvbvd" event={"ID":"acd08608-79ba-4fd2-8dec-1409a5761af3","Type":"ContainerStarted","Data":"33d2b3aa80db2ffcd1d814dfd6c5f89a9e11395edde5c4a07fa749530b24232c"}
Sep 10 22:30:00.655898 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:30:00.655874    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-cleanup-996d51-29817990-k28h2" event={"ID":"70eb6cb4-d357-4c2c-b4ab-4eb978435598","Type":"ContainerStarted","Data":"697d3df7b58d4e9975a7f9b00fd68ba3f080b915bba37eb2da478c264669ad7b"}
Sep 10 22:30:00.845091 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:30:00.845067    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-996d51-dj5fq"
Sep 10 22:30:00.970423 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:30:00.970370    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3895c295-4058-4354-a386-7f54e623d507-maas-api-service-ca\") pod \"3895c295-4058-4354-a386-7f54e623d507\" (UID: \"3895c295-4058-4354-a386-7f54e623d507\") "
Sep 10 22:30:00.970674 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:30:00.970653    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3895c295-4058-4354-a386-7f54e623d507-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3895c295-4058-4354-a386-7f54e623d507" (UID: "3895c295-4058-4354-a386-7f54e623d507"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 22:30:01.071777 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:30:01.071753    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3895c295-4058-4354-a386-7f54e623d507-maas-api-service-ca\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:30:01.665584 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:30:01.665554    2582 generic.go:356] "Generic (PLEG): container finished" podID="c4268668-fa77-4642-a070-051e83e655f9" containerID="403d010af83008b2522446eeb5dd09e2072e7c023c758673eb75670149466299" exitCode=0
Sep 10 22:30:01.665937 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:30:01.665639    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a12859-29817990-fj4bh" event={"ID":"c4268668-fa77-4642-a070-051e83e655f9","Type":"ContainerDied","Data":"403d010af83008b2522446eeb5dd09e2072e7c023c758673eb75670149466299"}
Sep 10 22:30:01.667159 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:30:01.667133    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-79434a-29817990-rvbvd" event={"ID":"acd08608-79ba-4fd2-8dec-1409a5761af3","Type":"ContainerStarted","Data":"1dc994243a44662aad1960e484814f99042ce77febb2e0f92f366e026f817c77"}
Sep 10 22:30:01.668609 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:30:01.668588    2582 generic.go:356] "Generic (PLEG): container finished" podID="70eb6cb4-d357-4c2c-b4ab-4eb978435598" containerID="c7371960bd92cb885df4973495465e83617416c1a7d3a65b692aaa4184f444e6" exitCode=0
Sep 10 22:30:01.668716 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:30:01.668659    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-cleanup-996d51-29817990-k28h2" event={"ID":"70eb6cb4-d357-4c2c-b4ab-4eb978435598","Type":"ContainerDied","Data":"c7371960bd92cb885df4973495465e83617416c1a7d3a65b692aaa4184f444e6"}
Sep 10 22:30:01.669807 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:30:01.669791    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-996d51-dj5fq"
Sep 10 22:30:01.669901 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:30:01.669817    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-996d51-dj5fq" event={"ID":"3895c295-4058-4354-a386-7f54e623d507","Type":"ContainerDied","Data":"f63e13710638e2dc3028a6db1b148073b226567f6605bc0a0bd618560f48cb6b"}
Sep 10 22:30:01.669901 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:30:01.669869    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f63e13710638e2dc3028a6db1b148073b226567f6605bc0a0bd618560f48cb6b"
Sep 10 22:30:01.671399 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:30:01.671377    2582 generic.go:356] "Generic (PLEG): container finished" podID="b5d85362-c5b9-4fca-84b4-78e231c8528d" containerID="c7ead10f9f8ce114531a32dc6033a44a1d9922ca0efd0e6c0504603d75334a00" exitCode=0
Sep 10 22:30:01.671505 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:30:01.671400    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ec6d47-29817990-nrp45" event={"ID":"b5d85362-c5b9-4fca-84b4-78e231c8528d","Type":"ContainerDied","Data":"c7ead10f9f8ce114531a32dc6033a44a1d9922ca0efd0e6c0504603d75334a00"}
Sep 10 22:30:01.719657 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:30:01.719615    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-79434a-29817990-rvbvd" podStartSLOduration=1.7196033750000002 podStartE2EDuration="1.719603375s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:30:01.717636345 +0000 UTC m=+1722.673704052" watchObservedRunningTime="2026-09-10 22:30:01.719603375 +0000 UTC m=+1722.675671081"
Sep 10 22:30:02.678415 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:30:02.678383    2582 generic.go:356] "Generic (PLEG): container finished" podID="acd08608-79ba-4fd2-8dec-1409a5761af3" containerID="1dc994243a44662aad1960e484814f99042ce77febb2e0f92f366e026f817c77" exitCode=0
Sep 10 22:30:02.678883 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:30:02.678449    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-79434a-29817990-rvbvd" event={"ID":"acd08608-79ba-4fd2-8dec-1409a5761af3","Type":"ContainerDied","Data":"1dc994243a44662aad1960e484814f99042ce77febb2e0f92f366e026f817c77"}
Sep 10 22:30:02.822650 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:30:02.822622    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-cleanup-996d51-29817990-k28h2"
Sep 10 22:30:02.859275 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:30:02.859251    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a12859-29817990-fj4bh"
Sep 10 22:30:02.870910 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:30:02.870891    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ec6d47-29817990-nrp45"
Sep 10 22:30:03.684231 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:30:03.684211    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a12859-29817990-fj4bh"
Sep 10 22:30:03.684597 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:30:03.684214    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a12859-29817990-fj4bh" event={"ID":"c4268668-fa77-4642-a070-051e83e655f9","Type":"ContainerDied","Data":"b0564824d88301eeb7992d9a55e5d088c1d03a770ffcef9e0a1e7d6eb8077aa1"}
Sep 10 22:30:03.684597 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:30:03.684343    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b0564824d88301eeb7992d9a55e5d088c1d03a770ffcef9e0a1e7d6eb8077aa1"
Sep 10 22:30:03.685582 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:30:03.685567    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-cleanup-996d51-29817990-k28h2"
Sep 10 22:30:03.685674 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:30:03.685613    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-cleanup-996d51-29817990-k28h2" event={"ID":"70eb6cb4-d357-4c2c-b4ab-4eb978435598","Type":"ContainerDied","Data":"697d3df7b58d4e9975a7f9b00fd68ba3f080b915bba37eb2da478c264669ad7b"}
Sep 10 22:30:03.685674 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:30:03.685651    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="697d3df7b58d4e9975a7f9b00fd68ba3f080b915bba37eb2da478c264669ad7b"
Sep 10 22:30:03.687078 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:30:03.687056    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ec6d47-29817990-nrp45"
Sep 10 22:30:03.687188 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:30:03.687085    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ec6d47-29817990-nrp45" event={"ID":"b5d85362-c5b9-4fca-84b4-78e231c8528d","Type":"ContainerDied","Data":"ae7836a7b305f608dd026a462616985398c10d98d60c4542217983036b5ae18e"}
Sep 10 22:30:03.687188 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:30:03.687117    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ae7836a7b305f608dd026a462616985398c10d98d60c4542217983036b5ae18e"
Sep 10 22:30:03.815547 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:30:03.815523    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-79434a-29817990-rvbvd"
Sep 10 22:30:03.967294 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:30:03.967259    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-cleanup-996d51-29817990-k28h2"]
Sep 10 22:30:03.976239 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:30:03.976214    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-cleanup-996d51-29817990-k28h2"]
Sep 10 22:30:04.015448 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:30:04.015423    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-996d51-85cd468b6-5mn2m"]
Sep 10 22:30:04.015650 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:30:04.015628    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-996d51-85cd468b6-5mn2m" podUID="4e310f75-5591-4fc2-9bc7-6e6bd1ae7e81" containerName="payload-pre-processing" containerID="cri-o://8b1db90676b0cb931b835af0f1d7720d4a4dff23f4eab54eeb81300d9a6fc633" gracePeriod=30
Sep 10 22:30:04.226196 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:30:04.226161    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-996d51-dj5fq"]
Sep 10 22:30:04.232712 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:30:04.232667    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-996d51-dj5fq"]
Sep 10 22:30:04.273908 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:30:04.273836    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-996d51-85cd468b6-5mn2m"
Sep 10 22:30:04.400281 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:30:04.400196    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fbw6j\" (UniqueName: \"kubernetes.io/projected/4e310f75-5591-4fc2-9bc7-6e6bd1ae7e81-kube-api-access-fbw6j\") pod \"4e310f75-5591-4fc2-9bc7-6e6bd1ae7e81\" (UID: \"4e310f75-5591-4fc2-9bc7-6e6bd1ae7e81\") "
Sep 10 22:30:04.400281 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:30:04.400278    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e310f75-5591-4fc2-9bc7-6e6bd1ae7e81-plugins-config-volume\") pod \"4e310f75-5591-4fc2-9bc7-6e6bd1ae7e81\" (UID: \"4e310f75-5591-4fc2-9bc7-6e6bd1ae7e81\") "
Sep 10 22:30:04.400662 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:30:04.400637    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e310f75-5591-4fc2-9bc7-6e6bd1ae7e81-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4e310f75-5591-4fc2-9bc7-6e6bd1ae7e81" (UID: "4e310f75-5591-4fc2-9bc7-6e6bd1ae7e81"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 22:30:04.402355 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:30:04.402332    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e310f75-5591-4fc2-9bc7-6e6bd1ae7e81-kube-api-access-fbw6j" (OuterVolumeSpecName: "kube-api-access-fbw6j") pod "4e310f75-5591-4fc2-9bc7-6e6bd1ae7e81" (UID: "4e310f75-5591-4fc2-9bc7-6e6bd1ae7e81"). InnerVolumeSpecName "kube-api-access-fbw6j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 22:30:04.501606 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:30:04.501582    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e310f75-5591-4fc2-9bc7-6e6bd1ae7e81-plugins-config-volume\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:30:04.501606 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:30:04.501605    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fbw6j\" (UniqueName: \"kubernetes.io/projected/4e310f75-5591-4fc2-9bc7-6e6bd1ae7e81-kube-api-access-fbw6j\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:30:04.692670 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:30:04.692642    2582 generic.go:356] "Generic (PLEG): container finished" podID="4e310f75-5591-4fc2-9bc7-6e6bd1ae7e81" containerID="8b1db90676b0cb931b835af0f1d7720d4a4dff23f4eab54eeb81300d9a6fc633" exitCode=0
Sep 10 22:30:04.693057 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:30:04.692681    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-996d51-85cd468b6-5mn2m"
Sep 10 22:30:04.693057 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:30:04.692730    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-996d51-85cd468b6-5mn2m" event={"ID":"4e310f75-5591-4fc2-9bc7-6e6bd1ae7e81","Type":"ContainerDied","Data":"8b1db90676b0cb931b835af0f1d7720d4a4dff23f4eab54eeb81300d9a6fc633"}
Sep 10 22:30:04.693057 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:30:04.692788    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-996d51-85cd468b6-5mn2m" event={"ID":"4e310f75-5591-4fc2-9bc7-6e6bd1ae7e81","Type":"ContainerDied","Data":"f8ad8d6347f00a9971ab9461511ff08988760f878cf5095505a82323dad2314a"}
Sep 10 22:30:04.693057 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:30:04.692842    2582 scope.go:117] "RemoveContainer" containerID="8b1db90676b0cb931b835af0f1d7720d4a4dff23f4eab54eeb81300d9a6fc633"
Sep 10 22:30:04.694423 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:30:04.694405    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-79434a-29817990-rvbvd"
Sep 10 22:30:04.694423 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:30:04.694407    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-79434a-29817990-rvbvd" event={"ID":"acd08608-79ba-4fd2-8dec-1409a5761af3","Type":"ContainerDied","Data":"33d2b3aa80db2ffcd1d814dfd6c5f89a9e11395edde5c4a07fa749530b24232c"}
Sep 10 22:30:04.694612 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:30:04.694458    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="33d2b3aa80db2ffcd1d814dfd6c5f89a9e11395edde5c4a07fa749530b24232c"
Sep 10 22:30:04.701730 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:30:04.701641    2582 scope.go:117] "RemoveContainer" containerID="8b1db90676b0cb931b835af0f1d7720d4a4dff23f4eab54eeb81300d9a6fc633"
Sep 10 22:30:04.702065 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:30:04.702044    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b1db90676b0cb931b835af0f1d7720d4a4dff23f4eab54eeb81300d9a6fc633\": container with ID starting with 8b1db90676b0cb931b835af0f1d7720d4a4dff23f4eab54eeb81300d9a6fc633 not found: ID does not exist" containerID="8b1db90676b0cb931b835af0f1d7720d4a4dff23f4eab54eeb81300d9a6fc633"
Sep 10 22:30:04.702160 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:30:04.702070    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b1db90676b0cb931b835af0f1d7720d4a4dff23f4eab54eeb81300d9a6fc633"} err="failed to get container status \"8b1db90676b0cb931b835af0f1d7720d4a4dff23f4eab54eeb81300d9a6fc633\": rpc error: code = NotFound desc = could not find container \"8b1db90676b0cb931b835af0f1d7720d4a4dff23f4eab54eeb81300d9a6fc633\": container with ID starting with 8b1db90676b0cb931b835af0f1d7720d4a4dff23f4eab54eeb81300d9a6fc633 not found: ID does not exist"
Sep 10 22:30:04.715936 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:30:04.715911    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-996d51-85cd468b6-5mn2m"]
Sep 10 22:30:04.720166 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:30:04.720145    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-996d51-85cd468b6-5mn2m"]
Sep 10 22:30:05.498567 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:30:05.498537    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3895c295-4058-4354-a386-7f54e623d507" path="/var/lib/kubelet/pods/3895c295-4058-4354-a386-7f54e623d507/volumes"
Sep 10 22:30:05.498921 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:30:05.498906    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4e310f75-5591-4fc2-9bc7-6e6bd1ae7e81" path="/var/lib/kubelet/pods/4e310f75-5591-4fc2-9bc7-6e6bd1ae7e81/volumes"
Sep 10 22:30:05.499179 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:30:05.499167    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="70eb6cb4-d357-4c2c-b4ab-4eb978435598" path="/var/lib/kubelet/pods/70eb6cb4-d357-4c2c-b4ab-4eb978435598/volumes"
Sep 10 22:30:14.337923 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:30:14.337884    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a12859-22jvv"]
Sep 10 22:30:14.338305 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:30:14.338241    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3895c295-4058-4354-a386-7f54e623d507" containerName="revoke-keys"
Sep 10 22:30:14.338305 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:30:14.338253    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="3895c295-4058-4354-a386-7f54e623d507" containerName="revoke-keys"
Sep 10 22:30:14.338305 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:30:14.338261    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70eb6cb4-d357-4c2c-b4ab-4eb978435598" containerName="cleanup"
Sep 10 22:30:14.338305 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:30:14.338266    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="70eb6cb4-d357-4c2c-b4ab-4eb978435598" containerName="cleanup"
Sep 10 22:30:14.338305 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:30:14.338275    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e310f75-5591-4fc2-9bc7-6e6bd1ae7e81" containerName="payload-pre-processing"
Sep 10 22:30:14.338305 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:30:14.338281    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e310f75-5591-4fc2-9bc7-6e6bd1ae7e81" containerName="payload-pre-processing"
Sep 10 22:30:14.338305 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:30:14.338287    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5d85362-c5b9-4fca-84b4-78e231c8528d" containerName="cleanup"
Sep 10 22:30:14.338305 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:30:14.338292    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5d85362-c5b9-4fca-84b4-78e231c8528d" containerName="cleanup"
Sep 10 22:30:14.338305 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:30:14.338306    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4268668-fa77-4642-a070-051e83e655f9" containerName="cleanup"
Sep 10 22:30:14.338305 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:30:14.338311    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4268668-fa77-4642-a070-051e83e655f9" containerName="cleanup"
Sep 10 22:30:14.338601 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:30:14.338318    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="acd08608-79ba-4fd2-8dec-1409a5761af3" containerName="cleanup"
Sep 10 22:30:14.338601 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:30:14.338323    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="acd08608-79ba-4fd2-8dec-1409a5761af3" containerName="cleanup"
Sep 10 22:30:14.338601 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:30:14.338375    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="70eb6cb4-d357-4c2c-b4ab-4eb978435598" containerName="cleanup"
Sep 10 22:30:14.338601 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:30:14.338383    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="3895c295-4058-4354-a386-7f54e623d507" containerName="revoke-keys"
Sep 10 22:30:14.338601 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:30:14.338389    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e310f75-5591-4fc2-9bc7-6e6bd1ae7e81" containerName="payload-pre-processing"
Sep 10 22:30:14.338601 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:30:14.338397    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="acd08608-79ba-4fd2-8dec-1409a5761af3" containerName="cleanup"
Sep 10 22:30:14.338601 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:30:14.338404    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="b5d85362-c5b9-4fca-84b4-78e231c8528d" containerName="cleanup"
Sep 10 22:30:14.338601 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:30:14.338410    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="c4268668-fa77-4642-a070-051e83e655f9" containerName="cleanup"
Sep 10 22:30:14.342658 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:30:14.342634    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a12859-22jvv"]
Sep 10 22:30:14.342814 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:30:14.342713    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a12859-22jvv"
Sep 10 22:30:14.345002 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:30:14.344978    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 22:30:14.345121 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:30:14.344979    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8hjrv\""
Sep 10 22:30:14.482534 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:30:14.482506    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6af2ab1f-e94e-4e2a-aa8b-1a397cf00c1e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-a12859-22jvv\" (UID: \"6af2ab1f-e94e-4e2a-aa8b-1a397cf00c1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a12859-22jvv"
Sep 10 22:30:14.583061 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:30:14.583035    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6af2ab1f-e94e-4e2a-aa8b-1a397cf00c1e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-a12859-22jvv\" (UID: \"6af2ab1f-e94e-4e2a-aa8b-1a397cf00c1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a12859-22jvv"
Sep 10 22:30:14.583599 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:30:14.583581    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6af2ab1f-e94e-4e2a-aa8b-1a397cf00c1e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-a12859-22jvv\" (UID: \"6af2ab1f-e94e-4e2a-aa8b-1a397cf00c1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a12859-22jvv"
Sep 10 22:30:14.654117 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:30:14.654096    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a12859-22jvv"
Sep 10 22:30:14.924606 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:30:14.924435    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a12859-22jvv"]
Sep 10 22:30:14.927185 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:30:14.927155    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6af2ab1f_e94e_4e2a_aa8b_1a397cf00c1e.slice/crio-2bda9ef3b73d9c28bea122e38dbc229a52b7c258d5d040907fabd957ec742b3e WatchSource:0}: Error finding container 2bda9ef3b73d9c28bea122e38dbc229a52b7c258d5d040907fabd957ec742b3e: Status 404 returned error can't find the container with id 2bda9ef3b73d9c28bea122e38dbc229a52b7c258d5d040907fabd957ec742b3e
Sep 10 22:30:15.754919 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:30:15.754885    2582 generic.go:356] "Generic (PLEG): container finished" podID="6af2ab1f-e94e-4e2a-aa8b-1a397cf00c1e" containerID="f440881c3576a59ae9d42b3377df9112e236875d576f531006b199f7553def85" exitCode=0
Sep 10 22:30:15.755253 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:30:15.754973    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a12859-22jvv" event={"ID":"6af2ab1f-e94e-4e2a-aa8b-1a397cf00c1e","Type":"ContainerDied","Data":"f440881c3576a59ae9d42b3377df9112e236875d576f531006b199f7553def85"}
Sep 10 22:30:15.755253 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:30:15.755040    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a12859-22jvv" event={"ID":"6af2ab1f-e94e-4e2a-aa8b-1a397cf00c1e","Type":"ContainerStarted","Data":"2bda9ef3b73d9c28bea122e38dbc229a52b7c258d5d040907fabd957ec742b3e"}
Sep 10 22:30:16.885051 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:30:16.885023    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a12859-22jvv"
Sep 10 22:30:17.003914 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:30:17.003887    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6af2ab1f-e94e-4e2a-aa8b-1a397cf00c1e-maas-api-service-ca\") pod \"6af2ab1f-e94e-4e2a-aa8b-1a397cf00c1e\" (UID: \"6af2ab1f-e94e-4e2a-aa8b-1a397cf00c1e\") "
Sep 10 22:30:17.004242 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:30:17.004216    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6af2ab1f-e94e-4e2a-aa8b-1a397cf00c1e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6af2ab1f-e94e-4e2a-aa8b-1a397cf00c1e" (UID: "6af2ab1f-e94e-4e2a-aa8b-1a397cf00c1e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 22:30:17.104955 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:30:17.104894    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6af2ab1f-e94e-4e2a-aa8b-1a397cf00c1e-maas-api-service-ca\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:30:17.766934 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:30:17.766872    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a12859-22jvv"
Sep 10 22:30:17.767064 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:30:17.766947    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a12859-22jvv" event={"ID":"6af2ab1f-e94e-4e2a-aa8b-1a397cf00c1e","Type":"ContainerDied","Data":"2bda9ef3b73d9c28bea122e38dbc229a52b7c258d5d040907fabd957ec742b3e"}
Sep 10 22:30:17.767064 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:30:17.766986    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2bda9ef3b73d9c28bea122e38dbc229a52b7c258d5d040907fabd957ec742b3e"
Sep 10 22:30:29.997604 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:30:29.997569    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a12859-29817990-fj4bh"]
Sep 10 22:30:30.001839 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:30:30.001814    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a12859-29817990-fj4bh"]
Sep 10 22:30:30.212711 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:30:30.212676    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a12859-22jvv"]
Sep 10 22:30:30.215964 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:30:30.215938    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a12859-22jvv"]
Sep 10 22:30:31.499014 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:30:31.498976    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6af2ab1f-e94e-4e2a-aa8b-1a397cf00c1e" path="/var/lib/kubelet/pods/6af2ab1f-e94e-4e2a-aa8b-1a397cf00c1e/volumes"
Sep 10 22:30:31.499389 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:30:31.499259    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c4268668-fa77-4642-a070-051e83e655f9" path="/var/lib/kubelet/pods/c4268668-fa77-4642-a070-051e83e655f9/volumes"
Sep 10 22:30:35.571637 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:30:35.571606    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-a12859/ipp-route-a12859-kserve-7dc58c5bc9-lqtq9"]
Sep 10 22:30:50.590455 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:30:50.590417    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-a12859/ipp-route-a12859-kserve-7dc58c5bc9-lqtq9" podUID="0cb70f46-10a3-4c47-87a4-85cf4e69847d" containerName="main" containerID="cri-o://9e163ecec7b4597e12264d8d86a7887440352f62e489c21e6e4427dfd083b20c" gracePeriod=45
Sep 10 22:30:50.843179 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:30:50.843119    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-a12859/ipp-route-a12859-kserve-7dc58c5bc9-lqtq9"
Sep 10 22:30:50.871047 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:30:50.871019    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0cb70f46-10a3-4c47-87a4-85cf4e69847d-home\") pod \"0cb70f46-10a3-4c47-87a4-85cf4e69847d\" (UID: \"0cb70f46-10a3-4c47-87a4-85cf4e69847d\") "
Sep 10 22:30:50.871214 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:30:50.871087    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0cb70f46-10a3-4c47-87a4-85cf4e69847d-tmp-dir\") pod \"0cb70f46-10a3-4c47-87a4-85cf4e69847d\" (UID: \"0cb70f46-10a3-4c47-87a4-85cf4e69847d\") "
Sep 10 22:30:50.871214 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:30:50.871140    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0cb70f46-10a3-4c47-87a4-85cf4e69847d-tls-certs\") pod \"0cb70f46-10a3-4c47-87a4-85cf4e69847d\" (UID: \"0cb70f46-10a3-4c47-87a4-85cf4e69847d\") "
Sep 10 22:30:50.871214 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:30:50.871186    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0cb70f46-10a3-4c47-87a4-85cf4e69847d-model-cache\") pod \"0cb70f46-10a3-4c47-87a4-85cf4e69847d\" (UID: \"0cb70f46-10a3-4c47-87a4-85cf4e69847d\") "
Sep 10 22:30:50.871395 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:30:50.871220    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z4vk6\" (UniqueName: \"kubernetes.io/projected/0cb70f46-10a3-4c47-87a4-85cf4e69847d-kube-api-access-z4vk6\") pod \"0cb70f46-10a3-4c47-87a4-85cf4e69847d\" (UID: \"0cb70f46-10a3-4c47-87a4-85cf4e69847d\") "
Sep 10 22:30:50.871395 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:30:50.871277    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0cb70f46-10a3-4c47-87a4-85cf4e69847d-dshm\") pod \"0cb70f46-10a3-4c47-87a4-85cf4e69847d\" (UID: \"0cb70f46-10a3-4c47-87a4-85cf4e69847d\") "
Sep 10 22:30:50.871495 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:30:50.871270    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cb70f46-10a3-4c47-87a4-85cf4e69847d-home" (OuterVolumeSpecName: "home") pod "0cb70f46-10a3-4c47-87a4-85cf4e69847d" (UID: "0cb70f46-10a3-4c47-87a4-85cf4e69847d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 22:30:50.871495 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:30:50.871413    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cb70f46-10a3-4c47-87a4-85cf4e69847d-model-cache" (OuterVolumeSpecName: "model-cache") pod "0cb70f46-10a3-4c47-87a4-85cf4e69847d" (UID: "0cb70f46-10a3-4c47-87a4-85cf4e69847d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 22:30:50.871495 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:30:50.871312    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cb70f46-10a3-4c47-87a4-85cf4e69847d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0cb70f46-10a3-4c47-87a4-85cf4e69847d" (UID: "0cb70f46-10a3-4c47-87a4-85cf4e69847d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 22:30:50.871665 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:30:50.871646    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0cb70f46-10a3-4c47-87a4-85cf4e69847d-home\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:30:50.871732 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:30:50.871669    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0cb70f46-10a3-4c47-87a4-85cf4e69847d-tmp-dir\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:30:50.871732 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:30:50.871684    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0cb70f46-10a3-4c47-87a4-85cf4e69847d-model-cache\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:30:50.873602 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:30:50.873569    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cb70f46-10a3-4c47-87a4-85cf4e69847d-dshm" (OuterVolumeSpecName: "dshm") pod "0cb70f46-10a3-4c47-87a4-85cf4e69847d" (UID: "0cb70f46-10a3-4c47-87a4-85cf4e69847d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 22:30:50.873919 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:30:50.873888    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0cb70f46-10a3-4c47-87a4-85cf4e69847d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0cb70f46-10a3-4c47-87a4-85cf4e69847d" (UID: "0cb70f46-10a3-4c47-87a4-85cf4e69847d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 22:30:50.873919 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:30:50.873894    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cb70f46-10a3-4c47-87a4-85cf4e69847d-kube-api-access-z4vk6" (OuterVolumeSpecName: "kube-api-access-z4vk6") pod "0cb70f46-10a3-4c47-87a4-85cf4e69847d" (UID: "0cb70f46-10a3-4c47-87a4-85cf4e69847d"). InnerVolumeSpecName "kube-api-access-z4vk6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 22:30:50.934967 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:30:50.934935    2582 generic.go:356] "Generic (PLEG): container finished" podID="0cb70f46-10a3-4c47-87a4-85cf4e69847d" containerID="9e163ecec7b4597e12264d8d86a7887440352f62e489c21e6e4427dfd083b20c" exitCode=0
Sep 10 22:30:50.935110 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:30:50.934981    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-a12859/ipp-route-a12859-kserve-7dc58c5bc9-lqtq9"
Sep 10 22:30:50.935110 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:30:50.935023    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-a12859/ipp-route-a12859-kserve-7dc58c5bc9-lqtq9" event={"ID":"0cb70f46-10a3-4c47-87a4-85cf4e69847d","Type":"ContainerDied","Data":"9e163ecec7b4597e12264d8d86a7887440352f62e489c21e6e4427dfd083b20c"}
Sep 10 22:30:50.935110 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:30:50.935063    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-a12859/ipp-route-a12859-kserve-7dc58c5bc9-lqtq9" event={"ID":"0cb70f46-10a3-4c47-87a4-85cf4e69847d","Type":"ContainerDied","Data":"1ffd52690972857025c9bdbbfa9c90880a6c37a908b9de9d01fe429f80e962f4"}
Sep 10 22:30:50.935110 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:30:50.935083    2582 scope.go:117] "RemoveContainer" containerID="9e163ecec7b4597e12264d8d86a7887440352f62e489c21e6e4427dfd083b20c"
Sep 10 22:30:50.944875 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:30:50.944837    2582 scope.go:117] "RemoveContainer" containerID="9e163ecec7b4597e12264d8d86a7887440352f62e489c21e6e4427dfd083b20c"
Sep 10 22:30:50.945147 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:30:50.945128    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e163ecec7b4597e12264d8d86a7887440352f62e489c21e6e4427dfd083b20c\": container with ID starting with 9e163ecec7b4597e12264d8d86a7887440352f62e489c21e6e4427dfd083b20c not found: ID does not exist" containerID="9e163ecec7b4597e12264d8d86a7887440352f62e489c21e6e4427dfd083b20c"
Sep 10 22:30:50.945198 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:30:50.945155    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e163ecec7b4597e12264d8d86a7887440352f62e489c21e6e4427dfd083b20c"} err="failed to get container status \"9e163ecec7b4597e12264d8d86a7887440352f62e489c21e6e4427dfd083b20c\": rpc error: code = NotFound desc = could not find container \"9e163ecec7b4597e12264d8d86a7887440352f62e489c21e6e4427dfd083b20c\": container with ID starting with 9e163ecec7b4597e12264d8d86a7887440352f62e489c21e6e4427dfd083b20c not found: ID does not exist"
Sep 10 22:30:50.955752 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:30:50.955727    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-a12859/ipp-route-a12859-kserve-7dc58c5bc9-lqtq9"]
Sep 10 22:30:50.962744 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:30:50.962718    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-a12859/ipp-route-a12859-kserve-7dc58c5bc9-lqtq9"]
Sep 10 22:30:50.972970 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:30:50.972948    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0cb70f46-10a3-4c47-87a4-85cf4e69847d-tls-certs\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:30:50.972970 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:30:50.972970    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z4vk6\" (UniqueName: \"kubernetes.io/projected/0cb70f46-10a3-4c47-87a4-85cf4e69847d-kube-api-access-z4vk6\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:30:50.973130 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:30:50.972980    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0cb70f46-10a3-4c47-87a4-85cf4e69847d-dshm\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:30:51.498529 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:30:51.498493    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0cb70f46-10a3-4c47-87a4-85cf4e69847d" path="/var/lib/kubelet/pods/0cb70f46-10a3-4c47-87a4-85cf4e69847d/volumes"
Sep 10 22:31:19.328729 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:31:19.328688    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-79434a-xlcf6"]
Sep 10 22:31:19.329115 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:31:19.329046    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0cb70f46-10a3-4c47-87a4-85cf4e69847d" containerName="main"
Sep 10 22:31:19.329115 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:31:19.329058    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="0cb70f46-10a3-4c47-87a4-85cf4e69847d" containerName="main"
Sep 10 22:31:19.329115 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:31:19.329079    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6af2ab1f-e94e-4e2a-aa8b-1a397cf00c1e" containerName="revoke-keys"
Sep 10 22:31:19.329115 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:31:19.329084    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="6af2ab1f-e94e-4e2a-aa8b-1a397cf00c1e" containerName="revoke-keys"
Sep 10 22:31:19.329246 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:31:19.329141    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="6af2ab1f-e94e-4e2a-aa8b-1a397cf00c1e" containerName="revoke-keys"
Sep 10 22:31:19.329246 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:31:19.329152    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="0cb70f46-10a3-4c47-87a4-85cf4e69847d" containerName="main"
Sep 10 22:31:19.332137 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:31:19.332119    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-79434a-xlcf6"
Sep 10 22:31:19.334160 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:31:19.334139    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 22:31:19.334233 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:31:19.334164    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8hjrv\""
Sep 10 22:31:19.338340 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:31:19.338318    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-79434a-xlcf6"]
Sep 10 22:31:19.399231 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:31:19.399206    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffc78611-91a9-4dbb-9398-7a9a16e96a12-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-79434a-xlcf6\" (UID: \"ffc78611-91a9-4dbb-9398-7a9a16e96a12\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-79434a-xlcf6"
Sep 10 22:31:19.500312 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:31:19.500291    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffc78611-91a9-4dbb-9398-7a9a16e96a12-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-79434a-xlcf6\" (UID: \"ffc78611-91a9-4dbb-9398-7a9a16e96a12\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-79434a-xlcf6"
Sep 10 22:31:19.502386 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:31:19.502369    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 22:31:19.511948 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:31:19.511928    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffc78611-91a9-4dbb-9398-7a9a16e96a12-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-79434a-xlcf6\" (UID: \"ffc78611-91a9-4dbb-9398-7a9a16e96a12\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-79434a-xlcf6"
Sep 10 22:31:19.567661 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:31:19.567640    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-w6vlb_8517c9a7-382a-40fd-bc52-e6da90ebc584/console-operator/2.log"
Sep 10 22:31:19.572096 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:31:19.572072    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-w6vlb_8517c9a7-382a-40fd-bc52-e6da90ebc584/console-operator/2.log"
Sep 10 22:31:19.646523 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:31:19.646464    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8hjrv\""
Sep 10 22:31:19.654111 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:31:19.654087    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-79434a-xlcf6"
Sep 10 22:31:19.922748 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:31:19.922723    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-79434a-xlcf6"]
Sep 10 22:31:19.924895 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:31:19.924867    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffc78611_91a9_4dbb_9398_7a9a16e96a12.slice/crio-28a505226b99855828b9183e68fa13c6f866ff0583f344c2b89334dfd8e4c779 WatchSource:0}: Error finding container 28a505226b99855828b9183e68fa13c6f866ff0583f344c2b89334dfd8e4c779: Status 404 returned error can't find the container with id 28a505226b99855828b9183e68fa13c6f866ff0583f344c2b89334dfd8e4c779
Sep 10 22:31:20.087546 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:31:20.087512    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-79434a-xlcf6" event={"ID":"ffc78611-91a9-4dbb-9398-7a9a16e96a12","Type":"ContainerStarted","Data":"28a505226b99855828b9183e68fa13c6f866ff0583f344c2b89334dfd8e4c779"}
Sep 10 22:31:21.093561 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:31:21.093526    2582 generic.go:356] "Generic (PLEG): container finished" podID="ffc78611-91a9-4dbb-9398-7a9a16e96a12" containerID="674d6658a33b05d889a577d5d1db090b34527bcc01e013f1e048e3898283757d" exitCode=0
Sep 10 22:31:21.093561 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:31:21.093564    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-79434a-xlcf6" event={"ID":"ffc78611-91a9-4dbb-9398-7a9a16e96a12","Type":"ContainerDied","Data":"674d6658a33b05d889a577d5d1db090b34527bcc01e013f1e048e3898283757d"}
Sep 10 22:31:22.224222 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:31:22.224197    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-79434a-xlcf6"
Sep 10 22:31:22.325143 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:31:22.325111    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffc78611-91a9-4dbb-9398-7a9a16e96a12-maas-api-service-ca\") pod \"ffc78611-91a9-4dbb-9398-7a9a16e96a12\" (UID: \"ffc78611-91a9-4dbb-9398-7a9a16e96a12\") "
Sep 10 22:31:22.325486 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:31:22.325467    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffc78611-91a9-4dbb-9398-7a9a16e96a12-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ffc78611-91a9-4dbb-9398-7a9a16e96a12" (UID: "ffc78611-91a9-4dbb-9398-7a9a16e96a12"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 22:31:22.425995 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:31:22.425972    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffc78611-91a9-4dbb-9398-7a9a16e96a12-maas-api-service-ca\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:31:23.105185 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:31:23.105156    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-79434a-xlcf6"
Sep 10 22:31:23.105355 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:31:23.105188    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-79434a-xlcf6" event={"ID":"ffc78611-91a9-4dbb-9398-7a9a16e96a12","Type":"ContainerDied","Data":"28a505226b99855828b9183e68fa13c6f866ff0583f344c2b89334dfd8e4c779"}
Sep 10 22:31:23.105355 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:31:23.105234    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="28a505226b99855828b9183e68fa13c6f866ff0583f344c2b89334dfd8e4c779"
Sep 10 22:31:30.000338 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:31:30.000303    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-79434a-29817990-rvbvd"]
Sep 10 22:31:30.003357 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:31:30.003331    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-79434a-29817990-rvbvd"]
Sep 10 22:31:30.038379 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:31:30.038347    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-79434a-85db7444f4-bcj4j"]
Sep 10 22:31:30.038629 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:31:30.038575    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-79434a-85db7444f4-bcj4j" podUID="5ceabc6f-b6e9-4a9b-96b4-637182881a75" containerName="payload-pre-processing" containerID="cri-o://0142fe0cbc095c9f2fc18a548827932f3824f4cc2f2e41bbd93a51a6d0efa2f5" gracePeriod=30
Sep 10 22:31:30.226399 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:31:30.226351    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-79434a-xlcf6"]
Sep 10 22:31:30.228985 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:31:30.228959    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-79434a-xlcf6"]
Sep 10 22:31:30.287574 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:31:30.287554    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-79434a-85db7444f4-bcj4j"
Sep 10 22:31:30.389542 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:31:30.389482    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lc88f\" (UniqueName: \"kubernetes.io/projected/5ceabc6f-b6e9-4a9b-96b4-637182881a75-kube-api-access-lc88f\") pod \"5ceabc6f-b6e9-4a9b-96b4-637182881a75\" (UID: \"5ceabc6f-b6e9-4a9b-96b4-637182881a75\") "
Sep 10 22:31:30.389725 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:31:30.389631    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ceabc6f-b6e9-4a9b-96b4-637182881a75-plugins-config-volume\") pod \"5ceabc6f-b6e9-4a9b-96b4-637182881a75\" (UID: \"5ceabc6f-b6e9-4a9b-96b4-637182881a75\") "
Sep 10 22:31:30.390011 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:31:30.389989    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ceabc6f-b6e9-4a9b-96b4-637182881a75-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5ceabc6f-b6e9-4a9b-96b4-637182881a75" (UID: "5ceabc6f-b6e9-4a9b-96b4-637182881a75"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 22:31:30.391657 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:31:30.391634    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ceabc6f-b6e9-4a9b-96b4-637182881a75-kube-api-access-lc88f" (OuterVolumeSpecName: "kube-api-access-lc88f") pod "5ceabc6f-b6e9-4a9b-96b4-637182881a75" (UID: "5ceabc6f-b6e9-4a9b-96b4-637182881a75"). InnerVolumeSpecName "kube-api-access-lc88f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 22:31:30.490585 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:31:30.490546    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ceabc6f-b6e9-4a9b-96b4-637182881a75-plugins-config-volume\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:31:30.490585 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:31:30.490571    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lc88f\" (UniqueName: \"kubernetes.io/projected/5ceabc6f-b6e9-4a9b-96b4-637182881a75-kube-api-access-lc88f\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:31:31.147903 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:31:31.147846    2582 generic.go:356] "Generic (PLEG): container finished" podID="5ceabc6f-b6e9-4a9b-96b4-637182881a75" containerID="0142fe0cbc095c9f2fc18a548827932f3824f4cc2f2e41bbd93a51a6d0efa2f5" exitCode=0
Sep 10 22:31:31.148392 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:31:31.147912    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-79434a-85db7444f4-bcj4j"
Sep 10 22:31:31.148392 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:31:31.147931    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-79434a-85db7444f4-bcj4j" event={"ID":"5ceabc6f-b6e9-4a9b-96b4-637182881a75","Type":"ContainerDied","Data":"0142fe0cbc095c9f2fc18a548827932f3824f4cc2f2e41bbd93a51a6d0efa2f5"}
Sep 10 22:31:31.148392 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:31:31.147987    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-79434a-85db7444f4-bcj4j" event={"ID":"5ceabc6f-b6e9-4a9b-96b4-637182881a75","Type":"ContainerDied","Data":"4f30bcf1cafc0370987bd38ffa623deb3feb6f59678496366a515ea9edf132d2"}
Sep 10 22:31:31.148392 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:31:31.148019    2582 scope.go:117] "RemoveContainer" containerID="0142fe0cbc095c9f2fc18a548827932f3824f4cc2f2e41bbd93a51a6d0efa2f5"
Sep 10 22:31:31.157143 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:31:31.157122    2582 scope.go:117] "RemoveContainer" containerID="0142fe0cbc095c9f2fc18a548827932f3824f4cc2f2e41bbd93a51a6d0efa2f5"
Sep 10 22:31:31.157418 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:31:31.157397    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0142fe0cbc095c9f2fc18a548827932f3824f4cc2f2e41bbd93a51a6d0efa2f5\": container with ID starting with 0142fe0cbc095c9f2fc18a548827932f3824f4cc2f2e41bbd93a51a6d0efa2f5 not found: ID does not exist" containerID="0142fe0cbc095c9f2fc18a548827932f3824f4cc2f2e41bbd93a51a6d0efa2f5"
Sep 10 22:31:31.157494 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:31:31.157426    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0142fe0cbc095c9f2fc18a548827932f3824f4cc2f2e41bbd93a51a6d0efa2f5"} err="failed to get container status \"0142fe0cbc095c9f2fc18a548827932f3824f4cc2f2e41bbd93a51a6d0efa2f5\": rpc error: code = NotFound desc = could not find container \"0142fe0cbc095c9f2fc18a548827932f3824f4cc2f2e41bbd93a51a6d0efa2f5\": container with ID starting with 0142fe0cbc095c9f2fc18a548827932f3824f4cc2f2e41bbd93a51a6d0efa2f5 not found: ID does not exist"
Sep 10 22:31:31.167691 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:31:31.167672    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-79434a-85db7444f4-bcj4j"]
Sep 10 22:31:31.176260 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:31:31.176223    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-79434a-85db7444f4-bcj4j"]
Sep 10 22:31:31.498657 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:31:31.498623    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5ceabc6f-b6e9-4a9b-96b4-637182881a75" path="/var/lib/kubelet/pods/5ceabc6f-b6e9-4a9b-96b4-637182881a75/volumes"
Sep 10 22:31:31.498922 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:31:31.498907    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="acd08608-79ba-4fd2-8dec-1409a5761af3" path="/var/lib/kubelet/pods/acd08608-79ba-4fd2-8dec-1409a5761af3/volumes"
Sep 10 22:31:31.499169 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:31:31.499157    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ffc78611-91a9-4dbb-9398-7a9a16e96a12" path="/var/lib/kubelet/pods/ffc78611-91a9-4dbb-9398-7a9a16e96a12/volumes"
Sep 10 22:31:37.327219 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:31:37.327185    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-79434a-openshift-default-8f46dd48-r6zgk"]
Sep 10 22:31:37.327784 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:31:37.327511    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-79434a-openshift-default-8f46dd48-r6zgk" podUID="0f9549c1-10dc-4700-804f-82d6b1241447" containerName="istio-proxy" containerID="cri-o://541f181436b6e3889bfbee0bd2c0536345eabff04e57049039941ce0f774dbce" gracePeriod=30
Sep 10 22:31:37.704713 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:31:37.704672    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-d07dfb-7tcm8"]
Sep 10 22:31:37.705095 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:31:37.705078    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ceabc6f-b6e9-4a9b-96b4-637182881a75" containerName="payload-pre-processing"
Sep 10 22:31:37.705095 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:31:37.705097    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ceabc6f-b6e9-4a9b-96b4-637182881a75" containerName="payload-pre-processing"
Sep 10 22:31:37.705252 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:31:37.705109    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ffc78611-91a9-4dbb-9398-7a9a16e96a12" containerName="revoke-keys"
Sep 10 22:31:37.705252 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:31:37.705117    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="ffc78611-91a9-4dbb-9398-7a9a16e96a12" containerName="revoke-keys"
Sep 10 22:31:37.705252 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:31:37.705232    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="ffc78611-91a9-4dbb-9398-7a9a16e96a12" containerName="revoke-keys"
Sep 10 22:31:37.705252 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:31:37.705247    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ceabc6f-b6e9-4a9b-96b4-637182881a75" containerName="payload-pre-processing"
Sep 10 22:31:37.709365 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:31:37.709344    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-d07dfb-7tcm8"
Sep 10 22:31:37.711348 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:31:37.711326    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 22:31:37.711469 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:31:37.711324    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8hjrv\""
Sep 10 22:31:37.715137 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:31:37.715111    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-d07dfb-7tcm8"]
Sep 10 22:31:37.847930 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:31:37.847901    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/45ad2ba5-16bf-4d2f-b91a-a488686a526b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-d07dfb-7tcm8\" (UID: \"45ad2ba5-16bf-4d2f-b91a-a488686a526b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-d07dfb-7tcm8"
Sep 10 22:31:37.948979 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:31:37.948948    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/45ad2ba5-16bf-4d2f-b91a-a488686a526b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-d07dfb-7tcm8\" (UID: \"45ad2ba5-16bf-4d2f-b91a-a488686a526b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-d07dfb-7tcm8"
Sep 10 22:31:37.949585 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:31:37.949566    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/45ad2ba5-16bf-4d2f-b91a-a488686a526b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-d07dfb-7tcm8\" (UID: \"45ad2ba5-16bf-4d2f-b91a-a488686a526b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-d07dfb-7tcm8"
Sep 10 22:31:38.020019 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:31:38.019959    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-d07dfb-7tcm8"
Sep 10 22:31:38.287030 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:31:38.287004    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-d07dfb-7tcm8"]
Sep 10 22:31:39.191426 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:31:39.191394    2582 generic.go:356] "Generic (PLEG): container finished" podID="45ad2ba5-16bf-4d2f-b91a-a488686a526b" containerID="9f7375517642f25cbecc9971068a1f2b4b450fb6c312ad546365393b02b70645" exitCode=0
Sep 10 22:31:39.191790 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:31:39.191477    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-d07dfb-7tcm8" event={"ID":"45ad2ba5-16bf-4d2f-b91a-a488686a526b","Type":"ContainerDied","Data":"9f7375517642f25cbecc9971068a1f2b4b450fb6c312ad546365393b02b70645"}
Sep 10 22:31:39.191790 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:31:39.191522    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-d07dfb-7tcm8" event={"ID":"45ad2ba5-16bf-4d2f-b91a-a488686a526b","Type":"ContainerStarted","Data":"9af3b22a63b3b2756cc0135afa10bdf85fc6ff235b30b592efb6841b35eaaf8c"}
Sep 10 22:31:40.320156 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:31:40.320133    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-d07dfb-7tcm8"
Sep 10 22:31:40.468164 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:31:40.468086    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/45ad2ba5-16bf-4d2f-b91a-a488686a526b-maas-api-service-ca\") pod \"45ad2ba5-16bf-4d2f-b91a-a488686a526b\" (UID: \"45ad2ba5-16bf-4d2f-b91a-a488686a526b\") "
Sep 10 22:31:40.468434 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:31:40.468411    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45ad2ba5-16bf-4d2f-b91a-a488686a526b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "45ad2ba5-16bf-4d2f-b91a-a488686a526b" (UID: "45ad2ba5-16bf-4d2f-b91a-a488686a526b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 22:31:40.568918 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:31:40.568891    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/45ad2ba5-16bf-4d2f-b91a-a488686a526b-maas-api-service-ca\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:31:41.203148 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:31:41.203112    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-d07dfb-7tcm8" event={"ID":"45ad2ba5-16bf-4d2f-b91a-a488686a526b","Type":"ContainerDied","Data":"9af3b22a63b3b2756cc0135afa10bdf85fc6ff235b30b592efb6841b35eaaf8c"}
Sep 10 22:31:41.203307 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:31:41.203162    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9af3b22a63b3b2756cc0135afa10bdf85fc6ff235b30b592efb6841b35eaaf8c"
Sep 10 22:31:41.203307 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:31:41.203126    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-d07dfb-7tcm8"
Sep 10 22:31:42.596876 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:31:42.596776    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-79434a-openshift-default-8f46dd48-r6zgk"
Sep 10 22:31:42.789358 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:31:42.789322    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f9549c1-10dc-4700-804f-82d6b1241447-istiod-ca-cert\") pod \"0f9549c1-10dc-4700-804f-82d6b1241447\" (UID: \"0f9549c1-10dc-4700-804f-82d6b1241447\") "
Sep 10 22:31:42.789547 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:31:42.789398    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f9549c1-10dc-4700-804f-82d6b1241447-workload-certs\") pod \"0f9549c1-10dc-4700-804f-82d6b1241447\" (UID: \"0f9549c1-10dc-4700-804f-82d6b1241447\") "
Sep 10 22:31:42.789547 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:31:42.789424    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-62p6p\" (UniqueName: \"kubernetes.io/projected/0f9549c1-10dc-4700-804f-82d6b1241447-kube-api-access-62p6p\") pod \"0f9549c1-10dc-4700-804f-82d6b1241447\" (UID: \"0f9549c1-10dc-4700-804f-82d6b1241447\") "
Sep 10 22:31:42.789547 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:31:42.789461    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f9549c1-10dc-4700-804f-82d6b1241447-istio-data\") pod \"0f9549c1-10dc-4700-804f-82d6b1241447\" (UID: \"0f9549c1-10dc-4700-804f-82d6b1241447\") "
Sep 10 22:31:42.789547 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:31:42.789497    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f9549c1-10dc-4700-804f-82d6b1241447-workload-socket\") pod \"0f9549c1-10dc-4700-804f-82d6b1241447\" (UID: \"0f9549c1-10dc-4700-804f-82d6b1241447\") "
Sep 10 22:31:42.789767 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:31:42.789547    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f9549c1-10dc-4700-804f-82d6b1241447-istio-podinfo\") pod \"0f9549c1-10dc-4700-804f-82d6b1241447\" (UID: \"0f9549c1-10dc-4700-804f-82d6b1241447\") "
Sep 10 22:31:42.789767 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:31:42.789570    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f9549c1-10dc-4700-804f-82d6b1241447-istio-token\") pod \"0f9549c1-10dc-4700-804f-82d6b1241447\" (UID: \"0f9549c1-10dc-4700-804f-82d6b1241447\") "
Sep 10 22:31:42.789767 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:31:42.789599    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f9549c1-10dc-4700-804f-82d6b1241447-istio-envoy\") pod \"0f9549c1-10dc-4700-804f-82d6b1241447\" (UID: \"0f9549c1-10dc-4700-804f-82d6b1241447\") "
Sep 10 22:31:42.789767 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:31:42.789660    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f9549c1-10dc-4700-804f-82d6b1241447-credential-socket\") pod \"0f9549c1-10dc-4700-804f-82d6b1241447\" (UID: \"0f9549c1-10dc-4700-804f-82d6b1241447\") "
Sep 10 22:31:42.789767 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:31:42.789729    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f9549c1-10dc-4700-804f-82d6b1241447-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0f9549c1-10dc-4700-804f-82d6b1241447" (UID: "0f9549c1-10dc-4700-804f-82d6b1241447"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 22:31:42.790069 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:31:42.789802    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f9549c1-10dc-4700-804f-82d6b1241447-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0f9549c1-10dc-4700-804f-82d6b1241447" (UID: "0f9549c1-10dc-4700-804f-82d6b1241447"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 22:31:42.790069 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:31:42.789815    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f9549c1-10dc-4700-804f-82d6b1241447-istio-data" (OuterVolumeSpecName: "istio-data") pod "0f9549c1-10dc-4700-804f-82d6b1241447" (UID: "0f9549c1-10dc-4700-804f-82d6b1241447"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 22:31:42.790069 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:31:42.789877    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f9549c1-10dc-4700-804f-82d6b1241447-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0f9549c1-10dc-4700-804f-82d6b1241447" (UID: "0f9549c1-10dc-4700-804f-82d6b1241447"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 22:31:42.790069 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:31:42.790010    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f9549c1-10dc-4700-804f-82d6b1241447-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0f9549c1-10dc-4700-804f-82d6b1241447" (UID: "0f9549c1-10dc-4700-804f-82d6b1241447"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 22:31:42.790265 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:31:42.790090    2582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f9549c1-10dc-4700-804f-82d6b1241447-istio-data\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:31:42.790265 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:31:42.790114    2582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f9549c1-10dc-4700-804f-82d6b1241447-workload-socket\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:31:42.790265 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:31:42.790130    2582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f9549c1-10dc-4700-804f-82d6b1241447-credential-socket\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:31:42.790265 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:31:42.790145    2582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f9549c1-10dc-4700-804f-82d6b1241447-istiod-ca-cert\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:31:42.790265 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:31:42.790158    2582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f9549c1-10dc-4700-804f-82d6b1241447-workload-certs\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:31:42.791833 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:31:42.791797    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f9549c1-10dc-4700-804f-82d6b1241447-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0f9549c1-10dc-4700-804f-82d6b1241447" (UID: "0f9549c1-10dc-4700-804f-82d6b1241447"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 22:31:42.791833 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:31:42.791810    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f9549c1-10dc-4700-804f-82d6b1241447-kube-api-access-62p6p" (OuterVolumeSpecName: "kube-api-access-62p6p") pod "0f9549c1-10dc-4700-804f-82d6b1241447" (UID: "0f9549c1-10dc-4700-804f-82d6b1241447"). InnerVolumeSpecName "kube-api-access-62p6p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 22:31:42.792118 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:31:42.792096    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f9549c1-10dc-4700-804f-82d6b1241447-istio-token" (OuterVolumeSpecName: "istio-token") pod "0f9549c1-10dc-4700-804f-82d6b1241447" (UID: "0f9549c1-10dc-4700-804f-82d6b1241447"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 22:31:42.792192 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:31:42.792107    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0f9549c1-10dc-4700-804f-82d6b1241447-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0f9549c1-10dc-4700-804f-82d6b1241447" (UID: "0f9549c1-10dc-4700-804f-82d6b1241447"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 22:31:42.891513 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:31:42.891455    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-62p6p\" (UniqueName: \"kubernetes.io/projected/0f9549c1-10dc-4700-804f-82d6b1241447-kube-api-access-62p6p\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:31:42.891513 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:31:42.891478    2582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f9549c1-10dc-4700-804f-82d6b1241447-istio-podinfo\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:31:42.891513 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:31:42.891488    2582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f9549c1-10dc-4700-804f-82d6b1241447-istio-token\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:31:42.891513 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:31:42.891497    2582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f9549c1-10dc-4700-804f-82d6b1241447-istio-envoy\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:31:43.214640 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:31:43.214608    2582 generic.go:356] "Generic (PLEG): container finished" podID="0f9549c1-10dc-4700-804f-82d6b1241447" containerID="541f181436b6e3889bfbee0bd2c0536345eabff04e57049039941ce0f774dbce" exitCode=0
Sep 10 22:31:43.214835 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:31:43.214655    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-79434a-openshift-default-8f46dd48-r6zgk"
Sep 10 22:31:43.214835 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:31:43.214689    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-79434a-openshift-default-8f46dd48-r6zgk" event={"ID":"0f9549c1-10dc-4700-804f-82d6b1241447","Type":"ContainerDied","Data":"541f181436b6e3889bfbee0bd2c0536345eabff04e57049039941ce0f774dbce"}
Sep 10 22:31:43.214835 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:31:43.214759    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-79434a-openshift-default-8f46dd48-r6zgk" event={"ID":"0f9549c1-10dc-4700-804f-82d6b1241447","Type":"ContainerDied","Data":"6eec2aac898bc4363734147008359a6e6e3892605529ae054f1f57a9fa9f87a3"}
Sep 10 22:31:43.214835 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:31:43.214788    2582 scope.go:117] "RemoveContainer" containerID="541f181436b6e3889bfbee0bd2c0536345eabff04e57049039941ce0f774dbce"
Sep 10 22:31:43.223936 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:31:43.223916    2582 scope.go:117] "RemoveContainer" containerID="17371bb521b421da173fb53058dd7f51bf314045f2d05b446d18f8ef9791ae72"
Sep 10 22:31:43.231914 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:31:43.231898    2582 scope.go:117] "RemoveContainer" containerID="541f181436b6e3889bfbee0bd2c0536345eabff04e57049039941ce0f774dbce"
Sep 10 22:31:43.232160 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:31:43.232141    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"541f181436b6e3889bfbee0bd2c0536345eabff04e57049039941ce0f774dbce\": container with ID starting with 541f181436b6e3889bfbee0bd2c0536345eabff04e57049039941ce0f774dbce not found: ID does not exist" containerID="541f181436b6e3889bfbee0bd2c0536345eabff04e57049039941ce0f774dbce"
Sep 10 22:31:43.232207 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:31:43.232166    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"541f181436b6e3889bfbee0bd2c0536345eabff04e57049039941ce0f774dbce"} err="failed to get container status \"541f181436b6e3889bfbee0bd2c0536345eabff04e57049039941ce0f774dbce\": rpc error: code = NotFound desc = could not find container \"541f181436b6e3889bfbee0bd2c0536345eabff04e57049039941ce0f774dbce\": container with ID starting with 541f181436b6e3889bfbee0bd2c0536345eabff04e57049039941ce0f774dbce not found: ID does not exist"
Sep 10 22:31:43.232207 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:31:43.232180    2582 scope.go:117] "RemoveContainer" containerID="17371bb521b421da173fb53058dd7f51bf314045f2d05b446d18f8ef9791ae72"
Sep 10 22:31:43.232451 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:31:43.232432    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17371bb521b421da173fb53058dd7f51bf314045f2d05b446d18f8ef9791ae72\": container with ID starting with 17371bb521b421da173fb53058dd7f51bf314045f2d05b446d18f8ef9791ae72 not found: ID does not exist" containerID="17371bb521b421da173fb53058dd7f51bf314045f2d05b446d18f8ef9791ae72"
Sep 10 22:31:43.232504 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:31:43.232456    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17371bb521b421da173fb53058dd7f51bf314045f2d05b446d18f8ef9791ae72"} err="failed to get container status \"17371bb521b421da173fb53058dd7f51bf314045f2d05b446d18f8ef9791ae72\": rpc error: code = NotFound desc = could not find container \"17371bb521b421da173fb53058dd7f51bf314045f2d05b446d18f8ef9791ae72\": container with ID starting with 17371bb521b421da173fb53058dd7f51bf314045f2d05b446d18f8ef9791ae72 not found: ID does not exist"
Sep 10 22:31:43.248928 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:31:43.248900    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-79434a-openshift-default-8f46dd48-r6zgk"]
Sep 10 22:31:43.257294 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:31:43.257269    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-79434a-openshift-default-8f46dd48-r6zgk"]
Sep 10 22:31:43.497597 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:31:43.497521    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0f9549c1-10dc-4700-804f-82d6b1241447" path="/var/lib/kubelet/pods/0f9549c1-10dc-4700-804f-82d6b1241447/volumes"
Sep 10 22:31:48.323265 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:31:48.323226    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-d07dfb-5bf7b6b89c-8psmq"]
Sep 10 22:31:48.323833 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:31:48.323519    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-d07dfb-5bf7b6b89c-8psmq" podUID="0ae76787-4199-472e-8858-ab3c1d35724d" containerName="payload-processing" containerID="cri-o://0718e2c82b0185c2db907febc8d092f7e46f61a6d34e4271983a9c5aa75dd4a8" gracePeriod=30
Sep 10 22:31:48.511543 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:31:48.511485    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-d07dfb-7tcm8"]
Sep 10 22:31:48.514807 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:31:48.514775    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-d07dfb-7tcm8"]
Sep 10 22:31:48.573709 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:31:48.573660    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-d07dfb-5bf7b6b89c-8psmq"
Sep 10 22:31:48.633110 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:31:48.633074    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ae76787-4199-472e-8858-ab3c1d35724d-plugins-config-volume\") pod \"0ae76787-4199-472e-8858-ab3c1d35724d\" (UID: \"0ae76787-4199-472e-8858-ab3c1d35724d\") "
Sep 10 22:31:48.633278 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:31:48.633152    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lcz8p\" (UniqueName: \"kubernetes.io/projected/0ae76787-4199-472e-8858-ab3c1d35724d-kube-api-access-lcz8p\") pod \"0ae76787-4199-472e-8858-ab3c1d35724d\" (UID: \"0ae76787-4199-472e-8858-ab3c1d35724d\") "
Sep 10 22:31:48.633486 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:31:48.633460    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ae76787-4199-472e-8858-ab3c1d35724d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0ae76787-4199-472e-8858-ab3c1d35724d" (UID: "0ae76787-4199-472e-8858-ab3c1d35724d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 22:31:48.635589 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:31:48.635558    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ae76787-4199-472e-8858-ab3c1d35724d-kube-api-access-lcz8p" (OuterVolumeSpecName: "kube-api-access-lcz8p") pod "0ae76787-4199-472e-8858-ab3c1d35724d" (UID: "0ae76787-4199-472e-8858-ab3c1d35724d"). InnerVolumeSpecName "kube-api-access-lcz8p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 22:31:48.734319 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:31:48.734292    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ae76787-4199-472e-8858-ab3c1d35724d-plugins-config-volume\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:31:48.734319 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:31:48.734317    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lcz8p\" (UniqueName: \"kubernetes.io/projected/0ae76787-4199-472e-8858-ab3c1d35724d-kube-api-access-lcz8p\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:31:49.256605 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:31:49.256569    2582 generic.go:356] "Generic (PLEG): container finished" podID="0ae76787-4199-472e-8858-ab3c1d35724d" containerID="0718e2c82b0185c2db907febc8d092f7e46f61a6d34e4271983a9c5aa75dd4a8" exitCode=0
Sep 10 22:31:49.256786 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:31:49.256612    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-d07dfb-5bf7b6b89c-8psmq"
Sep 10 22:31:49.256786 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:31:49.256619    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-d07dfb-5bf7b6b89c-8psmq" event={"ID":"0ae76787-4199-472e-8858-ab3c1d35724d","Type":"ContainerDied","Data":"0718e2c82b0185c2db907febc8d092f7e46f61a6d34e4271983a9c5aa75dd4a8"}
Sep 10 22:31:49.256786 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:31:49.256671    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-d07dfb-5bf7b6b89c-8psmq" event={"ID":"0ae76787-4199-472e-8858-ab3c1d35724d","Type":"ContainerDied","Data":"5c9d6f8f3f00a2ae88d7e7f94512ffc44d59480ab1808a61da904a3dcbb7eae0"}
Sep 10 22:31:49.256786 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:31:49.256699    2582 scope.go:117] "RemoveContainer" containerID="0718e2c82b0185c2db907febc8d092f7e46f61a6d34e4271983a9c5aa75dd4a8"
Sep 10 22:31:49.265700 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:31:49.265683    2582 scope.go:117] "RemoveContainer" containerID="0718e2c82b0185c2db907febc8d092f7e46f61a6d34e4271983a9c5aa75dd4a8"
Sep 10 22:31:49.265978 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:31:49.265961    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0718e2c82b0185c2db907febc8d092f7e46f61a6d34e4271983a9c5aa75dd4a8\": container with ID starting with 0718e2c82b0185c2db907febc8d092f7e46f61a6d34e4271983a9c5aa75dd4a8 not found: ID does not exist" containerID="0718e2c82b0185c2db907febc8d092f7e46f61a6d34e4271983a9c5aa75dd4a8"
Sep 10 22:31:49.266034 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:31:49.265985    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0718e2c82b0185c2db907febc8d092f7e46f61a6d34e4271983a9c5aa75dd4a8"} err="failed to get container status \"0718e2c82b0185c2db907febc8d092f7e46f61a6d34e4271983a9c5aa75dd4a8\": rpc error: code = NotFound desc = could not find container \"0718e2c82b0185c2db907febc8d092f7e46f61a6d34e4271983a9c5aa75dd4a8\": container with ID starting with 0718e2c82b0185c2db907febc8d092f7e46f61a6d34e4271983a9c5aa75dd4a8 not found: ID does not exist"
Sep 10 22:31:49.276917 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:31:49.276891    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-d07dfb-5bf7b6b89c-8psmq"]
Sep 10 22:31:49.284136 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:31:49.284115    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-d07dfb-5bf7b6b89c-8psmq"]
Sep 10 22:31:49.499170 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:31:49.499144    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0ae76787-4199-472e-8858-ab3c1d35724d" path="/var/lib/kubelet/pods/0ae76787-4199-472e-8858-ab3c1d35724d/volumes"
Sep 10 22:31:49.499478 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:31:49.499443    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="45ad2ba5-16bf-4d2f-b91a-a488686a526b" path="/var/lib/kubelet/pods/45ad2ba5-16bf-4d2f-b91a-a488686a526b/volumes"
Sep 10 22:31:55.111272 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:31:55.111240    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-d07dfb-openshift-default-6b77565746-fcc6m"]
Sep 10 22:31:55.111726 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:31:55.111506    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-d07dfb-openshift-default-6b77565746-fcc6m" podUID="de372118-61dc-45ee-b7db-ae68c945f4bc" containerName="istio-proxy" containerID="cri-o://5e7d852998864b7edc92d018366c6b7ad51789086bf8d41ee71aadc41dff14fd" gracePeriod=30
Sep 10 22:31:55.465864 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:31:55.465827    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ec6d47-zk8m6"]
Sep 10 22:31:55.466165 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:31:55.466152    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ae76787-4199-472e-8858-ab3c1d35724d" containerName="payload-processing"
Sep 10 22:31:55.466210 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:31:55.466167    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ae76787-4199-472e-8858-ab3c1d35724d" containerName="payload-processing"
Sep 10 22:31:55.466210 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:31:55.466180    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f9549c1-10dc-4700-804f-82d6b1241447" containerName="istio-proxy"
Sep 10 22:31:55.466210 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:31:55.466186    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f9549c1-10dc-4700-804f-82d6b1241447" containerName="istio-proxy"
Sep 10 22:31:55.466210 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:31:55.466197    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f9549c1-10dc-4700-804f-82d6b1241447" containerName="istio-proxy"
Sep 10 22:31:55.466210 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:31:55.466202    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f9549c1-10dc-4700-804f-82d6b1241447" containerName="istio-proxy"
Sep 10 22:31:55.466371 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:31:55.466215    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45ad2ba5-16bf-4d2f-b91a-a488686a526b" containerName="revoke-keys"
Sep 10 22:31:55.466371 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:31:55.466220    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="45ad2ba5-16bf-4d2f-b91a-a488686a526b" containerName="revoke-keys"
Sep 10 22:31:55.466371 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:31:55.466271    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="45ad2ba5-16bf-4d2f-b91a-a488686a526b" containerName="revoke-keys"
Sep 10 22:31:55.466371 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:31:55.466281    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f9549c1-10dc-4700-804f-82d6b1241447" containerName="istio-proxy"
Sep 10 22:31:55.466371 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:31:55.466287    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="0ae76787-4199-472e-8858-ab3c1d35724d" containerName="payload-processing"
Sep 10 22:31:55.466371 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:31:55.466292    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f9549c1-10dc-4700-804f-82d6b1241447" containerName="istio-proxy"
Sep 10 22:31:55.469279 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:31:55.469263    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ec6d47-zk8m6"
Sep 10 22:31:55.471536 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:31:55.471512    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 22:31:55.471536 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:31:55.471525    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8hjrv\""
Sep 10 22:31:55.478012 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:31:55.477979    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ec6d47-zk8m6"]
Sep 10 22:31:55.486987 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:31:55.486959    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0e1a9ec-3818-4a93-9a4a-f20dca93cd32-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-ec6d47-zk8m6\" (UID: \"a0e1a9ec-3818-4a93-9a4a-f20dca93cd32\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ec6d47-zk8m6"
Sep 10 22:31:55.588334 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:31:55.588305    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0e1a9ec-3818-4a93-9a4a-f20dca93cd32-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-ec6d47-zk8m6\" (UID: \"a0e1a9ec-3818-4a93-9a4a-f20dca93cd32\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ec6d47-zk8m6"
Sep 10 22:31:55.588913 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:31:55.588894    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0e1a9ec-3818-4a93-9a4a-f20dca93cd32-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-ec6d47-zk8m6\" (UID: \"a0e1a9ec-3818-4a93-9a4a-f20dca93cd32\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ec6d47-zk8m6"
Sep 10 22:31:55.780920 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:31:55.780838    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ec6d47-zk8m6"
Sep 10 22:31:56.051516 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:31:56.051473    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ec6d47-zk8m6"]
Sep 10 22:31:56.053794 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:31:56.053768    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0e1a9ec_3818_4a93_9a4a_f20dca93cd32.slice/crio-596d669f6324440a204b1faa4ca8ff4adccb6c8fd5897518bb5a6e1f814bb57b WatchSource:0}: Error finding container 596d669f6324440a204b1faa4ca8ff4adccb6c8fd5897518bb5a6e1f814bb57b: Status 404 returned error can't find the container with id 596d669f6324440a204b1faa4ca8ff4adccb6c8fd5897518bb5a6e1f814bb57b
Sep 10 22:31:56.293396 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:31:56.293368    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ec6d47-zk8m6" event={"ID":"a0e1a9ec-3818-4a93-9a4a-f20dca93cd32","Type":"ContainerStarted","Data":"596d669f6324440a204b1faa4ca8ff4adccb6c8fd5897518bb5a6e1f814bb57b"}
Sep 10 22:31:57.299999 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:31:57.299966    2582 generic.go:356] "Generic (PLEG): container finished" podID="a0e1a9ec-3818-4a93-9a4a-f20dca93cd32" containerID="1848cf449aef81519b9f8d9996e0f91c40f8e9249f18bbe2299e169c62896d58" exitCode=0
Sep 10 22:31:57.300568 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:31:57.300056    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ec6d47-zk8m6" event={"ID":"a0e1a9ec-3818-4a93-9a4a-f20dca93cd32","Type":"ContainerDied","Data":"1848cf449aef81519b9f8d9996e0f91c40f8e9249f18bbe2299e169c62896d58"}
Sep 10 22:31:58.430767 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:31:58.430745    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ec6d47-zk8m6"
Sep 10 22:31:58.513017 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:31:58.512990    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0e1a9ec-3818-4a93-9a4a-f20dca93cd32-maas-api-service-ca\") pod \"a0e1a9ec-3818-4a93-9a4a-f20dca93cd32\" (UID: \"a0e1a9ec-3818-4a93-9a4a-f20dca93cd32\") "
Sep 10 22:31:58.513340 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:31:58.513317    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0e1a9ec-3818-4a93-9a4a-f20dca93cd32-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a0e1a9ec-3818-4a93-9a4a-f20dca93cd32" (UID: "a0e1a9ec-3818-4a93-9a4a-f20dca93cd32"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 22:31:58.613548 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:31:58.613487    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0e1a9ec-3818-4a93-9a4a-f20dca93cd32-maas-api-service-ca\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:31:59.311767 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:31:59.311729    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ec6d47-zk8m6" event={"ID":"a0e1a9ec-3818-4a93-9a4a-f20dca93cd32","Type":"ContainerDied","Data":"596d669f6324440a204b1faa4ca8ff4adccb6c8fd5897518bb5a6e1f814bb57b"}
Sep 10 22:31:59.311954 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:31:59.311798    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="596d669f6324440a204b1faa4ca8ff4adccb6c8fd5897518bb5a6e1f814bb57b"
Sep 10 22:31:59.311954 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:31:59.311737    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ec6d47-zk8m6"
Sep 10 22:32:00.318664 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:32:00.318636    2582 generic.go:356] "Generic (PLEG): container finished" podID="de372118-61dc-45ee-b7db-ae68c945f4bc" containerID="5e7d852998864b7edc92d018366c6b7ad51789086bf8d41ee71aadc41dff14fd" exitCode=0
Sep 10 22:32:00.319065 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:32:00.318709    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-d07dfb-openshift-default-6b77565746-fcc6m" event={"ID":"de372118-61dc-45ee-b7db-ae68c945f4bc","Type":"ContainerDied","Data":"5e7d852998864b7edc92d018366c6b7ad51789086bf8d41ee71aadc41dff14fd"}
Sep 10 22:32:00.319065 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:32:00.318775    2582 scope.go:117] "RemoveContainer" containerID="3c34f92956b034b46b4c0cf7fa6d3987dbb3a0524d809205dce916ca197fd271"
Sep 10 22:32:00.354916 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:32:00.354891    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-d07dfb-openshift-default-6b77565746-fcc6m"
Sep 10 22:32:00.428037 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:32:00.428011    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de372118-61dc-45ee-b7db-ae68c945f4bc-istio-podinfo\") pod \"de372118-61dc-45ee-b7db-ae68c945f4bc\" (UID: \"de372118-61dc-45ee-b7db-ae68c945f4bc\") "
Sep 10 22:32:00.428169 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:32:00.428050    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de372118-61dc-45ee-b7db-ae68c945f4bc-istio-token\") pod \"de372118-61dc-45ee-b7db-ae68c945f4bc\" (UID: \"de372118-61dc-45ee-b7db-ae68c945f4bc\") "
Sep 10 22:32:00.428169 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:32:00.428076    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de372118-61dc-45ee-b7db-ae68c945f4bc-workload-socket\") pod \"de372118-61dc-45ee-b7db-ae68c945f4bc\" (UID: \"de372118-61dc-45ee-b7db-ae68c945f4bc\") "
Sep 10 22:32:00.428169 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:32:00.428107    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de372118-61dc-45ee-b7db-ae68c945f4bc-workload-certs\") pod \"de372118-61dc-45ee-b7db-ae68c945f4bc\" (UID: \"de372118-61dc-45ee-b7db-ae68c945f4bc\") "
Sep 10 22:32:00.428169 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:32:00.428128    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de372118-61dc-45ee-b7db-ae68c945f4bc-istio-data\") pod \"de372118-61dc-45ee-b7db-ae68c945f4bc\" (UID: \"de372118-61dc-45ee-b7db-ae68c945f4bc\") "
Sep 10 22:32:00.428169 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:32:00.428147    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mf4r6\" (UniqueName: \"kubernetes.io/projected/de372118-61dc-45ee-b7db-ae68c945f4bc-kube-api-access-mf4r6\") pod \"de372118-61dc-45ee-b7db-ae68c945f4bc\" (UID: \"de372118-61dc-45ee-b7db-ae68c945f4bc\") "
Sep 10 22:32:00.428169 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:32:00.428165    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de372118-61dc-45ee-b7db-ae68c945f4bc-istiod-ca-cert\") pod \"de372118-61dc-45ee-b7db-ae68c945f4bc\" (UID: \"de372118-61dc-45ee-b7db-ae68c945f4bc\") "
Sep 10 22:32:00.428494 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:32:00.428186    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de372118-61dc-45ee-b7db-ae68c945f4bc-istio-envoy\") pod \"de372118-61dc-45ee-b7db-ae68c945f4bc\" (UID: \"de372118-61dc-45ee-b7db-ae68c945f4bc\") "
Sep 10 22:32:00.428494 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:32:00.428213    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de372118-61dc-45ee-b7db-ae68c945f4bc-credential-socket\") pod \"de372118-61dc-45ee-b7db-ae68c945f4bc\" (UID: \"de372118-61dc-45ee-b7db-ae68c945f4bc\") "
Sep 10 22:32:00.428494 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:32:00.428389    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de372118-61dc-45ee-b7db-ae68c945f4bc-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "de372118-61dc-45ee-b7db-ae68c945f4bc" (UID: "de372118-61dc-45ee-b7db-ae68c945f4bc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 22:32:00.428494 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:32:00.428415    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de372118-61dc-45ee-b7db-ae68c945f4bc-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "de372118-61dc-45ee-b7db-ae68c945f4bc" (UID: "de372118-61dc-45ee-b7db-ae68c945f4bc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 22:32:00.428494 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:32:00.428453    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de372118-61dc-45ee-b7db-ae68c945f4bc-istio-data" (OuterVolumeSpecName: "istio-data") pod "de372118-61dc-45ee-b7db-ae68c945f4bc" (UID: "de372118-61dc-45ee-b7db-ae68c945f4bc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 22:32:00.428757 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:32:00.428504    2582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de372118-61dc-45ee-b7db-ae68c945f4bc-workload-socket\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:32:00.428757 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:32:00.428522    2582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de372118-61dc-45ee-b7db-ae68c945f4bc-workload-certs\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:32:00.428757 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:32:00.428564    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de372118-61dc-45ee-b7db-ae68c945f4bc-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "de372118-61dc-45ee-b7db-ae68c945f4bc" (UID: "de372118-61dc-45ee-b7db-ae68c945f4bc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 22:32:00.428757 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:32:00.428607    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de372118-61dc-45ee-b7db-ae68c945f4bc-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "de372118-61dc-45ee-b7db-ae68c945f4bc" (UID: "de372118-61dc-45ee-b7db-ae68c945f4bc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 22:32:00.430250 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:32:00.430229    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/de372118-61dc-45ee-b7db-ae68c945f4bc-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "de372118-61dc-45ee-b7db-ae68c945f4bc" (UID: "de372118-61dc-45ee-b7db-ae68c945f4bc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 22:32:00.430415 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:32:00.430390    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de372118-61dc-45ee-b7db-ae68c945f4bc-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "de372118-61dc-45ee-b7db-ae68c945f4bc" (UID: "de372118-61dc-45ee-b7db-ae68c945f4bc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 22:32:00.430415 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:32:00.430402    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de372118-61dc-45ee-b7db-ae68c945f4bc-kube-api-access-mf4r6" (OuterVolumeSpecName: "kube-api-access-mf4r6") pod "de372118-61dc-45ee-b7db-ae68c945f4bc" (UID: "de372118-61dc-45ee-b7db-ae68c945f4bc"). InnerVolumeSpecName "kube-api-access-mf4r6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 22:32:00.430494 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:32:00.430469    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de372118-61dc-45ee-b7db-ae68c945f4bc-istio-token" (OuterVolumeSpecName: "istio-token") pod "de372118-61dc-45ee-b7db-ae68c945f4bc" (UID: "de372118-61dc-45ee-b7db-ae68c945f4bc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 22:32:00.529018 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:32:00.528998    2582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de372118-61dc-45ee-b7db-ae68c945f4bc-istio-data\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:32:00.529018 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:32:00.529017    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mf4r6\" (UniqueName: \"kubernetes.io/projected/de372118-61dc-45ee-b7db-ae68c945f4bc-kube-api-access-mf4r6\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:32:00.529123 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:32:00.529026    2582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de372118-61dc-45ee-b7db-ae68c945f4bc-istiod-ca-cert\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:32:00.529123 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:32:00.529035    2582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de372118-61dc-45ee-b7db-ae68c945f4bc-istio-envoy\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:32:00.529123 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:32:00.529043    2582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de372118-61dc-45ee-b7db-ae68c945f4bc-credential-socket\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:32:00.529123 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:32:00.529053    2582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de372118-61dc-45ee-b7db-ae68c945f4bc-istio-podinfo\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:32:00.529123 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:32:00.529061    2582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de372118-61dc-45ee-b7db-ae68c945f4bc-istio-token\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:32:01.325696 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:32:01.325666    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-d07dfb-openshift-default-6b77565746-fcc6m" event={"ID":"de372118-61dc-45ee-b7db-ae68c945f4bc","Type":"ContainerDied","Data":"ec21f9dce953617a37f1ed4e8783bee23374da4e1113d892d950e58474dfa66f"}
Sep 10 22:32:01.325696 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:32:01.325696    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-d07dfb-openshift-default-6b77565746-fcc6m"
Sep 10 22:32:01.326189 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:32:01.325725    2582 scope.go:117] "RemoveContainer" containerID="5e7d852998864b7edc92d018366c6b7ad51789086bf8d41ee71aadc41dff14fd"
Sep 10 22:32:01.349316 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:32:01.349289    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-d07dfb-openshift-default-6b77565746-fcc6m"]
Sep 10 22:32:01.357491 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:32:01.357471    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-d07dfb-openshift-default-6b77565746-fcc6m"]
Sep 10 22:32:01.498037 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:32:01.498010    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="de372118-61dc-45ee-b7db-ae68c945f4bc" path="/var/lib/kubelet/pods/de372118-61dc-45ee-b7db-ae68c945f4bc/volumes"
Sep 10 22:32:06.033006 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:32:06.032969    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ec6d47-29817990-nrp45"]
Sep 10 22:32:06.035808 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:32:06.035784    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ec6d47-29817990-nrp45"]
Sep 10 22:32:06.067436 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:32:06.067409    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ec6d47-684fb785b6-llwjf"]
Sep 10 22:32:06.067643 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:32:06.067614    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ec6d47-684fb785b6-llwjf" podUID="8269d673-185f-421c-b5bb-ca7d77226be6" containerName="payload-pre-processing" containerID="cri-o://552930ed95112188648b5d43832e95f56e61e37e4a0c051b2561464ff77211fe" gracePeriod=30
Sep 10 22:32:06.265141 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:32:06.265111    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ec6d47-zk8m6"]
Sep 10 22:32:06.268415 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:32:06.268389    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ec6d47-zk8m6"]
Sep 10 22:32:06.313951 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:32:06.313930    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ec6d47-684fb785b6-llwjf"
Sep 10 22:32:06.353649 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:32:06.353615    2582 generic.go:356] "Generic (PLEG): container finished" podID="8269d673-185f-421c-b5bb-ca7d77226be6" containerID="552930ed95112188648b5d43832e95f56e61e37e4a0c051b2561464ff77211fe" exitCode=0
Sep 10 22:32:06.353757 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:32:06.353654    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ec6d47-684fb785b6-llwjf"
Sep 10 22:32:06.353757 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:32:06.353688    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ec6d47-684fb785b6-llwjf" event={"ID":"8269d673-185f-421c-b5bb-ca7d77226be6","Type":"ContainerDied","Data":"552930ed95112188648b5d43832e95f56e61e37e4a0c051b2561464ff77211fe"}
Sep 10 22:32:06.353757 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:32:06.353737    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ec6d47-684fb785b6-llwjf" event={"ID":"8269d673-185f-421c-b5bb-ca7d77226be6","Type":"ContainerDied","Data":"82a750da4a2fbb05d2fa7868f745382aceb8709247617841e635c9c01bc10a4e"}
Sep 10 22:32:06.353901 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:32:06.353759    2582 scope.go:117] "RemoveContainer" containerID="552930ed95112188648b5d43832e95f56e61e37e4a0c051b2561464ff77211fe"
Sep 10 22:32:06.362025 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:32:06.362005    2582 scope.go:117] "RemoveContainer" containerID="552930ed95112188648b5d43832e95f56e61e37e4a0c051b2561464ff77211fe"
Sep 10 22:32:06.362279 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:32:06.362260    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"552930ed95112188648b5d43832e95f56e61e37e4a0c051b2561464ff77211fe\": container with ID starting with 552930ed95112188648b5d43832e95f56e61e37e4a0c051b2561464ff77211fe not found: ID does not exist" containerID="552930ed95112188648b5d43832e95f56e61e37e4a0c051b2561464ff77211fe"
Sep 10 22:32:06.362337 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:32:06.362292    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"552930ed95112188648b5d43832e95f56e61e37e4a0c051b2561464ff77211fe"} err="failed to get container status \"552930ed95112188648b5d43832e95f56e61e37e4a0c051b2561464ff77211fe\": rpc error: code = NotFound desc = could not find container \"552930ed95112188648b5d43832e95f56e61e37e4a0c051b2561464ff77211fe\": container with ID starting with 552930ed95112188648b5d43832e95f56e61e37e4a0c051b2561464ff77211fe not found: ID does not exist"
Sep 10 22:32:06.367601 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:32:06.367583    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c7kb6\" (UniqueName: \"kubernetes.io/projected/8269d673-185f-421c-b5bb-ca7d77226be6-kube-api-access-c7kb6\") pod \"8269d673-185f-421c-b5bb-ca7d77226be6\" (UID: \"8269d673-185f-421c-b5bb-ca7d77226be6\") "
Sep 10 22:32:06.367670 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:32:06.367642    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8269d673-185f-421c-b5bb-ca7d77226be6-plugins-config-volume\") pod \"8269d673-185f-421c-b5bb-ca7d77226be6\" (UID: \"8269d673-185f-421c-b5bb-ca7d77226be6\") "
Sep 10 22:32:06.368001 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:32:06.367977    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8269d673-185f-421c-b5bb-ca7d77226be6-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8269d673-185f-421c-b5bb-ca7d77226be6" (UID: "8269d673-185f-421c-b5bb-ca7d77226be6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 22:32:06.369524 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:32:06.369500    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8269d673-185f-421c-b5bb-ca7d77226be6-kube-api-access-c7kb6" (OuterVolumeSpecName: "kube-api-access-c7kb6") pod "8269d673-185f-421c-b5bb-ca7d77226be6" (UID: "8269d673-185f-421c-b5bb-ca7d77226be6"). InnerVolumeSpecName "kube-api-access-c7kb6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 22:32:06.469181 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:32:06.469147    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c7kb6\" (UniqueName: \"kubernetes.io/projected/8269d673-185f-421c-b5bb-ca7d77226be6-kube-api-access-c7kb6\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:32:06.469181 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:32:06.469175    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8269d673-185f-421c-b5bb-ca7d77226be6-plugins-config-volume\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:32:06.676729 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:32:06.676697    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ec6d47-684fb785b6-llwjf"]
Sep 10 22:32:06.684716 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:32:06.684692    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ec6d47-684fb785b6-llwjf"]
Sep 10 22:32:07.497529 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:32:07.497496    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8269d673-185f-421c-b5bb-ca7d77226be6" path="/var/lib/kubelet/pods/8269d673-185f-421c-b5bb-ca7d77226be6/volumes"
Sep 10 22:32:07.497912 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:32:07.497802    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a0e1a9ec-3818-4a93-9a4a-f20dca93cd32" path="/var/lib/kubelet/pods/a0e1a9ec-3818-4a93-9a4a-f20dca93cd32/volumes"
Sep 10 22:32:07.498090 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:32:07.498076    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b5d85362-c5b9-4fca-84b4-78e231c8528d" path="/var/lib/kubelet/pods/b5d85362-c5b9-4fca-84b4-78e231c8528d/volumes"
Sep 10 22:32:13.423428 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:32:13.423392    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-ec6d47-openshift-default-c4db8956d-h7bnw"]
Sep 10 22:32:13.424057 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:32:13.423614    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-ec6d47-openshift-default-c4db8956d-h7bnw" podUID="aeeca89b-577d-4dd5-990f-cf0706a478e7" containerName="istio-proxy" containerID="cri-o://b0f64faaa8af94d43a965943c4e76e9eb074bea1454151c00249c8ec4eac8bd2" gracePeriod=30
Sep 10 22:32:18.582542 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:32:18.582509    2582 patch_prober.go:28] interesting pod/e2e-shared-b-w3-ec6d47-openshift-default-c4db8956d-h7bnw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Sep 10 22:32:18.582915 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:32:18.582576    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-ec6d47-openshift-default-c4db8956d-h7bnw" podUID="aeeca89b-577d-4dd5-990f-cf0706a478e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Sep 10 22:32:18.688691 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:32:18.688629    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-ec6d47-openshift-default-c4db8956d-h7bnw"
Sep 10 22:32:18.768342 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:32:18.768319    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m7ftf\" (UniqueName: \"kubernetes.io/projected/aeeca89b-577d-4dd5-990f-cf0706a478e7-kube-api-access-m7ftf\") pod \"aeeca89b-577d-4dd5-990f-cf0706a478e7\" (UID: \"aeeca89b-577d-4dd5-990f-cf0706a478e7\") "
Sep 10 22:32:18.768474 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:32:18.768352    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aeeca89b-577d-4dd5-990f-cf0706a478e7-workload-certs\") pod \"aeeca89b-577d-4dd5-990f-cf0706a478e7\" (UID: \"aeeca89b-577d-4dd5-990f-cf0706a478e7\") "
Sep 10 22:32:18.768474 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:32:18.768368    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aeeca89b-577d-4dd5-990f-cf0706a478e7-istiod-ca-cert\") pod \"aeeca89b-577d-4dd5-990f-cf0706a478e7\" (UID: \"aeeca89b-577d-4dd5-990f-cf0706a478e7\") "
Sep 10 22:32:18.768474 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:32:18.768389    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aeeca89b-577d-4dd5-990f-cf0706a478e7-istio-data\") pod \"aeeca89b-577d-4dd5-990f-cf0706a478e7\" (UID: \"aeeca89b-577d-4dd5-990f-cf0706a478e7\") "
Sep 10 22:32:18.768474 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:32:18.768414    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aeeca89b-577d-4dd5-990f-cf0706a478e7-istio-envoy\") pod \"aeeca89b-577d-4dd5-990f-cf0706a478e7\" (UID: \"aeeca89b-577d-4dd5-990f-cf0706a478e7\") "
Sep 10 22:32:18.768474 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:32:18.768438    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aeeca89b-577d-4dd5-990f-cf0706a478e7-istio-podinfo\") pod \"aeeca89b-577d-4dd5-990f-cf0706a478e7\" (UID: \"aeeca89b-577d-4dd5-990f-cf0706a478e7\") "
Sep 10 22:32:18.768474 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:32:18.768470    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aeeca89b-577d-4dd5-990f-cf0706a478e7-workload-socket\") pod \"aeeca89b-577d-4dd5-990f-cf0706a478e7\" (UID: \"aeeca89b-577d-4dd5-990f-cf0706a478e7\") "
Sep 10 22:32:18.768780 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:32:18.768571    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aeeca89b-577d-4dd5-990f-cf0706a478e7-credential-socket\") pod \"aeeca89b-577d-4dd5-990f-cf0706a478e7\" (UID: \"aeeca89b-577d-4dd5-990f-cf0706a478e7\") "
Sep 10 22:32:18.768780 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:32:18.768596    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aeeca89b-577d-4dd5-990f-cf0706a478e7-istio-token\") pod \"aeeca89b-577d-4dd5-990f-cf0706a478e7\" (UID: \"aeeca89b-577d-4dd5-990f-cf0706a478e7\") "
Sep 10 22:32:18.768780 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:32:18.768665    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aeeca89b-577d-4dd5-990f-cf0706a478e7-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "aeeca89b-577d-4dd5-990f-cf0706a478e7" (UID: "aeeca89b-577d-4dd5-990f-cf0706a478e7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 22:32:18.768978 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:32:18.768801    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aeeca89b-577d-4dd5-990f-cf0706a478e7-istio-data" (OuterVolumeSpecName: "istio-data") pod "aeeca89b-577d-4dd5-990f-cf0706a478e7" (UID: "aeeca89b-577d-4dd5-990f-cf0706a478e7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 22:32:18.768978 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:32:18.768840    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aeeca89b-577d-4dd5-990f-cf0706a478e7-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "aeeca89b-577d-4dd5-990f-cf0706a478e7" (UID: "aeeca89b-577d-4dd5-990f-cf0706a478e7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 22:32:18.768978 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:32:18.768924    2582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aeeca89b-577d-4dd5-990f-cf0706a478e7-workload-certs\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:32:18.768978 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:32:18.768945    2582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aeeca89b-577d-4dd5-990f-cf0706a478e7-istiod-ca-cert\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:32:18.768978 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:32:18.768958    2582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aeeca89b-577d-4dd5-990f-cf0706a478e7-istio-data\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:32:18.769243 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:32:18.769032    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aeeca89b-577d-4dd5-990f-cf0706a478e7-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "aeeca89b-577d-4dd5-990f-cf0706a478e7" (UID: "aeeca89b-577d-4dd5-990f-cf0706a478e7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 22:32:18.769243 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:32:18.769143    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aeeca89b-577d-4dd5-990f-cf0706a478e7-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "aeeca89b-577d-4dd5-990f-cf0706a478e7" (UID: "aeeca89b-577d-4dd5-990f-cf0706a478e7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 22:32:18.770721 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:32:18.770690    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aeeca89b-577d-4dd5-990f-cf0706a478e7-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "aeeca89b-577d-4dd5-990f-cf0706a478e7" (UID: "aeeca89b-577d-4dd5-990f-cf0706a478e7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 22:32:18.770962 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:32:18.770944    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aeeca89b-577d-4dd5-990f-cf0706a478e7-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "aeeca89b-577d-4dd5-990f-cf0706a478e7" (UID: "aeeca89b-577d-4dd5-990f-cf0706a478e7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 22:32:18.771163 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:32:18.771146    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aeeca89b-577d-4dd5-990f-cf0706a478e7-istio-token" (OuterVolumeSpecName: "istio-token") pod "aeeca89b-577d-4dd5-990f-cf0706a478e7" (UID: "aeeca89b-577d-4dd5-990f-cf0706a478e7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 22:32:18.771233 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:32:18.771164    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aeeca89b-577d-4dd5-990f-cf0706a478e7-kube-api-access-m7ftf" (OuterVolumeSpecName: "kube-api-access-m7ftf") pod "aeeca89b-577d-4dd5-990f-cf0706a478e7" (UID: "aeeca89b-577d-4dd5-990f-cf0706a478e7"). InnerVolumeSpecName "kube-api-access-m7ftf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 22:32:18.869578 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:32:18.869555    2582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aeeca89b-577d-4dd5-990f-cf0706a478e7-istio-envoy\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:32:18.869578 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:32:18.869576    2582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aeeca89b-577d-4dd5-990f-cf0706a478e7-istio-podinfo\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:32:18.869709 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:32:18.869585    2582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aeeca89b-577d-4dd5-990f-cf0706a478e7-workload-socket\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:32:18.869709 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:32:18.869594    2582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aeeca89b-577d-4dd5-990f-cf0706a478e7-credential-socket\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:32:18.869709 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:32:18.869603    2582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aeeca89b-577d-4dd5-990f-cf0706a478e7-istio-token\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:32:18.869709 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:32:18.869612    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7ftf\" (UniqueName: \"kubernetes.io/projected/aeeca89b-577d-4dd5-990f-cf0706a478e7-kube-api-access-m7ftf\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:32:19.425318 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:32:19.425285    2582 generic.go:356] "Generic (PLEG): container finished" podID="aeeca89b-577d-4dd5-990f-cf0706a478e7" containerID="b0f64faaa8af94d43a965943c4e76e9eb074bea1454151c00249c8ec4eac8bd2" exitCode=0
Sep 10 22:32:19.425502 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:32:19.425325    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-ec6d47-openshift-default-c4db8956d-h7bnw"
Sep 10 22:32:19.425502 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:32:19.425367    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-ec6d47-openshift-default-c4db8956d-h7bnw" event={"ID":"aeeca89b-577d-4dd5-990f-cf0706a478e7","Type":"ContainerDied","Data":"b0f64faaa8af94d43a965943c4e76e9eb074bea1454151c00249c8ec4eac8bd2"}
Sep 10 22:32:19.425502 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:32:19.425424    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-ec6d47-openshift-default-c4db8956d-h7bnw" event={"ID":"aeeca89b-577d-4dd5-990f-cf0706a478e7","Type":"ContainerDied","Data":"2be0514d432c3a5fdadda359c3c91464fcd89d8d481e17b02c9c25787a4b284d"}
Sep 10 22:32:19.425502 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:32:19.425453    2582 scope.go:117] "RemoveContainer" containerID="b0f64faaa8af94d43a965943c4e76e9eb074bea1454151c00249c8ec4eac8bd2"
Sep 10 22:32:19.434926 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:32:19.434908    2582 scope.go:117] "RemoveContainer" containerID="226a5112c7d11f88754a06fa72a7e3f8d259a7a9a675f80157904c106b40a68b"
Sep 10 22:32:19.442995 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:32:19.442979    2582 scope.go:117] "RemoveContainer" containerID="b0f64faaa8af94d43a965943c4e76e9eb074bea1454151c00249c8ec4eac8bd2"
Sep 10 22:32:19.443261 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:32:19.443240    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0f64faaa8af94d43a965943c4e76e9eb074bea1454151c00249c8ec4eac8bd2\": container with ID starting with b0f64faaa8af94d43a965943c4e76e9eb074bea1454151c00249c8ec4eac8bd2 not found: ID does not exist" containerID="b0f64faaa8af94d43a965943c4e76e9eb074bea1454151c00249c8ec4eac8bd2"
Sep 10 22:32:19.443333 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:32:19.443271    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0f64faaa8af94d43a965943c4e76e9eb074bea1454151c00249c8ec4eac8bd2"} err="failed to get container status \"b0f64faaa8af94d43a965943c4e76e9eb074bea1454151c00249c8ec4eac8bd2\": rpc error: code = NotFound desc = could not find container \"b0f64faaa8af94d43a965943c4e76e9eb074bea1454151c00249c8ec4eac8bd2\": container with ID starting with b0f64faaa8af94d43a965943c4e76e9eb074bea1454151c00249c8ec4eac8bd2 not found: ID does not exist"
Sep 10 22:32:19.443333 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:32:19.443296    2582 scope.go:117] "RemoveContainer" containerID="226a5112c7d11f88754a06fa72a7e3f8d259a7a9a675f80157904c106b40a68b"
Sep 10 22:32:19.443540 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:32:19.443523    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"226a5112c7d11f88754a06fa72a7e3f8d259a7a9a675f80157904c106b40a68b\": container with ID starting with 226a5112c7d11f88754a06fa72a7e3f8d259a7a9a675f80157904c106b40a68b not found: ID does not exist" containerID="226a5112c7d11f88754a06fa72a7e3f8d259a7a9a675f80157904c106b40a68b"
Sep 10 22:32:19.443581 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:32:19.443546    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"226a5112c7d11f88754a06fa72a7e3f8d259a7a9a675f80157904c106b40a68b"} err="failed to get container status \"226a5112c7d11f88754a06fa72a7e3f8d259a7a9a675f80157904c106b40a68b\": rpc error: code = NotFound desc = could not find container \"226a5112c7d11f88754a06fa72a7e3f8d259a7a9a675f80157904c106b40a68b\": container with ID starting with 226a5112c7d11f88754a06fa72a7e3f8d259a7a9a675f80157904c106b40a68b not found: ID does not exist"
Sep 10 22:32:19.449909 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:32:19.449889    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-ec6d47-openshift-default-c4db8956d-h7bnw"]
Sep 10 22:32:19.460154 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:32:19.460133    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-ec6d47-openshift-default-c4db8956d-h7bnw"]
Sep 10 22:32:19.498296 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:32:19.498275    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aeeca89b-577d-4dd5-990f-cf0706a478e7" path="/var/lib/kubelet/pods/aeeca89b-577d-4dd5-990f-cf0706a478e7/volumes"
Sep 10 22:32:19.827116 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:32:19.827043    2582 scope.go:117] "RemoveContainer" containerID="65846ba9a26559973e19f432a7542f4bf9f6ba76d062de0b23a296997c6588b4"
Sep 10 22:33:19.868758 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:33:19.868726    2582 scope.go:117] "RemoveContainer" containerID="4d2281781f17e3ed1ef1845906783312afd04605be6bdf559a71c70cca96353a"
Sep 10 22:33:19.876914 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:33:19.876893    2582 scope.go:117] "RemoveContainer" containerID="dbb3869315d03be99af693f8b9d55b12221bcaafc52b4972cebf9045f7c47def"
Sep 10 22:33:19.884415 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:33:19.884398    2582 scope.go:117] "RemoveContainer" containerID="e9e139b6ac11ddd62a7d177c09ad2800930cc69deb79aff54c367075629aff0f"
Sep 10 22:33:19.891336 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:33:19.891322    2582 scope.go:117] "RemoveContainer" containerID="3b73caebf662d7a931f61c9503957ba1f3bde86d7f8400fa94c9e5ba23c22bd7"
Sep 10 22:36:19.595070 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:36:19.595036    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-w6vlb_8517c9a7-382a-40fd-bc52-e6da90ebc584/console-operator/2.log"
Sep 10 22:36:19.600277 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:36:19.600252    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-w6vlb_8517c9a7-382a-40fd-bc52-e6da90ebc584/console-operator/2.log"
Sep 10 22:36:19.930751 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:36:19.930725    2582 scope.go:117] "RemoveContainer" containerID="1dc994243a44662aad1960e484814f99042ce77febb2e0f92f366e026f817c77"
Sep 10 22:36:19.941146 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:36:19.941125    2582 scope.go:117] "RemoveContainer" containerID="f440881c3576a59ae9d42b3377df9112e236875d576f531006b199f7553def85"
Sep 10 22:36:19.948510 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:36:19.948492    2582 scope.go:117] "RemoveContainer" containerID="403d010af83008b2522446eeb5dd09e2072e7c023c758673eb75670149466299"
Sep 10 22:36:19.955255 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:36:19.955234    2582 scope.go:117] "RemoveContainer" containerID="c7ead10f9f8ce114531a32dc6033a44a1d9922ca0efd0e6c0504603d75334a00"
Sep 10 22:36:19.961748 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:36:19.961728    2582 scope.go:117] "RemoveContainer" containerID="c7371960bd92cb885df4973495465e83617416c1a7d3a65b692aaa4184f444e6"
Sep 10 22:36:19.968287 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:36:19.968268    2582 scope.go:117] "RemoveContainer" containerID="925d30d80ec7826367d46e18732b4e9f03dc969a731e144580b5e2eedb2569f9"
Sep 10 22:36:52.228635 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:36:52.228597    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-b3d40012-kserve-59ffd46f87-g9ktf"]
Sep 10 22:36:52.229137 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:36:52.228905    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0e1a9ec-3818-4a93-9a4a-f20dca93cd32" containerName="revoke-keys"
Sep 10 22:36:52.229137 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:36:52.228917    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0e1a9ec-3818-4a93-9a4a-f20dca93cd32" containerName="revoke-keys"
Sep 10 22:36:52.229137 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:36:52.228925    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de372118-61dc-45ee-b7db-ae68c945f4bc" containerName="istio-proxy"
Sep 10 22:36:52.229137 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:36:52.228931    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="de372118-61dc-45ee-b7db-ae68c945f4bc" containerName="istio-proxy"
Sep 10 22:36:52.229137 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:36:52.228940    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8269d673-185f-421c-b5bb-ca7d77226be6" containerName="payload-pre-processing"
Sep 10 22:36:52.229137 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:36:52.228946    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="8269d673-185f-421c-b5bb-ca7d77226be6" containerName="payload-pre-processing"
Sep 10 22:36:52.229137 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:36:52.228956    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aeeca89b-577d-4dd5-990f-cf0706a478e7" containerName="istio-proxy"
Sep 10 22:36:52.229137 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:36:52.228961    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="aeeca89b-577d-4dd5-990f-cf0706a478e7" containerName="istio-proxy"
Sep 10 22:36:52.229137 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:36:52.229017    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="de372118-61dc-45ee-b7db-ae68c945f4bc" containerName="istio-proxy"
Sep 10 22:36:52.229137 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:36:52.229025    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="aeeca89b-577d-4dd5-990f-cf0706a478e7" containerName="istio-proxy"
Sep 10 22:36:52.229137 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:36:52.229031    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="8269d673-185f-421c-b5bb-ca7d77226be6" containerName="payload-pre-processing"
Sep 10 22:36:52.229137 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:36:52.229037    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="a0e1a9ec-3818-4a93-9a4a-f20dca93cd32" containerName="revoke-keys"
Sep 10 22:36:52.231889 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:36:52.231868    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-b3d40012-kserve-59ffd46f87-g9ktf"
Sep 10 22:36:52.240752 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:36:52.240731    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-b3d40012-kserve-self-signed-certs\""
Sep 10 22:36:52.260760 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:36:52.260736    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-b3d40012-kserve-59ffd46f87-g9ktf"]
Sep 10 22:36:52.323275 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:36:52.323247    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/043154d6-6bdf-44f2-a043-5855251fe7a6-tmp-dir\") pod \"e2e-same-id-a-b3d40012-kserve-59ffd46f87-g9ktf\" (UID: \"043154d6-6bdf-44f2-a043-5855251fe7a6\") " pod="llm/e2e-same-id-a-b3d40012-kserve-59ffd46f87-g9ktf"
Sep 10 22:36:52.323412 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:36:52.323294    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/043154d6-6bdf-44f2-a043-5855251fe7a6-tls-certs\") pod \"e2e-same-id-a-b3d40012-kserve-59ffd46f87-g9ktf\" (UID: \"043154d6-6bdf-44f2-a043-5855251fe7a6\") " pod="llm/e2e-same-id-a-b3d40012-kserve-59ffd46f87-g9ktf"
Sep 10 22:36:52.323412 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:36:52.323317    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/043154d6-6bdf-44f2-a043-5855251fe7a6-dshm\") pod \"e2e-same-id-a-b3d40012-kserve-59ffd46f87-g9ktf\" (UID: \"043154d6-6bdf-44f2-a043-5855251fe7a6\") " pod="llm/e2e-same-id-a-b3d40012-kserve-59ffd46f87-g9ktf"
Sep 10 22:36:52.323412 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:36:52.323366    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/043154d6-6bdf-44f2-a043-5855251fe7a6-home\") pod \"e2e-same-id-a-b3d40012-kserve-59ffd46f87-g9ktf\" (UID: \"043154d6-6bdf-44f2-a043-5855251fe7a6\") " pod="llm/e2e-same-id-a-b3d40012-kserve-59ffd46f87-g9ktf"
Sep 10 22:36:52.323412 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:36:52.323404    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbwbm\" (UniqueName: \"kubernetes.io/projected/043154d6-6bdf-44f2-a043-5855251fe7a6-kube-api-access-lbwbm\") pod \"e2e-same-id-a-b3d40012-kserve-59ffd46f87-g9ktf\" (UID: \"043154d6-6bdf-44f2-a043-5855251fe7a6\") " pod="llm/e2e-same-id-a-b3d40012-kserve-59ffd46f87-g9ktf"
Sep 10 22:36:52.323593 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:36:52.323510    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/043154d6-6bdf-44f2-a043-5855251fe7a6-model-cache\") pod \"e2e-same-id-a-b3d40012-kserve-59ffd46f87-g9ktf\" (UID: \"043154d6-6bdf-44f2-a043-5855251fe7a6\") " pod="llm/e2e-same-id-a-b3d40012-kserve-59ffd46f87-g9ktf"
Sep 10 22:36:52.424739 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:36:52.424708    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/043154d6-6bdf-44f2-a043-5855251fe7a6-tmp-dir\") pod \"e2e-same-id-a-b3d40012-kserve-59ffd46f87-g9ktf\" (UID: \"043154d6-6bdf-44f2-a043-5855251fe7a6\") " pod="llm/e2e-same-id-a-b3d40012-kserve-59ffd46f87-g9ktf"
Sep 10 22:36:52.424889 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:36:52.424751    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/043154d6-6bdf-44f2-a043-5855251fe7a6-tls-certs\") pod \"e2e-same-id-a-b3d40012-kserve-59ffd46f87-g9ktf\" (UID: \"043154d6-6bdf-44f2-a043-5855251fe7a6\") " pod="llm/e2e-same-id-a-b3d40012-kserve-59ffd46f87-g9ktf"
Sep 10 22:36:52.424889 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:36:52.424777    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/043154d6-6bdf-44f2-a043-5855251fe7a6-dshm\") pod \"e2e-same-id-a-b3d40012-kserve-59ffd46f87-g9ktf\" (UID: \"043154d6-6bdf-44f2-a043-5855251fe7a6\") " pod="llm/e2e-same-id-a-b3d40012-kserve-59ffd46f87-g9ktf"
Sep 10 22:36:52.424889 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:36:52.424803    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/043154d6-6bdf-44f2-a043-5855251fe7a6-home\") pod \"e2e-same-id-a-b3d40012-kserve-59ffd46f87-g9ktf\" (UID: \"043154d6-6bdf-44f2-a043-5855251fe7a6\") " pod="llm/e2e-same-id-a-b3d40012-kserve-59ffd46f87-g9ktf"
Sep 10 22:36:52.424889 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:36:52.424842    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbwbm\" (UniqueName: \"kubernetes.io/projected/043154d6-6bdf-44f2-a043-5855251fe7a6-kube-api-access-lbwbm\") pod \"e2e-same-id-a-b3d40012-kserve-59ffd46f87-g9ktf\" (UID: \"043154d6-6bdf-44f2-a043-5855251fe7a6\") " pod="llm/e2e-same-id-a-b3d40012-kserve-59ffd46f87-g9ktf"
Sep 10 22:36:52.425128 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:36:52.424949    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/043154d6-6bdf-44f2-a043-5855251fe7a6-model-cache\") pod \"e2e-same-id-a-b3d40012-kserve-59ffd46f87-g9ktf\" (UID: \"043154d6-6bdf-44f2-a043-5855251fe7a6\") " pod="llm/e2e-same-id-a-b3d40012-kserve-59ffd46f87-g9ktf"
Sep 10 22:36:52.425189 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:36:52.425162    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/043154d6-6bdf-44f2-a043-5855251fe7a6-tmp-dir\") pod \"e2e-same-id-a-b3d40012-kserve-59ffd46f87-g9ktf\" (UID: \"043154d6-6bdf-44f2-a043-5855251fe7a6\") " pod="llm/e2e-same-id-a-b3d40012-kserve-59ffd46f87-g9ktf"
Sep 10 22:36:52.425253 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:36:52.425232    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/043154d6-6bdf-44f2-a043-5855251fe7a6-home\") pod \"e2e-same-id-a-b3d40012-kserve-59ffd46f87-g9ktf\" (UID: \"043154d6-6bdf-44f2-a043-5855251fe7a6\") " pod="llm/e2e-same-id-a-b3d40012-kserve-59ffd46f87-g9ktf"
Sep 10 22:36:52.425315 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:36:52.425275    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/043154d6-6bdf-44f2-a043-5855251fe7a6-model-cache\") pod \"e2e-same-id-a-b3d40012-kserve-59ffd46f87-g9ktf\" (UID: \"043154d6-6bdf-44f2-a043-5855251fe7a6\") " pod="llm/e2e-same-id-a-b3d40012-kserve-59ffd46f87-g9ktf"
Sep 10 22:36:52.427021 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:36:52.426987    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/043154d6-6bdf-44f2-a043-5855251fe7a6-dshm\") pod \"e2e-same-id-a-b3d40012-kserve-59ffd46f87-g9ktf\" (UID: \"043154d6-6bdf-44f2-a043-5855251fe7a6\") " pod="llm/e2e-same-id-a-b3d40012-kserve-59ffd46f87-g9ktf"
Sep 10 22:36:52.427185 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:36:52.427168    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/043154d6-6bdf-44f2-a043-5855251fe7a6-tls-certs\") pod \"e2e-same-id-a-b3d40012-kserve-59ffd46f87-g9ktf\" (UID: \"043154d6-6bdf-44f2-a043-5855251fe7a6\") " pod="llm/e2e-same-id-a-b3d40012-kserve-59ffd46f87-g9ktf"
Sep 10 22:36:52.445719 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:36:52.445693    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbwbm\" (UniqueName: \"kubernetes.io/projected/043154d6-6bdf-44f2-a043-5855251fe7a6-kube-api-access-lbwbm\") pod \"e2e-same-id-a-b3d40012-kserve-59ffd46f87-g9ktf\" (UID: \"043154d6-6bdf-44f2-a043-5855251fe7a6\") " pod="llm/e2e-same-id-a-b3d40012-kserve-59ffd46f87-g9ktf"
Sep 10 22:36:52.541013 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:36:52.540944    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-b3d40012-kserve-59ffd46f87-g9ktf"
Sep 10 22:36:52.810617 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:36:52.810513    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-b3d40012-kserve-59ffd46f87-g9ktf"]
Sep 10 22:36:52.813633 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:36:52.813604    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod043154d6_6bdf_44f2_a043_5855251fe7a6.slice/crio-af61ae5c4546917112aec7dbd115cd1bcb88cd9f7d55256631c887f846aac340 WatchSource:0}: Error finding container af61ae5c4546917112aec7dbd115cd1bcb88cd9f7d55256631c887f846aac340: Status 404 returned error can't find the container with id af61ae5c4546917112aec7dbd115cd1bcb88cd9f7d55256631c887f846aac340
Sep 10 22:36:53.686994 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:36:53.686957    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-b3d40012-kserve-59ffd46f87-g9ktf" event={"ID":"043154d6-6bdf-44f2-a043-5855251fe7a6","Type":"ContainerStarted","Data":"a964a88296388f8384355f1941605ea4fb7cbbd7c245fafb4eeedacfa49b2a9d"}
Sep 10 22:36:53.686994 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:36:53.686999    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-b3d40012-kserve-59ffd46f87-g9ktf" event={"ID":"043154d6-6bdf-44f2-a043-5855251fe7a6","Type":"ContainerStarted","Data":"af61ae5c4546917112aec7dbd115cd1bcb88cd9f7d55256631c887f846aac340"}
Sep 10 22:36:53.707869 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:36:53.707801    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-b3d40012-kserve-59ffd46f87-g9ktf" podStartSLOduration=1.707786337 podStartE2EDuration="1.707786337s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 22:36:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:36:53.706021458 +0000 UTC m=+2134.662089177" watchObservedRunningTime="2026-09-10 22:36:53.707786337 +0000 UTC m=+2134.663854042"
Sep 10 22:37:02.541901 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:37:02.541868    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-b3d40012-kserve-59ffd46f87-g9ktf"
Sep 10 22:37:02.542340 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:37:02.541917    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-b3d40012-kserve-59ffd46f87-g9ktf"
Sep 10 22:37:02.554289 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:37:02.554265    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-b3d40012-kserve-59ffd46f87-g9ktf"
Sep 10 22:37:02.734896 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:37:02.734873    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-b3d40012-kserve-59ffd46f87-g9ktf"
Sep 10 22:37:10.098310 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:37:10.098276    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-b3d40012-kserve-69c7bb9bb7-m8mml"]
Sep 10 22:37:10.098669 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:37:10.098558    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de372118-61dc-45ee-b7db-ae68c945f4bc" containerName="istio-proxy"
Sep 10 22:37:10.098669 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:37:10.098569    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="de372118-61dc-45ee-b7db-ae68c945f4bc" containerName="istio-proxy"
Sep 10 22:37:10.098669 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:37:10.098591    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aeeca89b-577d-4dd5-990f-cf0706a478e7" containerName="istio-proxy"
Sep 10 22:37:10.098669 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:37:10.098600    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="aeeca89b-577d-4dd5-990f-cf0706a478e7" containerName="istio-proxy"
Sep 10 22:37:10.098887 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:37:10.098674    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="de372118-61dc-45ee-b7db-ae68c945f4bc" containerName="istio-proxy"
Sep 10 22:37:10.098887 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:37:10.098682    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="aeeca89b-577d-4dd5-990f-cf0706a478e7" containerName="istio-proxy"
Sep 10 22:37:10.101545 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:37:10.101528    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-b3d40012-kserve-69c7bb9bb7-m8mml"
Sep 10 22:37:10.103593 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:37:10.103566    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-b3d40012-kserve-self-signed-certs\""
Sep 10 22:37:10.109568 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:37:10.109546    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-b3d40012-kserve-69c7bb9bb7-m8mml"]
Sep 10 22:37:10.268560 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:37:10.268528    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a67318e3-2c53-4f0d-94ad-ee190d7e631d-tls-certs\") pod \"e2e-same-id-b-b3d40012-kserve-69c7bb9bb7-m8mml\" (UID: \"a67318e3-2c53-4f0d-94ad-ee190d7e631d\") " pod="llm/e2e-same-id-b-b3d40012-kserve-69c7bb9bb7-m8mml"
Sep 10 22:37:10.268560 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:37:10.268561    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a67318e3-2c53-4f0d-94ad-ee190d7e631d-tmp-dir\") pod \"e2e-same-id-b-b3d40012-kserve-69c7bb9bb7-m8mml\" (UID: \"a67318e3-2c53-4f0d-94ad-ee190d7e631d\") " pod="llm/e2e-same-id-b-b3d40012-kserve-69c7bb9bb7-m8mml"
Sep 10 22:37:10.268786 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:37:10.268591    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a67318e3-2c53-4f0d-94ad-ee190d7e631d-home\") pod \"e2e-same-id-b-b3d40012-kserve-69c7bb9bb7-m8mml\" (UID: \"a67318e3-2c53-4f0d-94ad-ee190d7e631d\") " pod="llm/e2e-same-id-b-b3d40012-kserve-69c7bb9bb7-m8mml"
Sep 10 22:37:10.268786 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:37:10.268697    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a67318e3-2c53-4f0d-94ad-ee190d7e631d-model-cache\") pod \"e2e-same-id-b-b3d40012-kserve-69c7bb9bb7-m8mml\" (UID: \"a67318e3-2c53-4f0d-94ad-ee190d7e631d\") " pod="llm/e2e-same-id-b-b3d40012-kserve-69c7bb9bb7-m8mml"
Sep 10 22:37:10.268786 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:37:10.268719    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v649p\" (UniqueName: \"kubernetes.io/projected/a67318e3-2c53-4f0d-94ad-ee190d7e631d-kube-api-access-v649p\") pod \"e2e-same-id-b-b3d40012-kserve-69c7bb9bb7-m8mml\" (UID: \"a67318e3-2c53-4f0d-94ad-ee190d7e631d\") " pod="llm/e2e-same-id-b-b3d40012-kserve-69c7bb9bb7-m8mml"
Sep 10 22:37:10.268786 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:37:10.268747    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a67318e3-2c53-4f0d-94ad-ee190d7e631d-dshm\") pod \"e2e-same-id-b-b3d40012-kserve-69c7bb9bb7-m8mml\" (UID: \"a67318e3-2c53-4f0d-94ad-ee190d7e631d\") " pod="llm/e2e-same-id-b-b3d40012-kserve-69c7bb9bb7-m8mml"
Sep 10 22:37:10.370029 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:37:10.369960    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a67318e3-2c53-4f0d-94ad-ee190d7e631d-model-cache\") pod \"e2e-same-id-b-b3d40012-kserve-69c7bb9bb7-m8mml\" (UID: \"a67318e3-2c53-4f0d-94ad-ee190d7e631d\") " pod="llm/e2e-same-id-b-b3d40012-kserve-69c7bb9bb7-m8mml"
Sep 10 22:37:10.370029 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:37:10.369991    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v649p\" (UniqueName: \"kubernetes.io/projected/a67318e3-2c53-4f0d-94ad-ee190d7e631d-kube-api-access-v649p\") pod \"e2e-same-id-b-b3d40012-kserve-69c7bb9bb7-m8mml\" (UID: \"a67318e3-2c53-4f0d-94ad-ee190d7e631d\") " pod="llm/e2e-same-id-b-b3d40012-kserve-69c7bb9bb7-m8mml"
Sep 10 22:37:10.370029 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:37:10.370017    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a67318e3-2c53-4f0d-94ad-ee190d7e631d-dshm\") pod \"e2e-same-id-b-b3d40012-kserve-69c7bb9bb7-m8mml\" (UID: \"a67318e3-2c53-4f0d-94ad-ee190d7e631d\") " pod="llm/e2e-same-id-b-b3d40012-kserve-69c7bb9bb7-m8mml"
Sep 10 22:37:10.370281 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:37:10.370045    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a67318e3-2c53-4f0d-94ad-ee190d7e631d-tls-certs\") pod \"e2e-same-id-b-b3d40012-kserve-69c7bb9bb7-m8mml\" (UID: \"a67318e3-2c53-4f0d-94ad-ee190d7e631d\") " pod="llm/e2e-same-id-b-b3d40012-kserve-69c7bb9bb7-m8mml"
Sep 10 22:37:10.370281 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:37:10.370071    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a67318e3-2c53-4f0d-94ad-ee190d7e631d-tmp-dir\") pod \"e2e-same-id-b-b3d40012-kserve-69c7bb9bb7-m8mml\" (UID: \"a67318e3-2c53-4f0d-94ad-ee190d7e631d\") " pod="llm/e2e-same-id-b-b3d40012-kserve-69c7bb9bb7-m8mml"
Sep 10 22:37:10.370281 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:37:10.370120    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a67318e3-2c53-4f0d-94ad-ee190d7e631d-home\") pod \"e2e-same-id-b-b3d40012-kserve-69c7bb9bb7-m8mml\" (UID: \"a67318e3-2c53-4f0d-94ad-ee190d7e631d\") " pod="llm/e2e-same-id-b-b3d40012-kserve-69c7bb9bb7-m8mml"
Sep 10 22:37:10.370445 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:37:10.370417    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a67318e3-2c53-4f0d-94ad-ee190d7e631d-model-cache\") pod \"e2e-same-id-b-b3d40012-kserve-69c7bb9bb7-m8mml\" (UID: \"a67318e3-2c53-4f0d-94ad-ee190d7e631d\") " pod="llm/e2e-same-id-b-b3d40012-kserve-69c7bb9bb7-m8mml"
Sep 10 22:37:10.370491 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:37:10.370467    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a67318e3-2c53-4f0d-94ad-ee190d7e631d-tmp-dir\") pod \"e2e-same-id-b-b3d40012-kserve-69c7bb9bb7-m8mml\" (UID: \"a67318e3-2c53-4f0d-94ad-ee190d7e631d\") " pod="llm/e2e-same-id-b-b3d40012-kserve-69c7bb9bb7-m8mml"
Sep 10 22:37:10.370527 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:37:10.370512    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a67318e3-2c53-4f0d-94ad-ee190d7e631d-home\") pod \"e2e-same-id-b-b3d40012-kserve-69c7bb9bb7-m8mml\" (UID: \"a67318e3-2c53-4f0d-94ad-ee190d7e631d\") " pod="llm/e2e-same-id-b-b3d40012-kserve-69c7bb9bb7-m8mml"
Sep 10 22:37:10.372342 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:37:10.372317    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a67318e3-2c53-4f0d-94ad-ee190d7e631d-dshm\") pod \"e2e-same-id-b-b3d40012-kserve-69c7bb9bb7-m8mml\" (UID: \"a67318e3-2c53-4f0d-94ad-ee190d7e631d\") " pod="llm/e2e-same-id-b-b3d40012-kserve-69c7bb9bb7-m8mml"
Sep 10 22:37:10.372578 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:37:10.372561    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a67318e3-2c53-4f0d-94ad-ee190d7e631d-tls-certs\") pod \"e2e-same-id-b-b3d40012-kserve-69c7bb9bb7-m8mml\" (UID: \"a67318e3-2c53-4f0d-94ad-ee190d7e631d\") " pod="llm/e2e-same-id-b-b3d40012-kserve-69c7bb9bb7-m8mml"
Sep 10 22:37:10.377269 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:37:10.377248    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v649p\" (UniqueName: \"kubernetes.io/projected/a67318e3-2c53-4f0d-94ad-ee190d7e631d-kube-api-access-v649p\") pod \"e2e-same-id-b-b3d40012-kserve-69c7bb9bb7-m8mml\" (UID: \"a67318e3-2c53-4f0d-94ad-ee190d7e631d\") " pod="llm/e2e-same-id-b-b3d40012-kserve-69c7bb9bb7-m8mml"
Sep 10 22:37:10.411657 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:37:10.411634    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-b3d40012-kserve-69c7bb9bb7-m8mml"
Sep 10 22:37:10.688379 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:37:10.688210    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-b3d40012-kserve-69c7bb9bb7-m8mml"]
Sep 10 22:37:10.691001 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:37:10.690978    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda67318e3_2c53_4f0d_94ad_ee190d7e631d.slice/crio-a9c83ba222cc9adb902bc364d29a39e199afc1b85279c6ca45b30401e79521e9 WatchSource:0}: Error finding container a9c83ba222cc9adb902bc364d29a39e199afc1b85279c6ca45b30401e79521e9: Status 404 returned error can't find the container with id a9c83ba222cc9adb902bc364d29a39e199afc1b85279c6ca45b30401e79521e9
Sep 10 22:37:10.758724 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:37:10.758691    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-b3d40012-kserve-69c7bb9bb7-m8mml" event={"ID":"a67318e3-2c53-4f0d-94ad-ee190d7e631d","Type":"ContainerStarted","Data":"01465aa5acf74884bb719c381163b3a0fe9129ba04275e0f668021af399eae6f"}
Sep 10 22:37:10.758873 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:37:10.758730    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-b3d40012-kserve-69c7bb9bb7-m8mml" event={"ID":"a67318e3-2c53-4f0d-94ad-ee190d7e631d","Type":"ContainerStarted","Data":"a9c83ba222cc9adb902bc364d29a39e199afc1b85279c6ca45b30401e79521e9"}
Sep 10 22:37:10.777531 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:37:10.777490    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-b3d40012-kserve-69c7bb9bb7-m8mml" podStartSLOduration=0.777474524 podStartE2EDuration="777.474524ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 22:37:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:37:10.775263562 +0000 UTC m=+2151.731331278" watchObservedRunningTime="2026-09-10 22:37:10.777474524 +0000 UTC m=+2151.733542252"
Sep 10 22:37:19.999448 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:37:19.999375    2582 scope.go:117] "RemoveContainer" containerID="674d6658a33b05d889a577d5d1db090b34527bcc01e013f1e048e3898283757d"
Sep 10 22:37:20.412829 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:37:20.412797    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-b3d40012-kserve-69c7bb9bb7-m8mml"
Sep 10 22:37:20.412995 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:37:20.412861    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-b3d40012-kserve-69c7bb9bb7-m8mml"
Sep 10 22:37:20.425596 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:37:20.425574    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-b3d40012-kserve-69c7bb9bb7-m8mml"
Sep 10 22:37:20.815469 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:37:20.815392    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-b3d40012-kserve-69c7bb9bb7-m8mml"
Sep 10 22:37:45.295625 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:37:45.295587    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-b3d40012-kserve-59ffd46f87-g9ktf"]
Sep 10 22:37:46.253394 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:37:46.253363    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-b3d40012-kserve-69c7bb9bb7-m8mml"]
Sep 10 22:38:00.313651 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:38:00.313612    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-b3d40012-kserve-59ffd46f87-g9ktf" podUID="043154d6-6bdf-44f2-a043-5855251fe7a6" containerName="main" containerID="cri-o://a964a88296388f8384355f1941605ea4fb7cbbd7c245fafb4eeedacfa49b2a9d" gracePeriod=45
Sep 10 22:38:00.556865 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:38:00.556821    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-b3d40012-kserve-59ffd46f87-g9ktf"
Sep 10 22:38:00.645661 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:38:00.645583    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/043154d6-6bdf-44f2-a043-5855251fe7a6-model-cache\") pod \"043154d6-6bdf-44f2-a043-5855251fe7a6\" (UID: \"043154d6-6bdf-44f2-a043-5855251fe7a6\") "
Sep 10 22:38:00.645661 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:38:00.645635    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/043154d6-6bdf-44f2-a043-5855251fe7a6-tmp-dir\") pod \"043154d6-6bdf-44f2-a043-5855251fe7a6\" (UID: \"043154d6-6bdf-44f2-a043-5855251fe7a6\") "
Sep 10 22:38:00.645891 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:38:00.645677    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/043154d6-6bdf-44f2-a043-5855251fe7a6-dshm\") pod \"043154d6-6bdf-44f2-a043-5855251fe7a6\" (UID: \"043154d6-6bdf-44f2-a043-5855251fe7a6\") "
Sep 10 22:38:00.645891 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:38:00.645706    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/043154d6-6bdf-44f2-a043-5855251fe7a6-tls-certs\") pod \"043154d6-6bdf-44f2-a043-5855251fe7a6\" (UID: \"043154d6-6bdf-44f2-a043-5855251fe7a6\") "
Sep 10 22:38:00.645891 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:38:00.645749    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lbwbm\" (UniqueName: \"kubernetes.io/projected/043154d6-6bdf-44f2-a043-5855251fe7a6-kube-api-access-lbwbm\") pod \"043154d6-6bdf-44f2-a043-5855251fe7a6\" (UID: \"043154d6-6bdf-44f2-a043-5855251fe7a6\") "
Sep 10 22:38:00.645891 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:38:00.645790    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/043154d6-6bdf-44f2-a043-5855251fe7a6-home\") pod \"043154d6-6bdf-44f2-a043-5855251fe7a6\" (UID: \"043154d6-6bdf-44f2-a043-5855251fe7a6\") "
Sep 10 22:38:00.646107 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:38:00.645885    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/043154d6-6bdf-44f2-a043-5855251fe7a6-model-cache" (OuterVolumeSpecName: "model-cache") pod "043154d6-6bdf-44f2-a043-5855251fe7a6" (UID: "043154d6-6bdf-44f2-a043-5855251fe7a6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 22:38:00.646107 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:38:00.645937    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/043154d6-6bdf-44f2-a043-5855251fe7a6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "043154d6-6bdf-44f2-a043-5855251fe7a6" (UID: "043154d6-6bdf-44f2-a043-5855251fe7a6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 22:38:00.646107 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:38:00.646032    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/043154d6-6bdf-44f2-a043-5855251fe7a6-model-cache\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:38:00.646222 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:38:00.646072    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/043154d6-6bdf-44f2-a043-5855251fe7a6-tmp-dir\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:38:00.646222 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:38:00.646133    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/043154d6-6bdf-44f2-a043-5855251fe7a6-home" (OuterVolumeSpecName: "home") pod "043154d6-6bdf-44f2-a043-5855251fe7a6" (UID: "043154d6-6bdf-44f2-a043-5855251fe7a6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 22:38:00.648047 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:38:00.648023    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/043154d6-6bdf-44f2-a043-5855251fe7a6-kube-api-access-lbwbm" (OuterVolumeSpecName: "kube-api-access-lbwbm") pod "043154d6-6bdf-44f2-a043-5855251fe7a6" (UID: "043154d6-6bdf-44f2-a043-5855251fe7a6"). InnerVolumeSpecName "kube-api-access-lbwbm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 22:38:00.648225 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:38:00.648208    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/043154d6-6bdf-44f2-a043-5855251fe7a6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "043154d6-6bdf-44f2-a043-5855251fe7a6" (UID: "043154d6-6bdf-44f2-a043-5855251fe7a6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 22:38:00.648225 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:38:00.648214    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/043154d6-6bdf-44f2-a043-5855251fe7a6-dshm" (OuterVolumeSpecName: "dshm") pod "043154d6-6bdf-44f2-a043-5855251fe7a6" (UID: "043154d6-6bdf-44f2-a043-5855251fe7a6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 22:38:00.746632 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:38:00.746606    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbwbm\" (UniqueName: \"kubernetes.io/projected/043154d6-6bdf-44f2-a043-5855251fe7a6-kube-api-access-lbwbm\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:38:00.746632 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:38:00.746629    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/043154d6-6bdf-44f2-a043-5855251fe7a6-home\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:38:00.746769 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:38:00.746639    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/043154d6-6bdf-44f2-a043-5855251fe7a6-dshm\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:38:00.746769 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:38:00.746647    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/043154d6-6bdf-44f2-a043-5855251fe7a6-tls-certs\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:38:00.970824 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:38:00.970790    2582 generic.go:356] "Generic (PLEG): container finished" podID="043154d6-6bdf-44f2-a043-5855251fe7a6" containerID="a964a88296388f8384355f1941605ea4fb7cbbd7c245fafb4eeedacfa49b2a9d" exitCode=0
Sep 10 22:38:00.970983 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:38:00.970839    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-b3d40012-kserve-59ffd46f87-g9ktf" event={"ID":"043154d6-6bdf-44f2-a043-5855251fe7a6","Type":"ContainerDied","Data":"a964a88296388f8384355f1941605ea4fb7cbbd7c245fafb4eeedacfa49b2a9d"}
Sep 10 22:38:00.970983 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:38:00.970899    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-b3d40012-kserve-59ffd46f87-g9ktf" event={"ID":"043154d6-6bdf-44f2-a043-5855251fe7a6","Type":"ContainerDied","Data":"af61ae5c4546917112aec7dbd115cd1bcb88cd9f7d55256631c887f846aac340"}
Sep 10 22:38:00.970983 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:38:00.970917    2582 scope.go:117] "RemoveContainer" containerID="a964a88296388f8384355f1941605ea4fb7cbbd7c245fafb4eeedacfa49b2a9d"
Sep 10 22:38:00.970983 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:38:00.970863    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-b3d40012-kserve-59ffd46f87-g9ktf"
Sep 10 22:38:00.979674 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:38:00.979657    2582 scope.go:117] "RemoveContainer" containerID="a964a88296388f8384355f1941605ea4fb7cbbd7c245fafb4eeedacfa49b2a9d"
Sep 10 22:38:00.979950 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:38:00.979923    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a964a88296388f8384355f1941605ea4fb7cbbd7c245fafb4eeedacfa49b2a9d\": container with ID starting with a964a88296388f8384355f1941605ea4fb7cbbd7c245fafb4eeedacfa49b2a9d not found: ID does not exist" containerID="a964a88296388f8384355f1941605ea4fb7cbbd7c245fafb4eeedacfa49b2a9d"
Sep 10 22:38:00.980059 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:38:00.979958    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a964a88296388f8384355f1941605ea4fb7cbbd7c245fafb4eeedacfa49b2a9d"} err="failed to get container status \"a964a88296388f8384355f1941605ea4fb7cbbd7c245fafb4eeedacfa49b2a9d\": rpc error: code = NotFound desc = could not find container \"a964a88296388f8384355f1941605ea4fb7cbbd7c245fafb4eeedacfa49b2a9d\": container with ID starting with a964a88296388f8384355f1941605ea4fb7cbbd7c245fafb4eeedacfa49b2a9d not found: ID does not exist"
Sep 10 22:38:00.994376 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:38:00.994348    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-b3d40012-kserve-59ffd46f87-g9ktf"]
Sep 10 22:38:01.002823 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:38:01.002800    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-b3d40012-kserve-59ffd46f87-g9ktf"]
Sep 10 22:38:01.271487 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:38:01.271399    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-b3d40012-kserve-69c7bb9bb7-m8mml" podUID="a67318e3-2c53-4f0d-94ad-ee190d7e631d" containerName="main" containerID="cri-o://01465aa5acf74884bb719c381163b3a0fe9129ba04275e0f668021af399eae6f" gracePeriod=45
Sep 10 22:38:01.498208 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:38:01.498173    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="043154d6-6bdf-44f2-a043-5855251fe7a6" path="/var/lib/kubelet/pods/043154d6-6bdf-44f2-a043-5855251fe7a6/volumes"
Sep 10 22:38:01.517736 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:38:01.517714    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-b3d40012-kserve-69c7bb9bb7-m8mml"
Sep 10 22:38:01.653680 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:38:01.653618    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a67318e3-2c53-4f0d-94ad-ee190d7e631d-model-cache\") pod \"a67318e3-2c53-4f0d-94ad-ee190d7e631d\" (UID: \"a67318e3-2c53-4f0d-94ad-ee190d7e631d\") "
Sep 10 22:38:01.653680 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:38:01.653653    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a67318e3-2c53-4f0d-94ad-ee190d7e631d-tmp-dir\") pod \"a67318e3-2c53-4f0d-94ad-ee190d7e631d\" (UID: \"a67318e3-2c53-4f0d-94ad-ee190d7e631d\") "
Sep 10 22:38:01.653680 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:38:01.653674    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a67318e3-2c53-4f0d-94ad-ee190d7e631d-dshm\") pod \"a67318e3-2c53-4f0d-94ad-ee190d7e631d\" (UID: \"a67318e3-2c53-4f0d-94ad-ee190d7e631d\") "
Sep 10 22:38:01.653967 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:38:01.653709    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v649p\" (UniqueName: \"kubernetes.io/projected/a67318e3-2c53-4f0d-94ad-ee190d7e631d-kube-api-access-v649p\") pod \"a67318e3-2c53-4f0d-94ad-ee190d7e631d\" (UID: \"a67318e3-2c53-4f0d-94ad-ee190d7e631d\") "
Sep 10 22:38:01.653967 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:38:01.653739    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a67318e3-2c53-4f0d-94ad-ee190d7e631d-tls-certs\") pod \"a67318e3-2c53-4f0d-94ad-ee190d7e631d\" (UID: \"a67318e3-2c53-4f0d-94ad-ee190d7e631d\") "
Sep 10 22:38:01.653967 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:38:01.653797    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a67318e3-2c53-4f0d-94ad-ee190d7e631d-home\") pod \"a67318e3-2c53-4f0d-94ad-ee190d7e631d\" (UID: \"a67318e3-2c53-4f0d-94ad-ee190d7e631d\") "
Sep 10 22:38:01.653967 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:38:01.653915    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a67318e3-2c53-4f0d-94ad-ee190d7e631d-model-cache" (OuterVolumeSpecName: "model-cache") pod "a67318e3-2c53-4f0d-94ad-ee190d7e631d" (UID: "a67318e3-2c53-4f0d-94ad-ee190d7e631d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 22:38:01.653967 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:38:01.653931    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a67318e3-2c53-4f0d-94ad-ee190d7e631d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a67318e3-2c53-4f0d-94ad-ee190d7e631d" (UID: "a67318e3-2c53-4f0d-94ad-ee190d7e631d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 22:38:01.654227 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:38:01.654148    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a67318e3-2c53-4f0d-94ad-ee190d7e631d-model-cache\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:38:01.654227 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:38:01.654163    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a67318e3-2c53-4f0d-94ad-ee190d7e631d-tmp-dir\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:38:01.654227 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:38:01.654184    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a67318e3-2c53-4f0d-94ad-ee190d7e631d-home" (OuterVolumeSpecName: "home") pod "a67318e3-2c53-4f0d-94ad-ee190d7e631d" (UID: "a67318e3-2c53-4f0d-94ad-ee190d7e631d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 22:38:01.655733 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:38:01.655706    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a67318e3-2c53-4f0d-94ad-ee190d7e631d-dshm" (OuterVolumeSpecName: "dshm") pod "a67318e3-2c53-4f0d-94ad-ee190d7e631d" (UID: "a67318e3-2c53-4f0d-94ad-ee190d7e631d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 22:38:01.655840 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:38:01.655820    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a67318e3-2c53-4f0d-94ad-ee190d7e631d-kube-api-access-v649p" (OuterVolumeSpecName: "kube-api-access-v649p") pod "a67318e3-2c53-4f0d-94ad-ee190d7e631d" (UID: "a67318e3-2c53-4f0d-94ad-ee190d7e631d"). InnerVolumeSpecName "kube-api-access-v649p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 22:38:01.655927 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:38:01.655826    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a67318e3-2c53-4f0d-94ad-ee190d7e631d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a67318e3-2c53-4f0d-94ad-ee190d7e631d" (UID: "a67318e3-2c53-4f0d-94ad-ee190d7e631d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 22:38:01.754633 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:38:01.754610    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v649p\" (UniqueName: \"kubernetes.io/projected/a67318e3-2c53-4f0d-94ad-ee190d7e631d-kube-api-access-v649p\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:38:01.754633 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:38:01.754632    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a67318e3-2c53-4f0d-94ad-ee190d7e631d-tls-certs\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:38:01.754772 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:38:01.754642    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a67318e3-2c53-4f0d-94ad-ee190d7e631d-home\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:38:01.754772 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:38:01.754649    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a67318e3-2c53-4f0d-94ad-ee190d7e631d-dshm\") on node \"ip-10-0-134-144.ec2.internal\" DevicePath \"\""
Sep 10 22:38:01.977118 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:38:01.977055    2582 generic.go:356] "Generic (PLEG): container finished" podID="a67318e3-2c53-4f0d-94ad-ee190d7e631d" containerID="01465aa5acf74884bb719c381163b3a0fe9129ba04275e0f668021af399eae6f" exitCode=0
Sep 10 22:38:01.977118 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:38:01.977095    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-b3d40012-kserve-69c7bb9bb7-m8mml"
Sep 10 22:38:01.977265 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:38:01.977098    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-b3d40012-kserve-69c7bb9bb7-m8mml" event={"ID":"a67318e3-2c53-4f0d-94ad-ee190d7e631d","Type":"ContainerDied","Data":"01465aa5acf74884bb719c381163b3a0fe9129ba04275e0f668021af399eae6f"}
Sep 10 22:38:01.977265 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:38:01.977218    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-b3d40012-kserve-69c7bb9bb7-m8mml" event={"ID":"a67318e3-2c53-4f0d-94ad-ee190d7e631d","Type":"ContainerDied","Data":"a9c83ba222cc9adb902bc364d29a39e199afc1b85279c6ca45b30401e79521e9"}
Sep 10 22:38:01.977265 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:38:01.977242    2582 scope.go:117] "RemoveContainer" containerID="01465aa5acf74884bb719c381163b3a0fe9129ba04275e0f668021af399eae6f"
Sep 10 22:38:01.985767 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:38:01.985750    2582 scope.go:117] "RemoveContainer" containerID="01465aa5acf74884bb719c381163b3a0fe9129ba04275e0f668021af399eae6f"
Sep 10 22:38:01.986128 ip-10-0-134-144 kubenswrapper[2582]: E0910 22:38:01.986107    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01465aa5acf74884bb719c381163b3a0fe9129ba04275e0f668021af399eae6f\": container with ID starting with 01465aa5acf74884bb719c381163b3a0fe9129ba04275e0f668021af399eae6f not found: ID does not exist" containerID="01465aa5acf74884bb719c381163b3a0fe9129ba04275e0f668021af399eae6f"
Sep 10 22:38:01.986195 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:38:01.986135    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01465aa5acf74884bb719c381163b3a0fe9129ba04275e0f668021af399eae6f"} err="failed to get container status \"01465aa5acf74884bb719c381163b3a0fe9129ba04275e0f668021af399eae6f\": rpc error: code = NotFound desc = could not find container \"01465aa5acf74884bb719c381163b3a0fe9129ba04275e0f668021af399eae6f\": container with ID starting with 01465aa5acf74884bb719c381163b3a0fe9129ba04275e0f668021af399eae6f not found: ID does not exist"
Sep 10 22:38:01.997809 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:38:01.997784    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-b3d40012-kserve-69c7bb9bb7-m8mml"]
Sep 10 22:38:02.004602 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:38:02.004579    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-b3d40012-kserve-69c7bb9bb7-m8mml"]
Sep 10 22:38:03.498074 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:38:03.498038    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a67318e3-2c53-4f0d-94ad-ee190d7e631d" path="/var/lib/kubelet/pods/a67318e3-2c53-4f0d-94ad-ee190d7e631d/volumes"
Sep 10 22:38:13.785303 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:38:13.785270    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-559477f47-7d4w8"]
Sep 10 22:38:13.785667 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:38:13.785583    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a67318e3-2c53-4f0d-94ad-ee190d7e631d" containerName="main"
Sep 10 22:38:13.785667 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:38:13.785594    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="a67318e3-2c53-4f0d-94ad-ee190d7e631d" containerName="main"
Sep 10 22:38:13.785667 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:38:13.785615    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="043154d6-6bdf-44f2-a043-5855251fe7a6" containerName="main"
Sep 10 22:38:13.785667 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:38:13.785621    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="043154d6-6bdf-44f2-a043-5855251fe7a6" containerName="main"
Sep 10 22:38:13.785667 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:38:13.785669    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="043154d6-6bdf-44f2-a043-5855251fe7a6" containerName="main"
Sep 10 22:38:13.785834 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:38:13.785677    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="a67318e3-2c53-4f0d-94ad-ee190d7e631d" containerName="main"
Sep 10 22:38:13.789932 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:38:13.789914    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-559477f47-7d4w8"
Sep 10 22:38:13.797641 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:38:13.797621    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 22:38:13.798172 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:38:13.797999    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-spztp\""
Sep 10 22:38:13.798172 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:38:13.798086    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 10 22:38:13.798172 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:38:13.798106    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 10 22:38:13.798172 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:38:13.798140    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 10 22:38:13.807895 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:38:13.807876    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-559477f47-7d4w8"]
Sep 10 22:38:13.940886 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:38:13.940839    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52158f1c-66cf-4a36-ba96-844d9da60ace-metrics-certs\") pod \"maas-api-559477f47-7d4w8\" (UID: \"52158f1c-66cf-4a36-ba96-844d9da60ace\") " pod="odh-ai-gateway-infra/maas-api-559477f47-7d4w8"
Sep 10 22:38:13.941026 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:38:13.940916    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52158f1c-66cf-4a36-ba96-844d9da60ace-openshift-service-ca\") pod \"maas-api-559477f47-7d4w8\" (UID: \"52158f1c-66cf-4a36-ba96-844d9da60ace\") " pod="odh-ai-gateway-infra/maas-api-559477f47-7d4w8"
Sep 10 22:38:13.941026 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:38:13.940940    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52158f1c-66cf-4a36-ba96-844d9da60ace-maas-api-tls\") pod \"maas-api-559477f47-7d4w8\" (UID: \"52158f1c-66cf-4a36-ba96-844d9da60ace\") " pod="odh-ai-gateway-infra/maas-api-559477f47-7d4w8"
Sep 10 22:38:13.941026 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:38:13.940959    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bz56h\" (UniqueName: \"kubernetes.io/projected/52158f1c-66cf-4a36-ba96-844d9da60ace-kube-api-access-bz56h\") pod \"maas-api-559477f47-7d4w8\" (UID: \"52158f1c-66cf-4a36-ba96-844d9da60ace\") " pod="odh-ai-gateway-infra/maas-api-559477f47-7d4w8"
Sep 10 22:38:14.041358 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:38:14.041293    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52158f1c-66cf-4a36-ba96-844d9da60ace-openshift-service-ca\") pod \"maas-api-559477f47-7d4w8\" (UID: \"52158f1c-66cf-4a36-ba96-844d9da60ace\") " pod="odh-ai-gateway-infra/maas-api-559477f47-7d4w8"
Sep 10 22:38:14.041358 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:38:14.041329    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52158f1c-66cf-4a36-ba96-844d9da60ace-maas-api-tls\") pod \"maas-api-559477f47-7d4w8\" (UID: \"52158f1c-66cf-4a36-ba96-844d9da60ace\") " pod="odh-ai-gateway-infra/maas-api-559477f47-7d4w8"
Sep 10 22:38:14.041358 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:38:14.041347    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bz56h\" (UniqueName: \"kubernetes.io/projected/52158f1c-66cf-4a36-ba96-844d9da60ace-kube-api-access-bz56h\") pod \"maas-api-559477f47-7d4w8\" (UID: \"52158f1c-66cf-4a36-ba96-844d9da60ace\") " pod="odh-ai-gateway-infra/maas-api-559477f47-7d4w8"
Sep 10 22:38:14.041607 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:38:14.041476    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52158f1c-66cf-4a36-ba96-844d9da60ace-metrics-certs\") pod \"maas-api-559477f47-7d4w8\" (UID: \"52158f1c-66cf-4a36-ba96-844d9da60ace\") " pod="odh-ai-gateway-infra/maas-api-559477f47-7d4w8"
Sep 10 22:38:14.041979 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:38:14.041948    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52158f1c-66cf-4a36-ba96-844d9da60ace-openshift-service-ca\") pod \"maas-api-559477f47-7d4w8\" (UID: \"52158f1c-66cf-4a36-ba96-844d9da60ace\") " pod="odh-ai-gateway-infra/maas-api-559477f47-7d4w8"
Sep 10 22:38:14.043795 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:38:14.043772    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52158f1c-66cf-4a36-ba96-844d9da60ace-metrics-certs\") pod \"maas-api-559477f47-7d4w8\" (UID: \"52158f1c-66cf-4a36-ba96-844d9da60ace\") " pod="odh-ai-gateway-infra/maas-api-559477f47-7d4w8"
Sep 10 22:38:14.043910 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:38:14.043807    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52158f1c-66cf-4a36-ba96-844d9da60ace-maas-api-tls\") pod \"maas-api-559477f47-7d4w8\" (UID: \"52158f1c-66cf-4a36-ba96-844d9da60ace\") " pod="odh-ai-gateway-infra/maas-api-559477f47-7d4w8"
Sep 10 22:38:14.048949 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:38:14.048929    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bz56h\" (UniqueName: \"kubernetes.io/projected/52158f1c-66cf-4a36-ba96-844d9da60ace-kube-api-access-bz56h\") pod \"maas-api-559477f47-7d4w8\" (UID: \"52158f1c-66cf-4a36-ba96-844d9da60ace\") " pod="odh-ai-gateway-infra/maas-api-559477f47-7d4w8"
Sep 10 22:38:14.100026 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:38:14.100003    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-559477f47-7d4w8"
Sep 10 22:38:14.372635 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:38:14.372609    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-559477f47-7d4w8"]
Sep 10 22:38:14.374579 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:38:14.374552    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52158f1c_66cf_4a36_ba96_844d9da60ace.slice/crio-5fa8709ee01050ede7a9f283f33a1d5aa05cc9baef414d2a3f664c5fa4fa2b1c WatchSource:0}: Error finding container 5fa8709ee01050ede7a9f283f33a1d5aa05cc9baef414d2a3f664c5fa4fa2b1c: Status 404 returned error can't find the container with id 5fa8709ee01050ede7a9f283f33a1d5aa05cc9baef414d2a3f664c5fa4fa2b1c
Sep 10 22:38:14.375840 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:38:14.375820    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 22:38:15.037220 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:38:15.037140    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-559477f47-7d4w8" event={"ID":"52158f1c-66cf-4a36-ba96-844d9da60ace","Type":"ContainerStarted","Data":"fbad1d86758ceda0ee1ebfb4fd16898a37310c547128b51a371c0ed807583d9a"}
Sep 10 22:38:15.037220 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:38:15.037217    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-559477f47-7d4w8" event={"ID":"52158f1c-66cf-4a36-ba96-844d9da60ace","Type":"ContainerStarted","Data":"5fa8709ee01050ede7a9f283f33a1d5aa05cc9baef414d2a3f664c5fa4fa2b1c"}
Sep 10 22:38:15.037624 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:38:15.037292    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-559477f47-7d4w8"
Sep 10 22:38:15.077410 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:38:15.077369    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-559477f47-7d4w8" podStartSLOduration=1.690637489 podStartE2EDuration="2.077357253s" totalImagesPullingTime="386.719764ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 22:38:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:38:15.076042452 +0000 UTC m=+2216.032110158" watchObservedRunningTime="2026-09-10 22:38:15.077357253 +0000 UTC m=+2216.033424958"
Sep 10 22:38:20.017005 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:38:20.016975    2582 scope.go:117] "RemoveContainer" containerID="1848cf449aef81519b9f8d9996e0f91c40f8e9249f18bbe2299e169c62896d58"
Sep 10 22:38:20.025236 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:38:20.025214    2582 scope.go:117] "RemoveContainer" containerID="9f7375517642f25cbecc9971068a1f2b4b450fb6c312ad546365393b02b70645"
Sep 10 22:38:21.047528 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:38:21.047499    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-559477f47-7d4w8"
Sep 10 22:38:37.445829 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:38:37.445798    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6b96dfc79b-g9q5h_6bae3cbf-19db-4a96-bb9d-870154b86bc3/authorino/0.log"
Sep 10 22:38:43.201820 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:38:43.201726    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-r2b97_6760a7a5-e21c-4606-89a3-f78554990043/server/0.log"
Sep 10 22:38:43.326232 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:38:43.326202    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-8fs8w_fadc83c6-d3c0-42fd-bf1c-b9137805a88a/manager/0.log"
Sep 10 22:38:43.717998 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:38:43.717974    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-76d44b6d5b-vjglf_d16e4185-bc86-4442-81a4-9ca18f1b49d2/copy-manifests/0.log"
Sep 10 22:38:43.725803 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:38:43.725772    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-76d44b6d5b-vjglf_d16e4185-bc86-4442-81a4-9ca18f1b49d2/manager/0.log"
Sep 10 22:38:44.076329 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:38:44.076239    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-559477f47-7d4w8_52158f1c-66cf-4a36-ba96-844d9da60ace/maas-api/0.log"
Sep 10 22:38:45.346321 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:38:45.346286    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6b96dfc79b-g9q5h_6bae3cbf-19db-4a96-bb9d-870154b86bc3/authorino/0.log"
Sep 10 22:38:46.417697 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:38:46.417659    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-q8r9d_d2892c71-f59d-454f-81dc-168d4259d13f/istio-proxy/0.log"
Sep 10 22:38:46.777064 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:38:46.776985    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-69ccfddbf-qbrxk_eff019c7-6e59-467a-a49e-8b761744ac47/kube-auth-proxy/0.log"
Sep 10 22:38:47.304410 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:38:47.304381    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-66fc76d68b-5wp97_97c76d14-c853-48fa-bb5c-b7c553b4f899/router/0.log"
Sep 10 22:38:47.741600 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:38:47.741562    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-4c59l_68299c8c-a2fe-4c4e-ad50-9d2bf78d13f5/main/0.log"
Sep 10 22:38:48.301612 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:38:48.301578    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-wb4r5_18e39a69-1b48-40ca-9438-6210181c9f8c/main/0.log"
Sep 10 22:38:57.435628 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:38:57.435595    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-n9r9x_0d6f601b-532d-4f1f-9580-977997448ea9/global-pull-secret-syncer/0.log"
Sep 10 22:38:57.527619 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:38:57.527583    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-cjdvg_f0c8381d-52b6-4511-90be-59aebfef283a/konnectivity-agent/0.log"
Sep 10 22:38:57.634803 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:38:57.634777    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-134-144.ec2.internal_f31b7187b87e2b509b6e74131f844880/haproxy/0.log"
Sep 10 22:39:02.309573 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:39:02.309539    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6b96dfc79b-g9q5h_6bae3cbf-19db-4a96-bb9d-870154b86bc3/authorino/0.log"
Sep 10 22:39:05.460428 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:39:05.460395    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-kcl7j_a43f9f15-f001-4c47-8676-2cdd638c206a/node-exporter/0.log"
Sep 10 22:39:05.492534 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:39:05.492499    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-kcl7j_a43f9f15-f001-4c47-8676-2cdd638c206a/kube-rbac-proxy/0.log"
Sep 10 22:39:05.521784 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:39:05.521764    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-kcl7j_a43f9f15-f001-4c47-8676-2cdd638c206a/init-textfile/0.log"
Sep 10 22:39:07.004146 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:39:07.004116    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-8dpvh/perf-node-gather-daemonset-6dznh"]
Sep 10 22:39:07.006966 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:39:07.006948    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8dpvh/perf-node-gather-daemonset-6dznh"
Sep 10 22:39:07.009634 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:39:07.009611    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-8dpvh\"/\"openshift-service-ca.crt\""
Sep 10 22:39:07.009750 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:39:07.009657    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-8dpvh\"/\"kube-root-ca.crt\""
Sep 10 22:39:07.010394 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:39:07.010377    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-8dpvh\"/\"default-dockercfg-6cqgb\""
Sep 10 22:39:07.020290 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:39:07.020270    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8dpvh/perf-node-gather-daemonset-6dznh"]
Sep 10 22:39:07.030283 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:39:07.030257    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/90aa9776-fa8a-40d4-bc9f-c5cbc2b43ccc-proc\") pod \"perf-node-gather-daemonset-6dznh\" (UID: \"90aa9776-fa8a-40d4-bc9f-c5cbc2b43ccc\") " pod="openshift-must-gather-8dpvh/perf-node-gather-daemonset-6dznh"
Sep 10 22:39:07.030398 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:39:07.030297    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/90aa9776-fa8a-40d4-bc9f-c5cbc2b43ccc-podres\") pod \"perf-node-gather-daemonset-6dznh\" (UID: \"90aa9776-fa8a-40d4-bc9f-c5cbc2b43ccc\") " pod="openshift-must-gather-8dpvh/perf-node-gather-daemonset-6dznh"
Sep 10 22:39:07.030398 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:39:07.030386    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/90aa9776-fa8a-40d4-bc9f-c5cbc2b43ccc-lib-modules\") pod \"perf-node-gather-daemonset-6dznh\" (UID: \"90aa9776-fa8a-40d4-bc9f-c5cbc2b43ccc\") " pod="openshift-must-gather-8dpvh/perf-node-gather-daemonset-6dznh"
Sep 10 22:39:07.030488 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:39:07.030424    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/90aa9776-fa8a-40d4-bc9f-c5cbc2b43ccc-sys\") pod \"perf-node-gather-daemonset-6dznh\" (UID: \"90aa9776-fa8a-40d4-bc9f-c5cbc2b43ccc\") " pod="openshift-must-gather-8dpvh/perf-node-gather-daemonset-6dznh"
Sep 10 22:39:07.030488 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:39:07.030466    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtqvz\" (UniqueName: \"kubernetes.io/projected/90aa9776-fa8a-40d4-bc9f-c5cbc2b43ccc-kube-api-access-gtqvz\") pod \"perf-node-gather-daemonset-6dznh\" (UID: \"90aa9776-fa8a-40d4-bc9f-c5cbc2b43ccc\") " pod="openshift-must-gather-8dpvh/perf-node-gather-daemonset-6dznh"
Sep 10 22:39:07.131457 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:39:07.131423    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/90aa9776-fa8a-40d4-bc9f-c5cbc2b43ccc-podres\") pod \"perf-node-gather-daemonset-6dznh\" (UID: \"90aa9776-fa8a-40d4-bc9f-c5cbc2b43ccc\") " pod="openshift-must-gather-8dpvh/perf-node-gather-daemonset-6dznh"
Sep 10 22:39:07.131665 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:39:07.131473    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/90aa9776-fa8a-40d4-bc9f-c5cbc2b43ccc-lib-modules\") pod \"perf-node-gather-daemonset-6dznh\" (UID: \"90aa9776-fa8a-40d4-bc9f-c5cbc2b43ccc\") " pod="openshift-must-gather-8dpvh/perf-node-gather-daemonset-6dznh"
Sep 10 22:39:07.131665 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:39:07.131492    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/90aa9776-fa8a-40d4-bc9f-c5cbc2b43ccc-sys\") pod \"perf-node-gather-daemonset-6dznh\" (UID: \"90aa9776-fa8a-40d4-bc9f-c5cbc2b43ccc\") " pod="openshift-must-gather-8dpvh/perf-node-gather-daemonset-6dznh"
Sep 10 22:39:07.131665 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:39:07.131524    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtqvz\" (UniqueName: \"kubernetes.io/projected/90aa9776-fa8a-40d4-bc9f-c5cbc2b43ccc-kube-api-access-gtqvz\") pod \"perf-node-gather-daemonset-6dznh\" (UID: \"90aa9776-fa8a-40d4-bc9f-c5cbc2b43ccc\") " pod="openshift-must-gather-8dpvh/perf-node-gather-daemonset-6dznh"
Sep 10 22:39:07.131665 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:39:07.131560    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/90aa9776-fa8a-40d4-bc9f-c5cbc2b43ccc-proc\") pod \"perf-node-gather-daemonset-6dznh\" (UID: \"90aa9776-fa8a-40d4-bc9f-c5cbc2b43ccc\") " pod="openshift-must-gather-8dpvh/perf-node-gather-daemonset-6dznh"
Sep 10 22:39:07.131665 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:39:07.131632    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/90aa9776-fa8a-40d4-bc9f-c5cbc2b43ccc-sys\") pod \"perf-node-gather-daemonset-6dznh\" (UID: \"90aa9776-fa8a-40d4-bc9f-c5cbc2b43ccc\") " pod="openshift-must-gather-8dpvh/perf-node-gather-daemonset-6dznh"
Sep 10 22:39:07.131665 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:39:07.131652    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/90aa9776-fa8a-40d4-bc9f-c5cbc2b43ccc-proc\") pod \"perf-node-gather-daemonset-6dznh\" (UID: \"90aa9776-fa8a-40d4-bc9f-c5cbc2b43ccc\") " pod="openshift-must-gather-8dpvh/perf-node-gather-daemonset-6dznh"
Sep 10 22:39:07.131935 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:39:07.131631    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/90aa9776-fa8a-40d4-bc9f-c5cbc2b43ccc-podres\") pod \"perf-node-gather-daemonset-6dznh\" (UID: \"90aa9776-fa8a-40d4-bc9f-c5cbc2b43ccc\") " pod="openshift-must-gather-8dpvh/perf-node-gather-daemonset-6dznh"
Sep 10 22:39:07.131935 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:39:07.131632    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/90aa9776-fa8a-40d4-bc9f-c5cbc2b43ccc-lib-modules\") pod \"perf-node-gather-daemonset-6dznh\" (UID: \"90aa9776-fa8a-40d4-bc9f-c5cbc2b43ccc\") " pod="openshift-must-gather-8dpvh/perf-node-gather-daemonset-6dznh"
Sep 10 22:39:07.139701 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:39:07.139679    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtqvz\" (UniqueName: \"kubernetes.io/projected/90aa9776-fa8a-40d4-bc9f-c5cbc2b43ccc-kube-api-access-gtqvz\") pod \"perf-node-gather-daemonset-6dznh\" (UID: \"90aa9776-fa8a-40d4-bc9f-c5cbc2b43ccc\") " pod="openshift-must-gather-8dpvh/perf-node-gather-daemonset-6dznh"
Sep 10 22:39:07.316796 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:39:07.316705    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8dpvh/perf-node-gather-daemonset-6dznh"
Sep 10 22:39:07.582929 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:39:07.582904    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8dpvh/perf-node-gather-daemonset-6dznh"]
Sep 10 22:39:07.585310 ip-10-0-134-144 kubenswrapper[2582]: W0910 22:39:07.585283    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod90aa9776_fa8a_40d4_bc9f_c5cbc2b43ccc.slice/crio-51748ca9a234e213d8137d9fcceed2a34e57d0953b2460834d3e08595f0009f0 WatchSource:0}: Error finding container 51748ca9a234e213d8137d9fcceed2a34e57d0953b2460834d3e08595f0009f0: Status 404 returned error can't find the container with id 51748ca9a234e213d8137d9fcceed2a34e57d0953b2460834d3e08595f0009f0
Sep 10 22:39:08.212640 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:39:08.212612    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-w6vlb_8517c9a7-382a-40fd-bc52-e6da90ebc584/console-operator/2.log"
Sep 10 22:39:08.220313 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:39:08.220291    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-w6vlb_8517c9a7-382a-40fd-bc52-e6da90ebc584/console-operator/3.log"
Sep 10 22:39:08.267130 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:39:08.267105    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8dpvh/perf-node-gather-daemonset-6dznh" event={"ID":"90aa9776-fa8a-40d4-bc9f-c5cbc2b43ccc","Type":"ContainerStarted","Data":"fdde7693fb77068c54298a5f58a965de3f2f8e98f7411a4253af912e0a6f2947"}
Sep 10 22:39:08.267320 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:39:08.267157    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8dpvh/perf-node-gather-daemonset-6dznh" event={"ID":"90aa9776-fa8a-40d4-bc9f-c5cbc2b43ccc","Type":"ContainerStarted","Data":"51748ca9a234e213d8137d9fcceed2a34e57d0953b2460834d3e08595f0009f0"}
Sep 10 22:39:08.267320 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:39:08.267200    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-8dpvh/perf-node-gather-daemonset-6dznh"
Sep 10 22:39:08.283264 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:39:08.283226    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-8dpvh/perf-node-gather-daemonset-6dznh" podStartSLOduration=2.283211997 podStartE2EDuration="2.283211997s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 22:39:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:39:08.282111186 +0000 UTC m=+2269.238178891" watchObservedRunningTime="2026-09-10 22:39:08.283211997 +0000 UTC m=+2269.239279702"
Sep 10 22:39:10.519590 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:39:10.519561    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-xck4c_9d4e1da6-4cc3-41a7-a8be-1fdb06fd6759/dns/0.log"
Sep 10 22:39:10.540541 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:39:10.540512    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-xck4c_9d4e1da6-4cc3-41a7-a8be-1fdb06fd6759/kube-rbac-proxy/0.log"
Sep 10 22:39:10.680008 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:39:10.679986    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-ncrzz_16caf96d-d15f-45e5-b003-1aec5786ea58/dns-node-resolver/0.log"
Sep 10 22:39:11.328128 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:39:11.328105    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-74bb6_f430e7fc-0f5e-48e2-9ea6-f97c277cdb06/node-ca/0.log"
Sep 10 22:39:12.474032 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:39:12.474000    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-q8r9d_d2892c71-f59d-454f-81dc-168d4259d13f/istio-proxy/0.log"
Sep 10 22:39:12.762298 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:39:12.762210    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-69ccfddbf-qbrxk_eff019c7-6e59-467a-a49e-8b761744ac47/kube-auth-proxy/0.log"
Sep 10 22:39:12.922630 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:39:12.922596    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-66fc76d68b-5wp97_97c76d14-c853-48fa-bb5c-b7c553b4f899/router/0.log"
Sep 10 22:39:13.877498 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:39:13.877468    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-5p6sf_b8b665fe-9846-4157-8922-433c2e13f6fa/serve-healthcheck-canary/0.log"
Sep 10 22:39:14.280730 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:39:14.280703    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-8dpvh/perf-node-gather-daemonset-6dznh"
Sep 10 22:39:14.392942 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:39:14.392911    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-bhv4x_3aaaf117-e959-49e2-93f0-342a657439c5/insights-operator/0.log"
Sep 10 22:39:14.393453 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:39:14.393413    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-bhv4x_3aaaf117-e959-49e2-93f0-342a657439c5/insights-operator/1.log"
Sep 10 22:39:14.579037 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:39:14.578957    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hrpd5_29d71552-94a7-4491-aced-1141665a8b0b/kube-rbac-proxy/0.log"
Sep 10 22:39:14.603127 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:39:14.603098    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hrpd5_29d71552-94a7-4491-aced-1141665a8b0b/exporter/0.log"
Sep 10 22:39:14.626334 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:39:14.626307    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hrpd5_29d71552-94a7-4491-aced-1141665a8b0b/extractor/0.log"
Sep 10 22:39:17.148040 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:39:17.148017    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-r2b97_6760a7a5-e21c-4606-89a3-f78554990043/server/0.log"
Sep 10 22:39:17.277566 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:39:17.277538    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-8fs8w_fadc83c6-d3c0-42fd-bf1c-b9137805a88a/manager/0.log"
Sep 10 22:39:17.499349 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:39:17.499303    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-76d44b6d5b-vjglf_d16e4185-bc86-4442-81a4-9ca18f1b49d2/manager/0.log"
Sep 10 22:39:17.520521 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:39:17.520498    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-76d44b6d5b-vjglf_d16e4185-bc86-4442-81a4-9ca18f1b49d2/copy-manifests/0.log"
Sep 10 22:39:18.687544 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:39:18.687511    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-q8r9d_d2892c71-f59d-454f-81dc-168d4259d13f/istio-proxy/0.log"
Sep 10 22:39:18.890928 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:39:18.890900    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-56f58b8775-bsmcq_a2719a25-d4d6-419f-8310-56426b57f105/manager/0.log"
Sep 10 22:39:18.943276 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:39:18.943199    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-69ccfddbf-qbrxk_eff019c7-6e59-467a-a49e-8b761744ac47/kube-auth-proxy/0.log"
Sep 10 22:39:18.945112 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:39:18.945090    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-m9dhp_45a6a874-366b-44c6-8e2d-59b370cbb643/openshift-lws-operator/0.log"
Sep 10 22:39:19.057753 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:39:19.057727    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-66fc76d68b-5wp97_97c76d14-c853-48fa-bb5c-b7c553b4f899/router/0.log"
Sep 10 22:39:20.563269 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:39:20.563235    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-q8r9d_d2892c71-f59d-454f-81dc-168d4259d13f/istio-proxy/0.log"
Sep 10 22:39:20.827541 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:39:20.827465    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-69ccfddbf-qbrxk_eff019c7-6e59-467a-a49e-8b761744ac47/kube-auth-proxy/0.log"
Sep 10 22:39:20.945149 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:39:20.945124    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-66fc76d68b-5wp97_97c76d14-c853-48fa-bb5c-b7c553b4f899/router/0.log"
Sep 10 22:39:24.198740 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:39:24.198660    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-hcz8r_759ec1b7-bee2-429e-aac1-2256e6e8dea4/migrator/0.log"
Sep 10 22:39:24.221598 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:39:24.221571    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-hcz8r_759ec1b7-bee2-429e-aac1-2256e6e8dea4/graceful-termination/0.log"
Sep 10 22:39:25.730121 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:39:25.730089    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-8rwdz_1731f853-3630-4b20-b78c-5243c08f20e4/kube-multus/0.log"
Sep 10 22:39:25.787145 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:39:25.787121    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dt25t_3cf12f53-0edf-43cc-913f-50b1669152b2/kube-multus-additional-cni-plugins/0.log"
Sep 10 22:39:25.847256 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:39:25.847229    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dt25t_3cf12f53-0edf-43cc-913f-50b1669152b2/egress-router-binary-copy/0.log"
Sep 10 22:39:25.931585 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:39:25.931556    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dt25t_3cf12f53-0edf-43cc-913f-50b1669152b2/cni-plugins/0.log"
Sep 10 22:39:26.007927 ip-10-0-134-144 kubenswrapper[2582]: I0910 22:39:26.007845    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dt25t_3cf12f53-0edf-43cc-913f-50b1669152b2/bond-cni-plugin/0.log"