Sep 17 03:17:25.121394 ip-10-0-139-53 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 17 03:17:25.121400 ip-10-0-139-53 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 17 03:17:25.121409 ip-10-0-139-53 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 17 03:17:25.121728 ip-10-0-139-53 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 17 03:17:35.306977 ip-10-0-139-53 systemd[1]: kubelet.service: Failed to schedule restart job: Unit openvswitch.service not found.
Sep 17 03:17:35.306989 ip-10-0-139-53 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot ce3cc11ffea34424b9b33eb75a20bffa --
Sep 17 03:19:24.424963 ip-10-0-139-53 systemd[1]: Starting Kubernetes Kubelet...
Sep 17 03:19:24.783280 ip-10-0-139-53 kubenswrapper[2552]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 03:19:24.783280 ip-10-0-139-53 kubenswrapper[2552]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 17 03:19:24.783280 ip-10-0-139-53 kubenswrapper[2552]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 03:19:24.783280 ip-10-0-139-53 kubenswrapper[2552]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 17 03:19:24.783280 ip-10-0-139-53 kubenswrapper[2552]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 03:19:24.784686 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.784570    2552 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 17 03:19:24.788979 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.788954    2552 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 03:19:24.788979 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.788975    2552 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 03:19:24.788979 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.788979    2552 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 03:19:24.788979 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.788982    2552 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 03:19:24.788979 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.788987    2552 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 03:19:24.789178 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.788990    2552 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 03:19:24.789178 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.788993    2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 03:19:24.789178 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.788996    2552 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 03:19:24.789178 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.788999    2552 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 03:19:24.789178 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.789002    2552 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 03:19:24.789178 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.789005    2552 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 03:19:24.789178 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.789007    2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 03:19:24.789178 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.789010    2552 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 03:19:24.789178 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.789016    2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 03:19:24.789178 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.789019    2552 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 03:19:24.789178 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.789023    2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 03:19:24.789178 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.789026    2552 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 03:19:24.789178 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.789029    2552 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 03:19:24.789178 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.789032    2552 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 03:19:24.789178 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.789035    2552 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 03:19:24.789178 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.789037    2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 03:19:24.789178 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.789040    2552 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 03:19:24.789178 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.789043    2552 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 03:19:24.789178 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.789046    2552 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 03:19:24.789178 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.789048    2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 03:19:24.789671 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.789054    2552 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 03:19:24.789671 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.789057    2552 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 03:19:24.789671 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.789059    2552 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 03:19:24.789671 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.789062    2552 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 03:19:24.789671 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.789065    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 03:19:24.789671 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.789068    2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 03:19:24.789671 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.789071    2552 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 03:19:24.789671 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.789074    2552 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 03:19:24.789671 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.789077    2552 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 03:19:24.789671 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.789079    2552 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 03:19:24.789671 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.789082    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 03:19:24.789671 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.789085    2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 03:19:24.789671 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.789088    2552 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 03:19:24.789671 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.789094    2552 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 03:19:24.789671 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.789097    2552 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 03:19:24.789671 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.789100    2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 03:19:24.789671 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.789103    2552 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 03:19:24.789671 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.789106    2552 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 03:19:24.789671 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.789108    2552 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 03:19:24.790135 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.789111    2552 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 03:19:24.790135 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.789116    2552 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 03:19:24.790135 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.789120    2552 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 03:19:24.790135 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.789123    2552 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 03:19:24.790135 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.789127    2552 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 03:19:24.790135 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.789132    2552 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 03:19:24.790135 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.789137    2552 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 03:19:24.790135 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.789141    2552 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 03:19:24.790135 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.789144    2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 03:19:24.790135 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.789147    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 03:19:24.790135 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.789150    2552 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 03:19:24.790135 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.789155    2552 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 03:19:24.790135 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.789159    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 03:19:24.790135 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.789163    2552 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 03:19:24.790135 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.789166    2552 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 03:19:24.790135 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.789169    2552 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 03:19:24.790135 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.789171    2552 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 03:19:24.790135 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.789174    2552 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 03:19:24.790135 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.789180    2552 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 03:19:24.790642 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.789183    2552 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 03:19:24.790642 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.789186    2552 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 03:19:24.790642 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.789190    2552 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 03:19:24.790642 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.789195    2552 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 03:19:24.790642 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.789198    2552 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 03:19:24.790642 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.789201    2552 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 03:19:24.790642 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.789205    2552 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 03:19:24.790642 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.789209    2552 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 03:19:24.790642 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.789212    2552 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 03:19:24.790642 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.789216    2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 03:19:24.790642 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.789221    2552 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 03:19:24.790642 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.789224    2552 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 03:19:24.790642 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.789227    2552 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 03:19:24.790642 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.789229    2552 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 03:19:24.790642 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.789233    2552 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 03:19:24.790642 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.789235    2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 03:19:24.790642 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.789238    2552 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 03:19:24.790642 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.789241    2552 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 03:19:24.790642 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.789244    2552 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 03:19:24.790642 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.789247    2552 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 03:19:24.791124 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.789250    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 03:19:24.791124 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.789252    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 03:19:24.791124 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.789255    2552 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 03:19:24.791124 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.789259    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 03:19:24.791124 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.789262    2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 03:19:24.791124 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.789265    2552 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 03:19:24.791124 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.789268    2552 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 03:19:24.791124 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.789270    2552 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 03:19:24.791124 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.789273    2552 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 03:19:24.791124 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.789276    2552 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 03:19:24.791124 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.789278    2552 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 03:19:24.791124 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.789281    2552 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 03:19:24.791124 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.789284    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 03:19:24.791124 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.789286    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 03:19:24.791124 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.789289    2552 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 03:19:24.791124 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.789294    2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 03:19:24.791124 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.790462    2552 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 03:19:24.791124 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.790473    2552 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 03:19:24.791124 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.790476    2552 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 03:19:24.791124 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.790480    2552 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 03:19:24.791613 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.790483    2552 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 03:19:24.791613 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.790486    2552 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 03:19:24.791613 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.790489    2552 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 03:19:24.791613 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.790491    2552 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 03:19:24.791613 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.790494    2552 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 03:19:24.791613 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.790497    2552 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 03:19:24.791613 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.790500    2552 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 03:19:24.791613 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.790503    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 03:19:24.791613 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.790505    2552 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 03:19:24.791613 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.790511    2552 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 03:19:24.791613 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.790515    2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 03:19:24.791613 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.790519    2552 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 03:19:24.791613 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.790522    2552 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 03:19:24.791613 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.790524    2552 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 03:19:24.791613 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.790529    2552 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 03:19:24.791613 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.790533    2552 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 03:19:24.791613 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.790537    2552 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 03:19:24.791613 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.790540    2552 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 03:19:24.791613 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.790543    2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 03:19:24.792092 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.790546    2552 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 03:19:24.792092 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.790549    2552 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 03:19:24.792092 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.790555    2552 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 03:19:24.792092 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.790558    2552 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 03:19:24.792092 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.790561    2552 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 03:19:24.792092 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.790564    2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 03:19:24.792092 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.790567    2552 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 03:19:24.792092 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.790570    2552 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 03:19:24.792092 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.790573    2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 03:19:24.792092 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.790576    2552 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 03:19:24.792092 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.790580    2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 03:19:24.792092 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.790584    2552 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 03:19:24.792092 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.790587    2552 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 03:19:24.792092 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.790590    2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 03:19:24.792092 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.790606    2552 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 03:19:24.792092 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.790612    2552 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 03:19:24.792092 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.790615    2552 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 03:19:24.792092 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.790618    2552 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 03:19:24.792092 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.790621    2552 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 03:19:24.792568 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.790624    2552 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 03:19:24.792568 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.790627    2552 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 03:19:24.792568 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.790629    2552 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 03:19:24.792568 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.790632    2552 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 03:19:24.792568 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.790635    2552 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 03:19:24.792568 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.790638    2552 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 03:19:24.792568 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.790641    2552 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 03:19:24.792568 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.790644    2552 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 03:19:24.792568 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.790650    2552 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 03:19:24.792568 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.790653    2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 03:19:24.792568 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.790656    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 03:19:24.792568 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.790659    2552 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 03:19:24.792568 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.790662    2552 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 03:19:24.792568 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.790665    2552 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 03:19:24.792568 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.790668    2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 03:19:24.792568 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.790671    2552 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 03:19:24.792568 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.790673    2552 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 03:19:24.792568 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.790676    2552 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 03:19:24.792568 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.790679    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 03:19:24.792568 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.790682    2552 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 03:19:24.793088 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.790687    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 03:19:24.793088 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.790690    2552 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 03:19:24.793088 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.790693    2552 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 03:19:24.793088 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.790696    2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 03:19:24.793088 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.790700    2552 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 03:19:24.793088 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.790702    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 03:19:24.793088 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.790705    2552 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 03:19:24.793088 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.790708    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 03:19:24.793088 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.790710    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 03:19:24.793088 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.790713    2552 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 03:19:24.793088 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.790716    2552 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 03:19:24.793088 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.790720    2552 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 03:19:24.793088 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.790727    2552 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 03:19:24.793088 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.790731    2552 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 03:19:24.793088 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.790735    2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 03:19:24.793088 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.790738    2552 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 03:19:24.793088 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.790741    2552 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 03:19:24.793088 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.790744    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 03:19:24.793088 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.790747    2552 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 03:19:24.793088 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.790751    2552 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 03:19:24.793583 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.790754    2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 03:19:24.793583 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.790757    2552 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 03:19:24.793583 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.790760    2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 03:19:24.793583 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.790763    2552 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 03:19:24.793583 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.790768    2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 03:19:24.793583 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.790771    2552 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 03:19:24.793583 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.790774    2552 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 03:19:24.793583 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.790777    2552 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 03:19:24.793583 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.790779    2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 03:19:24.793583 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.790782    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 03:19:24.793583 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.790785    2552 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 03:19:24.793583 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.790788    2552 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 03:19:24.793583 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.790790    2552 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 03:19:24.793583 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.790793    2552 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 03:19:24.793583 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.790796    2552 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 03:19:24.793583 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.790799    2552 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 03:19:24.793583 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.790802    2552 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 03:19:24.793583 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.791793    2552 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 03:19:24.793583 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.791799    2552 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 03:19:24.794068 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.791804    2552 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 03:19:24.794068 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.791808    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 03:19:24.794068 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.791811    2552 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 03:19:24.794068 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.791814    2552 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 03:19:24.794068 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.791818    2552 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 03:19:24.794068 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.791820    2552 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 03:19:24.794068 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.791823    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 03:19:24.794068 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.791826    2552 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 03:19:24.794068 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.791829    2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 03:19:24.794068 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.791831    2552 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 03:19:24.794068 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.791837    2552 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 03:19:24.794068 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.791840    2552 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 03:19:24.794068 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.791843    2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 03:19:24.794068 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.791846    2552 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 03:19:24.794068 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.791849    2552 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 03:19:24.794068 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.791851    2552 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 03:19:24.794068 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.791854    2552 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 03:19:24.794068 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.791857    2552 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 03:19:24.794068 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.791859    2552 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 03:19:24.794527 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.791862    2552 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 03:19:24.794527 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.791865    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 03:19:24.794527 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.791868    2552 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 03:19:24.794527 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.791871    2552 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 03:19:24.794527 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.791876    2552 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 03:19:24.794527 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.791879    2552 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 03:19:24.794527 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.791882    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 03:19:24.794527 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.791884    2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 03:19:24.794527 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.791887    2552 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 03:19:24.794527 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.791890    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 03:19:24.794527 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.791893    2552 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 03:19:24.794527 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.791896    2552 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 03:19:24.794527 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.791898    2552 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 03:19:24.794527 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.791901    2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 03:19:24.794527 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.791904    2552 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 03:19:24.794527 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.791906    2552 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 03:19:24.794527 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.791909    2552 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 03:19:24.794527 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.791914    2552 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 03:19:24.794527 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.791917    2552 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 03:19:24.794527 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.791919    2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 03:19:24.795033 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.791922    2552 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 03:19:24.795033 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.791924    2552 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 03:19:24.795033 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.791927    2552 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 03:19:24.795033 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.791929    2552 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 03:19:24.795033 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.791932    2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 03:19:24.795033 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.791935    2552 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 03:19:24.795033 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.791938    2552 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 03:19:24.795033 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.791941    2552 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 03:19:24.795033 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.791945    2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 03:19:24.795033 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.791950    2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 03:19:24.795033 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.791952    2552 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 03:19:24.795033 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.791955    2552 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 03:19:24.795033 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.791958    2552 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 03:19:24.795033 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.791961    2552 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 03:19:24.795033 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.791964    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 03:19:24.795033 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.791967    2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 03:19:24.795033 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.791969    2552 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 03:19:24.795033 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.791972    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 03:19:24.795033 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.791975    2552 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 03:19:24.795033 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.791978    2552 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 03:19:24.795560 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.791981    2552 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 03:19:24.795560 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.791986    2552 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 03:19:24.795560 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.791988    2552 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 03:19:24.795560 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.791991    2552 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 03:19:24.795560 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.791994    2552 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 03:19:24.795560 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.791996    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 03:19:24.795560 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.791999    2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 03:19:24.795560 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.792002    2552 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 03:19:24.795560 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.792004    2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 03:19:24.795560 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.792007    2552 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 03:19:24.795560 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.792010    2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 03:19:24.795560 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.792013    2552 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 03:19:24.795560 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.792016    2552 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 03:19:24.795560 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.792018    2552 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 03:19:24.795560 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.792023    2552 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 03:19:24.795560 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.792026    2552 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 03:19:24.795560 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.792029    2552 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 03:19:24.795560 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.792033    2552 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 03:19:24.795560 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.792037    2552 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 03:19:24.796051 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.792042    2552 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 03:19:24.796051 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.792045    2552 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 03:19:24.796051 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.792048    2552 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 03:19:24.796051 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.792051    2552 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 03:19:24.796051 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.792053    2552 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 03:19:24.796051 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.792056    2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 03:19:24.796051 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.792062    2552 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 03:19:24.796051 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.792064    2552 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 03:19:24.796051 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.792067    2552 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 03:19:24.796051 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.792070    2552 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 03:19:24.796051 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.792072    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 03:19:24.796051 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.792076    2552 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 03:19:24.796051 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.792078    2552 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 03:19:24.796051 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.792081    2552 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 03:19:24.796051 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.792084    2552 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 03:19:24.796051 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.792086    2552 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 03:19:24.796051 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.792089    2552 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 03:19:24.796051 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.792092    2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 03:19:24.796051 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.792095    2552 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 03:19:24.796051 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.792343    2552 flags.go:64] FLAG: --address="0.0.0.0"
Sep 17 03:19:24.796536 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.792393    2552 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 17 03:19:24.796536 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.792400    2552 flags.go:64] FLAG: --anonymous-auth="true"
Sep 17 03:19:24.796536 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.792405    2552 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 17 03:19:24.796536 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.792416    2552 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 17 03:19:24.796536 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.792421    2552 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 17 03:19:24.796536 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.792426    2552 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 17 03:19:24.796536 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.792431    2552 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 17 03:19:24.796536 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.792434    2552 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 17 03:19:24.796536 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.792477    2552 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 17 03:19:24.796536 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.792482    2552 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 17 03:19:24.796536 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.792486    2552 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 17 03:19:24.796536 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.792489    2552 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 17 03:19:24.796536 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.792492    2552 flags.go:64] FLAG: --cgroup-root=""
Sep 17 03:19:24.796536 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.792498    2552 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 17 03:19:24.796536 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.792503    2552 flags.go:64] FLAG: --client-ca-file=""
Sep 17 03:19:24.796536 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.792507    2552 flags.go:64] FLAG: --cloud-provider="external"
Sep 17 03:19:24.796536 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.792512    2552 flags.go:64] FLAG: --cluster-dns="[]"
Sep 17 03:19:24.796536 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.792519    2552 flags.go:64] FLAG: --cluster-domain=""
Sep 17 03:19:24.796536 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.792522    2552 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 17 03:19:24.796536 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.792526    2552 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 17 03:19:24.796536 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.792530    2552 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 17 03:19:24.796536 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.792535    2552 flags.go:64] FLAG: --container-log-max-files="5"
Sep 17 03:19:24.796536 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.792541    2552 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 17 03:19:24.796536 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.792546    2552 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 17 03:19:24.797140 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.792550    2552 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 17 03:19:24.797140 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.792553    2552 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 17 03:19:24.797140 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.792556    2552 flags.go:64] FLAG: --contention-profiling="false"
Sep 17 03:19:24.797140 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.792559    2552 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 17 03:19:24.797140 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.792562    2552 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 17 03:19:24.797140 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.792565    2552 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 17 03:19:24.797140 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.792569    2552 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 17 03:19:24.797140 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.792573    2552 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 17 03:19:24.797140 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.792579    2552 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 17 03:19:24.797140 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.792582    2552 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 17 03:19:24.797140 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.792585    2552 flags.go:64] FLAG: --enable-load-reader="false"
Sep 17 03:19:24.797140 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.792588    2552 flags.go:64] FLAG: --enable-server="true"
Sep 17 03:19:24.797140 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.792591    2552 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 17 03:19:24.797140 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.792611    2552 flags.go:64] FLAG: --event-burst="100"
Sep 17 03:19:24.797140 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.792614    2552 flags.go:64] FLAG: --event-qps="50"
Sep 17 03:19:24.797140 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.792617    2552 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 17 03:19:24.797140 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.792620    2552 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 17 03:19:24.797140 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.792623    2552 flags.go:64] FLAG: --eviction-hard=""
Sep 17 03:19:24.797140 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.792627    2552 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 17 03:19:24.797140 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.792630    2552 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 17 03:19:24.797140 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.792633    2552 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 17 03:19:24.797140 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.792636    2552 flags.go:64] FLAG: --eviction-soft=""
Sep 17 03:19:24.797140 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.792639    2552 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 17 03:19:24.797140 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.792642    2552 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 17 03:19:24.797140 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.792647    2552 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 17 03:19:24.797808 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.792649    2552 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 17 03:19:24.797808 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.792652    2552 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 17 03:19:24.797808 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.792655    2552 flags.go:64] FLAG: --fail-swap-on="true"
Sep 17 03:19:24.797808 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.792658    2552 flags.go:64] FLAG: --feature-gates=""
Sep 17 03:19:24.797808 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.792662    2552 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 17 03:19:24.797808 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.792665    2552 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 17 03:19:24.797808 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.792670    2552 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 17 03:19:24.797808 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.792674    2552 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 17 03:19:24.797808 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.792677    2552 flags.go:64] FLAG: --healthz-port="10248"
Sep 17 03:19:24.797808 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.792679    2552 flags.go:64] FLAG: --help="false"
Sep 17 03:19:24.797808 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.792682    2552 flags.go:64] FLAG: --hostname-override="ip-10-0-139-53.ec2.internal"
Sep 17 03:19:24.797808 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.792685    2552 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 17 03:19:24.797808 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.792688    2552 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 17 03:19:24.797808 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.792691    2552 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 17 03:19:24.797808 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.792695    2552 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 17 03:19:24.797808 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.792699    2552 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 17 03:19:24.797808 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.792702    2552 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 17 03:19:24.797808 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.792706    2552 flags.go:64] FLAG: --image-service-endpoint=""
Sep 17 03:19:24.797808 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.792708    2552 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 17 03:19:24.797808 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.792711    2552 flags.go:64] FLAG: --kube-api-burst="100"
Sep 17 03:19:24.797808 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.792714    2552 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 17 03:19:24.797808 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.792717    2552 flags.go:64] FLAG: --kube-api-qps="50"
Sep 17 03:19:24.797808 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.792720    2552 flags.go:64] FLAG: --kube-reserved=""
Sep 17 03:19:24.797808 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.792724    2552 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 17 03:19:24.798393 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.792727    2552 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 17 03:19:24.798393 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.792730    2552 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 17 03:19:24.798393 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.792733    2552 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 17 03:19:24.798393 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.792735    2552 flags.go:64] FLAG: --lock-file=""
Sep 17 03:19:24.798393 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.792738    2552 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 17 03:19:24.798393 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.792741    2552 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 17 03:19:24.798393 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.792744    2552 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 17 03:19:24.798393 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.792749    2552 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 17 03:19:24.798393 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.792752    2552 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 17 03:19:24.798393 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.792755    2552 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 17 03:19:24.798393 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.792758    2552 flags.go:64] FLAG: --logging-format="text"
Sep 17 03:19:24.798393 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.792761    2552 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 17 03:19:24.798393 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.792764    2552 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 17 03:19:24.798393 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.792767    2552 flags.go:64] FLAG: --manifest-url=""
Sep 17 03:19:24.798393 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.792773    2552 flags.go:64] FLAG: --manifest-url-header=""
Sep 17 03:19:24.798393 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.792778    2552 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 17 03:19:24.798393 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.792781    2552 flags.go:64] FLAG: --max-open-files="1000000"
Sep 17 03:19:24.798393 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.792785    2552 flags.go:64] FLAG: --max-pods="110"
Sep 17 03:19:24.798393 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.792789    2552 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 17 03:19:24.798393 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.792792    2552 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 17 03:19:24.798393 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.792795    2552 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 17 03:19:24.798393 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.792799    2552 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 17 03:19:24.798393 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.792802    2552 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 17 03:19:24.798393 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.792807    2552 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 17 03:19:24.798393 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.792809    2552 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 17 03:19:24.799055 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.792818    2552 flags.go:64] FLAG: --node-status-max-images="50"
Sep 17 03:19:24.799055 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.792821    2552 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 17 03:19:24.799055 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.792824    2552 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 17 03:19:24.799055 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.792827    2552 flags.go:64] FLAG: --pod-cidr=""
Sep 17 03:19:24.799055 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.792830    2552 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 17 03:19:24.799055 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.792836    2552 flags.go:64] FLAG: --pod-manifest-path=""
Sep 17 03:19:24.799055 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.792839    2552 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 17 03:19:24.799055 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.792842    2552 flags.go:64] FLAG: --pods-per-core="0"
Sep 17 03:19:24.799055 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.792845    2552 flags.go:64] FLAG: --port="10250"
Sep 17 03:19:24.799055 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.792848    2552 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 17 03:19:24.799055 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.792851    2552 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-080bb1a2ce0391de2"
Sep 17 03:19:24.799055 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.792854    2552 flags.go:64] FLAG: --qos-reserved=""
Sep 17 03:19:24.799055 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.792857    2552 flags.go:64] FLAG: --read-only-port="10255"
Sep 17 03:19:24.799055 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.792860    2552 flags.go:64] FLAG: --register-node="true"
Sep 17 03:19:24.799055 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.792862    2552 flags.go:64] FLAG: --register-with-taints=""
Sep 17 03:19:24.799055 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.792866    2552 flags.go:64] FLAG: --registry-burst="10"
Sep 17 03:19:24.799055 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.792870    2552 flags.go:64] FLAG: --registry-qps="5"
Sep 17 03:19:24.799055 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.792873    2552 flags.go:64] FLAG: --reserved-cpus=""
Sep 17 03:19:24.799055 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.792875    2552 flags.go:64] FLAG: --reserved-memory=""
Sep 17 03:19:24.799055 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.792879    2552 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 17 03:19:24.799055 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.792883    2552 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 17 03:19:24.799055 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.792885    2552 flags.go:64] FLAG: --rotate-certificates="false"
Sep 17 03:19:24.799055 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.792891    2552 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 17 03:19:24.799055 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.792894    2552 flags.go:64] FLAG: --runonce="false"
Sep 17 03:19:24.799055 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.792897    2552 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 17 03:19:24.799706 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.792900    2552 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 17 03:19:24.799706 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.792903    2552 flags.go:64] FLAG: --seccomp-default="false"
Sep 17 03:19:24.799706 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.792906    2552 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 17 03:19:24.799706 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.792909    2552 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 17 03:19:24.799706 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.792912    2552 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 17 03:19:24.799706 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.792916    2552 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 17 03:19:24.799706 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.792919    2552 flags.go:64] FLAG: --storage-driver-password="root"
Sep 17 03:19:24.799706 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.792922    2552 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 17 03:19:24.799706 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.792925    2552 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 17 03:19:24.799706 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.792928    2552 flags.go:64] FLAG: --storage-driver-user="root"
Sep 17 03:19:24.799706 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.792931    2552 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 17 03:19:24.799706 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.792934    2552 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 17 03:19:24.799706 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.792937    2552 flags.go:64] FLAG: --system-cgroups=""
Sep 17 03:19:24.799706 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.792940    2552 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 17 03:19:24.799706 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.792946    2552 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 17 03:19:24.799706 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.792949    2552 flags.go:64] FLAG: --tls-cert-file=""
Sep 17 03:19:24.799706 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.792952    2552 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 17 03:19:24.799706 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.792956    2552 flags.go:64] FLAG: --tls-min-version=""
Sep 17 03:19:24.799706 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.792959    2552 flags.go:64] FLAG: --tls-private-key-file=""
Sep 17 03:19:24.799706 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.792961    2552 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 17 03:19:24.799706 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.792964    2552 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 17 03:19:24.799706 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.792967    2552 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 17 03:19:24.799706 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.792970    2552 flags.go:64] FLAG: --v="2"
Sep 17 03:19:24.799706 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.792973    2552 flags.go:64] FLAG: --version="false"
Sep 17 03:19:24.799706 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.792978    2552 flags.go:64] FLAG: --vmodule=""
Sep 17 03:19:24.800348 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.792981    2552 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 17 03:19:24.800348 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.792985    2552 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 17 03:19:24.800348 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.793118    2552 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 03:19:24.800348 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.793123    2552 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 03:19:24.800348 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.793128    2552 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 03:19:24.800348 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.793131    2552 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 03:19:24.800348 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.793134    2552 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 03:19:24.800348 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.793137    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 03:19:24.800348 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.793140    2552 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 03:19:24.800348 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.793145    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 03:19:24.800348 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.793150    2552 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 03:19:24.800348 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.793154    2552 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 03:19:24.800348 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.793158    2552 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 03:19:24.800348 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.793162    2552 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 03:19:24.800348 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.793166    2552 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 03:19:24.800348 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.793168    2552 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 03:19:24.800348 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.793173    2552 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 03:19:24.800348 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.793176    2552 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 03:19:24.800348 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.793180    2552 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 03:19:24.800919 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.793182    2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 03:19:24.800919 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.793185    2552 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 03:19:24.800919 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.793188    2552 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 03:19:24.800919 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.793191    2552 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 03:19:24.800919 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.793194    2552 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 03:19:24.800919 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.793197    2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 03:19:24.800919 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.793199    2552 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 03:19:24.800919 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.793202    2552 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 03:19:24.800919 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.793205    2552 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 03:19:24.800919 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.793207    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 03:19:24.800919 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.793210    2552 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 03:19:24.800919 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.793213    2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 03:19:24.800919 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.793215    2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 03:19:24.800919 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.793219    2552 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 03:19:24.800919 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.793221    2552 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 03:19:24.800919 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.793224    2552 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 03:19:24.800919 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.793226    2552 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 03:19:24.800919 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.793231    2552 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 03:19:24.800919 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.793233    2552 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 03:19:24.800919 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.793236    2552 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 03:19:24.801423 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.793238    2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 03:19:24.801423 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.793241    2552 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 03:19:24.801423 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.793243    2552 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 03:19:24.801423 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.793246    2552 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 03:19:24.801423 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.793249    2552 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 03:19:24.801423 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.793252    2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 03:19:24.801423 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.793254    2552 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 03:19:24.801423 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.793257    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 03:19:24.801423 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.793259    2552 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 03:19:24.801423 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.793263    2552 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 03:19:24.801423 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.793265    2552 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 03:19:24.801423 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.793268    2552 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 03:19:24.801423 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.793271    2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 03:19:24.801423 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.793273    2552 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 03:19:24.801423 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.793276    2552 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 03:19:24.801423 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.793278    2552 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 03:19:24.801423 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.793281    2552 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 03:19:24.801423 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.793283    2552 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 03:19:24.801423 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.793286    2552 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 03:19:24.801423 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.793289    2552 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 03:19:24.802018 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.793291    2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 03:19:24.802018 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.793294    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 03:19:24.802018 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.793296    2552 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 03:19:24.802018 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.793299    2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 03:19:24.802018 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.793301    2552 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 03:19:24.802018 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.793304    2552 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 03:19:24.802018 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.793307    2552 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 03:19:24.802018 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.793309    2552 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 03:19:24.802018 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.793312    2552 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 03:19:24.802018 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.793316    2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 03:19:24.802018 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.793319    2552 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 03:19:24.802018 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.793321    2552 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 03:19:24.802018 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.793324    2552 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 03:19:24.802018 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.793326    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 03:19:24.802018 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.793329    2552 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 03:19:24.802018 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.793331    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 03:19:24.802018 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.793334    2552 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 03:19:24.802018 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.793337    2552 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 03:19:24.802018 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.793339    2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 03:19:24.802018 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.793342    2552 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 03:19:24.802514 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.793345    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 03:19:24.802514 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.793347    2552 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 03:19:24.802514 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.793350    2552 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 03:19:24.802514 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.793353    2552 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 03:19:24.802514 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.793356    2552 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 03:19:24.802514 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.793358    2552 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 03:19:24.802514 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.793361    2552 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 03:19:24.802514 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.793363    2552 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 03:19:24.802514 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.793366    2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 03:19:24.802514 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.793369    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 03:19:24.802514 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.793371    2552 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 03:19:24.802514 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.793376    2552 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 03:19:24.802514 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.793379    2552 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 03:19:24.802514 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.793383    2552 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 03:19:24.802514 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.793385    2552 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 03:19:24.802514 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.793388    2552 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 03:19:24.802514 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.793391    2552 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 03:19:24.802514 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.793394    2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 03:19:24.802514 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.793397    2552 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 03:19:24.803042 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.793400    2552 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 03:19:24.803042 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.793402    2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 03:19:24.803042 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.793406    2552 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 03:19:24.803042 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.794022    2552 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 03:19:24.803042 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.801510    2552 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 17 03:19:24.803042 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.801529    2552 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 17 03:19:24.803042 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.801555    2552 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 17 03:19:24.803042 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.801561    2552 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 17 03:19:24.803042 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.801641    2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 03:19:24.803042 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.801647    2552 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 03:19:24.803042 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.801650    2552 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 03:19:24.803042 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.801653    2552 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 03:19:24.803042 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.801656    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 03:19:24.803042 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.801659    2552 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 03:19:24.803398 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.801663    2552 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 03:19:24.803398 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.801666    2552 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 03:19:24.803398 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.801669    2552 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 03:19:24.803398 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.801672    2552 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 03:19:24.803398 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.801675    2552 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 03:19:24.803398 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.801678    2552 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 03:19:24.803398 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.801680    2552 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 03:19:24.803398 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.801683    2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 03:19:24.803398 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.801686    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 03:19:24.803398 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.801688    2552 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 03:19:24.803398 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.801691    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 03:19:24.803398 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.801694    2552 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 03:19:24.803398 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.801696    2552 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 03:19:24.803398 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.801699    2552 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 03:19:24.803398 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.801703    2552 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 03:19:24.803398 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.801708    2552 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 03:19:24.803398 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.801711    2552 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 03:19:24.803398 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.801714    2552 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 03:19:24.803398 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.801718    2552 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 03:19:24.803881 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.801720    2552 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 03:19:24.803881 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.801724    2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 03:19:24.803881 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.801726    2552 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 03:19:24.803881 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.801730    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 03:19:24.803881 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.801733    2552 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 03:19:24.803881 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.801736    2552 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 03:19:24.803881 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.801739    2552 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 03:19:24.803881 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.801741    2552 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 03:19:24.803881 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.801744    2552 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 03:19:24.803881 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.801746    2552 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 03:19:24.803881 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.801749    2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 03:19:24.803881 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.801752    2552 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 03:19:24.803881 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.801755    2552 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 03:19:24.803881 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.801757    2552 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 03:19:24.803881 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.801760    2552 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 03:19:24.803881 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.801762    2552 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 03:19:24.803881 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.801765    2552 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 03:19:24.803881 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.801767    2552 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 03:19:24.803881 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.801770    2552 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 03:19:24.803881 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.801773    2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 03:19:24.804372 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.801777    2552 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 03:19:24.804372 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.801780    2552 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 03:19:24.804372 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.801783    2552 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 03:19:24.804372 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.801786    2552 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 03:19:24.804372 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.801789    2552 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 03:19:24.804372 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.801792    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 03:19:24.804372 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.801795    2552 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 03:19:24.804372 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.801798    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 03:19:24.804372 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.801802    2552 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 03:19:24.804372 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.801805    2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 03:19:24.804372 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.801808    2552 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 03:19:24.804372 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.801810    2552 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 03:19:24.804372 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.801814    2552 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 03:19:24.804372 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.801817    2552 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 03:19:24.804372 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.801820    2552 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 03:19:24.804372 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.801823    2552 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 03:19:24.804372 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.801826    2552 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 03:19:24.804372 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.801830    2552 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 03:19:24.804372 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.801834    2552 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 03:19:24.804372 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.801837    2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 03:19:24.804898 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.801840    2552 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 03:19:24.804898 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.801843    2552 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 03:19:24.804898 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.801846    2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 03:19:24.804898 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.801848    2552 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 03:19:24.804898 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.801851    2552 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 03:19:24.804898 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.801853    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 03:19:24.804898 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.801856    2552 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 03:19:24.804898 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.801859    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 03:19:24.804898 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.801862    2552 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 03:19:24.804898 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.801864    2552 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 03:19:24.804898 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.801867    2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 03:19:24.804898 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.801870    2552 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 03:19:24.804898 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.801872    2552 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 03:19:24.804898 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.801875    2552 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 03:19:24.804898 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.801877    2552 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 03:19:24.804898 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.801880    2552 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 03:19:24.804898 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.801882    2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 03:19:24.804898 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.801885    2552 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 03:19:24.804898 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.801887    2552 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 03:19:24.804898 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.801890    2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 03:19:24.805390 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.801893    2552 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 03:19:24.805390 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.801896    2552 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 03:19:24.805390 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.801898    2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 03:19:24.805390 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.801901    2552 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 03:19:24.805390 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.801904    2552 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 03:19:24.805390 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.801907    2552 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 03:19:24.805390 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.801910    2552 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 03:19:24.805390 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.801913    2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 03:19:24.805390 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.801915    2552 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 03:19:24.805390 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.801918    2552 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 03:19:24.805390 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.801921    2552 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 03:19:24.805390 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.801923    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 03:19:24.805390 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.801926    2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 03:19:24.805390 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.801928    2552 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 03:19:24.805390 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.801934    2552 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 03:19:24.805790 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.802060    2552 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 03:19:24.805790 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.802065    2552 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 03:19:24.805790 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.802069    2552 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 03:19:24.805790 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.802071    2552 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 03:19:24.805790 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.802074    2552 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 03:19:24.805790 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.802077    2552 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 03:19:24.805790 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.802080    2552 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 03:19:24.805790 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.802083    2552 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 03:19:24.805790 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.802086    2552 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 03:19:24.805790 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.802088    2552 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 03:19:24.805790 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.802091    2552 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 03:19:24.805790 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.802094    2552 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 03:19:24.805790 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.802096    2552 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 03:19:24.805790 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.802099    2552 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 03:19:24.805790 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.802101    2552 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 03:19:24.805790 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.802104    2552 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 03:19:24.805790 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.802106    2552 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 03:19:24.805790 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.802109    2552 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 03:19:24.805790 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.802112    2552 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 03:19:24.805790 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.802115    2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 03:19:24.806273 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.802117    2552 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 03:19:24.806273 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.802120    2552 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 03:19:24.806273 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.802122    2552 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 03:19:24.806273 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.802126    2552 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 03:19:24.806273 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.802129    2552 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 03:19:24.806273 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.802132    2552 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 03:19:24.806273 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.802135    2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 03:19:24.806273 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.802137    2552 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 03:19:24.806273 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.802140    2552 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 03:19:24.806273 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.802143    2552 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 03:19:24.806273 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.802145    2552 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 03:19:24.806273 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.802148    2552 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 03:19:24.806273 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.802151    2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 03:19:24.806273 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.802153    2552 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 03:19:24.806273 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.802156    2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 03:19:24.806273 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.802159    2552 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 03:19:24.806273 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.802161    2552 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 03:19:24.806273 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.802163    2552 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 03:19:24.806273 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.802166    2552 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 03:19:24.806755 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.802169    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 03:19:24.806755 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.802173    2552 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 03:19:24.806755 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.802178    2552 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 03:19:24.806755 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.802181    2552 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 03:19:24.806755 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.802184    2552 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 03:19:24.806755 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.802187    2552 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 03:19:24.806755 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.802190    2552 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 03:19:24.806755 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.802193    2552 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 03:19:24.806755 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.802196    2552 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 03:19:24.806755 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.802198    2552 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 03:19:24.806755 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.802201    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 03:19:24.806755 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.802203    2552 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 03:19:24.806755 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.802206    2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 03:19:24.806755 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.802208    2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 03:19:24.806755 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.802211    2552 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 03:19:24.806755 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.802214    2552 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 03:19:24.806755 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.802216    2552 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 03:19:24.806755 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.802220    2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 03:19:24.806755 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.802223    2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 03:19:24.807219 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.802225    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 03:19:24.807219 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.802228    2552 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 03:19:24.807219 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.802231    2552 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 03:19:24.807219 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.802233    2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 03:19:24.807219 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.802236    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 03:19:24.807219 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.802238    2552 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 03:19:24.807219 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.802241    2552 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 03:19:24.807219 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.802243    2552 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 03:19:24.807219 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.802246    2552 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 03:19:24.807219 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.802249    2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 03:19:24.807219 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.802251    2552 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 03:19:24.807219 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.802254    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 03:19:24.807219 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.802256    2552 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 03:19:24.807219 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.802259    2552 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 03:19:24.807219 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.802261    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 03:19:24.807219 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.802264    2552 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 03:19:24.807219 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.802267    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 03:19:24.807219 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.802270    2552 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 03:19:24.807219 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.802273    2552 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 03:19:24.807776 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.802277    2552 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 03:19:24.807776 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.802280    2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 03:19:24.807776 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.802283    2552 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 03:19:24.807776 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.802285    2552 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 03:19:24.807776 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.802289    2552 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 03:19:24.807776 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.802291    2552 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 03:19:24.807776 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.802294    2552 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 03:19:24.807776 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.802297    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 03:19:24.807776 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.802299    2552 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 03:19:24.807776 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.802302    2552 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 03:19:24.807776 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.802305    2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 03:19:24.807776 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.802308    2552 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 03:19:24.807776 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.802320    2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 03:19:24.807776 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.802323    2552 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 03:19:24.807776 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.802326    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 03:19:24.807776 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.802328    2552 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 03:19:24.807776 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.802331    2552 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 03:19:24.807776 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.802333    2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 03:19:24.807776 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.802336    2552 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 03:19:24.807776 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.802339    2552 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 03:19:24.808310 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.802341    2552 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 03:19:24.808310 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:24.802344    2552 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 03:19:24.808310 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.802349    2552 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 03:19:24.808310 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.802491    2552 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 17 03:19:24.808310 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.804798    2552 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 17 03:19:24.808310 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.805653    2552 server.go:1019] "Starting client certificate rotation"
Sep 17 03:19:24.808310 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.805759    2552 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 03:19:24.808310 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.805802    2552 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 03:19:24.808310 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.808066    2552 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 03:19:24.811441 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.811420    2552 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 03:19:24.820535 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.820513    2552 log.go:25] "Validated CRI v1 runtime API"
Sep 17 03:19:24.824145 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.824119    2552 log.go:25] "Validated CRI v1 image API"
Sep 17 03:19:24.827118 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.827088    2552 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 17 03:19:24.831961 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.831932    2552 fs.go:135] Filesystem UUIDs: map[0c1e12a9-b518-4a3c-956d-5c4fff0bdb36:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 b5f94db5-6d70-4bd4-9356-7d45c9b26c5f:/dev/nvme0n1p4]
Sep 17 03:19:24.831961 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.831956    2552 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 17 03:19:24.834792 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.834768    2552 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 03:19:24.838179 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.838047    2552 manager.go:216] Machine: {Timestamp:2026-09-17 03:19:24.836770458 +0000 UTC m=+0.303177974 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3095897 MemoryCapacity:33164472320 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2b7b7964c2630d928cc882d07fd800 SystemUUID:ec2b7b79-64c2-630d-928c-c882d07fd800 BootID:ce3cc11f-fea3-4424-b9b3-3eb75a20bffa Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582234112 Type:vfs Inodes:4048397 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632894464 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:f4:87:e2:db Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:f4:87:e2:db Speed:0 Mtu:9001} {Name:ovs-system MacAddress:5a:b7:76:08:a5:ef Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164472320 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 17 03:19:24.838179 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.838169    2552 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 17 03:19:24.838288 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.838275    2552 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 17 03:19:24.840553 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.840517    2552 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 17 03:19:24.840713 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.840557    2552 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-53.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 17 03:19:24.841304 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.841291    2552 topology_manager.go:138] "Creating topology manager with none policy"
Sep 17 03:19:24.841337 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.841307    2552 container_manager_linux.go:309] "Creating device plugin manager"
Sep 17 03:19:24.841337 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.841316    2552 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 03:19:24.841958 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.841946    2552 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 03:19:24.842886 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.842874    2552 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 17 03:19:24.847019 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.847002    2552 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 03:19:24.847161 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.847151    2552 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 17 03:19:24.847191 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.847187    2552 kubelet.go:486] "Attempting to sync node with API server"
Sep 17 03:19:24.847217 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.847200    2552 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 17 03:19:24.847217 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.847213    2552 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 17 03:19:24.847293 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.847224    2552 kubelet.go:398] "Adding apiserver pod source"
Sep 17 03:19:24.847293 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.847236    2552 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 17 03:19:24.847687 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.847669    2552 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 03:19:24.847742 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.847697    2552 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 03:19:24.849716 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.849697    2552 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 17 03:19:24.852146 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.852126    2552 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 17 03:19:24.852239 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.852167    2552 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 17 03:19:24.853853 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.853837    2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 17 03:19:24.853853 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.853859    2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 17 03:19:24.854001 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.853869    2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 17 03:19:24.854001 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.853877    2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 17 03:19:24.854001 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.853890    2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 17 03:19:24.854001 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.853898    2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 17 03:19:24.854001 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.853920    2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 17 03:19:24.854001 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.853928    2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 17 03:19:24.854001 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.853938    2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 17 03:19:24.854001 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.853946    2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 17 03:19:24.854001 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.853970    2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 17 03:19:24.854001 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.853979    2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 17 03:19:24.854676 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.854664    2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 17 03:19:24.854727 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.854679    2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 17 03:19:24.857641 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.857587    2552 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-139-53.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 17 03:19:24.857762 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:19:24.857674    2552 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-139-53.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 17 03:19:24.857762 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:19:24.857665    2552 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 17 03:19:24.858858 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.858843    2552 server.go:1268] "Started kubelet"
Sep 17 03:19:24.858950 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.858899    2552 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 17 03:19:24.859062 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.859004    2552 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 17 03:19:24.859101 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.859087    2552 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 17 03:19:24.859521 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.859504    2552 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-jcz5c"
Sep 17 03:19:24.859800 ip-10-0-139-53 systemd[1]: Started Kubernetes Kubelet.
Sep 17 03:19:24.860975 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.860940    2552 server.go:310] "Adding debug handlers to kubelet server"
Sep 17 03:19:24.861205 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.861101    2552 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 17 03:19:24.864111 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:19:24.864087    2552 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 17 03:19:24.866361 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:19:24.865555    2552 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-53.ec2.internal.18d5fd82ba0fd3b8  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-53.ec2.internal,UID:ip-10-0-139-53.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-139-53.ec2.internal,},FirstTimestamp:2026-09-17 03:19:24.858815416 +0000 UTC m=+0.325222932,LastTimestamp:2026-09-17 03:19:24.858815416 +0000 UTC m=+0.325222932,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-53.ec2.internal,}"
Sep 17 03:19:24.866702 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.866686    2552 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-jcz5c"
Sep 17 03:19:24.866912 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.866893    2552 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 17 03:19:24.867391 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.867375    2552 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 17 03:19:24.867504 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:19:24.867479    2552 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-53.ec2.internal\" not found"
Sep 17 03:19:24.868039 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.868021    2552 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 17 03:19:24.868108 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.868042    2552 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 17 03:19:24.868967 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.868018    2552 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 17 03:19:24.869102 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.869089    2552 reconstruct.go:97] "Volume reconstruction finished"
Sep 17 03:19:24.869163 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.869103    2552 reconciler.go:29] "Reconciler: start to sync state"
Sep 17 03:19:24.869295 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.869277    2552 factory.go:55] Registering systemd factory
Sep 17 03:19:24.869374 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.869325    2552 factory.go:223] Registration of the systemd container factory successfully
Sep 17 03:19:24.869709 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.869692    2552 factory.go:153] Registering CRI-O factory
Sep 17 03:19:24.869785 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.869712    2552 factory.go:223] Registration of the crio container factory successfully
Sep 17 03:19:24.869785 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.869768    2552 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 17 03:19:24.869871 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.869786    2552 factory.go:103] Registering Raw factory
Sep 17 03:19:24.869871 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.869811    2552 manager.go:1196] Started watching for new ooms in manager
Sep 17 03:19:24.870192 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.870179    2552 manager.go:349] Starting recovery of all containers
Sep 17 03:19:24.871975 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:19:24.871951    2552 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-139-53.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 17 03:19:24.872269 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:19:24.872128    2552 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 17 03:19:24.882350 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.882156    2552 manager.go:354] Recovery completed
Sep 17 03:19:24.887325 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.887310    2552 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 03:19:24.889407 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.889387    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-53.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 03:19:24.889407 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.889418    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-53.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 03:19:24.889586 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.889428    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-53.ec2.internal" event="NodeHasSufficientPID"
Sep 17 03:19:24.890123 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.890104    2552 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 17 03:19:24.890123 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.890119    2552 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 17 03:19:24.890250 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.890136    2552 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 03:19:24.890250 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.890120    2552 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 03:19:24.891153 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.891136    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-53.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 03:19:24.891240 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.891165    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-53.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 03:19:24.891240 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.891175    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-53.ec2.internal" event="NodeHasSufficientPID"
Sep 17 03:19:24.892288 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.892274    2552 policy_none.go:49] "None policy: Start"
Sep 17 03:19:24.892359 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.892292    2552 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 17 03:19:24.892359 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.892303    2552 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 17 03:19:24.894342 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.894327    2552 policy_none.go:47] "Start"
Sep 17 03:19:24.932407 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.932378    2552 manager.go:337] "Starting Device Plugin manager"
Sep 17 03:19:24.954003 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:19:24.932522    2552 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 17 03:19:24.954003 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.932539    2552 server.go:87] "Starting device plugin registration server"
Sep 17 03:19:24.954003 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.932807    2552 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 17 03:19:24.954003 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.932824    2552 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 17 03:19:24.954003 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.932948    2552 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 17 03:19:24.954003 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.933588    2552 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 17 03:19:24.954003 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.933624    2552 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 17 03:19:24.954003 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:19:24.933661    2552 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 17 03:19:24.954003 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:19:24.933710    2552 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-139-53.ec2.internal\" not found"
Sep 17 03:19:24.977434 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.977399    2552 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 17 03:19:24.978759 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.978738    2552 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 17 03:19:24.978846 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.978816    2552 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 17 03:19:24.978902 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.978849    2552 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 17 03:19:24.978902 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:19:24.978890    2552 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 17 03:19:24.981442 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:24.981414    2552 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 03:19:25.033561 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:25.033475    2552 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 03:19:25.035155 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:25.035134    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-53.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 03:19:25.035259 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:25.035167    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-53.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 03:19:25.035259 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:25.035181    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-53.ec2.internal" event="NodeHasSufficientPID"
Sep 17 03:19:25.035259 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:25.035208    2552 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-139-53.ec2.internal"
Sep 17 03:19:25.041627 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:25.041590    2552 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-139-53.ec2.internal"
Sep 17 03:19:25.041627 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:19:25.041631    2552 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-139-53.ec2.internal\": node \"ip-10-0-139-53.ec2.internal\" not found"
Sep 17 03:19:25.058099 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:19:25.058072    2552 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-53.ec2.internal\" not found"
Sep 17 03:19:25.079925 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:25.079894    2552 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-53.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-139-53.ec2.internal"]
Sep 17 03:19:25.080055 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:25.079982    2552 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 03:19:25.082093 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:25.082078    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-53.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 03:19:25.082151 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:25.082110    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-53.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 03:19:25.082151 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:25.082123    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-53.ec2.internal" event="NodeHasSufficientPID"
Sep 17 03:19:25.084560 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:25.084536    2552 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 03:19:25.084685 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:25.084663    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-53.ec2.internal"
Sep 17 03:19:25.084728 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:25.084697    2552 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 03:19:25.085482 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:25.085468    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-53.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 03:19:25.085564 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:25.085495    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-53.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 03:19:25.085564 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:25.085505    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-53.ec2.internal" event="NodeHasSufficientPID"
Sep 17 03:19:25.085564 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:25.085472    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-53.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 03:19:25.085564 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:25.085565    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-53.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 03:19:25.085721 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:25.085575    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-53.ec2.internal" event="NodeHasSufficientPID"
Sep 17 03:19:25.087964 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:25.087947    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-53.ec2.internal"
Sep 17 03:19:25.088081 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:25.087978    2552 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 03:19:25.088873 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:25.088856    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-53.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 03:19:25.088958 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:25.088885    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-53.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 03:19:25.088958 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:25.088895    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-53.ec2.internal" event="NodeHasSufficientPID"
Sep 17 03:19:25.119420 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:19:25.119385    2552 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-53.ec2.internal\" not found" node="ip-10-0-139-53.ec2.internal"
Sep 17 03:19:25.123926 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:19:25.123904    2552 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-53.ec2.internal\" not found" node="ip-10-0-139-53.ec2.internal"
Sep 17 03:19:25.158972 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:19:25.158941    2552 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-53.ec2.internal\" not found"
Sep 17 03:19:25.169853 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:25.169814    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ce8d1a2b33a64a4f4ec3c7eebb5340d9-config\") pod \"kube-apiserver-proxy-ip-10-0-139-53.ec2.internal\" (UID: \"ce8d1a2b33a64a4f4ec3c7eebb5340d9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-53.ec2.internal"
Sep 17 03:19:25.169853 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:25.169855    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a15e02334a54e08adac5d9c153f86344-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-53.ec2.internal\" (UID: \"a15e02334a54e08adac5d9c153f86344\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-53.ec2.internal"
Sep 17 03:19:25.170065 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:25.169873    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a15e02334a54e08adac5d9c153f86344-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-53.ec2.internal\" (UID: \"a15e02334a54e08adac5d9c153f86344\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-53.ec2.internal"
Sep 17 03:19:25.259884 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:19:25.259837    2552 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-53.ec2.internal\" not found"
Sep 17 03:19:25.270260 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:25.270226    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ce8d1a2b33a64a4f4ec3c7eebb5340d9-config\") pod \"kube-apiserver-proxy-ip-10-0-139-53.ec2.internal\" (UID: \"ce8d1a2b33a64a4f4ec3c7eebb5340d9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-53.ec2.internal"
Sep 17 03:19:25.270260 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:25.270264    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a15e02334a54e08adac5d9c153f86344-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-53.ec2.internal\" (UID: \"a15e02334a54e08adac5d9c153f86344\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-53.ec2.internal"
Sep 17 03:19:25.270439 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:25.270281    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a15e02334a54e08adac5d9c153f86344-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-53.ec2.internal\" (UID: \"a15e02334a54e08adac5d9c153f86344\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-53.ec2.internal"
Sep 17 03:19:25.270439 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:25.270328    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a15e02334a54e08adac5d9c153f86344-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-53.ec2.internal\" (UID: \"a15e02334a54e08adac5d9c153f86344\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-53.ec2.internal"
Sep 17 03:19:25.270439 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:25.270335    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a15e02334a54e08adac5d9c153f86344-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-53.ec2.internal\" (UID: \"a15e02334a54e08adac5d9c153f86344\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-53.ec2.internal"
Sep 17 03:19:25.270439 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:25.270353    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ce8d1a2b33a64a4f4ec3c7eebb5340d9-config\") pod \"kube-apiserver-proxy-ip-10-0-139-53.ec2.internal\" (UID: \"ce8d1a2b33a64a4f4ec3c7eebb5340d9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-53.ec2.internal"
Sep 17 03:19:25.360709 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:19:25.360628    2552 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-53.ec2.internal\" not found"
Sep 17 03:19:25.425136 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:25.425096    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-53.ec2.internal"
Sep 17 03:19:25.427492 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:25.427469    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-53.ec2.internal"
Sep 17 03:19:25.461565 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:19:25.461520    2552 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-53.ec2.internal\" not found"
Sep 17 03:19:25.564768 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:19:25.564724    2552 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-53.ec2.internal\" not found"
Sep 17 03:19:25.665074 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:19:25.664997    2552 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-53.ec2.internal\" not found"
Sep 17 03:19:25.766790 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:19:25.766748    2552 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-53.ec2.internal\" not found"
Sep 17 03:19:25.807229 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:25.807187    2552 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 17 03:19:25.807946 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:25.807360    2552 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 17 03:19:25.866972 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:19:25.866932    2552 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-53.ec2.internal\" not found"
Sep 17 03:19:25.873026 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:25.872988    2552 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-16 03:14:25 +0000 UTC" deadline="2028-06-06 18:02:59.607760516 +0000 UTC"
Sep 17 03:19:25.873026 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:25.873018    2552 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15086h43m33.734746109s"
Sep 17 03:19:25.969155 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:19:25.969055    2552 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-53.ec2.internal\" not found"
Sep 17 03:19:26.019039 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.019007    2552 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 03:19:26.069221 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:19:26.069179    2552 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-53.ec2.internal\" not found"
Sep 17 03:19:26.169623 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:19:26.169573    2552 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-53.ec2.internal\" not found"
Sep 17 03:19:26.218442 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.218410    2552 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 03:19:26.253369 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:26.253326    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce8d1a2b33a64a4f4ec3c7eebb5340d9.slice/crio-9a59679dbff14a7fc737771279c85b6250ac88e937c6a530edb5e0b4e4c95a5e WatchSource:0}: Error finding container 9a59679dbff14a7fc737771279c85b6250ac88e937c6a530edb5e0b4e4c95a5e: Status 404 returned error can't find the container with id 9a59679dbff14a7fc737771279c85b6250ac88e937c6a530edb5e0b4e4c95a5e
Sep 17 03:19:26.253956 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:26.253936    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda15e02334a54e08adac5d9c153f86344.slice/crio-bf6862b1ff1d082d2d51ac628f9f3c1bc4463c855c3e3b0911957dabfdd772a3 WatchSource:0}: Error finding container bf6862b1ff1d082d2d51ac628f9f3c1bc4463c855c3e3b0911957dabfdd772a3: Status 404 returned error can't find the container with id bf6862b1ff1d082d2d51ac628f9f3c1bc4463c855c3e3b0911957dabfdd772a3
Sep 17 03:19:26.259337 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.259318    2552 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 03:19:26.269769 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:19:26.269731    2552 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-53.ec2.internal\" not found"
Sep 17 03:19:26.370564 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:19:26.370525    2552 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-53.ec2.internal\" not found"
Sep 17 03:19:26.416047 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.416015    2552 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 03:19:26.469007 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.468958    2552 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-53.ec2.internal"
Sep 17 03:19:26.481713 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.481669    2552 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 03:19:26.483510 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.483490    2552 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-53.ec2.internal"
Sep 17 03:19:26.490375 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.490354    2552 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 03:19:26.849000 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.848969    2552 apiserver.go:52] "Watching apiserver"
Sep 17 03:19:26.858140 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.858099    2552 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 17 03:19:26.858783 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.858756    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-operator/iptables-alerter-mvmw5","kube-system/global-pull-secret-syncer-4944l","kube-system/kube-apiserver-proxy-ip-10-0-139-53.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qspfk","openshift-image-registry/node-ca-rld2q","openshift-multus/multus-additional-cni-plugins-nkrxz","openshift-multus/multus-wds8n","openshift-ovn-kubernetes/ovnkube-node-ldhl4","kube-system/konnectivity-agent-8mzx5","openshift-cluster-node-tuning-operator/tuned-vkdcx","openshift-dns/node-resolver-gxmbv","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-53.ec2.internal","openshift-multus/network-metrics-daemon-xl9wh","openshift-network-diagnostics/network-check-target-9d4m6"]
Sep 17 03:19:26.861695 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.861665    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-wds8n"
Sep 17 03:19:26.864606 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.864574    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 17 03:19:26.864824 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.864807    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 17 03:19:26.865416 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.865392    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xl9wh"
Sep 17 03:19:26.865507 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.865470    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 17 03:19:26.865507 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:19:26.865486    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xl9wh" podUID="2758c0c8-e1ff-454a-9f0b-43cc8e1f8609"
Sep 17 03:19:26.865622 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.865552    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 17 03:19:26.865622 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.865471    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-br6zv\""
Sep 17 03:19:26.867233 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.867202    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9d4m6"
Sep 17 03:19:26.867378 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:19:26.867276    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9d4m6" podUID="9e5c90e7-d9db-4205-9bc6-c6790b31e53f"
Sep 17 03:19:26.867378 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.867327    2552 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 17 03:19:26.869878 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.869852    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qspfk"
Sep 17 03:19:26.872726 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.872665    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-rld2q"
Sep 17 03:19:26.875650 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.874590    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 17 03:19:26.875650 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.875588    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-fsfpl\""
Sep 17 03:19:26.875854 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.875782    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 17 03:19:26.875914 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.875870    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 17 03:19:26.876329 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.876105    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 17 03:19:26.876329 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.876140    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-nkrxz"
Sep 17 03:19:26.878315 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.876562    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-qz5vk\""
Sep 17 03:19:26.878315 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.876772    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 17 03:19:26.878315 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.876832    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 17 03:19:26.878315 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.877155    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c6a03737-feb7-48d2-82ba-5a2188f80980-multus-conf-dir\") pod \"multus-wds8n\" (UID: \"c6a03737-feb7-48d2-82ba-5a2188f80980\") " pod="openshift-multus/multus-wds8n"
Sep 17 03:19:26.878315 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.877189    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/27724468-aead-4a4a-9ec1-c11d8fa2bb7c-socket-dir\") pod \"aws-ebs-csi-driver-node-qspfk\" (UID: \"27724468-aead-4a4a-9ec1-c11d8fa2bb7c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qspfk"
Sep 17 03:19:26.878315 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.877217    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/27724468-aead-4a4a-9ec1-c11d8fa2bb7c-device-dir\") pod \"aws-ebs-csi-driver-node-qspfk\" (UID: \"27724468-aead-4a4a-9ec1-c11d8fa2bb7c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qspfk"
Sep 17 03:19:26.878315 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.877268    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c6a03737-feb7-48d2-82ba-5a2188f80980-host-var-lib-kubelet\") pod \"multus-wds8n\" (UID: \"c6a03737-feb7-48d2-82ba-5a2188f80980\") " pod="openshift-multus/multus-wds8n"
Sep 17 03:19:26.878315 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.877345    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c6a03737-feb7-48d2-82ba-5a2188f80980-system-cni-dir\") pod \"multus-wds8n\" (UID: \"c6a03737-feb7-48d2-82ba-5a2188f80980\") " pod="openshift-multus/multus-wds8n"
Sep 17 03:19:26.878315 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.877392    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vs279\" (UniqueName: \"kubernetes.io/projected/27724468-aead-4a4a-9ec1-c11d8fa2bb7c-kube-api-access-vs279\") pod \"aws-ebs-csi-driver-node-qspfk\" (UID: \"27724468-aead-4a4a-9ec1-c11d8fa2bb7c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qspfk"
Sep 17 03:19:26.878315 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.877429    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c6a03737-feb7-48d2-82ba-5a2188f80980-multus-cni-dir\") pod \"multus-wds8n\" (UID: \"c6a03737-feb7-48d2-82ba-5a2188f80980\") " pod="openshift-multus/multus-wds8n"
Sep 17 03:19:26.878315 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.877466    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c6a03737-feb7-48d2-82ba-5a2188f80980-cni-binary-copy\") pod \"multus-wds8n\" (UID: \"c6a03737-feb7-48d2-82ba-5a2188f80980\") " pod="openshift-multus/multus-wds8n"
Sep 17 03:19:26.878315 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.877497    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c6a03737-feb7-48d2-82ba-5a2188f80980-etc-kubernetes\") pod \"multus-wds8n\" (UID: \"c6a03737-feb7-48d2-82ba-5a2188f80980\") " pod="openshift-multus/multus-wds8n"
Sep 17 03:19:26.878315 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.877524    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c6a03737-feb7-48d2-82ba-5a2188f80980-cnibin\") pod \"multus-wds8n\" (UID: \"c6a03737-feb7-48d2-82ba-5a2188f80980\") " pod="openshift-multus/multus-wds8n"
Sep 17 03:19:26.878315 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.877558    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c6a03737-feb7-48d2-82ba-5a2188f80980-host-run-k8s-cni-cncf-io\") pod \"multus-wds8n\" (UID: \"c6a03737-feb7-48d2-82ba-5a2188f80980\") " pod="openshift-multus/multus-wds8n"
Sep 17 03:19:26.878315 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.877590    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c6a03737-feb7-48d2-82ba-5a2188f80980-hostroot\") pod \"multus-wds8n\" (UID: \"c6a03737-feb7-48d2-82ba-5a2188f80980\") " pod="openshift-multus/multus-wds8n"
Sep 17 03:19:26.878315 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.877770    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2758c0c8-e1ff-454a-9f0b-43cc8e1f8609-metrics-certs\") pod \"network-metrics-daemon-xl9wh\" (UID: \"2758c0c8-e1ff-454a-9f0b-43cc8e1f8609\") " pod="openshift-multus/network-metrics-daemon-xl9wh"
Sep 17 03:19:26.878315 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.877819    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c6a03737-feb7-48d2-82ba-5a2188f80980-host-run-netns\") pod \"multus-wds8n\" (UID: \"c6a03737-feb7-48d2-82ba-5a2188f80980\") " pod="openshift-multus/multus-wds8n"
Sep 17 03:19:26.878315 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.877848    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/27724468-aead-4a4a-9ec1-c11d8fa2bb7c-registration-dir\") pod \"aws-ebs-csi-driver-node-qspfk\" (UID: \"27724468-aead-4a4a-9ec1-c11d8fa2bb7c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qspfk"
Sep 17 03:19:26.879398 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.877885    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/27724468-aead-4a4a-9ec1-c11d8fa2bb7c-sys-fs\") pod \"aws-ebs-csi-driver-node-qspfk\" (UID: \"27724468-aead-4a4a-9ec1-c11d8fa2bb7c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qspfk"
Sep 17 03:19:26.879398 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.877920    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/27724468-aead-4a4a-9ec1-c11d8fa2bb7c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qspfk\" (UID: \"27724468-aead-4a4a-9ec1-c11d8fa2bb7c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qspfk"
Sep 17 03:19:26.879398 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.877953    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/27724468-aead-4a4a-9ec1-c11d8fa2bb7c-etc-selinux\") pod \"aws-ebs-csi-driver-node-qspfk\" (UID: \"27724468-aead-4a4a-9ec1-c11d8fa2bb7c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qspfk"
Sep 17 03:19:26.879398 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.877985    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c6a03737-feb7-48d2-82ba-5a2188f80980-host-var-lib-cni-multus\") pod \"multus-wds8n\" (UID: \"c6a03737-feb7-48d2-82ba-5a2188f80980\") " pod="openshift-multus/multus-wds8n"
Sep 17 03:19:26.879398 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.878504    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c6a03737-feb7-48d2-82ba-5a2188f80980-multus-daemon-config\") pod \"multus-wds8n\" (UID: \"c6a03737-feb7-48d2-82ba-5a2188f80980\") " pod="openshift-multus/multus-wds8n"
Sep 17 03:19:26.879398 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.878555    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c6a03737-feb7-48d2-82ba-5a2188f80980-host-run-multus-certs\") pod \"multus-wds8n\" (UID: \"c6a03737-feb7-48d2-82ba-5a2188f80980\") " pod="openshift-multus/multus-wds8n"
Sep 17 03:19:26.879398 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.878584    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2d4rr\" (UniqueName: \"kubernetes.io/projected/c6a03737-feb7-48d2-82ba-5a2188f80980-kube-api-access-2d4rr\") pod \"multus-wds8n\" (UID: \"c6a03737-feb7-48d2-82ba-5a2188f80980\") " pod="openshift-multus/multus-wds8n"
Sep 17 03:19:26.879398 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.878626    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlt4p\" (UniqueName: \"kubernetes.io/projected/2758c0c8-e1ff-454a-9f0b-43cc8e1f8609-kube-api-access-mlt4p\") pod \"network-metrics-daemon-xl9wh\" (UID: \"2758c0c8-e1ff-454a-9f0b-43cc8e1f8609\") " pod="openshift-multus/network-metrics-daemon-xl9wh"
Sep 17 03:19:26.879398 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.878653    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmr6b\" (UniqueName: \"kubernetes.io/projected/9e5c90e7-d9db-4205-9bc6-c6790b31e53f-kube-api-access-nmr6b\") pod \"network-check-target-9d4m6\" (UID: \"9e5c90e7-d9db-4205-9bc6-c6790b31e53f\") " pod="openshift-network-diagnostics/network-check-target-9d4m6"
Sep 17 03:19:26.879398 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.878682    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c6a03737-feb7-48d2-82ba-5a2188f80980-multus-socket-dir-parent\") pod \"multus-wds8n\" (UID: \"c6a03737-feb7-48d2-82ba-5a2188f80980\") " pod="openshift-multus/multus-wds8n"
Sep 17 03:19:26.879398 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.878705    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c6a03737-feb7-48d2-82ba-5a2188f80980-host-var-lib-cni-bin\") pod \"multus-wds8n\" (UID: \"c6a03737-feb7-48d2-82ba-5a2188f80980\") " pod="openshift-multus/multus-wds8n"
Sep 17 03:19:26.879398 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.879155    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-ll4qf\""
Sep 17 03:19:26.879398 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.879202    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 17 03:19:26.879398 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.879155    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 17 03:19:26.880996 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.880946    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-8mzx5"
Sep 17 03:19:26.881507 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.881488    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-mvmw5"
Sep 17 03:19:26.883007 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.882982    2552 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 03:19:26.884200 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.884180    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4944l"
Sep 17 03:19:26.884300 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:19:26.884262    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4944l" podUID="d1adf33b-0171-446e-aabe-f60fb74197e3"
Sep 17 03:19:26.884610 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.884585    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-5vpjv\""
Sep 17 03:19:26.884968 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.884952    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 17 03:19:26.885182 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.885162    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 17 03:19:26.885266 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.885228    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 17 03:19:26.885352 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.885267    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 17 03:19:26.885415 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.885267    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 17 03:19:26.885415 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.885385    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-6ncgd\""
Sep 17 03:19:26.888972 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.888950    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-vkdcx"
Sep 17 03:19:26.892021 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.891997    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 17 03:19:26.892207 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.892152    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-gxmbv"
Sep 17 03:19:26.892345 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.892326    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-7vs6m\""
Sep 17 03:19:26.892465 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.892445    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 17 03:19:26.894023 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.894005    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-ldhl4"
Sep 17 03:19:26.894980 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.894959    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 17 03:19:26.895098 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.895079    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 17 03:19:26.895226 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.895179    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-j56rg\""
Sep 17 03:19:26.896792 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.896764    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-c8mtd\""
Sep 17 03:19:26.896888 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.896826    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 17 03:19:26.898295 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.898274    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 17 03:19:26.898377 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.898326    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 17 03:19:26.898589 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.898572    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 17 03:19:26.898631 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.898584    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 17 03:19:26.898885 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.898862    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 17 03:19:26.901232 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.901212    2552 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-v6plx"
Sep 17 03:19:26.906646 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.906597    2552 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-v6plx"
Sep 17 03:19:26.970644 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.970613    2552 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 17 03:19:26.979989 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.979864    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d3e92f88-fcb1-4d0f-b653-77b7ec5681fd-run-openvswitch\") pod \"ovnkube-node-ldhl4\" (UID: \"d3e92f88-fcb1-4d0f-b653-77b7ec5681fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldhl4"
Sep 17 03:19:26.979989 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.979927    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d3e92f88-fcb1-4d0f-b653-77b7ec5681fd-log-socket\") pod \"ovnkube-node-ldhl4\" (UID: \"d3e92f88-fcb1-4d0f-b653-77b7ec5681fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldhl4"
Sep 17 03:19:26.979989 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.979985    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c6a03737-feb7-48d2-82ba-5a2188f80980-cnibin\") pod \"multus-wds8n\" (UID: \"c6a03737-feb7-48d2-82ba-5a2188f80980\") " pod="openshift-multus/multus-wds8n"
Sep 17 03:19:26.980291 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.980050    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c6a03737-feb7-48d2-82ba-5a2188f80980-cnibin\") pod \"multus-wds8n\" (UID: \"c6a03737-feb7-48d2-82ba-5a2188f80980\") " pod="openshift-multus/multus-wds8n"
Sep 17 03:19:26.980291 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.980087    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c6a03737-feb7-48d2-82ba-5a2188f80980-host-run-k8s-cni-cncf-io\") pod \"multus-wds8n\" (UID: \"c6a03737-feb7-48d2-82ba-5a2188f80980\") " pod="openshift-multus/multus-wds8n"
Sep 17 03:19:26.980291 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.980151    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c6a03737-feb7-48d2-82ba-5a2188f80980-hostroot\") pod \"multus-wds8n\" (UID: \"c6a03737-feb7-48d2-82ba-5a2188f80980\") " pod="openshift-multus/multus-wds8n"
Sep 17 03:19:26.980291 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.980174    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2758c0c8-e1ff-454a-9f0b-43cc8e1f8609-metrics-certs\") pod \"network-metrics-daemon-xl9wh\" (UID: \"2758c0c8-e1ff-454a-9f0b-43cc8e1f8609\") " pod="openshift-multus/network-metrics-daemon-xl9wh"
Sep 17 03:19:26.980291 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.980194    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4e98ab7d-ed30-48e6-9751-cc40209037aa-etc-sysctl-d\") pod \"tuned-vkdcx\" (UID: \"4e98ab7d-ed30-48e6-9751-cc40209037aa\") " pod="openshift-cluster-node-tuning-operator/tuned-vkdcx"
Sep 17 03:19:26.980291 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.980216    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c6a03737-feb7-48d2-82ba-5a2188f80980-host-run-netns\") pod \"multus-wds8n\" (UID: \"c6a03737-feb7-48d2-82ba-5a2188f80980\") " pod="openshift-multus/multus-wds8n"
Sep 17 03:19:26.980291 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.980234    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/27724468-aead-4a4a-9ec1-c11d8fa2bb7c-sys-fs\") pod \"aws-ebs-csi-driver-node-qspfk\" (UID: \"27724468-aead-4a4a-9ec1-c11d8fa2bb7c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qspfk"
Sep 17 03:19:26.980291 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.980244    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c6a03737-feb7-48d2-82ba-5a2188f80980-hostroot\") pod \"multus-wds8n\" (UID: \"c6a03737-feb7-48d2-82ba-5a2188f80980\") " pod="openshift-multus/multus-wds8n"
Sep 17 03:19:26.980740 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.980297    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c6a03737-feb7-48d2-82ba-5a2188f80980-host-run-k8s-cni-cncf-io\") pod \"multus-wds8n\" (UID: \"c6a03737-feb7-48d2-82ba-5a2188f80980\") " pod="openshift-multus/multus-wds8n"
Sep 17 03:19:26.980740 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.980250    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2a4ab360-f144-4999-97dc-6796bd4a44c2-agent-certs\") pod \"konnectivity-agent-8mzx5\" (UID: \"2a4ab360-f144-4999-97dc-6796bd4a44c2\") " pod="kube-system/konnectivity-agent-8mzx5"
Sep 17 03:19:26.980740 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.980337    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d3e92f88-fcb1-4d0f-b653-77b7ec5681fd-host-run-netns\") pod \"ovnkube-node-ldhl4\" (UID: \"d3e92f88-fcb1-4d0f-b653-77b7ec5681fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldhl4"
Sep 17 03:19:26.980740 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.980355    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c6a03737-feb7-48d2-82ba-5a2188f80980-host-run-netns\") pod \"multus-wds8n\" (UID: \"c6a03737-feb7-48d2-82ba-5a2188f80980\") " pod="openshift-multus/multus-wds8n"
Sep 17 03:19:26.980740 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.980362    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d3e92f88-fcb1-4d0f-b653-77b7ec5681fd-host-cni-netd\") pod \"ovnkube-node-ldhl4\" (UID: \"d3e92f88-fcb1-4d0f-b653-77b7ec5681fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldhl4"
Sep 17 03:19:26.980740 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:19:26.980359    2552 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 03:19:26.980740 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.980386    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d3e92f88-fcb1-4d0f-b653-77b7ec5681fd-env-overrides\") pod \"ovnkube-node-ldhl4\" (UID: \"d3e92f88-fcb1-4d0f-b653-77b7ec5681fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldhl4"
Sep 17 03:19:26.980740 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.980412    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/27724468-aead-4a4a-9ec1-c11d8fa2bb7c-etc-selinux\") pod \"aws-ebs-csi-driver-node-qspfk\" (UID: \"27724468-aead-4a4a-9ec1-c11d8fa2bb7c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qspfk"
Sep 17 03:19:26.980740 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.980416    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/27724468-aead-4a4a-9ec1-c11d8fa2bb7c-sys-fs\") pod \"aws-ebs-csi-driver-node-qspfk\" (UID: \"27724468-aead-4a4a-9ec1-c11d8fa2bb7c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qspfk"
Sep 17 03:19:26.980740 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:19:26.980443    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2758c0c8-e1ff-454a-9f0b-43cc8e1f8609-metrics-certs podName:2758c0c8-e1ff-454a-9f0b-43cc8e1f8609 nodeName:}" failed. No retries permitted until 2026-09-17 03:19:27.480418082 +0000 UTC m=+2.946825591 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2758c0c8-e1ff-454a-9f0b-43cc8e1f8609-metrics-certs") pod "network-metrics-daemon-xl9wh" (UID: "2758c0c8-e1ff-454a-9f0b-43cc8e1f8609") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 03:19:26.980740 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.980481    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/27724468-aead-4a4a-9ec1-c11d8fa2bb7c-etc-selinux\") pod \"aws-ebs-csi-driver-node-qspfk\" (UID: \"27724468-aead-4a4a-9ec1-c11d8fa2bb7c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qspfk"
Sep 17 03:19:26.980740 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.980506    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c6a03737-feb7-48d2-82ba-5a2188f80980-multus-daemon-config\") pod \"multus-wds8n\" (UID: \"c6a03737-feb7-48d2-82ba-5a2188f80980\") " pod="openshift-multus/multus-wds8n"
Sep 17 03:19:26.980740 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.980536    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmr6b\" (UniqueName: \"kubernetes.io/projected/9e5c90e7-d9db-4205-9bc6-c6790b31e53f-kube-api-access-nmr6b\") pod \"network-check-target-9d4m6\" (UID: \"9e5c90e7-d9db-4205-9bc6-c6790b31e53f\") " pod="openshift-network-diagnostics/network-check-target-9d4m6"
Sep 17 03:19:26.980740 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.980646    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0cc4aedf-1bb8-47c6-88f2-266bbbf99058-cnibin\") pod \"multus-additional-cni-plugins-nkrxz\" (UID: \"0cc4aedf-1bb8-47c6-88f2-266bbbf99058\") " pod="openshift-multus/multus-additional-cni-plugins-nkrxz"
Sep 17 03:19:26.980740 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.980681    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/045af8b4-e20b-46e7-bbcf-b5465710791d-host-slash\") pod \"iptables-alerter-mvmw5\" (UID: \"045af8b4-e20b-46e7-bbcf-b5465710791d\") " pod="openshift-network-operator/iptables-alerter-mvmw5"
Sep 17 03:19:26.980740 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.980716    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c6a03737-feb7-48d2-82ba-5a2188f80980-multus-socket-dir-parent\") pod \"multus-wds8n\" (UID: \"c6a03737-feb7-48d2-82ba-5a2188f80980\") " pod="openshift-multus/multus-wds8n"
Sep 17 03:19:26.981574 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.980757    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d1adf33b-0171-446e-aabe-f60fb74197e3-dbus\") pod \"global-pull-secret-syncer-4944l\" (UID: \"d1adf33b-0171-446e-aabe-f60fb74197e3\") " pod="kube-system/global-pull-secret-syncer-4944l"
Sep 17 03:19:26.981574 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.980785    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d1adf33b-0171-446e-aabe-f60fb74197e3-original-pull-secret\") pod \"global-pull-secret-syncer-4944l\" (UID: \"d1adf33b-0171-446e-aabe-f60fb74197e3\") " pod="kube-system/global-pull-secret-syncer-4944l"
Sep 17 03:19:26.981574 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.980872    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d3e92f88-fcb1-4d0f-b653-77b7ec5681fd-systemd-units\") pod \"ovnkube-node-ldhl4\" (UID: \"d3e92f88-fcb1-4d0f-b653-77b7ec5681fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldhl4"
Sep 17 03:19:26.981574 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.980910    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d3e92f88-fcb1-4d0f-b653-77b7ec5681fd-ovn-node-metrics-cert\") pod \"ovnkube-node-ldhl4\" (UID: \"d3e92f88-fcb1-4d0f-b653-77b7ec5681fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldhl4"
Sep 17 03:19:26.981574 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.980975    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c6a03737-feb7-48d2-82ba-5a2188f80980-multus-socket-dir-parent\") pod \"multus-wds8n\" (UID: \"c6a03737-feb7-48d2-82ba-5a2188f80980\") " pod="openshift-multus/multus-wds8n"
Sep 17 03:19:26.981574 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.980990    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c6a03737-feb7-48d2-82ba-5a2188f80980-multus-conf-dir\") pod \"multus-wds8n\" (UID: \"c6a03737-feb7-48d2-82ba-5a2188f80980\") " pod="openshift-multus/multus-wds8n"
Sep 17 03:19:26.981574 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.981019    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/27724468-aead-4a4a-9ec1-c11d8fa2bb7c-socket-dir\") pod \"aws-ebs-csi-driver-node-qspfk\" (UID: \"27724468-aead-4a4a-9ec1-c11d8fa2bb7c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qspfk"
Sep 17 03:19:26.981574 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.981044    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0cc4aedf-1bb8-47c6-88f2-266bbbf99058-system-cni-dir\") pod \"multus-additional-cni-plugins-nkrxz\" (UID: \"0cc4aedf-1bb8-47c6-88f2-266bbbf99058\") " pod="openshift-multus/multus-additional-cni-plugins-nkrxz"
Sep 17 03:19:26.981574 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.981069    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0cc4aedf-1bb8-47c6-88f2-266bbbf99058-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-nkrxz\" (UID: \"0cc4aedf-1bb8-47c6-88f2-266bbbf99058\") " pod="openshift-multus/multus-additional-cni-plugins-nkrxz"
Sep 17 03:19:26.981574 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.981096    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d3e92f88-fcb1-4d0f-b653-77b7ec5681fd-run-systemd\") pod \"ovnkube-node-ldhl4\" (UID: \"d3e92f88-fcb1-4d0f-b653-77b7ec5681fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldhl4"
Sep 17 03:19:26.981574 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.981098    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c6a03737-feb7-48d2-82ba-5a2188f80980-multus-conf-dir\") pod \"multus-wds8n\" (UID: \"c6a03737-feb7-48d2-82ba-5a2188f80980\") " pod="openshift-multus/multus-wds8n"
Sep 17 03:19:26.981574 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.981123    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d3e92f88-fcb1-4d0f-b653-77b7ec5681fd-host-run-ovn-kubernetes\") pod \"ovnkube-node-ldhl4\" (UID: \"d3e92f88-fcb1-4d0f-b653-77b7ec5681fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldhl4"
Sep 17 03:19:26.981574 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.981148    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4e98ab7d-ed30-48e6-9751-cc40209037aa-etc-kubernetes\") pod \"tuned-vkdcx\" (UID: \"4e98ab7d-ed30-48e6-9751-cc40209037aa\") " pod="openshift-cluster-node-tuning-operator/tuned-vkdcx"
Sep 17 03:19:26.981574 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.981177    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d3e92f88-fcb1-4d0f-b653-77b7ec5681fd-node-log\") pod \"ovnkube-node-ldhl4\" (UID: \"d3e92f88-fcb1-4d0f-b653-77b7ec5681fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldhl4"
Sep 17 03:19:26.981574 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.981182    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/27724468-aead-4a4a-9ec1-c11d8fa2bb7c-socket-dir\") pod \"aws-ebs-csi-driver-node-qspfk\" (UID: \"27724468-aead-4a4a-9ec1-c11d8fa2bb7c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qspfk"
Sep 17 03:19:26.981574 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.981200    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c6a03737-feb7-48d2-82ba-5a2188f80980-multus-daemon-config\") pod \"multus-wds8n\" (UID: \"c6a03737-feb7-48d2-82ba-5a2188f80980\") " pod="openshift-multus/multus-wds8n"
Sep 17 03:19:26.982418 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.981201    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d3e92f88-fcb1-4d0f-b653-77b7ec5681fd-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ldhl4\" (UID: \"d3e92f88-fcb1-4d0f-b653-77b7ec5681fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldhl4"
Sep 17 03:19:26.982418 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.981266    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d3e92f88-fcb1-4d0f-b653-77b7ec5681fd-ovnkube-script-lib\") pod \"ovnkube-node-ldhl4\" (UID: \"d3e92f88-fcb1-4d0f-b653-77b7ec5681fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldhl4"
Sep 17 03:19:26.982418 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.981299    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vs279\" (UniqueName: \"kubernetes.io/projected/27724468-aead-4a4a-9ec1-c11d8fa2bb7c-kube-api-access-vs279\") pod \"aws-ebs-csi-driver-node-qspfk\" (UID: \"27724468-aead-4a4a-9ec1-c11d8fa2bb7c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qspfk"
Sep 17 03:19:26.982418 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.981333    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4e98ab7d-ed30-48e6-9751-cc40209037aa-tmp\") pod \"tuned-vkdcx\" (UID: \"4e98ab7d-ed30-48e6-9751-cc40209037aa\") " pod="openshift-cluster-node-tuning-operator/tuned-vkdcx"
Sep 17 03:19:26.982418 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.981371    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c6a03737-feb7-48d2-82ba-5a2188f80980-cni-binary-copy\") pod \"multus-wds8n\" (UID: \"c6a03737-feb7-48d2-82ba-5a2188f80980\") " pod="openshift-multus/multus-wds8n"
Sep 17 03:19:26.982418 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.981396    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4e98ab7d-ed30-48e6-9751-cc40209037aa-etc-sysconfig\") pod \"tuned-vkdcx\" (UID: \"4e98ab7d-ed30-48e6-9751-cc40209037aa\") " pod="openshift-cluster-node-tuning-operator/tuned-vkdcx"
Sep 17 03:19:26.982418 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.981420    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4e98ab7d-ed30-48e6-9751-cc40209037aa-etc-tuned\") pod \"tuned-vkdcx\" (UID: \"4e98ab7d-ed30-48e6-9751-cc40209037aa\") " pod="openshift-cluster-node-tuning-operator/tuned-vkdcx"
Sep 17 03:19:26.982418 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.981443    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0cc4aedf-1bb8-47c6-88f2-266bbbf99058-tuning-conf-dir\") pod \"multus-additional-cni-plugins-nkrxz\" (UID: \"0cc4aedf-1bb8-47c6-88f2-266bbbf99058\") " pod="openshift-multus/multus-additional-cni-plugins-nkrxz"
Sep 17 03:19:26.982418 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.981468    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4e98ab7d-ed30-48e6-9751-cc40209037aa-etc-systemd\") pod \"tuned-vkdcx\" (UID: \"4e98ab7d-ed30-48e6-9751-cc40209037aa\") " pod="openshift-cluster-node-tuning-operator/tuned-vkdcx"
Sep 17 03:19:26.982418 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.981491    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4e98ab7d-ed30-48e6-9751-cc40209037aa-sys\") pod \"tuned-vkdcx\" (UID: \"4e98ab7d-ed30-48e6-9751-cc40209037aa\") " pod="openshift-cluster-node-tuning-operator/tuned-vkdcx"
Sep 17 03:19:26.982418 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.981514    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d3e92f88-fcb1-4d0f-b653-77b7ec5681fd-host-slash\") pod \"ovnkube-node-ldhl4\" (UID: \"d3e92f88-fcb1-4d0f-b653-77b7ec5681fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldhl4"
Sep 17 03:19:26.982418 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.981538    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/27724468-aead-4a4a-9ec1-c11d8fa2bb7c-registration-dir\") pod \"aws-ebs-csi-driver-node-qspfk\" (UID: \"27724468-aead-4a4a-9ec1-c11d8fa2bb7c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qspfk"
Sep 17 03:19:26.982418 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.981565    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0cc4aedf-1bb8-47c6-88f2-266bbbf99058-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-nkrxz\" (UID: \"0cc4aedf-1bb8-47c6-88f2-266bbbf99058\") " pod="openshift-multus/multus-additional-cni-plugins-nkrxz"
Sep 17 03:19:26.982418 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.981641    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/045af8b4-e20b-46e7-bbcf-b5465710791d-iptables-alerter-script\") pod \"iptables-alerter-mvmw5\" (UID: \"045af8b4-e20b-46e7-bbcf-b5465710791d\") " pod="openshift-network-operator/iptables-alerter-mvmw5"
Sep 17 03:19:26.982418 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.981673    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d1adf33b-0171-446e-aabe-f60fb74197e3-kubelet-config\") pod \"global-pull-secret-syncer-4944l\" (UID: \"d1adf33b-0171-446e-aabe-f60fb74197e3\") " pod="kube-system/global-pull-secret-syncer-4944l"
Sep 17 03:19:26.982418 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.981700    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/27724468-aead-4a4a-9ec1-c11d8fa2bb7c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qspfk\" (UID: \"27724468-aead-4a4a-9ec1-c11d8fa2bb7c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qspfk"
Sep 17 03:19:26.983186 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.981726    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/27724468-aead-4a4a-9ec1-c11d8fa2bb7c-registration-dir\") pod \"aws-ebs-csi-driver-node-qspfk\" (UID: \"27724468-aead-4a4a-9ec1-c11d8fa2bb7c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qspfk"
Sep 17 03:19:26.983186 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.981727    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c6a03737-feb7-48d2-82ba-5a2188f80980-host-var-lib-cni-multus\") pod \"multus-wds8n\" (UID: \"c6a03737-feb7-48d2-82ba-5a2188f80980\") " pod="openshift-multus/multus-wds8n"
Sep 17 03:19:26.983186 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.981787    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/27724468-aead-4a4a-9ec1-c11d8fa2bb7c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qspfk\" (UID: \"27724468-aead-4a4a-9ec1-c11d8fa2bb7c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qspfk"
Sep 17 03:19:26.983186 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.981841    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c6a03737-feb7-48d2-82ba-5a2188f80980-host-run-multus-certs\") pod \"multus-wds8n\" (UID: \"c6a03737-feb7-48d2-82ba-5a2188f80980\") " pod="openshift-multus/multus-wds8n"
Sep 17 03:19:26.983186 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.981862    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c6a03737-feb7-48d2-82ba-5a2188f80980-cni-binary-copy\") pod \"multus-wds8n\" (UID: \"c6a03737-feb7-48d2-82ba-5a2188f80980\") " pod="openshift-multus/multus-wds8n"
Sep 17 03:19:26.983186 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.981874    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2d4rr\" (UniqueName: \"kubernetes.io/projected/c6a03737-feb7-48d2-82ba-5a2188f80980-kube-api-access-2d4rr\") pod \"multus-wds8n\" (UID: \"c6a03737-feb7-48d2-82ba-5a2188f80980\") " pod="openshift-multus/multus-wds8n"
Sep 17 03:19:26.983186 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.981889    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c6a03737-feb7-48d2-82ba-5a2188f80980-host-run-multus-certs\") pod \"multus-wds8n\" (UID: \"c6a03737-feb7-48d2-82ba-5a2188f80980\") " pod="openshift-multus/multus-wds8n"
Sep 17 03:19:26.983186 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.981766    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c6a03737-feb7-48d2-82ba-5a2188f80980-host-var-lib-cni-multus\") pod \"multus-wds8n\" (UID: \"c6a03737-feb7-48d2-82ba-5a2188f80980\") " pod="openshift-multus/multus-wds8n"
Sep 17 03:19:26.983186 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.981914    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlt4p\" (UniqueName: \"kubernetes.io/projected/2758c0c8-e1ff-454a-9f0b-43cc8e1f8609-kube-api-access-mlt4p\") pod \"network-metrics-daemon-xl9wh\" (UID: \"2758c0c8-e1ff-454a-9f0b-43cc8e1f8609\") " pod="openshift-multus/network-metrics-daemon-xl9wh"
Sep 17 03:19:26.983186 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.981946    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0cc4aedf-1bb8-47c6-88f2-266bbbf99058-cni-binary-copy\") pod \"multus-additional-cni-plugins-nkrxz\" (UID: \"0cc4aedf-1bb8-47c6-88f2-266bbbf99058\") " pod="openshift-multus/multus-additional-cni-plugins-nkrxz"
Sep 17 03:19:26.983186 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.981995    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4e98ab7d-ed30-48e6-9751-cc40209037aa-etc-sysctl-conf\") pod \"tuned-vkdcx\" (UID: \"4e98ab7d-ed30-48e6-9751-cc40209037aa\") " pod="openshift-cluster-node-tuning-operator/tuned-vkdcx"
Sep 17 03:19:26.983186 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.982021    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4e98ab7d-ed30-48e6-9751-cc40209037aa-lib-modules\") pod \"tuned-vkdcx\" (UID: \"4e98ab7d-ed30-48e6-9751-cc40209037aa\") " pod="openshift-cluster-node-tuning-operator/tuned-vkdcx"
Sep 17 03:19:26.983186 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.982054    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c6a03737-feb7-48d2-82ba-5a2188f80980-host-var-lib-cni-bin\") pod \"multus-wds8n\" (UID: \"c6a03737-feb7-48d2-82ba-5a2188f80980\") " pod="openshift-multus/multus-wds8n"
Sep 17 03:19:26.983186 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.982082    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p99g5\" (UniqueName: \"kubernetes.io/projected/b7b79096-9bfc-4d82-9406-073a3f7961d0-kube-api-access-p99g5\") pod \"node-ca-rld2q\" (UID: \"b7b79096-9bfc-4d82-9406-073a3f7961d0\") " pod="openshift-image-registry/node-ca-rld2q"
Sep 17 03:19:26.983186 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.982107    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgbfd\" (UniqueName: \"kubernetes.io/projected/4e98ab7d-ed30-48e6-9751-cc40209037aa-kube-api-access-bgbfd\") pod \"tuned-vkdcx\" (UID: \"4e98ab7d-ed30-48e6-9751-cc40209037aa\") " pod="openshift-cluster-node-tuning-operator/tuned-vkdcx"
Sep 17 03:19:26.983186 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.982142    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ggvx\" (UniqueName: \"kubernetes.io/projected/b3bac7f7-fbb1-4cec-8a26-046b702f4d70-kube-api-access-7ggvx\") pod \"node-resolver-gxmbv\" (UID: \"b3bac7f7-fbb1-4cec-8a26-046b702f4d70\") " pod="openshift-dns/node-resolver-gxmbv"
Sep 17 03:19:26.983186 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.982173    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d3e92f88-fcb1-4d0f-b653-77b7ec5681fd-host-cni-bin\") pod \"ovnkube-node-ldhl4\" (UID: \"d3e92f88-fcb1-4d0f-b653-77b7ec5681fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldhl4"
Sep 17 03:19:26.984070 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.982180    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c6a03737-feb7-48d2-82ba-5a2188f80980-host-var-lib-cni-bin\") pod \"multus-wds8n\" (UID: \"c6a03737-feb7-48d2-82ba-5a2188f80980\") " pod="openshift-multus/multus-wds8n"
Sep 17 03:19:26.984070 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.982207    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdkqz\" (UniqueName: \"kubernetes.io/projected/d3e92f88-fcb1-4d0f-b653-77b7ec5681fd-kube-api-access-fdkqz\") pod \"ovnkube-node-ldhl4\" (UID: \"d3e92f88-fcb1-4d0f-b653-77b7ec5681fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldhl4"
Sep 17 03:19:26.984070 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.982236    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/27724468-aead-4a4a-9ec1-c11d8fa2bb7c-device-dir\") pod \"aws-ebs-csi-driver-node-qspfk\" (UID: \"27724468-aead-4a4a-9ec1-c11d8fa2bb7c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qspfk"
Sep 17 03:19:26.984070 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.982282    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9wqn\" (UniqueName: \"kubernetes.io/projected/0cc4aedf-1bb8-47c6-88f2-266bbbf99058-kube-api-access-s9wqn\") pod \"multus-additional-cni-plugins-nkrxz\" (UID: \"0cc4aedf-1bb8-47c6-88f2-266bbbf99058\") " pod="openshift-multus/multus-additional-cni-plugins-nkrxz"
Sep 17 03:19:26.984070 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.982325    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4e98ab7d-ed30-48e6-9751-cc40209037aa-etc-modprobe-d\") pod \"tuned-vkdcx\" (UID: \"4e98ab7d-ed30-48e6-9751-cc40209037aa\") " pod="openshift-cluster-node-tuning-operator/tuned-vkdcx"
Sep 17 03:19:26.984070 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.982348    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4e98ab7d-ed30-48e6-9751-cc40209037aa-run\") pod \"tuned-vkdcx\" (UID: \"4e98ab7d-ed30-48e6-9751-cc40209037aa\") " pod="openshift-cluster-node-tuning-operator/tuned-vkdcx"
Sep 17 03:19:26.984070 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.982356    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/27724468-aead-4a4a-9ec1-c11d8fa2bb7c-device-dir\") pod \"aws-ebs-csi-driver-node-qspfk\" (UID: \"27724468-aead-4a4a-9ec1-c11d8fa2bb7c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qspfk"
Sep 17 03:19:26.984070 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.982371    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4e98ab7d-ed30-48e6-9751-cc40209037aa-host\") pod \"tuned-vkdcx\" (UID: \"4e98ab7d-ed30-48e6-9751-cc40209037aa\") " pod="openshift-cluster-node-tuning-operator/tuned-vkdcx"
Sep 17 03:19:26.984070 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.982395    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d3e92f88-fcb1-4d0f-b653-77b7ec5681fd-ovnkube-config\") pod \"ovnkube-node-ldhl4\" (UID: \"d3e92f88-fcb1-4d0f-b653-77b7ec5681fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldhl4"
Sep 17 03:19:26.984070 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.982527    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c6a03737-feb7-48d2-82ba-5a2188f80980-host-var-lib-kubelet\") pod \"multus-wds8n\" (UID: \"c6a03737-feb7-48d2-82ba-5a2188f80980\") " pod="openshift-multus/multus-wds8n"
Sep 17 03:19:26.984070 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.982564    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b7b79096-9bfc-4d82-9406-073a3f7961d0-serviceca\") pod \"node-ca-rld2q\" (UID: \"b7b79096-9bfc-4d82-9406-073a3f7961d0\") " pod="openshift-image-registry/node-ca-rld2q"
Sep 17 03:19:26.984070 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.982590    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbvh5\" (UniqueName: \"kubernetes.io/projected/045af8b4-e20b-46e7-bbcf-b5465710791d-kube-api-access-nbvh5\") pod \"iptables-alerter-mvmw5\" (UID: \"045af8b4-e20b-46e7-bbcf-b5465710791d\") " pod="openshift-network-operator/iptables-alerter-mvmw5"
Sep 17 03:19:26.984070 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.982610    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c6a03737-feb7-48d2-82ba-5a2188f80980-host-var-lib-kubelet\") pod \"multus-wds8n\" (UID: \"c6a03737-feb7-48d2-82ba-5a2188f80980\") " pod="openshift-multus/multus-wds8n"
Sep 17 03:19:26.984070 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.982615    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b3bac7f7-fbb1-4cec-8a26-046b702f4d70-hosts-file\") pod \"node-resolver-gxmbv\" (UID: \"b3bac7f7-fbb1-4cec-8a26-046b702f4d70\") " pod="openshift-dns/node-resolver-gxmbv"
Sep 17 03:19:26.984070 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.982661    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d3e92f88-fcb1-4d0f-b653-77b7ec5681fd-host-kubelet\") pod \"ovnkube-node-ldhl4\" (UID: \"d3e92f88-fcb1-4d0f-b653-77b7ec5681fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldhl4"
Sep 17 03:19:26.984070 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.982686    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d3e92f88-fcb1-4d0f-b653-77b7ec5681fd-var-lib-openvswitch\") pod \"ovnkube-node-ldhl4\" (UID: \"d3e92f88-fcb1-4d0f-b653-77b7ec5681fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldhl4"
Sep 17 03:19:26.984070 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.982710    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d3e92f88-fcb1-4d0f-b653-77b7ec5681fd-run-ovn\") pod \"ovnkube-node-ldhl4\" (UID: \"d3e92f88-fcb1-4d0f-b653-77b7ec5681fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldhl4"
Sep 17 03:19:26.985071 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.982739    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c6a03737-feb7-48d2-82ba-5a2188f80980-system-cni-dir\") pod \"multus-wds8n\" (UID: \"c6a03737-feb7-48d2-82ba-5a2188f80980\") " pod="openshift-multus/multus-wds8n"
Sep 17 03:19:26.985071 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.982764    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2a4ab360-f144-4999-97dc-6796bd4a44c2-konnectivity-ca\") pod \"konnectivity-agent-8mzx5\" (UID: \"2a4ab360-f144-4999-97dc-6796bd4a44c2\") " pod="kube-system/konnectivity-agent-8mzx5"
Sep 17 03:19:26.985071 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.982788    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b7b79096-9bfc-4d82-9406-073a3f7961d0-host\") pod \"node-ca-rld2q\" (UID: \"b7b79096-9bfc-4d82-9406-073a3f7961d0\") " pod="openshift-image-registry/node-ca-rld2q"
Sep 17 03:19:26.985071 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.982814    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c6a03737-feb7-48d2-82ba-5a2188f80980-multus-cni-dir\") pod \"multus-wds8n\" (UID: \"c6a03737-feb7-48d2-82ba-5a2188f80980\") " pod="openshift-multus/multus-wds8n"
Sep 17 03:19:26.985071 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.982838    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c6a03737-feb7-48d2-82ba-5a2188f80980-etc-kubernetes\") pod \"multus-wds8n\" (UID: \"c6a03737-feb7-48d2-82ba-5a2188f80980\") " pod="openshift-multus/multus-wds8n"
Sep 17 03:19:26.985071 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.982891    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4e98ab7d-ed30-48e6-9751-cc40209037aa-var-lib-kubelet\") pod \"tuned-vkdcx\" (UID: \"4e98ab7d-ed30-48e6-9751-cc40209037aa\") " pod="openshift-cluster-node-tuning-operator/tuned-vkdcx"
Sep 17 03:19:26.985071 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.982929    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3bac7f7-fbb1-4cec-8a26-046b702f4d70-tmp-dir\") pod \"node-resolver-gxmbv\" (UID: \"b3bac7f7-fbb1-4cec-8a26-046b702f4d70\") " pod="openshift-dns/node-resolver-gxmbv"
Sep 17 03:19:26.985071 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.982974    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d3e92f88-fcb1-4d0f-b653-77b7ec5681fd-etc-openvswitch\") pod \"ovnkube-node-ldhl4\" (UID: \"d3e92f88-fcb1-4d0f-b653-77b7ec5681fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldhl4"
Sep 17 03:19:26.985071 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.983120    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c6a03737-feb7-48d2-82ba-5a2188f80980-multus-cni-dir\") pod \"multus-wds8n\" (UID: \"c6a03737-feb7-48d2-82ba-5a2188f80980\") " pod="openshift-multus/multus-wds8n"
Sep 17 03:19:26.985071 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.983142    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c6a03737-feb7-48d2-82ba-5a2188f80980-system-cni-dir\") pod \"multus-wds8n\" (UID: \"c6a03737-feb7-48d2-82ba-5a2188f80980\") " pod="openshift-multus/multus-wds8n"
Sep 17 03:19:26.985071 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.983167    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c6a03737-feb7-48d2-82ba-5a2188f80980-etc-kubernetes\") pod \"multus-wds8n\" (UID: \"c6a03737-feb7-48d2-82ba-5a2188f80980\") " pod="openshift-multus/multus-wds8n"
Sep 17 03:19:26.986217 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.986144    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-53.ec2.internal" event={"ID":"ce8d1a2b33a64a4f4ec3c7eebb5340d9","Type":"ContainerStarted","Data":"9a59679dbff14a7fc737771279c85b6250ac88e937c6a530edb5e0b4e4c95a5e"}
Sep 17 03:19:26.987345 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:19:26.987321    2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 03:19:26.987474 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:19:26.987350    2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 03:19:26.987474 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:19:26.987370    2552 projected.go:196] Error preparing data for projected volume kube-api-access-nmr6b for pod openshift-network-diagnostics/network-check-target-9d4m6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 03:19:26.987474 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:19:26.987454    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9e5c90e7-d9db-4205-9bc6-c6790b31e53f-kube-api-access-nmr6b podName:9e5c90e7-d9db-4205-9bc6-c6790b31e53f nodeName:}" failed. No retries permitted until 2026-09-17 03:19:27.487433629 +0000 UTC m=+2.953841151 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-nmr6b" (UniqueName: "kubernetes.io/projected/9e5c90e7-d9db-4205-9bc6-c6790b31e53f-kube-api-access-nmr6b") pod "network-check-target-9d4m6" (UID: "9e5c90e7-d9db-4205-9bc6-c6790b31e53f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 03:19:26.988700 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.988382    2552 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 17 03:19:26.989307 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.988873    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-53.ec2.internal" event={"ID":"a15e02334a54e08adac5d9c153f86344","Type":"ContainerStarted","Data":"bf6862b1ff1d082d2d51ac628f9f3c1bc4463c855c3e3b0911957dabfdd772a3"}
Sep 17 03:19:26.992996 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.992916    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vs279\" (UniqueName: \"kubernetes.io/projected/27724468-aead-4a4a-9ec1-c11d8fa2bb7c-kube-api-access-vs279\") pod \"aws-ebs-csi-driver-node-qspfk\" (UID: \"27724468-aead-4a4a-9ec1-c11d8fa2bb7c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qspfk"
Sep 17 03:19:26.993586 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.993564    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2d4rr\" (UniqueName: \"kubernetes.io/projected/c6a03737-feb7-48d2-82ba-5a2188f80980-kube-api-access-2d4rr\") pod \"multus-wds8n\" (UID: \"c6a03737-feb7-48d2-82ba-5a2188f80980\") " pod="openshift-multus/multus-wds8n"
Sep 17 03:19:26.994251 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:26.994230    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlt4p\" (UniqueName: \"kubernetes.io/projected/2758c0c8-e1ff-454a-9f0b-43cc8e1f8609-kube-api-access-mlt4p\") pod \"network-metrics-daemon-xl9wh\" (UID: \"2758c0c8-e1ff-454a-9f0b-43cc8e1f8609\") " pod="openshift-multus/network-metrics-daemon-xl9wh"
Sep 17 03:19:27.004398 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:27.004368    2552 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 03:19:27.083394 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:27.083361    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d3e92f88-fcb1-4d0f-b653-77b7ec5681fd-etc-openvswitch\") pod \"ovnkube-node-ldhl4\" (UID: \"d3e92f88-fcb1-4d0f-b653-77b7ec5681fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldhl4"
Sep 17 03:19:27.083595 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:27.083409    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d3e92f88-fcb1-4d0f-b653-77b7ec5681fd-run-openvswitch\") pod \"ovnkube-node-ldhl4\" (UID: \"d3e92f88-fcb1-4d0f-b653-77b7ec5681fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldhl4"
Sep 17 03:19:27.083595 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:27.083440    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d3e92f88-fcb1-4d0f-b653-77b7ec5681fd-log-socket\") pod \"ovnkube-node-ldhl4\" (UID: \"d3e92f88-fcb1-4d0f-b653-77b7ec5681fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldhl4"
Sep 17 03:19:27.083595 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:27.083481    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4e98ab7d-ed30-48e6-9751-cc40209037aa-etc-sysctl-d\") pod \"tuned-vkdcx\" (UID: \"4e98ab7d-ed30-48e6-9751-cc40209037aa\") " pod="openshift-cluster-node-tuning-operator/tuned-vkdcx"
Sep 17 03:19:27.083595 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:27.083509    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2a4ab360-f144-4999-97dc-6796bd4a44c2-agent-certs\") pod \"konnectivity-agent-8mzx5\" (UID: \"2a4ab360-f144-4999-97dc-6796bd4a44c2\") " pod="kube-system/konnectivity-agent-8mzx5"
Sep 17 03:19:27.083595 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:27.083531    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d3e92f88-fcb1-4d0f-b653-77b7ec5681fd-host-run-netns\") pod \"ovnkube-node-ldhl4\" (UID: \"d3e92f88-fcb1-4d0f-b653-77b7ec5681fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldhl4"
Sep 17 03:19:27.083595 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:27.083554    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d3e92f88-fcb1-4d0f-b653-77b7ec5681fd-host-cni-netd\") pod \"ovnkube-node-ldhl4\" (UID: \"d3e92f88-fcb1-4d0f-b653-77b7ec5681fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldhl4"
Sep 17 03:19:27.083595 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:27.083575    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d3e92f88-fcb1-4d0f-b653-77b7ec5681fd-env-overrides\") pod \"ovnkube-node-ldhl4\" (UID: \"d3e92f88-fcb1-4d0f-b653-77b7ec5681fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldhl4"
Sep 17 03:19:27.084050 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:27.083615    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0cc4aedf-1bb8-47c6-88f2-266bbbf99058-cnibin\") pod \"multus-additional-cni-plugins-nkrxz\" (UID: \"0cc4aedf-1bb8-47c6-88f2-266bbbf99058\") " pod="openshift-multus/multus-additional-cni-plugins-nkrxz"
Sep 17 03:19:27.084050 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:27.083656    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/045af8b4-e20b-46e7-bbcf-b5465710791d-host-slash\") pod \"iptables-alerter-mvmw5\" (UID: \"045af8b4-e20b-46e7-bbcf-b5465710791d\") " pod="openshift-network-operator/iptables-alerter-mvmw5"
Sep 17 03:19:27.084050 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:27.083682    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d1adf33b-0171-446e-aabe-f60fb74197e3-dbus\") pod \"global-pull-secret-syncer-4944l\" (UID: \"d1adf33b-0171-446e-aabe-f60fb74197e3\") " pod="kube-system/global-pull-secret-syncer-4944l"
Sep 17 03:19:27.084050 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:27.083704    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d1adf33b-0171-446e-aabe-f60fb74197e3-original-pull-secret\") pod \"global-pull-secret-syncer-4944l\" (UID: \"d1adf33b-0171-446e-aabe-f60fb74197e3\") " pod="kube-system/global-pull-secret-syncer-4944l"
Sep 17 03:19:27.084050 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:27.083729    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d3e92f88-fcb1-4d0f-b653-77b7ec5681fd-systemd-units\") pod \"ovnkube-node-ldhl4\" (UID: \"d3e92f88-fcb1-4d0f-b653-77b7ec5681fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldhl4"
Sep 17 03:19:27.084050 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:27.083753    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d3e92f88-fcb1-4d0f-b653-77b7ec5681fd-ovn-node-metrics-cert\") pod \"ovnkube-node-ldhl4\" (UID: \"d3e92f88-fcb1-4d0f-b653-77b7ec5681fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldhl4"
Sep 17 03:19:27.084050 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:27.083779    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0cc4aedf-1bb8-47c6-88f2-266bbbf99058-system-cni-dir\") pod \"multus-additional-cni-plugins-nkrxz\" (UID: \"0cc4aedf-1bb8-47c6-88f2-266bbbf99058\") " pod="openshift-multus/multus-additional-cni-plugins-nkrxz"
Sep 17 03:19:27.084050 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:27.083787    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d3e92f88-fcb1-4d0f-b653-77b7ec5681fd-host-run-netns\") pod \"ovnkube-node-ldhl4\" (UID: \"d3e92f88-fcb1-4d0f-b653-77b7ec5681fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldhl4"
Sep 17 03:19:27.084050 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:27.083805    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0cc4aedf-1bb8-47c6-88f2-266bbbf99058-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-nkrxz\" (UID: \"0cc4aedf-1bb8-47c6-88f2-266bbbf99058\") " pod="openshift-multus/multus-additional-cni-plugins-nkrxz"
Sep 17 03:19:27.084050 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:27.083839    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d3e92f88-fcb1-4d0f-b653-77b7ec5681fd-etc-openvswitch\") pod \"ovnkube-node-ldhl4\" (UID: \"d3e92f88-fcb1-4d0f-b653-77b7ec5681fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldhl4"
Sep 17 03:19:27.084050 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:27.083833    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d3e92f88-fcb1-4d0f-b653-77b7ec5681fd-run-systemd\") pod \"ovnkube-node-ldhl4\" (UID: \"d3e92f88-fcb1-4d0f-b653-77b7ec5681fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldhl4"
Sep 17 03:19:27.084050 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:27.083876    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d3e92f88-fcb1-4d0f-b653-77b7ec5681fd-host-run-ovn-kubernetes\") pod \"ovnkube-node-ldhl4\" (UID: \"d3e92f88-fcb1-4d0f-b653-77b7ec5681fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldhl4"
Sep 17 03:19:27.084050 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:27.083887    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d3e92f88-fcb1-4d0f-b653-77b7ec5681fd-run-openvswitch\") pod \"ovnkube-node-ldhl4\" (UID: \"d3e92f88-fcb1-4d0f-b653-77b7ec5681fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldhl4"
Sep 17 03:19:27.084050 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:27.083904    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4e98ab7d-ed30-48e6-9751-cc40209037aa-etc-kubernetes\") pod \"tuned-vkdcx\" (UID: \"4e98ab7d-ed30-48e6-9751-cc40209037aa\") " pod="openshift-cluster-node-tuning-operator/tuned-vkdcx"
Sep 17 03:19:27.084050 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:27.083929    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d3e92f88-fcb1-4d0f-b653-77b7ec5681fd-log-socket\") pod \"ovnkube-node-ldhl4\" (UID: \"d3e92f88-fcb1-4d0f-b653-77b7ec5681fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldhl4"
Sep 17 03:19:27.084050 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:27.083929    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d3e92f88-fcb1-4d0f-b653-77b7ec5681fd-node-log\") pod \"ovnkube-node-ldhl4\" (UID: \"d3e92f88-fcb1-4d0f-b653-77b7ec5681fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldhl4"
Sep 17 03:19:27.084050 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:27.083970    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d3e92f88-fcb1-4d0f-b653-77b7ec5681fd-node-log\") pod \"ovnkube-node-ldhl4\" (UID: \"d3e92f88-fcb1-4d0f-b653-77b7ec5681fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldhl4"
Sep 17 03:19:27.084914 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:27.083968    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d3e92f88-fcb1-4d0f-b653-77b7ec5681fd-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ldhl4\" (UID: \"d3e92f88-fcb1-4d0f-b653-77b7ec5681fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldhl4"
Sep 17 03:19:27.084914 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:27.084003    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d3e92f88-fcb1-4d0f-b653-77b7ec5681fd-ovnkube-script-lib\") pod \"ovnkube-node-ldhl4\" (UID: \"d3e92f88-fcb1-4d0f-b653-77b7ec5681fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldhl4"
Sep 17 03:19:27.084914 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:27.084002    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d3e92f88-fcb1-4d0f-b653-77b7ec5681fd-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ldhl4\" (UID: \"d3e92f88-fcb1-4d0f-b653-77b7ec5681fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldhl4"
Sep 17 03:19:27.084914 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:27.084035    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4e98ab7d-ed30-48e6-9751-cc40209037aa-tmp\") pod \"tuned-vkdcx\" (UID: \"4e98ab7d-ed30-48e6-9751-cc40209037aa\") " pod="openshift-cluster-node-tuning-operator/tuned-vkdcx"
Sep 17 03:19:27.084914 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:27.084056    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d3e92f88-fcb1-4d0f-b653-77b7ec5681fd-host-cni-netd\") pod \"ovnkube-node-ldhl4\" (UID: \"d3e92f88-fcb1-4d0f-b653-77b7ec5681fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldhl4"
Sep 17 03:19:27.084914 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:27.084063    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4e98ab7d-ed30-48e6-9751-cc40209037aa-etc-sysconfig\") pod \"tuned-vkdcx\" (UID: \"4e98ab7d-ed30-48e6-9751-cc40209037aa\") " pod="openshift-cluster-node-tuning-operator/tuned-vkdcx"
Sep 17 03:19:27.084914 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:27.084088    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4e98ab7d-ed30-48e6-9751-cc40209037aa-etc-tuned\") pod \"tuned-vkdcx\" (UID: \"4e98ab7d-ed30-48e6-9751-cc40209037aa\") " pod="openshift-cluster-node-tuning-operator/tuned-vkdcx"
Sep 17 03:19:27.084914 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:27.084594    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d3e92f88-fcb1-4d0f-b653-77b7ec5681fd-env-overrides\") pod \"ovnkube-node-ldhl4\" (UID: \"d3e92f88-fcb1-4d0f-b653-77b7ec5681fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldhl4"
Sep 17 03:19:27.084914 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:27.084783    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4e98ab7d-ed30-48e6-9751-cc40209037aa-etc-sysctl-d\") pod \"tuned-vkdcx\" (UID: \"4e98ab7d-ed30-48e6-9751-cc40209037aa\") " pod="openshift-cluster-node-tuning-operator/tuned-vkdcx"
Sep 17 03:19:27.084914 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:27.084842    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0cc4aedf-1bb8-47c6-88f2-266bbbf99058-cnibin\") pod \"multus-additional-cni-plugins-nkrxz\" (UID: \"0cc4aedf-1bb8-47c6-88f2-266bbbf99058\") " pod="openshift-multus/multus-additional-cni-plugins-nkrxz"
Sep 17 03:19:27.084914 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:27.084843    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4e98ab7d-ed30-48e6-9751-cc40209037aa-etc-sysconfig\") pod \"tuned-vkdcx\" (UID: \"4e98ab7d-ed30-48e6-9751-cc40209037aa\") " pod="openshift-cluster-node-tuning-operator/tuned-vkdcx"
Sep 17 03:19:27.084914 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:27.084896    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d3e92f88-fcb1-4d0f-b653-77b7ec5681fd-run-systemd\") pod \"ovnkube-node-ldhl4\" (UID: \"d3e92f88-fcb1-4d0f-b653-77b7ec5681fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldhl4"
Sep 17 03:19:27.085510 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:27.084941    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d3e92f88-fcb1-4d0f-b653-77b7ec5681fd-host-run-ovn-kubernetes\") pod \"ovnkube-node-ldhl4\" (UID: \"d3e92f88-fcb1-4d0f-b653-77b7ec5681fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldhl4"
Sep 17 03:19:27.085510 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:27.084988    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4e98ab7d-ed30-48e6-9751-cc40209037aa-etc-kubernetes\") pod \"tuned-vkdcx\" (UID: \"4e98ab7d-ed30-48e6-9751-cc40209037aa\") " pod="openshift-cluster-node-tuning-operator/tuned-vkdcx"
Sep 17 03:19:27.085510 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:27.085004    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0cc4aedf-1bb8-47c6-88f2-266bbbf99058-tuning-conf-dir\") pod \"multus-additional-cni-plugins-nkrxz\" (UID: \"0cc4aedf-1bb8-47c6-88f2-266bbbf99058\") " pod="openshift-multus/multus-additional-cni-plugins-nkrxz"
Sep 17 03:19:27.085510 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:27.085050    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4e98ab7d-ed30-48e6-9751-cc40209037aa-etc-systemd\") pod \"tuned-vkdcx\" (UID: \"4e98ab7d-ed30-48e6-9751-cc40209037aa\") " pod="openshift-cluster-node-tuning-operator/tuned-vkdcx"
Sep 17 03:19:27.085510 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:27.085079    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4e98ab7d-ed30-48e6-9751-cc40209037aa-sys\") pod \"tuned-vkdcx\" (UID: \"4e98ab7d-ed30-48e6-9751-cc40209037aa\") " pod="openshift-cluster-node-tuning-operator/tuned-vkdcx"
Sep 17 03:19:27.085510 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:27.085104    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d3e92f88-fcb1-4d0f-b653-77b7ec5681fd-host-slash\") pod \"ovnkube-node-ldhl4\" (UID: \"d3e92f88-fcb1-4d0f-b653-77b7ec5681fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldhl4"
Sep 17 03:19:27.085510 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:27.085132    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0cc4aedf-1bb8-47c6-88f2-266bbbf99058-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-nkrxz\" (UID: \"0cc4aedf-1bb8-47c6-88f2-266bbbf99058\") " pod="openshift-multus/multus-additional-cni-plugins-nkrxz"
Sep 17 03:19:27.085510 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:27.085163    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/045af8b4-e20b-46e7-bbcf-b5465710791d-iptables-alerter-script\") pod \"iptables-alerter-mvmw5\" (UID: \"045af8b4-e20b-46e7-bbcf-b5465710791d\") " pod="openshift-network-operator/iptables-alerter-mvmw5"
Sep 17 03:19:27.085510 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:27.085188    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d1adf33b-0171-446e-aabe-f60fb74197e3-kubelet-config\") pod \"global-pull-secret-syncer-4944l\" (UID: \"d1adf33b-0171-446e-aabe-f60fb74197e3\") " pod="kube-system/global-pull-secret-syncer-4944l"
Sep 17 03:19:27.085510 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:27.085221    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0cc4aedf-1bb8-47c6-88f2-266bbbf99058-cni-binary-copy\") pod \"multus-additional-cni-plugins-nkrxz\" (UID: \"0cc4aedf-1bb8-47c6-88f2-266bbbf99058\") " pod="openshift-multus/multus-additional-cni-plugins-nkrxz"
Sep 17 03:19:27.085510 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:27.085248    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4e98ab7d-ed30-48e6-9751-cc40209037aa-etc-sysctl-conf\") pod \"tuned-vkdcx\" (UID: \"4e98ab7d-ed30-48e6-9751-cc40209037aa\") " pod="openshift-cluster-node-tuning-operator/tuned-vkdcx"
Sep 17 03:19:27.085510 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:27.085276    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4e98ab7d-ed30-48e6-9751-cc40209037aa-lib-modules\") pod \"tuned-vkdcx\" (UID: \"4e98ab7d-ed30-48e6-9751-cc40209037aa\") " pod="openshift-cluster-node-tuning-operator/tuned-vkdcx"
Sep 17 03:19:27.085510 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:27.085336    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p99g5\" (UniqueName: \"kubernetes.io/projected/b7b79096-9bfc-4d82-9406-073a3f7961d0-kube-api-access-p99g5\") pod \"node-ca-rld2q\" (UID: \"b7b79096-9bfc-4d82-9406-073a3f7961d0\") " pod="openshift-image-registry/node-ca-rld2q"
Sep 17 03:19:27.085510 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:27.085363    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgbfd\" (UniqueName: \"kubernetes.io/projected/4e98ab7d-ed30-48e6-9751-cc40209037aa-kube-api-access-bgbfd\") pod \"tuned-vkdcx\" (UID: \"4e98ab7d-ed30-48e6-9751-cc40209037aa\") " pod="openshift-cluster-node-tuning-operator/tuned-vkdcx"
Sep 17 03:19:27.085510 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:27.085395    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ggvx\" (UniqueName: \"kubernetes.io/projected/b3bac7f7-fbb1-4cec-8a26-046b702f4d70-kube-api-access-7ggvx\") pod \"node-resolver-gxmbv\" (UID: \"b3bac7f7-fbb1-4cec-8a26-046b702f4d70\") " pod="openshift-dns/node-resolver-gxmbv"
Sep 17 03:19:27.085510 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:27.085420    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d3e92f88-fcb1-4d0f-b653-77b7ec5681fd-host-cni-bin\") pod \"ovnkube-node-ldhl4\" (UID: \"d3e92f88-fcb1-4d0f-b653-77b7ec5681fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldhl4"
Sep 17 03:19:27.085510 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:27.085422    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0cc4aedf-1bb8-47c6-88f2-266bbbf99058-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-nkrxz\" (UID: \"0cc4aedf-1bb8-47c6-88f2-266bbbf99058\") " pod="openshift-multus/multus-additional-cni-plugins-nkrxz"
Sep 17 03:19:27.086368 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:27.085446    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdkqz\" (UniqueName: \"kubernetes.io/projected/d3e92f88-fcb1-4d0f-b653-77b7ec5681fd-kube-api-access-fdkqz\") pod \"ovnkube-node-ldhl4\" (UID: \"d3e92f88-fcb1-4d0f-b653-77b7ec5681fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldhl4"
Sep 17 03:19:27.086368 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:27.085497    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/045af8b4-e20b-46e7-bbcf-b5465710791d-host-slash\") pod \"iptables-alerter-mvmw5\" (UID: \"045af8b4-e20b-46e7-bbcf-b5465710791d\") " pod="openshift-network-operator/iptables-alerter-mvmw5"
Sep 17 03:19:27.086368 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:27.085532    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9wqn\" (UniqueName: \"kubernetes.io/projected/0cc4aedf-1bb8-47c6-88f2-266bbbf99058-kube-api-access-s9wqn\") pod \"multus-additional-cni-plugins-nkrxz\" (UID: \"0cc4aedf-1bb8-47c6-88f2-266bbbf99058\") " pod="openshift-multus/multus-additional-cni-plugins-nkrxz"
Sep 17 03:19:27.086368 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:27.085563    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4e98ab7d-ed30-48e6-9751-cc40209037aa-etc-modprobe-d\") pod \"tuned-vkdcx\" (UID: \"4e98ab7d-ed30-48e6-9751-cc40209037aa\") " pod="openshift-cluster-node-tuning-operator/tuned-vkdcx"
Sep 17 03:19:27.086368 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:27.085590    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4e98ab7d-ed30-48e6-9751-cc40209037aa-run\") pod \"tuned-vkdcx\" (UID: \"4e98ab7d-ed30-48e6-9751-cc40209037aa\") " pod="openshift-cluster-node-tuning-operator/tuned-vkdcx"
Sep 17 03:19:27.086368 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:27.085638    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4e98ab7d-ed30-48e6-9751-cc40209037aa-host\") pod \"tuned-vkdcx\" (UID: \"4e98ab7d-ed30-48e6-9751-cc40209037aa\") " pod="openshift-cluster-node-tuning-operator/tuned-vkdcx"
Sep 17 03:19:27.086368 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:27.085664    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d3e92f88-fcb1-4d0f-b653-77b7ec5681fd-ovnkube-config\") pod \"ovnkube-node-ldhl4\" (UID: \"d3e92f88-fcb1-4d0f-b653-77b7ec5681fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldhl4"
Sep 17 03:19:27.086368 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:27.085707    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b7b79096-9bfc-4d82-9406-073a3f7961d0-serviceca\") pod \"node-ca-rld2q\" (UID: \"b7b79096-9bfc-4d82-9406-073a3f7961d0\") " pod="openshift-image-registry/node-ca-rld2q"
Sep 17 03:19:27.086368 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:27.085734    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbvh5\" (UniqueName: \"kubernetes.io/projected/045af8b4-e20b-46e7-bbcf-b5465710791d-kube-api-access-nbvh5\") pod \"iptables-alerter-mvmw5\" (UID: \"045af8b4-e20b-46e7-bbcf-b5465710791d\") " pod="openshift-network-operator/iptables-alerter-mvmw5"
Sep 17 03:19:27.086368 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:27.085761    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b3bac7f7-fbb1-4cec-8a26-046b702f4d70-hosts-file\") pod \"node-resolver-gxmbv\" (UID: \"b3bac7f7-fbb1-4cec-8a26-046b702f4d70\") " pod="openshift-dns/node-resolver-gxmbv"
Sep 17 03:19:27.086368 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:27.085774    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d1adf33b-0171-446e-aabe-f60fb74197e3-kubelet-config\") pod \"global-pull-secret-syncer-4944l\" (UID: \"d1adf33b-0171-446e-aabe-f60fb74197e3\") " pod="kube-system/global-pull-secret-syncer-4944l"
Sep 17 03:19:27.086368 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:27.085793    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d3e92f88-fcb1-4d0f-b653-77b7ec5681fd-host-kubelet\") pod \"ovnkube-node-ldhl4\" (UID: \"d3e92f88-fcb1-4d0f-b653-77b7ec5681fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldhl4"
Sep 17 03:19:27.086368 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:27.085820    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d3e92f88-fcb1-4d0f-b653-77b7ec5681fd-var-lib-openvswitch\") pod \"ovnkube-node-ldhl4\" (UID: \"d3e92f88-fcb1-4d0f-b653-77b7ec5681fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldhl4"
Sep 17 03:19:27.086368 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:27.085849    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d3e92f88-fcb1-4d0f-b653-77b7ec5681fd-run-ovn\") pod \"ovnkube-node-ldhl4\" (UID: \"d3e92f88-fcb1-4d0f-b653-77b7ec5681fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldhl4"
Sep 17 03:19:27.086368 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:27.085895    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2a4ab360-f144-4999-97dc-6796bd4a44c2-konnectivity-ca\") pod \"konnectivity-agent-8mzx5\" (UID: \"2a4ab360-f144-4999-97dc-6796bd4a44c2\") " pod="kube-system/konnectivity-agent-8mzx5"
Sep 17 03:19:27.086368 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:27.085923    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b7b79096-9bfc-4d82-9406-073a3f7961d0-host\") pod \"node-ca-rld2q\" (UID: \"b7b79096-9bfc-4d82-9406-073a3f7961d0\") " pod="openshift-image-registry/node-ca-rld2q"
Sep 17 03:19:27.086368 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:27.085955    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4e98ab7d-ed30-48e6-9751-cc40209037aa-var-lib-kubelet\") pod \"tuned-vkdcx\" (UID: \"4e98ab7d-ed30-48e6-9751-cc40209037aa\") " pod="openshift-cluster-node-tuning-operator/tuned-vkdcx"
Sep 17 03:19:27.087249 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:27.085983    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3bac7f7-fbb1-4cec-8a26-046b702f4d70-tmp-dir\") pod \"node-resolver-gxmbv\" (UID: \"b3bac7f7-fbb1-4cec-8a26-046b702f4d70\") " pod="openshift-dns/node-resolver-gxmbv"
Sep 17 03:19:27.087249 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:27.086333    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d3e92f88-fcb1-4d0f-b653-77b7ec5681fd-ovnkube-script-lib\") pod \"ovnkube-node-ldhl4\" (UID: \"d3e92f88-fcb1-4d0f-b653-77b7ec5681fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldhl4"
Sep 17 03:19:27.087249 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:27.086344    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3bac7f7-fbb1-4cec-8a26-046b702f4d70-tmp-dir\") pod \"node-resolver-gxmbv\" (UID: \"b3bac7f7-fbb1-4cec-8a26-046b702f4d70\") " pod="openshift-dns/node-resolver-gxmbv"
Sep 17 03:19:27.087249 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:27.086497    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0cc4aedf-1bb8-47c6-88f2-266bbbf99058-tuning-conf-dir\") pod \"multus-additional-cni-plugins-nkrxz\" (UID: \"0cc4aedf-1bb8-47c6-88f2-266bbbf99058\") " pod="openshift-multus/multus-additional-cni-plugins-nkrxz"
Sep 17 03:19:27.087249 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:27.086527    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d1adf33b-0171-446e-aabe-f60fb74197e3-dbus\") pod \"global-pull-secret-syncer-4944l\" (UID: \"d1adf33b-0171-446e-aabe-f60fb74197e3\") " pod="kube-system/global-pull-secret-syncer-4944l"
Sep 17 03:19:27.087249 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:27.086562    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4e98ab7d-ed30-48e6-9751-cc40209037aa-etc-systemd\") pod \"tuned-vkdcx\" (UID: \"4e98ab7d-ed30-48e6-9751-cc40209037aa\") " pod="openshift-cluster-node-tuning-operator/tuned-vkdcx"
Sep 17 03:19:27.087249 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:27.086618    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4e98ab7d-ed30-48e6-9751-cc40209037aa-sys\") pod \"tuned-vkdcx\" (UID: \"4e98ab7d-ed30-48e6-9751-cc40209037aa\") " pod="openshift-cluster-node-tuning-operator/tuned-vkdcx"
Sep 17 03:19:27.087249 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:19:27.086630    2552 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 03:19:27.087249 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:27.086664    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d3e92f88-fcb1-4d0f-b653-77b7ec5681fd-host-slash\") pod \"ovnkube-node-ldhl4\" (UID: \"d3e92f88-fcb1-4d0f-b653-77b7ec5681fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldhl4"
Sep 17 03:19:27.087249 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:19:27.086711    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1adf33b-0171-446e-aabe-f60fb74197e3-original-pull-secret podName:d1adf33b-0171-446e-aabe-f60fb74197e3 nodeName:}" failed. No retries permitted until 2026-09-17 03:19:27.586689489 +0000 UTC m=+3.053097010 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d1adf33b-0171-446e-aabe-f60fb74197e3-original-pull-secret") pod "global-pull-secret-syncer-4944l" (UID: "d1adf33b-0171-446e-aabe-f60fb74197e3") : object "kube-system"/"original-pull-secret" not registered
Sep 17 03:19:27.087249 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:27.086760    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d3e92f88-fcb1-4d0f-b653-77b7ec5681fd-systemd-units\") pod \"ovnkube-node-ldhl4\" (UID: \"d3e92f88-fcb1-4d0f-b653-77b7ec5681fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldhl4"
Sep 17 03:19:27.087249 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:27.087159    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0cc4aedf-1bb8-47c6-88f2-266bbbf99058-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-nkrxz\" (UID: \"0cc4aedf-1bb8-47c6-88f2-266bbbf99058\") " pod="openshift-multus/multus-additional-cni-plugins-nkrxz"
Sep 17 03:19:27.087914 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:27.087665    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/045af8b4-e20b-46e7-bbcf-b5465710791d-iptables-alerter-script\") pod \"iptables-alerter-mvmw5\" (UID: \"045af8b4-e20b-46e7-bbcf-b5465710791d\") " pod="openshift-network-operator/iptables-alerter-mvmw5"
Sep 17 03:19:27.087914 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:27.087776    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4e98ab7d-ed30-48e6-9751-cc40209037aa-tmp\") pod \"tuned-vkdcx\" (UID: \"4e98ab7d-ed30-48e6-9751-cc40209037aa\") " pod="openshift-cluster-node-tuning-operator/tuned-vkdcx"
Sep 17 03:19:27.087914 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:27.087898    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0cc4aedf-1bb8-47c6-88f2-266bbbf99058-system-cni-dir\") pod \"multus-additional-cni-plugins-nkrxz\" (UID: \"0cc4aedf-1bb8-47c6-88f2-266bbbf99058\") " pod="openshift-multus/multus-additional-cni-plugins-nkrxz"
Sep 17 03:19:27.088086 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:27.088076    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4e98ab7d-ed30-48e6-9751-cc40209037aa-etc-modprobe-d\") pod \"tuned-vkdcx\" (UID: \"4e98ab7d-ed30-48e6-9751-cc40209037aa\") " pod="openshift-cluster-node-tuning-operator/tuned-vkdcx"
Sep 17 03:19:27.088140 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:27.088124    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4e98ab7d-ed30-48e6-9751-cc40209037aa-run\") pod \"tuned-vkdcx\" (UID: \"4e98ab7d-ed30-48e6-9751-cc40209037aa\") " pod="openshift-cluster-node-tuning-operator/tuned-vkdcx"
Sep 17 03:19:27.088200 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:27.088173    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4e98ab7d-ed30-48e6-9751-cc40209037aa-host\") pod \"tuned-vkdcx\" (UID: \"4e98ab7d-ed30-48e6-9751-cc40209037aa\") " pod="openshift-cluster-node-tuning-operator/tuned-vkdcx"
Sep 17 03:19:27.088488 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:27.088457    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0cc4aedf-1bb8-47c6-88f2-266bbbf99058-cni-binary-copy\") pod \"multus-additional-cni-plugins-nkrxz\" (UID: \"0cc4aedf-1bb8-47c6-88f2-266bbbf99058\") " pod="openshift-multus/multus-additional-cni-plugins-nkrxz"
Sep 17 03:19:27.088664 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:27.088640    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4e98ab7d-ed30-48e6-9751-cc40209037aa-etc-sysctl-conf\") pod \"tuned-vkdcx\" (UID: \"4e98ab7d-ed30-48e6-9751-cc40209037aa\") " pod="openshift-cluster-node-tuning-operator/tuned-vkdcx"
Sep 17 03:19:27.088759 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:27.088685    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d3e92f88-fcb1-4d0f-b653-77b7ec5681fd-ovnkube-config\") pod \"ovnkube-node-ldhl4\" (UID: \"d3e92f88-fcb1-4d0f-b653-77b7ec5681fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldhl4"
Sep 17 03:19:27.088825 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:27.088784    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4e98ab7d-ed30-48e6-9751-cc40209037aa-lib-modules\") pod \"tuned-vkdcx\" (UID: \"4e98ab7d-ed30-48e6-9751-cc40209037aa\") " pod="openshift-cluster-node-tuning-operator/tuned-vkdcx"
Sep 17 03:19:27.088880 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:27.088842    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d3e92f88-fcb1-4d0f-b653-77b7ec5681fd-run-ovn\") pod \"ovnkube-node-ldhl4\" (UID: \"d3e92f88-fcb1-4d0f-b653-77b7ec5681fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldhl4"
Sep 17 03:19:27.088932 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:27.088897    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b3bac7f7-fbb1-4cec-8a26-046b702f4d70-hosts-file\") pod \"node-resolver-gxmbv\" (UID: \"b3bac7f7-fbb1-4cec-8a26-046b702f4d70\") " pod="openshift-dns/node-resolver-gxmbv"
Sep 17 03:19:27.088989 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:27.088941    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d3e92f88-fcb1-4d0f-b653-77b7ec5681fd-host-kubelet\") pod \"ovnkube-node-ldhl4\" (UID: \"d3e92f88-fcb1-4d0f-b653-77b7ec5681fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldhl4"
Sep 17 03:19:27.089040 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:27.088986    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d3e92f88-fcb1-4d0f-b653-77b7ec5681fd-var-lib-openvswitch\") pod \"ovnkube-node-ldhl4\" (UID: \"d3e92f88-fcb1-4d0f-b653-77b7ec5681fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldhl4"
Sep 17 03:19:27.089096 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:27.089040    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b7b79096-9bfc-4d82-9406-073a3f7961d0-host\") pod \"node-ca-rld2q\" (UID: \"b7b79096-9bfc-4d82-9406-073a3f7961d0\") " pod="openshift-image-registry/node-ca-rld2q"
Sep 17 03:19:27.089096 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:27.089045    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b7b79096-9bfc-4d82-9406-073a3f7961d0-serviceca\") pod \"node-ca-rld2q\" (UID: \"b7b79096-9bfc-4d82-9406-073a3f7961d0\") " pod="openshift-image-registry/node-ca-rld2q"
Sep 17 03:19:27.089197 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:27.089097    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4e98ab7d-ed30-48e6-9751-cc40209037aa-var-lib-kubelet\") pod \"tuned-vkdcx\" (UID: \"4e98ab7d-ed30-48e6-9751-cc40209037aa\") " pod="openshift-cluster-node-tuning-operator/tuned-vkdcx"
Sep 17 03:19:27.089197 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:27.089154    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d3e92f88-fcb1-4d0f-b653-77b7ec5681fd-host-cni-bin\") pod \"ovnkube-node-ldhl4\" (UID: \"d3e92f88-fcb1-4d0f-b653-77b7ec5681fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldhl4"
Sep 17 03:19:27.089633 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:27.089542    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2a4ab360-f144-4999-97dc-6796bd4a44c2-konnectivity-ca\") pod \"konnectivity-agent-8mzx5\" (UID: \"2a4ab360-f144-4999-97dc-6796bd4a44c2\") " pod="kube-system/konnectivity-agent-8mzx5"
Sep 17 03:19:27.089953 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:27.089933    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2a4ab360-f144-4999-97dc-6796bd4a44c2-agent-certs\") pod \"konnectivity-agent-8mzx5\" (UID: \"2a4ab360-f144-4999-97dc-6796bd4a44c2\") " pod="kube-system/konnectivity-agent-8mzx5"
Sep 17 03:19:27.090247 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:27.090204    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4e98ab7d-ed30-48e6-9751-cc40209037aa-etc-tuned\") pod \"tuned-vkdcx\" (UID: \"4e98ab7d-ed30-48e6-9751-cc40209037aa\") " pod="openshift-cluster-node-tuning-operator/tuned-vkdcx"
Sep 17 03:19:27.092699 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:27.091869    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d3e92f88-fcb1-4d0f-b653-77b7ec5681fd-ovn-node-metrics-cert\") pod \"ovnkube-node-ldhl4\" (UID: \"d3e92f88-fcb1-4d0f-b653-77b7ec5681fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldhl4"
Sep 17 03:19:27.105433 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:27.105277    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ggvx\" (UniqueName: \"kubernetes.io/projected/b3bac7f7-fbb1-4cec-8a26-046b702f4d70-kube-api-access-7ggvx\") pod \"node-resolver-gxmbv\" (UID: \"b3bac7f7-fbb1-4cec-8a26-046b702f4d70\") " pod="openshift-dns/node-resolver-gxmbv"
Sep 17 03:19:27.105433 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:27.105296    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbvh5\" (UniqueName: \"kubernetes.io/projected/045af8b4-e20b-46e7-bbcf-b5465710791d-kube-api-access-nbvh5\") pod \"iptables-alerter-mvmw5\" (UID: \"045af8b4-e20b-46e7-bbcf-b5465710791d\") " pod="openshift-network-operator/iptables-alerter-mvmw5"
Sep 17 03:19:27.105433 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:27.105286    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9wqn\" (UniqueName: \"kubernetes.io/projected/0cc4aedf-1bb8-47c6-88f2-266bbbf99058-kube-api-access-s9wqn\") pod \"multus-additional-cni-plugins-nkrxz\" (UID: \"0cc4aedf-1bb8-47c6-88f2-266bbbf99058\") " pod="openshift-multus/multus-additional-cni-plugins-nkrxz"
Sep 17 03:19:27.106026 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:27.105995    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p99g5\" (UniqueName: \"kubernetes.io/projected/b7b79096-9bfc-4d82-9406-073a3f7961d0-kube-api-access-p99g5\") pod \"node-ca-rld2q\" (UID: \"b7b79096-9bfc-4d82-9406-073a3f7961d0\") " pod="openshift-image-registry/node-ca-rld2q"
Sep 17 03:19:27.107250 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:27.107194    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgbfd\" (UniqueName: \"kubernetes.io/projected/4e98ab7d-ed30-48e6-9751-cc40209037aa-kube-api-access-bgbfd\") pod \"tuned-vkdcx\" (UID: \"4e98ab7d-ed30-48e6-9751-cc40209037aa\") " pod="openshift-cluster-node-tuning-operator/tuned-vkdcx"
Sep 17 03:19:27.107699 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:27.107576    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdkqz\" (UniqueName: \"kubernetes.io/projected/d3e92f88-fcb1-4d0f-b653-77b7ec5681fd-kube-api-access-fdkqz\") pod \"ovnkube-node-ldhl4\" (UID: \"d3e92f88-fcb1-4d0f-b653-77b7ec5681fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-ldhl4"
Sep 17 03:19:27.181941 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:27.181721    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-wds8n"
Sep 17 03:19:27.194725 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:27.194690    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qspfk"
Sep 17 03:19:27.207655 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:27.207614    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-rld2q"
Sep 17 03:19:27.214033 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:27.213999    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-nkrxz"
Sep 17 03:19:27.224285 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:27.224252    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-8mzx5"
Sep 17 03:19:27.235518 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:27.235485    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-mvmw5"
Sep 17 03:19:27.242968 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:27.242920    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-vkdcx"
Sep 17 03:19:27.254324 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:27.254285    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-gxmbv"
Sep 17 03:19:27.263674 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:27.263622    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-ldhl4"
Sep 17 03:19:27.489243 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:27.489128    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2758c0c8-e1ff-454a-9f0b-43cc8e1f8609-metrics-certs\") pod \"network-metrics-daemon-xl9wh\" (UID: \"2758c0c8-e1ff-454a-9f0b-43cc8e1f8609\") " pod="openshift-multus/network-metrics-daemon-xl9wh"
Sep 17 03:19:27.489243 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:27.489196    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmr6b\" (UniqueName: \"kubernetes.io/projected/9e5c90e7-d9db-4205-9bc6-c6790b31e53f-kube-api-access-nmr6b\") pod \"network-check-target-9d4m6\" (UID: \"9e5c90e7-d9db-4205-9bc6-c6790b31e53f\") " pod="openshift-network-diagnostics/network-check-target-9d4m6"
Sep 17 03:19:27.489496 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:19:27.489318    2552 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 03:19:27.489496 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:19:27.489365    2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 03:19:27.489496 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:19:27.489386    2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 03:19:27.489496 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:19:27.489401    2552 projected.go:196] Error preparing data for projected volume kube-api-access-nmr6b for pod openshift-network-diagnostics/network-check-target-9d4m6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 03:19:27.489496 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:19:27.489412    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2758c0c8-e1ff-454a-9f0b-43cc8e1f8609-metrics-certs podName:2758c0c8-e1ff-454a-9f0b-43cc8e1f8609 nodeName:}" failed. No retries permitted until 2026-09-17 03:19:28.48938505 +0000 UTC m=+3.955792572 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2758c0c8-e1ff-454a-9f0b-43cc8e1f8609-metrics-certs") pod "network-metrics-daemon-xl9wh" (UID: "2758c0c8-e1ff-454a-9f0b-43cc8e1f8609") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 03:19:27.489496 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:19:27.489452    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9e5c90e7-d9db-4205-9bc6-c6790b31e53f-kube-api-access-nmr6b podName:9e5c90e7-d9db-4205-9bc6-c6790b31e53f nodeName:}" failed. No retries permitted until 2026-09-17 03:19:28.489438909 +0000 UTC m=+3.955846415 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-nmr6b" (UniqueName: "kubernetes.io/projected/9e5c90e7-d9db-4205-9bc6-c6790b31e53f-kube-api-access-nmr6b") pod "network-check-target-9d4m6" (UID: "9e5c90e7-d9db-4205-9bc6-c6790b31e53f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 03:19:27.589892 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:27.589848    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d1adf33b-0171-446e-aabe-f60fb74197e3-original-pull-secret\") pod \"global-pull-secret-syncer-4944l\" (UID: \"d1adf33b-0171-446e-aabe-f60fb74197e3\") " pod="kube-system/global-pull-secret-syncer-4944l"
Sep 17 03:19:27.590076 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:19:27.589987    2552 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 03:19:27.590076 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:19:27.590056    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1adf33b-0171-446e-aabe-f60fb74197e3-original-pull-secret podName:d1adf33b-0171-446e-aabe-f60fb74197e3 nodeName:}" failed. No retries permitted until 2026-09-17 03:19:28.590035896 +0000 UTC m=+4.056443400 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d1adf33b-0171-446e-aabe-f60fb74197e3-original-pull-secret") pod "global-pull-secret-syncer-4944l" (UID: "d1adf33b-0171-446e-aabe-f60fb74197e3") : object "kube-system"/"original-pull-secret" not registered
Sep 17 03:19:27.910452 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:27.910355    2552 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 03:14:27 +0000 UTC" deadline="2028-06-04 10:55:43.34176113 +0000 UTC"
Sep 17 03:19:27.910452 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:27.910448    2552 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15031h36m15.431317779s"
Sep 17 03:19:27.980148 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:27.980110    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xl9wh"
Sep 17 03:19:27.980333 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:19:27.980261    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xl9wh" podUID="2758c0c8-e1ff-454a-9f0b-43cc8e1f8609"
Sep 17 03:19:28.308050 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:28.308021    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod045af8b4_e20b_46e7_bbcf_b5465710791d.slice/crio-89fca065219e798c6afcadac139dc6712102eb54ed61d89a84debd2f0886cb3b WatchSource:0}: Error finding container 89fca065219e798c6afcadac139dc6712102eb54ed61d89a84debd2f0886cb3b: Status 404 returned error can't find the container with id 89fca065219e798c6afcadac139dc6712102eb54ed61d89a84debd2f0886cb3b
Sep 17 03:19:28.309789 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:28.309729    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27724468_aead_4a4a_9ec1_c11d8fa2bb7c.slice/crio-813dca49faca11d21b30232e7c47603bc0e7fae9badbe7f9b3844d12bdd53eda WatchSource:0}: Error finding container 813dca49faca11d21b30232e7c47603bc0e7fae9badbe7f9b3844d12bdd53eda: Status 404 returned error can't find the container with id 813dca49faca11d21b30232e7c47603bc0e7fae9badbe7f9b3844d12bdd53eda
Sep 17 03:19:28.311387 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:28.311358    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7b79096_9bfc_4d82_9406_073a3f7961d0.slice/crio-368f680d7206e24a75b692d24b905ebe4276e7cdb073a3541982525ce555b969 WatchSource:0}: Error finding container 368f680d7206e24a75b692d24b905ebe4276e7cdb073a3541982525ce555b969: Status 404 returned error can't find the container with id 368f680d7206e24a75b692d24b905ebe4276e7cdb073a3541982525ce555b969
Sep 17 03:19:28.312328 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:28.312172    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cc4aedf_1bb8_47c6_88f2_266bbbf99058.slice/crio-0a1ad05d18e1af66463c4a7fa96d73563eec03a4819b26949dc78a51b39d443e WatchSource:0}: Error finding container 0a1ad05d18e1af66463c4a7fa96d73563eec03a4819b26949dc78a51b39d443e: Status 404 returned error can't find the container with id 0a1ad05d18e1af66463c4a7fa96d73563eec03a4819b26949dc78a51b39d443e
Sep 17 03:19:28.315729 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:28.315696    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e98ab7d_ed30_48e6_9751_cc40209037aa.slice/crio-ca165128a8f934b3ce14718c33346cd22d25485690958da7d0051caab1f7e212 WatchSource:0}: Error finding container ca165128a8f934b3ce14718c33346cd22d25485690958da7d0051caab1f7e212: Status 404 returned error can't find the container with id ca165128a8f934b3ce14718c33346cd22d25485690958da7d0051caab1f7e212
Sep 17 03:19:28.316937 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:28.316911    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a4ab360_f144_4999_97dc_6796bd4a44c2.slice/crio-2239a3362d19d348a637308c7ad76ca838da6e1b467e4d95d12bee34d38fd634 WatchSource:0}: Error finding container 2239a3362d19d348a637308c7ad76ca838da6e1b467e4d95d12bee34d38fd634: Status 404 returned error can't find the container with id 2239a3362d19d348a637308c7ad76ca838da6e1b467e4d95d12bee34d38fd634
Sep 17 03:19:28.318476 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:28.318449    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6a03737_feb7_48d2_82ba_5a2188f80980.slice/crio-0c1433ae7395a50bc736324306aa604a7822d587c76a0176b274c625a5064040 WatchSource:0}: Error finding container 0c1433ae7395a50bc736324306aa604a7822d587c76a0176b274c625a5064040: Status 404 returned error can't find the container with id 0c1433ae7395a50bc736324306aa604a7822d587c76a0176b274c625a5064040
Sep 17 03:19:28.319604 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:28.319059    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3e92f88_fcb1_4d0f_b653_77b7ec5681fd.slice/crio-4d5d81bc0ce854d6018b75829f87a65738ccc2317403fb46d4c0eeefade21e26 WatchSource:0}: Error finding container 4d5d81bc0ce854d6018b75829f87a65738ccc2317403fb46d4c0eeefade21e26: Status 404 returned error can't find the container with id 4d5d81bc0ce854d6018b75829f87a65738ccc2317403fb46d4c0eeefade21e26
Sep 17 03:19:28.322392 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:19:28.321216    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3bac7f7_fbb1_4cec_8a26_046b702f4d70.slice/crio-dbdc777b1787343a9ebaa96b562828898b4ce652c6e97cce5640d6586d558977 WatchSource:0}: Error finding container dbdc777b1787343a9ebaa96b562828898b4ce652c6e97cce5640d6586d558977: Status 404 returned error can't find the container with id dbdc777b1787343a9ebaa96b562828898b4ce652c6e97cce5640d6586d558977
Sep 17 03:19:28.498907 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:28.498873    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmr6b\" (UniqueName: \"kubernetes.io/projected/9e5c90e7-d9db-4205-9bc6-c6790b31e53f-kube-api-access-nmr6b\") pod \"network-check-target-9d4m6\" (UID: \"9e5c90e7-d9db-4205-9bc6-c6790b31e53f\") " pod="openshift-network-diagnostics/network-check-target-9d4m6"
Sep 17 03:19:28.499072 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:28.498978    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2758c0c8-e1ff-454a-9f0b-43cc8e1f8609-metrics-certs\") pod \"network-metrics-daemon-xl9wh\" (UID: \"2758c0c8-e1ff-454a-9f0b-43cc8e1f8609\") " pod="openshift-multus/network-metrics-daemon-xl9wh"
Sep 17 03:19:28.499138 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:19:28.499071    2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 03:19:28.499138 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:19:28.499087    2552 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 03:19:28.499138 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:19:28.499097    2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 03:19:28.499138 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:19:28.499108    2552 projected.go:196] Error preparing data for projected volume kube-api-access-nmr6b for pod openshift-network-diagnostics/network-check-target-9d4m6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 03:19:28.499276 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:19:28.499153    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2758c0c8-e1ff-454a-9f0b-43cc8e1f8609-metrics-certs podName:2758c0c8-e1ff-454a-9f0b-43cc8e1f8609 nodeName:}" failed. No retries permitted until 2026-09-17 03:19:30.499137546 +0000 UTC m=+5.965545053 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2758c0c8-e1ff-454a-9f0b-43cc8e1f8609-metrics-certs") pod "network-metrics-daemon-xl9wh" (UID: "2758c0c8-e1ff-454a-9f0b-43cc8e1f8609") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 03:19:28.499276 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:19:28.499168    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9e5c90e7-d9db-4205-9bc6-c6790b31e53f-kube-api-access-nmr6b podName:9e5c90e7-d9db-4205-9bc6-c6790b31e53f nodeName:}" failed. No retries permitted until 2026-09-17 03:19:30.499161291 +0000 UTC m=+5.965568796 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-nmr6b" (UniqueName: "kubernetes.io/projected/9e5c90e7-d9db-4205-9bc6-c6790b31e53f-kube-api-access-nmr6b") pod "network-check-target-9d4m6" (UID: "9e5c90e7-d9db-4205-9bc6-c6790b31e53f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 03:19:28.599803 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:28.599765    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d1adf33b-0171-446e-aabe-f60fb74197e3-original-pull-secret\") pod \"global-pull-secret-syncer-4944l\" (UID: \"d1adf33b-0171-446e-aabe-f60fb74197e3\") " pod="kube-system/global-pull-secret-syncer-4944l"
Sep 17 03:19:28.599942 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:19:28.599928    2552 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 03:19:28.600011 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:19:28.600001    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1adf33b-0171-446e-aabe-f60fb74197e3-original-pull-secret podName:d1adf33b-0171-446e-aabe-f60fb74197e3 nodeName:}" failed. No retries permitted until 2026-09-17 03:19:30.599982833 +0000 UTC m=+6.066390340 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d1adf33b-0171-446e-aabe-f60fb74197e3-original-pull-secret") pod "global-pull-secret-syncer-4944l" (UID: "d1adf33b-0171-446e-aabe-f60fb74197e3") : object "kube-system"/"original-pull-secret" not registered
Sep 17 03:19:28.911416 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:28.911273    2552 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 03:14:27 +0000 UTC" deadline="2028-06-29 13:32:31.475736878 +0000 UTC"
Sep 17 03:19:28.911416 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:28.911317    2552 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15634h13m2.564426075s"
Sep 17 03:19:28.982540 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:28.982504    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9d4m6"
Sep 17 03:19:28.982736 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:19:28.982648    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9d4m6" podUID="9e5c90e7-d9db-4205-9bc6-c6790b31e53f"
Sep 17 03:19:28.982862 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:28.982747    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4944l"
Sep 17 03:19:28.982862 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:19:28.982832    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4944l" podUID="d1adf33b-0171-446e-aabe-f60fb74197e3"
Sep 17 03:19:29.002354 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:29.002315    2552 generic.go:356] "Generic (PLEG): container finished" podID="a15e02334a54e08adac5d9c153f86344" containerID="b7c2f3504693c988610f9b7fcb0aad6bbde3015a45a12ab7cfbea8cb364c8a09" exitCode=0
Sep 17 03:19:29.003023 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:29.002614    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-53.ec2.internal" event={"ID":"a15e02334a54e08adac5d9c153f86344","Type":"ContainerDied","Data":"b7c2f3504693c988610f9b7fcb0aad6bbde3015a45a12ab7cfbea8cb364c8a09"}
Sep 17 03:19:29.012749 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:29.012546    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-8mzx5" event={"ID":"2a4ab360-f144-4999-97dc-6796bd4a44c2","Type":"ContainerStarted","Data":"2239a3362d19d348a637308c7ad76ca838da6e1b467e4d95d12bee34d38fd634"}
Sep 17 03:19:29.022278 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:29.021990    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qspfk" event={"ID":"27724468-aead-4a4a-9ec1-c11d8fa2bb7c","Type":"ContainerStarted","Data":"813dca49faca11d21b30232e7c47603bc0e7fae9badbe7f9b3844d12bdd53eda"}
Sep 17 03:19:29.034669 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:29.034568    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-mvmw5" event={"ID":"045af8b4-e20b-46e7-bbcf-b5465710791d","Type":"ContainerStarted","Data":"89fca065219e798c6afcadac139dc6712102eb54ed61d89a84debd2f0886cb3b"}
Sep 17 03:19:29.042128 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:29.042083    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-53.ec2.internal" event={"ID":"ce8d1a2b33a64a4f4ec3c7eebb5340d9","Type":"ContainerStarted","Data":"132c3ef6387559df5edb33931855a9a8e97732424d78ed7847e34a98802b8a8b"}
Sep 17 03:19:29.059987 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:29.059943    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-gxmbv" event={"ID":"b3bac7f7-fbb1-4cec-8a26-046b702f4d70","Type":"ContainerStarted","Data":"dbdc777b1787343a9ebaa96b562828898b4ce652c6e97cce5640d6586d558977"}
Sep 17 03:19:29.067426 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:29.067331    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-wds8n" event={"ID":"c6a03737-feb7-48d2-82ba-5a2188f80980","Type":"ContainerStarted","Data":"0c1433ae7395a50bc736324306aa604a7822d587c76a0176b274c625a5064040"}
Sep 17 03:19:29.074595 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:29.074553    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ldhl4" event={"ID":"d3e92f88-fcb1-4d0f-b653-77b7ec5681fd","Type":"ContainerStarted","Data":"4d5d81bc0ce854d6018b75829f87a65738ccc2317403fb46d4c0eeefade21e26"}
Sep 17 03:19:29.086534 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:29.086489    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-vkdcx" event={"ID":"4e98ab7d-ed30-48e6-9751-cc40209037aa","Type":"ContainerStarted","Data":"ca165128a8f934b3ce14718c33346cd22d25485690958da7d0051caab1f7e212"}
Sep 17 03:19:29.102544 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:29.102502    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nkrxz" event={"ID":"0cc4aedf-1bb8-47c6-88f2-266bbbf99058","Type":"ContainerStarted","Data":"0a1ad05d18e1af66463c4a7fa96d73563eec03a4819b26949dc78a51b39d443e"}
Sep 17 03:19:29.122003 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:29.121955    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-rld2q" event={"ID":"b7b79096-9bfc-4d82-9406-073a3f7961d0","Type":"ContainerStarted","Data":"368f680d7206e24a75b692d24b905ebe4276e7cdb073a3541982525ce555b969"}
Sep 17 03:19:29.979928 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:29.979841    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xl9wh"
Sep 17 03:19:29.980405 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:19:29.979986    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xl9wh" podUID="2758c0c8-e1ff-454a-9f0b-43cc8e1f8609"
Sep 17 03:19:30.146786 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:30.146745    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-53.ec2.internal" event={"ID":"a15e02334a54e08adac5d9c153f86344","Type":"ContainerStarted","Data":"373c258861057d1380f3f980b5473f5e82baccba4401519b1a7d6b590a24aea3"}
Sep 17 03:19:30.163224 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:30.163151    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-53.ec2.internal" podStartSLOduration=3.163124139 podStartE2EDuration="3.163124139s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 03:19:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:19:30.161776986 +0000 UTC m=+5.628184516" watchObservedRunningTime="2026-09-17 03:19:30.163124139 +0000 UTC m=+5.629531670"
Sep 17 03:19:30.163505 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:30.163473    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-53.ec2.internal" podStartSLOduration=3.163459763 podStartE2EDuration="3.163459763s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 03:19:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:19:29.060182535 +0000 UTC m=+4.526590065" watchObservedRunningTime="2026-09-17 03:19:30.163459763 +0000 UTC m=+5.629867295"
Sep 17 03:19:30.518049 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:30.518003    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2758c0c8-e1ff-454a-9f0b-43cc8e1f8609-metrics-certs\") pod \"network-metrics-daemon-xl9wh\" (UID: \"2758c0c8-e1ff-454a-9f0b-43cc8e1f8609\") " pod="openshift-multus/network-metrics-daemon-xl9wh"
Sep 17 03:19:30.518249 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:30.518071    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmr6b\" (UniqueName: \"kubernetes.io/projected/9e5c90e7-d9db-4205-9bc6-c6790b31e53f-kube-api-access-nmr6b\") pod \"network-check-target-9d4m6\" (UID: \"9e5c90e7-d9db-4205-9bc6-c6790b31e53f\") " pod="openshift-network-diagnostics/network-check-target-9d4m6"
Sep 17 03:19:30.518360 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:19:30.518300    2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 03:19:30.518424 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:19:30.518371    2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 03:19:30.518424 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:19:30.518386    2552 projected.go:196] Error preparing data for projected volume kube-api-access-nmr6b for pod openshift-network-diagnostics/network-check-target-9d4m6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 03:19:30.518538 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:19:30.518454    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9e5c90e7-d9db-4205-9bc6-c6790b31e53f-kube-api-access-nmr6b podName:9e5c90e7-d9db-4205-9bc6-c6790b31e53f nodeName:}" failed. No retries permitted until 2026-09-17 03:19:34.518432783 +0000 UTC m=+9.984840303 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-nmr6b" (UniqueName: "kubernetes.io/projected/9e5c90e7-d9db-4205-9bc6-c6790b31e53f-kube-api-access-nmr6b") pod "network-check-target-9d4m6" (UID: "9e5c90e7-d9db-4205-9bc6-c6790b31e53f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 03:19:30.518953 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:19:30.518931    2552 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 03:19:30.519035 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:19:30.518996    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2758c0c8-e1ff-454a-9f0b-43cc8e1f8609-metrics-certs podName:2758c0c8-e1ff-454a-9f0b-43cc8e1f8609 nodeName:}" failed. No retries permitted until 2026-09-17 03:19:34.51897804 +0000 UTC m=+9.985385550 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2758c0c8-e1ff-454a-9f0b-43cc8e1f8609-metrics-certs") pod "network-metrics-daemon-xl9wh" (UID: "2758c0c8-e1ff-454a-9f0b-43cc8e1f8609") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 03:19:30.619209 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:30.619166    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d1adf33b-0171-446e-aabe-f60fb74197e3-original-pull-secret\") pod \"global-pull-secret-syncer-4944l\" (UID: \"d1adf33b-0171-446e-aabe-f60fb74197e3\") " pod="kube-system/global-pull-secret-syncer-4944l"
Sep 17 03:19:30.619422 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:19:30.619314    2552 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 03:19:30.619422 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:19:30.619367    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1adf33b-0171-446e-aabe-f60fb74197e3-original-pull-secret podName:d1adf33b-0171-446e-aabe-f60fb74197e3 nodeName:}" failed. No retries permitted until 2026-09-17 03:19:34.619353286 +0000 UTC m=+10.085760806 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d1adf33b-0171-446e-aabe-f60fb74197e3-original-pull-secret") pod "global-pull-secret-syncer-4944l" (UID: "d1adf33b-0171-446e-aabe-f60fb74197e3") : object "kube-system"/"original-pull-secret" not registered
Sep 17 03:19:30.979537 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:30.979446    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9d4m6"
Sep 17 03:19:30.979738 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:19:30.979600    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9d4m6" podUID="9e5c90e7-d9db-4205-9bc6-c6790b31e53f"
Sep 17 03:19:30.980118 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:30.980096    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4944l"
Sep 17 03:19:30.980513 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:19:30.980224    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4944l" podUID="d1adf33b-0171-446e-aabe-f60fb74197e3"
Sep 17 03:19:31.979676 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:31.979058    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xl9wh"
Sep 17 03:19:31.979676 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:19:31.979217    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xl9wh" podUID="2758c0c8-e1ff-454a-9f0b-43cc8e1f8609"
Sep 17 03:19:32.982449 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:32.982402    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9d4m6"
Sep 17 03:19:32.982885 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:19:32.982781    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9d4m6" podUID="9e5c90e7-d9db-4205-9bc6-c6790b31e53f"
Sep 17 03:19:32.983035 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:32.983003    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4944l"
Sep 17 03:19:32.983253 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:19:32.983219    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4944l" podUID="d1adf33b-0171-446e-aabe-f60fb74197e3"
Sep 17 03:19:33.979775 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:33.979676    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xl9wh"
Sep 17 03:19:33.979945 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:19:33.979851    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xl9wh" podUID="2758c0c8-e1ff-454a-9f0b-43cc8e1f8609"
Sep 17 03:19:34.555727 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:34.554838    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2758c0c8-e1ff-454a-9f0b-43cc8e1f8609-metrics-certs\") pod \"network-metrics-daemon-xl9wh\" (UID: \"2758c0c8-e1ff-454a-9f0b-43cc8e1f8609\") " pod="openshift-multus/network-metrics-daemon-xl9wh"
Sep 17 03:19:34.555727 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:34.554906    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmr6b\" (UniqueName: \"kubernetes.io/projected/9e5c90e7-d9db-4205-9bc6-c6790b31e53f-kube-api-access-nmr6b\") pod \"network-check-target-9d4m6\" (UID: \"9e5c90e7-d9db-4205-9bc6-c6790b31e53f\") " pod="openshift-network-diagnostics/network-check-target-9d4m6"
Sep 17 03:19:34.555727 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:19:34.555089    2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 03:19:34.555727 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:19:34.555113    2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 03:19:34.555727 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:19:34.555127    2552 projected.go:196] Error preparing data for projected volume kube-api-access-nmr6b for pod openshift-network-diagnostics/network-check-target-9d4m6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 03:19:34.555727 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:19:34.555193    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9e5c90e7-d9db-4205-9bc6-c6790b31e53f-kube-api-access-nmr6b podName:9e5c90e7-d9db-4205-9bc6-c6790b31e53f nodeName:}" failed. No retries permitted until 2026-09-17 03:19:42.555172484 +0000 UTC m=+18.021579998 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-nmr6b" (UniqueName: "kubernetes.io/projected/9e5c90e7-d9db-4205-9bc6-c6790b31e53f-kube-api-access-nmr6b") pod "network-check-target-9d4m6" (UID: "9e5c90e7-d9db-4205-9bc6-c6790b31e53f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 03:19:34.555727 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:19:34.555626    2552 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 03:19:34.555727 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:19:34.555690    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2758c0c8-e1ff-454a-9f0b-43cc8e1f8609-metrics-certs podName:2758c0c8-e1ff-454a-9f0b-43cc8e1f8609 nodeName:}" failed. No retries permitted until 2026-09-17 03:19:42.555659619 +0000 UTC m=+18.022067128 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2758c0c8-e1ff-454a-9f0b-43cc8e1f8609-metrics-certs") pod "network-metrics-daemon-xl9wh" (UID: "2758c0c8-e1ff-454a-9f0b-43cc8e1f8609") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 03:19:34.655831 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:34.655524    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d1adf33b-0171-446e-aabe-f60fb74197e3-original-pull-secret\") pod \"global-pull-secret-syncer-4944l\" (UID: \"d1adf33b-0171-446e-aabe-f60fb74197e3\") " pod="kube-system/global-pull-secret-syncer-4944l"
Sep 17 03:19:34.655831 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:19:34.655701    2552 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 03:19:34.655831 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:19:34.655778    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1adf33b-0171-446e-aabe-f60fb74197e3-original-pull-secret podName:d1adf33b-0171-446e-aabe-f60fb74197e3 nodeName:}" failed. No retries permitted until 2026-09-17 03:19:42.655757941 +0000 UTC m=+18.122165465 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d1adf33b-0171-446e-aabe-f60fb74197e3-original-pull-secret") pod "global-pull-secret-syncer-4944l" (UID: "d1adf33b-0171-446e-aabe-f60fb74197e3") : object "kube-system"/"original-pull-secret" not registered
Sep 17 03:19:34.980337 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:34.980250    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9d4m6"
Sep 17 03:19:34.980337 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:34.980303    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4944l"
Sep 17 03:19:34.980596 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:19:34.980501    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4944l" podUID="d1adf33b-0171-446e-aabe-f60fb74197e3"
Sep 17 03:19:34.980596 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:19:34.980414    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9d4m6" podUID="9e5c90e7-d9db-4205-9bc6-c6790b31e53f"
Sep 17 03:19:35.979451 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:35.979401    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xl9wh"
Sep 17 03:19:35.979963 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:19:35.979559    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xl9wh" podUID="2758c0c8-e1ff-454a-9f0b-43cc8e1f8609"
Sep 17 03:19:36.979258 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:36.979085    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9d4m6"
Sep 17 03:19:36.979258 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:19:36.979226    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9d4m6" podUID="9e5c90e7-d9db-4205-9bc6-c6790b31e53f"
Sep 17 03:19:36.979729 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:36.979565    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4944l"
Sep 17 03:19:36.979729 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:19:36.979684    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4944l" podUID="d1adf33b-0171-446e-aabe-f60fb74197e3"
Sep 17 03:19:37.979545 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:37.979503    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xl9wh"
Sep 17 03:19:37.979752 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:19:37.979680    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xl9wh" podUID="2758c0c8-e1ff-454a-9f0b-43cc8e1f8609"
Sep 17 03:19:38.979391 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:38.979351    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9d4m6"
Sep 17 03:19:38.979588 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:38.979399    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4944l"
Sep 17 03:19:38.979588 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:19:38.979483    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9d4m6" podUID="9e5c90e7-d9db-4205-9bc6-c6790b31e53f"
Sep 17 03:19:38.979725 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:19:38.979635    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4944l" podUID="d1adf33b-0171-446e-aabe-f60fb74197e3"
Sep 17 03:19:39.979787 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:39.979751    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xl9wh"
Sep 17 03:19:39.980270 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:19:39.979881    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xl9wh" podUID="2758c0c8-e1ff-454a-9f0b-43cc8e1f8609"
Sep 17 03:19:40.979568 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:40.979485    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9d4m6"
Sep 17 03:19:40.979739 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:19:40.979639    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9d4m6" podUID="9e5c90e7-d9db-4205-9bc6-c6790b31e53f"
Sep 17 03:19:40.979739 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:40.979693    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4944l"
Sep 17 03:19:40.980118 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:19:40.979816    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4944l" podUID="d1adf33b-0171-446e-aabe-f60fb74197e3"
Sep 17 03:19:41.979444 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:41.979408    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xl9wh"
Sep 17 03:19:41.979709 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:19:41.979546    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xl9wh" podUID="2758c0c8-e1ff-454a-9f0b-43cc8e1f8609"
Sep 17 03:19:42.619743 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:42.619706    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2758c0c8-e1ff-454a-9f0b-43cc8e1f8609-metrics-certs\") pod \"network-metrics-daemon-xl9wh\" (UID: \"2758c0c8-e1ff-454a-9f0b-43cc8e1f8609\") " pod="openshift-multus/network-metrics-daemon-xl9wh"
Sep 17 03:19:42.619743 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:42.619749    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmr6b\" (UniqueName: \"kubernetes.io/projected/9e5c90e7-d9db-4205-9bc6-c6790b31e53f-kube-api-access-nmr6b\") pod \"network-check-target-9d4m6\" (UID: \"9e5c90e7-d9db-4205-9bc6-c6790b31e53f\") " pod="openshift-network-diagnostics/network-check-target-9d4m6"
Sep 17 03:19:42.620219 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:19:42.619862    2552 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 03:19:42.620219 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:19:42.619885    2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 03:19:42.620219 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:19:42.619902    2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 03:19:42.620219 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:19:42.619915    2552 projected.go:196] Error preparing data for projected volume kube-api-access-nmr6b for pod openshift-network-diagnostics/network-check-target-9d4m6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 03:19:42.620219 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:19:42.619935    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2758c0c8-e1ff-454a-9f0b-43cc8e1f8609-metrics-certs podName:2758c0c8-e1ff-454a-9f0b-43cc8e1f8609 nodeName:}" failed. No retries permitted until 2026-09-17 03:19:58.619913949 +0000 UTC m=+34.086321465 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2758c0c8-e1ff-454a-9f0b-43cc8e1f8609-metrics-certs") pod "network-metrics-daemon-xl9wh" (UID: "2758c0c8-e1ff-454a-9f0b-43cc8e1f8609") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 03:19:42.620219 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:19:42.619950    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9e5c90e7-d9db-4205-9bc6-c6790b31e53f-kube-api-access-nmr6b podName:9e5c90e7-d9db-4205-9bc6-c6790b31e53f nodeName:}" failed. No retries permitted until 2026-09-17 03:19:58.619940967 +0000 UTC m=+34.086348491 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-nmr6b" (UniqueName: "kubernetes.io/projected/9e5c90e7-d9db-4205-9bc6-c6790b31e53f-kube-api-access-nmr6b") pod "network-check-target-9d4m6" (UID: "9e5c90e7-d9db-4205-9bc6-c6790b31e53f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 03:19:42.720991 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:42.720954    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d1adf33b-0171-446e-aabe-f60fb74197e3-original-pull-secret\") pod \"global-pull-secret-syncer-4944l\" (UID: \"d1adf33b-0171-446e-aabe-f60fb74197e3\") " pod="kube-system/global-pull-secret-syncer-4944l"
Sep 17 03:19:42.721182 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:19:42.721103    2552 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 03:19:42.721182 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:19:42.721171    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1adf33b-0171-446e-aabe-f60fb74197e3-original-pull-secret podName:d1adf33b-0171-446e-aabe-f60fb74197e3 nodeName:}" failed. No retries permitted until 2026-09-17 03:19:58.721155705 +0000 UTC m=+34.187563214 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d1adf33b-0171-446e-aabe-f60fb74197e3-original-pull-secret") pod "global-pull-secret-syncer-4944l" (UID: "d1adf33b-0171-446e-aabe-f60fb74197e3") : object "kube-system"/"original-pull-secret" not registered
Sep 17 03:19:42.979662 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:42.979569    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4944l"
Sep 17 03:19:42.979662 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:42.979589    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9d4m6"
Sep 17 03:19:42.979869 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:19:42.979719    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4944l" podUID="d1adf33b-0171-446e-aabe-f60fb74197e3"
Sep 17 03:19:42.979869 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:19:42.979851    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9d4m6" podUID="9e5c90e7-d9db-4205-9bc6-c6790b31e53f"
Sep 17 03:19:43.979569 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:43.979511    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xl9wh"
Sep 17 03:19:43.980057 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:19:43.979723    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xl9wh" podUID="2758c0c8-e1ff-454a-9f0b-43cc8e1f8609"
Sep 17 03:19:44.980287 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:44.980254    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9d4m6"
Sep 17 03:19:44.980740 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:19:44.980358    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9d4m6" podUID="9e5c90e7-d9db-4205-9bc6-c6790b31e53f"
Sep 17 03:19:44.980740 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:44.980427    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4944l"
Sep 17 03:19:44.980740 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:19:44.980543    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4944l" podUID="d1adf33b-0171-446e-aabe-f60fb74197e3"
Sep 17 03:19:45.980133 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:45.980096    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xl9wh"
Sep 17 03:19:45.980318 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:19:45.980220    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xl9wh" podUID="2758c0c8-e1ff-454a-9f0b-43cc8e1f8609"
Sep 17 03:19:46.979720 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:46.979434    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9d4m6"
Sep 17 03:19:46.979922 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:46.979479    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4944l"
Sep 17 03:19:46.979922 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:19:46.979839    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9d4m6" podUID="9e5c90e7-d9db-4205-9bc6-c6790b31e53f"
Sep 17 03:19:46.979922 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:19:46.979871    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4944l" podUID="d1adf33b-0171-446e-aabe-f60fb74197e3"
Sep 17 03:19:47.184922 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:47.184880    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-wds8n" event={"ID":"c6a03737-feb7-48d2-82ba-5a2188f80980","Type":"ContainerStarted","Data":"48843aa5f4128e40f4fab757c811228111fa15115b8f9905bcbbed539e5bcdee"}
Sep 17 03:19:47.187789 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:47.187759    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ldhl4" event={"ID":"d3e92f88-fcb1-4d0f-b653-77b7ec5681fd","Type":"ContainerStarted","Data":"0b43b1f4f83cb81f996d96b6ef06335b8d656a10cfd6b49f8301af26af9c84e0"}
Sep 17 03:19:47.187933 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:47.187841    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ldhl4" event={"ID":"d3e92f88-fcb1-4d0f-b653-77b7ec5681fd","Type":"ContainerStarted","Data":"37d4c590a0060bf2a3b560b57a7a48a65dbd986461d66d6c1a9f39797b5c366d"}
Sep 17 03:19:47.187933 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:47.187890    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ldhl4" event={"ID":"d3e92f88-fcb1-4d0f-b653-77b7ec5681fd","Type":"ContainerStarted","Data":"d37364393bf0b397a57227cba91e42823b577f2d5821c542c356c2ec25c33b24"}
Sep 17 03:19:47.187933 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:47.187920    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ldhl4" event={"ID":"d3e92f88-fcb1-4d0f-b653-77b7ec5681fd","Type":"ContainerStarted","Data":"d62ea56832b913ee34f2413c669a85dce66f725b5ecc4f3600f8f42834211fb7"}
Sep 17 03:19:47.188044 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:47.187958    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ldhl4" event={"ID":"d3e92f88-fcb1-4d0f-b653-77b7ec5681fd","Type":"ContainerStarted","Data":"cb32a8aa8cf33da004f2115d7f0d95263043151d5cec09e55e16dd2981336a20"}
Sep 17 03:19:47.189267 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:47.189241    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-vkdcx" event={"ID":"4e98ab7d-ed30-48e6-9751-cc40209037aa","Type":"ContainerStarted","Data":"b0e36f833fb7868a8e1e1d16d25c535b41b122ac6a38b17c878d479e620b0131"}
Sep 17 03:19:47.190839 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:47.190814    2552 generic.go:356] "Generic (PLEG): container finished" podID="0cc4aedf-1bb8-47c6-88f2-266bbbf99058" containerID="a332be73e1dca42535b2cb661ec27f8de2a77a436465eac8f3230794e27a32a4" exitCode=0
Sep 17 03:19:47.190935 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:47.190907    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nkrxz" event={"ID":"0cc4aedf-1bb8-47c6-88f2-266bbbf99058","Type":"ContainerDied","Data":"a332be73e1dca42535b2cb661ec27f8de2a77a436465eac8f3230794e27a32a4"}
Sep 17 03:19:47.192461 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:47.192436    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-rld2q" event={"ID":"b7b79096-9bfc-4d82-9406-073a3f7961d0","Type":"ContainerStarted","Data":"86e379e0450dfc103954b3c0b7fd8e6f9b2f40222d112f2f024c965931232359"}
Sep 17 03:19:47.194123 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:47.194096    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-8mzx5" event={"ID":"2a4ab360-f144-4999-97dc-6796bd4a44c2","Type":"ContainerStarted","Data":"297f43aea37106b7ed05e540532bb217cc11a2fbfc28a64334f4b0121b5e9ef8"}
Sep 17 03:19:47.195908 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:47.195873    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qspfk" event={"ID":"27724468-aead-4a4a-9ec1-c11d8fa2bb7c","Type":"ContainerStarted","Data":"facf84634997d82ce682e13f777d96f93079aec5fdaeabe4afbdbcd8605327da"}
Sep 17 03:19:47.199477 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:47.199446    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-gxmbv" event={"ID":"b3bac7f7-fbb1-4cec-8a26-046b702f4d70","Type":"ContainerStarted","Data":"0f178698782182b3587660f257b1cce4a0ab7fdc66f585c19ea97bd6a3f7a537"}
Sep 17 03:19:47.239517 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:47.239467    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-vkdcx" podStartSLOduration=3.899448832 podStartE2EDuration="22.239449581s" totalImagesPullingTime="18.340000749s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 03:19:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:19:47.22418509 +0000 UTC m=+22.690592615" watchObservedRunningTime="2026-09-17 03:19:47.239449581 +0000 UTC m=+22.705857165"
Sep 17 03:19:47.255034 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:47.254984    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-wds8n" podStartSLOduration=3.873680246 podStartE2EDuration="22.254970009s" totalImagesPullingTime="18.381289763s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 03:19:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:19:47.239256027 +0000 UTC m=+22.705663552" watchObservedRunningTime="2026-09-17 03:19:47.254970009 +0000 UTC m=+22.721377537"
Sep 17 03:19:47.255279 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:47.255261    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-gxmbv" podStartSLOduration=3.929665063 podStartE2EDuration="22.255255244s" totalImagesPullingTime="18.325590181s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 03:19:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:19:47.254625893 +0000 UTC m=+22.721033419" watchObservedRunningTime="2026-09-17 03:19:47.255255244 +0000 UTC m=+22.721662769"
Sep 17 03:19:47.268852 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:47.268801    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-rld2q" podStartSLOduration=3.935496301 podStartE2EDuration="22.268785302s" totalImagesPullingTime="18.333289001s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 03:19:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:19:47.268177512 +0000 UTC m=+22.734585039" watchObservedRunningTime="2026-09-17 03:19:47.268785302 +0000 UTC m=+22.735192828"
Sep 17 03:19:47.281307 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:47.281259    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-8mzx5" podStartSLOduration=11.402375162 podStartE2EDuration="22.281242044s" totalImagesPullingTime="10.878866882s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 03:19:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:19:47.281104377 +0000 UTC m=+22.747511904" watchObservedRunningTime="2026-09-17 03:19:47.281242044 +0000 UTC m=+22.747649569"
Sep 17 03:19:47.383870 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:47.383839    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mzx5_2a4ab360-f144-4999-97dc-6796bd4a44c2/konnectivity-agent/0.log" limit=1024
Sep 17 03:19:47.434989 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:47.434910    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mzx5_2a4ab360-f144-4999-97dc-6796bd4a44c2/konnectivity-agent/0.log" limit=1024
Sep 17 03:19:47.979273 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:47.979235    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xl9wh"
Sep 17 03:19:47.979516 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:19:47.979372    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xl9wh" podUID="2758c0c8-e1ff-454a-9f0b-43cc8e1f8609"
Sep 17 03:19:47.998919 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:47.998876    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-8mzx5"
Sep 17 03:19:47.999571 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:47.999548    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-8mzx5"
Sep 17 03:19:48.203785 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:48.203582    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-mvmw5" event={"ID":"045af8b4-e20b-46e7-bbcf-b5465710791d","Type":"ContainerStarted","Data":"c9ff4cdfa130512e012b0458350322ff9a2e549eed80eea7bb675ee2e2589fd8"}
Sep 17 03:19:48.206917 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:48.206883    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ldhl4" event={"ID":"d3e92f88-fcb1-4d0f-b653-77b7ec5681fd","Type":"ContainerStarted","Data":"a6b962ca608341c2eaa163da2958bc095ba5f61d64bd19db3ab9bdd0ef70ac79"}
Sep 17 03:19:48.225008 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:48.224944    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-mvmw5" podStartSLOduration=4.888661056 podStartE2EDuration="23.2249197s" totalImagesPullingTime="18.336258644s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 03:19:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:19:48.224289109 +0000 UTC m=+23.690696636" watchObservedRunningTime="2026-09-17 03:19:48.2249197 +0000 UTC m=+23.691327226"
Sep 17 03:19:48.238225 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:48.238109    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mzx5_2a4ab360-f144-4999-97dc-6796bd4a44c2/konnectivity-agent/0.log" limit=1024
Sep 17 03:19:48.259520 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:48.259497    2552 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 17 03:19:48.339013 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:48.338977    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mzx5_2a4ab360-f144-4999-97dc-6796bd4a44c2/konnectivity-agent/0.log" limit=1024
Sep 17 03:19:48.439470 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:48.439432    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mzx5_2a4ab360-f144-4999-97dc-6796bd4a44c2/konnectivity-agent/0.log" limit=1024
Sep 17 03:19:48.487898 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:48.487859    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mzx5_2a4ab360-f144-4999-97dc-6796bd4a44c2/konnectivity-agent/0.log" limit=1024
Sep 17 03:19:48.702287 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:48.702247    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mzx5_2a4ab360-f144-4999-97dc-6796bd4a44c2/konnectivity-agent/0.log" limit=1024
Sep 17 03:19:48.757253 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:48.757217    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mzx5_2a4ab360-f144-4999-97dc-6796bd4a44c2/konnectivity-agent/0.log" limit=1024
Sep 17 03:19:48.949592 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:48.949406    2552 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-17T03:19:48.259514763Z","UUID":"b4a4a120-ef8d-4ed4-bc90-53fde529f80e","Handler":null,"Name":"","Endpoint":""}
Sep 17 03:19:48.953249 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:48.953165    2552 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 17 03:19:48.953249 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:48.953208    2552 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 17 03:19:48.979938 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:48.979439    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4944l"
Sep 17 03:19:48.979938 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:48.979439    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9d4m6"
Sep 17 03:19:48.979938 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:19:48.979566    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4944l" podUID="d1adf33b-0171-446e-aabe-f60fb74197e3"
Sep 17 03:19:48.979938 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:19:48.979728    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9d4m6" podUID="9e5c90e7-d9db-4205-9bc6-c6790b31e53f"
Sep 17 03:19:49.211723 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:49.211621    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qspfk" event={"ID":"27724468-aead-4a4a-9ec1-c11d8fa2bb7c","Type":"ContainerStarted","Data":"f7f8e1fe4e3762ebf95984d4c7cffb8b74f1f7d35e7ab95869e634bb791be183"}
Sep 17 03:19:49.211723 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:49.211663    2552 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 17 03:19:49.980081 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:49.980037    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xl9wh"
Sep 17 03:19:49.980282 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:19:49.980164    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xl9wh" podUID="2758c0c8-e1ff-454a-9f0b-43cc8e1f8609"
Sep 17 03:19:50.215518 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:50.215486    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qspfk" event={"ID":"27724468-aead-4a4a-9ec1-c11d8fa2bb7c","Type":"ContainerStarted","Data":"ca2abdb3e0bbe73234dd90ddce3694711a52daf64c6ec29b3ac8754a2f0e063e"}
Sep 17 03:19:50.222310 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:50.222277    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ldhl4" event={"ID":"d3e92f88-fcb1-4d0f-b653-77b7ec5681fd","Type":"ContainerStarted","Data":"9385db3c8f74d178c292e573cb152191ac4c57e05375062e6f6a31aeb44fc5ea"}
Sep 17 03:19:50.232438 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:50.232333    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qspfk" podStartSLOduration=4.410606518 podStartE2EDuration="25.232318439s" totalImagesPullingTime="20.821711921s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 03:19:25 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:19:50.232101861 +0000 UTC m=+25.698509388" watchObservedRunningTime="2026-09-17 03:19:50.232318439 +0000 UTC m=+25.698725963"
Sep 17 03:19:50.603292 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:50.603255    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-8mzx5"
Sep 17 03:19:50.603497 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:50.603392    2552 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 17 03:19:50.603827 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:50.603803    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-8mzx5"
Sep 17 03:19:50.661169 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:50.661130    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mzx5_2a4ab360-f144-4999-97dc-6796bd4a44c2/konnectivity-agent/0.log" limit=1024
Sep 17 03:19:50.979685 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:50.979583    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4944l"
Sep 17 03:19:50.979685 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:50.979583    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9d4m6"
Sep 17 03:19:50.979906 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:19:50.979743    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4944l" podUID="d1adf33b-0171-446e-aabe-f60fb74197e3"
Sep 17 03:19:50.979906 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:19:50.979843    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9d4m6" podUID="9e5c90e7-d9db-4205-9bc6-c6790b31e53f"
Sep 17 03:19:51.969448 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:51.969352    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mzx5_2a4ab360-f144-4999-97dc-6796bd4a44c2/konnectivity-agent/0.log" limit=1024
Sep 17 03:19:51.979271 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:51.979233    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xl9wh"
Sep 17 03:19:51.979470 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:19:51.979382    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xl9wh" podUID="2758c0c8-e1ff-454a-9f0b-43cc8e1f8609"
Sep 17 03:19:52.069910 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:52.069877    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mzx5_2a4ab360-f144-4999-97dc-6796bd4a44c2/konnectivity-agent/0.log" limit=1024
Sep 17 03:19:52.982433 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:52.982228    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4944l"
Sep 17 03:19:52.983035 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:52.982295    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9d4m6"
Sep 17 03:19:52.983035 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:19:52.982511    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4944l" podUID="d1adf33b-0171-446e-aabe-f60fb74197e3"
Sep 17 03:19:52.983035 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:19:52.982609    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9d4m6" podUID="9e5c90e7-d9db-4205-9bc6-c6790b31e53f"
Sep 17 03:19:53.233568 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:53.233482    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ldhl4" event={"ID":"d3e92f88-fcb1-4d0f-b653-77b7ec5681fd","Type":"ContainerStarted","Data":"ddf38d65fe886f1a717f5a7756a00b0871a695905f6c4f11d3532176758e360a"}
Sep 17 03:19:53.233773 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:53.233746    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ldhl4"
Sep 17 03:19:53.235521 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:53.235490    2552 generic.go:356] "Generic (PLEG): container finished" podID="0cc4aedf-1bb8-47c6-88f2-266bbbf99058" containerID="e420319c17ba6deb680ea88d0a4a5353fd7014dbad81d245b2e3b1e0d302b3cd" exitCode=0
Sep 17 03:19:53.235656 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:53.235538    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nkrxz" event={"ID":"0cc4aedf-1bb8-47c6-88f2-266bbbf99058","Type":"ContainerDied","Data":"e420319c17ba6deb680ea88d0a4a5353fd7014dbad81d245b2e3b1e0d302b3cd"}
Sep 17 03:19:53.250826 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:53.250798    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ldhl4"
Sep 17 03:19:53.306796 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:53.306744    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-ldhl4" podStartSLOduration=9.922505162 podStartE2EDuration="28.306727887s" totalImagesPullingTime="18.384222725s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 03:19:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:19:53.272903628 +0000 UTC m=+28.739311157" watchObservedRunningTime="2026-09-17 03:19:53.306727887 +0000 UTC m=+28.773135414"
Sep 17 03:19:53.979525 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:53.979433    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xl9wh"
Sep 17 03:19:53.988062 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:19:53.979574    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xl9wh" podUID="2758c0c8-e1ff-454a-9f0b-43cc8e1f8609"
Sep 17 03:19:54.238624 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:54.238541    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ldhl4"
Sep 17 03:19:54.238763 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:54.238627    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ldhl4"
Sep 17 03:19:54.254224 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:54.254197    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ldhl4"
Sep 17 03:19:54.622508 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:54.622315    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-9d4m6"]
Sep 17 03:19:54.622687 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:54.622639    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9d4m6"
Sep 17 03:19:54.622782 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:19:54.622751    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9d4m6" podUID="9e5c90e7-d9db-4205-9bc6-c6790b31e53f"
Sep 17 03:19:54.625573 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:54.625543    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-4944l"]
Sep 17 03:19:54.625800 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:54.625681    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4944l"
Sep 17 03:19:54.625877 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:19:54.625808    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4944l" podUID="d1adf33b-0171-446e-aabe-f60fb74197e3"
Sep 17 03:19:54.626282 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:54.626257    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-xl9wh"]
Sep 17 03:19:54.626408 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:54.626362    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xl9wh"
Sep 17 03:19:54.626498 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:19:54.626466    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xl9wh" podUID="2758c0c8-e1ff-454a-9f0b-43cc8e1f8609"
Sep 17 03:19:55.242952 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:55.242916    2552 generic.go:356] "Generic (PLEG): container finished" podID="0cc4aedf-1bb8-47c6-88f2-266bbbf99058" containerID="80f92e2de00782473f44c1942baee0ab979d0b0decb83d9cfa08b7f51daa10fe" exitCode=0
Sep 17 03:19:55.243345 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:55.243005    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nkrxz" event={"ID":"0cc4aedf-1bb8-47c6-88f2-266bbbf99058","Type":"ContainerDied","Data":"80f92e2de00782473f44c1942baee0ab979d0b0decb83d9cfa08b7f51daa10fe"}
Sep 17 03:19:55.980068 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:55.980038    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9d4m6"
Sep 17 03:19:55.980222 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:19:55.980144    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9d4m6" podUID="9e5c90e7-d9db-4205-9bc6-c6790b31e53f"
Sep 17 03:19:55.980222 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:55.980202    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4944l"
Sep 17 03:19:55.980341 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:19:55.980322    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4944l" podUID="d1adf33b-0171-446e-aabe-f60fb74197e3"
Sep 17 03:19:56.982318 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:56.982289    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xl9wh"
Sep 17 03:19:56.983072 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:19:56.982417    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xl9wh" podUID="2758c0c8-e1ff-454a-9f0b-43cc8e1f8609"
Sep 17 03:19:57.252159 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:57.252126    2552 generic.go:356] "Generic (PLEG): container finished" podID="0cc4aedf-1bb8-47c6-88f2-266bbbf99058" containerID="dbbe29f43259ad1f9355aa6554cc5691065bdaabe8ffe3c270d1d2952f504e4a" exitCode=0
Sep 17 03:19:57.252413 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:57.252173    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nkrxz" event={"ID":"0cc4aedf-1bb8-47c6-88f2-266bbbf99058","Type":"ContainerDied","Data":"dbbe29f43259ad1f9355aa6554cc5691065bdaabe8ffe3c270d1d2952f504e4a"}
Sep 17 03:19:57.979701 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:57.979655    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4944l"
Sep 17 03:19:57.979701 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:57.979672    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9d4m6"
Sep 17 03:19:57.980030 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:19:57.979779    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4944l" podUID="d1adf33b-0171-446e-aabe-f60fb74197e3"
Sep 17 03:19:57.980030 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:19:57.979906    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9d4m6" podUID="9e5c90e7-d9db-4205-9bc6-c6790b31e53f"
Sep 17 03:19:58.641320 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:58.641285    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2758c0c8-e1ff-454a-9f0b-43cc8e1f8609-metrics-certs\") pod \"network-metrics-daemon-xl9wh\" (UID: \"2758c0c8-e1ff-454a-9f0b-43cc8e1f8609\") " pod="openshift-multus/network-metrics-daemon-xl9wh"
Sep 17 03:19:58.641808 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:58.641336    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmr6b\" (UniqueName: \"kubernetes.io/projected/9e5c90e7-d9db-4205-9bc6-c6790b31e53f-kube-api-access-nmr6b\") pod \"network-check-target-9d4m6\" (UID: \"9e5c90e7-d9db-4205-9bc6-c6790b31e53f\") " pod="openshift-network-diagnostics/network-check-target-9d4m6"
Sep 17 03:19:58.641808 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:19:58.641467    2552 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 03:19:58.641808 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:19:58.641481    2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 03:19:58.641808 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:19:58.641503    2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 03:19:58.641808 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:19:58.641516    2552 projected.go:196] Error preparing data for projected volume kube-api-access-nmr6b for pod openshift-network-diagnostics/network-check-target-9d4m6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 03:19:58.641808 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:19:58.641550    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2758c0c8-e1ff-454a-9f0b-43cc8e1f8609-metrics-certs podName:2758c0c8-e1ff-454a-9f0b-43cc8e1f8609 nodeName:}" failed. No retries permitted until 2026-09-17 03:20:30.641528537 +0000 UTC m=+66.107936057 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2758c0c8-e1ff-454a-9f0b-43cc8e1f8609-metrics-certs") pod "network-metrics-daemon-xl9wh" (UID: "2758c0c8-e1ff-454a-9f0b-43cc8e1f8609") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 03:19:58.641808 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:19:58.641572    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9e5c90e7-d9db-4205-9bc6-c6790b31e53f-kube-api-access-nmr6b podName:9e5c90e7-d9db-4205-9bc6-c6790b31e53f nodeName:}" failed. No retries permitted until 2026-09-17 03:20:30.641560545 +0000 UTC m=+66.107968061 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-nmr6b" (UniqueName: "kubernetes.io/projected/9e5c90e7-d9db-4205-9bc6-c6790b31e53f-kube-api-access-nmr6b") pod "network-check-target-9d4m6" (UID: "9e5c90e7-d9db-4205-9bc6-c6790b31e53f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 03:19:58.741983 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:58.741941    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d1adf33b-0171-446e-aabe-f60fb74197e3-original-pull-secret\") pod \"global-pull-secret-syncer-4944l\" (UID: \"d1adf33b-0171-446e-aabe-f60fb74197e3\") " pod="kube-system/global-pull-secret-syncer-4944l"
Sep 17 03:19:58.742208 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:19:58.742133    2552 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 03:19:58.742267 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:19:58.742205    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1adf33b-0171-446e-aabe-f60fb74197e3-original-pull-secret podName:d1adf33b-0171-446e-aabe-f60fb74197e3 nodeName:}" failed. No retries permitted until 2026-09-17 03:20:30.742189139 +0000 UTC m=+66.208596647 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d1adf33b-0171-446e-aabe-f60fb74197e3-original-pull-secret") pod "global-pull-secret-syncer-4944l" (UID: "d1adf33b-0171-446e-aabe-f60fb74197e3") : object "kube-system"/"original-pull-secret" not registered
Sep 17 03:19:58.979765 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:58.979682    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xl9wh"
Sep 17 03:19:58.979933 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:19:58.979820    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xl9wh" podUID="2758c0c8-e1ff-454a-9f0b-43cc8e1f8609"
Sep 17 03:19:59.377015 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:59.376989    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-53.ec2.internal" event="NodeReady"
Sep 17 03:19:59.377293 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:59.377117    2552 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 17 03:19:59.416107 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:59.416065    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-vb2fm"]
Sep 17 03:19:59.431813 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:59.431778    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-tbtsz"]
Sep 17 03:19:59.431998 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:59.431934    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-vb2fm"
Sep 17 03:19:59.435173 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:59.435143    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-8xsvg\""
Sep 17 03:19:59.435173 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:59.435153    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 17 03:19:59.435375 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:59.435283    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 17 03:19:59.441992 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:59.441967    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-vb2fm"]
Sep 17 03:19:59.442111 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:59.442000    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-tbtsz"]
Sep 17 03:19:59.442152 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:59.442114    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-tbtsz"
Sep 17 03:19:59.444516 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:59.444492    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-tddkt\""
Sep 17 03:19:59.444664 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:59.444523    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 17 03:19:59.444664 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:59.444587    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 17 03:19:59.444784 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:59.444671    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 17 03:19:59.548099 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:59.548050    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6k9h\" (UniqueName: \"kubernetes.io/projected/a7d922cf-74cc-4afb-b900-b21d63d4a225-kube-api-access-m6k9h\") pod \"dns-default-vb2fm\" (UID: \"a7d922cf-74cc-4afb-b900-b21d63d4a225\") " pod="openshift-dns/dns-default-vb2fm"
Sep 17 03:19:59.548342 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:59.548131    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a7d922cf-74cc-4afb-b900-b21d63d4a225-config-volume\") pod \"dns-default-vb2fm\" (UID: \"a7d922cf-74cc-4afb-b900-b21d63d4a225\") " pod="openshift-dns/dns-default-vb2fm"
Sep 17 03:19:59.548342 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:59.548166    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1a74da51-05c3-4067-b3a5-233734643f6e-cert\") pod \"ingress-canary-tbtsz\" (UID: \"1a74da51-05c3-4067-b3a5-233734643f6e\") " pod="openshift-ingress-canary/ingress-canary-tbtsz"
Sep 17 03:19:59.548342 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:59.548211    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a7d922cf-74cc-4afb-b900-b21d63d4a225-metrics-tls\") pod \"dns-default-vb2fm\" (UID: \"a7d922cf-74cc-4afb-b900-b21d63d4a225\") " pod="openshift-dns/dns-default-vb2fm"
Sep 17 03:19:59.548342 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:59.548297    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdzm9\" (UniqueName: \"kubernetes.io/projected/1a74da51-05c3-4067-b3a5-233734643f6e-kube-api-access-pdzm9\") pod \"ingress-canary-tbtsz\" (UID: \"1a74da51-05c3-4067-b3a5-233734643f6e\") " pod="openshift-ingress-canary/ingress-canary-tbtsz"
Sep 17 03:19:59.548342 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:59.548327    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7d922cf-74cc-4afb-b900-b21d63d4a225-tmp-dir\") pod \"dns-default-vb2fm\" (UID: \"a7d922cf-74cc-4afb-b900-b21d63d4a225\") " pod="openshift-dns/dns-default-vb2fm"
Sep 17 03:19:59.649372 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:59.649279    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdzm9\" (UniqueName: \"kubernetes.io/projected/1a74da51-05c3-4067-b3a5-233734643f6e-kube-api-access-pdzm9\") pod \"ingress-canary-tbtsz\" (UID: \"1a74da51-05c3-4067-b3a5-233734643f6e\") " pod="openshift-ingress-canary/ingress-canary-tbtsz"
Sep 17 03:19:59.649372 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:59.649325    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7d922cf-74cc-4afb-b900-b21d63d4a225-tmp-dir\") pod \"dns-default-vb2fm\" (UID: \"a7d922cf-74cc-4afb-b900-b21d63d4a225\") " pod="openshift-dns/dns-default-vb2fm"
Sep 17 03:19:59.649372 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:59.649359    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6k9h\" (UniqueName: \"kubernetes.io/projected/a7d922cf-74cc-4afb-b900-b21d63d4a225-kube-api-access-m6k9h\") pod \"dns-default-vb2fm\" (UID: \"a7d922cf-74cc-4afb-b900-b21d63d4a225\") " pod="openshift-dns/dns-default-vb2fm"
Sep 17 03:19:59.649914 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:59.649393    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a7d922cf-74cc-4afb-b900-b21d63d4a225-config-volume\") pod \"dns-default-vb2fm\" (UID: \"a7d922cf-74cc-4afb-b900-b21d63d4a225\") " pod="openshift-dns/dns-default-vb2fm"
Sep 17 03:19:59.649914 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:59.649421    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1a74da51-05c3-4067-b3a5-233734643f6e-cert\") pod \"ingress-canary-tbtsz\" (UID: \"1a74da51-05c3-4067-b3a5-233734643f6e\") " pod="openshift-ingress-canary/ingress-canary-tbtsz"
Sep 17 03:19:59.649914 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:59.649460    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a7d922cf-74cc-4afb-b900-b21d63d4a225-metrics-tls\") pod \"dns-default-vb2fm\" (UID: \"a7d922cf-74cc-4afb-b900-b21d63d4a225\") " pod="openshift-dns/dns-default-vb2fm"
Sep 17 03:19:59.649914 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:19:59.649580    2552 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 03:19:59.649914 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:19:59.649664    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7d922cf-74cc-4afb-b900-b21d63d4a225-metrics-tls podName:a7d922cf-74cc-4afb-b900-b21d63d4a225 nodeName:}" failed. No retries permitted until 2026-09-17 03:20:00.149645827 +0000 UTC m=+35.616053349 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a7d922cf-74cc-4afb-b900-b21d63d4a225-metrics-tls") pod "dns-default-vb2fm" (UID: "a7d922cf-74cc-4afb-b900-b21d63d4a225") : secret "dns-default-metrics-tls" not found
Sep 17 03:19:59.649914 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:59.649847    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7d922cf-74cc-4afb-b900-b21d63d4a225-tmp-dir\") pod \"dns-default-vb2fm\" (UID: \"a7d922cf-74cc-4afb-b900-b21d63d4a225\") " pod="openshift-dns/dns-default-vb2fm"
Sep 17 03:19:59.650137 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:59.650108    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a7d922cf-74cc-4afb-b900-b21d63d4a225-config-volume\") pod \"dns-default-vb2fm\" (UID: \"a7d922cf-74cc-4afb-b900-b21d63d4a225\") " pod="openshift-dns/dns-default-vb2fm"
Sep 17 03:19:59.654752 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:59.654523    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1a74da51-05c3-4067-b3a5-233734643f6e-cert\") pod \"ingress-canary-tbtsz\" (UID: \"1a74da51-05c3-4067-b3a5-233734643f6e\") " pod="openshift-ingress-canary/ingress-canary-tbtsz"
Sep 17 03:19:59.658521 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:59.658488    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6k9h\" (UniqueName: \"kubernetes.io/projected/a7d922cf-74cc-4afb-b900-b21d63d4a225-kube-api-access-m6k9h\") pod \"dns-default-vb2fm\" (UID: \"a7d922cf-74cc-4afb-b900-b21d63d4a225\") " pod="openshift-dns/dns-default-vb2fm"
Sep 17 03:19:59.662072 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:59.662042    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdzm9\" (UniqueName: \"kubernetes.io/projected/1a74da51-05c3-4067-b3a5-233734643f6e-kube-api-access-pdzm9\") pod \"ingress-canary-tbtsz\" (UID: \"1a74da51-05c3-4067-b3a5-233734643f6e\") " pod="openshift-ingress-canary/ingress-canary-tbtsz"
Sep 17 03:19:59.753209 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:59.753172    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-tbtsz"
Sep 17 03:19:59.979398 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:59.979270    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4944l"
Sep 17 03:19:59.979398 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:59.979291    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9d4m6"
Sep 17 03:19:59.982039 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:59.982007    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 03:19:59.982039 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:59.982041    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-d2k6b\""
Sep 17 03:19:59.982205 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:59.982078    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 17 03:19:59.982205 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:19:59.982007    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 17 03:20:00.067317 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:00.067267    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-tbtsz"]
Sep 17 03:20:00.081954 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:20:00.081916    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a74da51_05c3_4067_b3a5_233734643f6e.slice/crio-cf4305cf953adea32fadda41d4cfa96020cc320dd9f242b93cfad1940c9fe499 WatchSource:0}: Error finding container cf4305cf953adea32fadda41d4cfa96020cc320dd9f242b93cfad1940c9fe499: Status 404 returned error can't find the container with id cf4305cf953adea32fadda41d4cfa96020cc320dd9f242b93cfad1940c9fe499
Sep 17 03:20:00.153026 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:00.152991    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a7d922cf-74cc-4afb-b900-b21d63d4a225-metrics-tls\") pod \"dns-default-vb2fm\" (UID: \"a7d922cf-74cc-4afb-b900-b21d63d4a225\") " pod="openshift-dns/dns-default-vb2fm"
Sep 17 03:20:00.153231 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:20:00.153154    2552 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 03:20:00.153271 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:20:00.153234    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7d922cf-74cc-4afb-b900-b21d63d4a225-metrics-tls podName:a7d922cf-74cc-4afb-b900-b21d63d4a225 nodeName:}" failed. No retries permitted until 2026-09-17 03:20:01.153211776 +0000 UTC m=+36.619619285 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a7d922cf-74cc-4afb-b900-b21d63d4a225-metrics-tls") pod "dns-default-vb2fm" (UID: "a7d922cf-74cc-4afb-b900-b21d63d4a225") : secret "dns-default-metrics-tls" not found
Sep 17 03:20:00.261948 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:00.261905    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-tbtsz" event={"ID":"1a74da51-05c3-4067-b3a5-233734643f6e","Type":"ContainerStarted","Data":"cf4305cf953adea32fadda41d4cfa96020cc320dd9f242b93cfad1940c9fe499"}
Sep 17 03:20:00.979769 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:00.979729    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xl9wh"
Sep 17 03:20:00.982558 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:00.982520    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 03:20:00.982558 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:00.982537    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-tmrnq\""
Sep 17 03:20:01.162245 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:01.162206    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a7d922cf-74cc-4afb-b900-b21d63d4a225-metrics-tls\") pod \"dns-default-vb2fm\" (UID: \"a7d922cf-74cc-4afb-b900-b21d63d4a225\") " pod="openshift-dns/dns-default-vb2fm"
Sep 17 03:20:01.162446 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:20:01.162333    2552 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 03:20:01.162446 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:20:01.162415    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7d922cf-74cc-4afb-b900-b21d63d4a225-metrics-tls podName:a7d922cf-74cc-4afb-b900-b21d63d4a225 nodeName:}" failed. No retries permitted until 2026-09-17 03:20:03.162395105 +0000 UTC m=+38.628802633 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a7d922cf-74cc-4afb-b900-b21d63d4a225-metrics-tls") pod "dns-default-vb2fm" (UID: "a7d922cf-74cc-4afb-b900-b21d63d4a225") : secret "dns-default-metrics-tls" not found
Sep 17 03:20:02.271739 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:02.271506    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nkrxz" event={"ID":"0cc4aedf-1bb8-47c6-88f2-266bbbf99058","Type":"ContainerStarted","Data":"b16b8c5e551667e2d459878400a9c23a3b712862cb3bbc312370cdecafc87054"}
Sep 17 03:20:03.178507 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:03.178461    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a7d922cf-74cc-4afb-b900-b21d63d4a225-metrics-tls\") pod \"dns-default-vb2fm\" (UID: \"a7d922cf-74cc-4afb-b900-b21d63d4a225\") " pod="openshift-dns/dns-default-vb2fm"
Sep 17 03:20:03.178727 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:20:03.178665    2552 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 03:20:03.178800 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:20:03.178745    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7d922cf-74cc-4afb-b900-b21d63d4a225-metrics-tls podName:a7d922cf-74cc-4afb-b900-b21d63d4a225 nodeName:}" failed. No retries permitted until 2026-09-17 03:20:07.178724266 +0000 UTC m=+42.645131771 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a7d922cf-74cc-4afb-b900-b21d63d4a225-metrics-tls") pod "dns-default-vb2fm" (UID: "a7d922cf-74cc-4afb-b900-b21d63d4a225") : secret "dns-default-metrics-tls" not found
Sep 17 03:20:03.277482 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:03.277449    2552 generic.go:356] "Generic (PLEG): container finished" podID="0cc4aedf-1bb8-47c6-88f2-266bbbf99058" containerID="b16b8c5e551667e2d459878400a9c23a3b712862cb3bbc312370cdecafc87054" exitCode=0
Sep 17 03:20:03.277940 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:03.277515    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nkrxz" event={"ID":"0cc4aedf-1bb8-47c6-88f2-266bbbf99058","Type":"ContainerDied","Data":"b16b8c5e551667e2d459878400a9c23a3b712862cb3bbc312370cdecafc87054"}
Sep 17 03:20:04.283403 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:04.283328    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nkrxz" event={"ID":"0cc4aedf-1bb8-47c6-88f2-266bbbf99058","Type":"ContainerStarted","Data":"ae728833939312c3991f775adc673134da0d9c47c3a82ec0972c42e0122dd100"}
Sep 17 03:20:04.852154 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:04.850320    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-kz6hm"]
Sep 17 03:20:04.867026 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:04.866977    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-srhm6"]
Sep 17 03:20:04.867231 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:04.867107    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-kz6hm"
Sep 17 03:20:04.870215 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:04.870114    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-9zmtc\""
Sep 17 03:20:04.894134 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:04.894096    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-vtbq6"]
Sep 17 03:20:04.894297 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:04.894213    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-srhm6"
Sep 17 03:20:04.896911 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:04.896881    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Sep 17 03:20:04.897067 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:04.896889    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-5cb5z\""
Sep 17 03:20:04.897067 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:04.896892    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Sep 17 03:20:04.918197 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:04.918155    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-n77tm"]
Sep 17 03:20:04.918347 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:04.918311    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-vtbq6"
Sep 17 03:20:04.921637 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:04.921610    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 17 03:20:04.921877 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:04.921857    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Sep 17 03:20:04.921965 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:04.921948    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 17 03:20:04.923194 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:04.923166    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-z2l2n\""
Sep 17 03:20:04.923340 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:04.923210    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Sep 17 03:20:04.941162 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:04.941136    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-kz6hm"]
Sep 17 03:20:04.941162 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:04.941171    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-srhm6"]
Sep 17 03:20:04.941409 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:04.941243    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-n77tm"
Sep 17 03:20:04.941409 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:04.941269    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-n77tm"]
Sep 17 03:20:04.941409 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:04.941294    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-vtbq6"]
Sep 17 03:20:04.943888 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:04.943862    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Sep 17 03:20:04.944044 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:04.943943    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Sep 17 03:20:04.944044 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:04.943950    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Sep 17 03:20:04.944216 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:04.944195    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-v7jkc\""
Sep 17 03:20:04.948644 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:04.948460    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-qtch6"]
Sep 17 03:20:04.966903 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:04.966870    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-khv5k"]
Sep 17 03:20:04.967079 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:04.966974    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-qtch6"
Sep 17 03:20:04.969358 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:04.969332    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-sw29n\""
Sep 17 03:20:04.969523 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:04.969426    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Sep 17 03:20:04.969523 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:04.969436    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Sep 17 03:20:04.993448 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:04.993412    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lv7jb\" (UniqueName: \"kubernetes.io/projected/bf40e783-559e-4067-b28b-5016ab6fb8c8-kube-api-access-lv7jb\") pod \"volume-data-source-validator-86695b7cc7-srhm6\" (UID: \"bf40e783-559e-4067-b28b-5016ab6fb8c8\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-srhm6"
Sep 17 03:20:04.993638 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:04.993464    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wng2g\" (UniqueName: \"kubernetes.io/projected/0ef4d12b-4a1d-451c-996f-79c43615c307-kube-api-access-wng2g\") pod \"network-check-source-6f8b768455-kz6hm\" (UID: \"0ef4d12b-4a1d-451c-996f-79c43615c307\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-kz6hm"
Sep 17 03:20:04.997345 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:04.997307    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-khv5k"
Sep 17 03:20:05.000090 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:05.000048    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Sep 17 03:20:05.000090 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:05.000069    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Sep 17 03:20:05.000090 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:05.000053    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Sep 17 03:20:05.000376 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:05.000058    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Sep 17 03:20:05.000376 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:05.000207    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-7zks5\""
Sep 17 03:20:05.001168 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:05.001143    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-qtch6"]
Sep 17 03:20:05.001299 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:05.001187    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-khv5k"]
Sep 17 03:20:05.049210 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:05.049177    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5844b856db-68rbk"]
Sep 17 03:20:05.068149 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:05.068120    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-56f7fb996f-x5tjj"]
Sep 17 03:20:05.068319 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:05.068264    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-68rbk"
Sep 17 03:20:05.072591 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:05.072565    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Sep 17 03:20:05.072730 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:05.072711    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Sep 17 03:20:05.073139 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:05.073124    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 17 03:20:05.073207 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:05.073134    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 17 03:20:05.073799 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:05.073578    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-djrbb\""
Sep 17 03:20:05.079781 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:05.079749    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Sep 17 03:20:05.088367 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:05.088333    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-68rbk"]
Sep 17 03:20:05.088367 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:05.088370    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-56f7fb996f-x5tjj"]
Sep 17 03:20:05.088586 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:05.088459    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-56f7fb996f-x5tjj"
Sep 17 03:20:05.090901 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:05.090875    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 17 03:20:05.090901 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:05.090893    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 17 03:20:05.091086 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:05.090917    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 17 03:20:05.091086 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:05.090936    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-lh58r\""
Sep 17 03:20:05.093942 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:05.093907    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhmn5\" (UniqueName: \"kubernetes.io/projected/91c3490d-d515-4cc2-bafb-aa6c999ac3d0-kube-api-access-mhmn5\") pod \"kube-storage-version-migrator-operator-6c8f84f897-khv5k\" (UID: \"91c3490d-d515-4cc2-bafb-aa6c999ac3d0\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-khv5k"
Sep 17 03:20:05.094119 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:05.094005    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/226fdca4-147c-4de3-acf2-7bf0b2e871ba-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-n77tm\" (UID: \"226fdca4-147c-4de3-acf2-7bf0b2e871ba\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-n77tm"
Sep 17 03:20:05.094191 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:05.094125    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/4f8da691-30ee-4f3a-b816-d4d7b7a4b0a0-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-vtbq6\" (UID: \"4f8da691-30ee-4f3a-b816-d4d7b7a4b0a0\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-vtbq6"
Sep 17 03:20:05.094191 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:05.094178    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/88221296-e256-4d59-aba5-9465442a5727-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-qtch6\" (UID: \"88221296-e256-4d59-aba5-9465442a5727\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-qtch6"
Sep 17 03:20:05.094301 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:05.094213    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ths5\" (UniqueName: \"kubernetes.io/projected/226fdca4-147c-4de3-acf2-7bf0b2e871ba-kube-api-access-9ths5\") pod \"cluster-samples-operator-86499c89c4-n77tm\" (UID: \"226fdca4-147c-4de3-acf2-7bf0b2e871ba\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-n77tm"
Sep 17 03:20:05.094301 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:05.094262    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lv7jb\" (UniqueName: \"kubernetes.io/projected/bf40e783-559e-4067-b28b-5016ab6fb8c8-kube-api-access-lv7jb\") pod \"volume-data-source-validator-86695b7cc7-srhm6\" (UID: \"bf40e783-559e-4067-b28b-5016ab6fb8c8\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-srhm6"
Sep 17 03:20:05.094301 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:05.094286    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hr9f\" (UniqueName: \"kubernetes.io/projected/4f8da691-30ee-4f3a-b816-d4d7b7a4b0a0-kube-api-access-6hr9f\") pod \"cluster-monitoring-operator-6cd954b897-vtbq6\" (UID: \"4f8da691-30ee-4f3a-b816-d4d7b7a4b0a0\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-vtbq6"
Sep 17 03:20:05.094457 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:05.094310    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/91c3490d-d515-4cc2-bafb-aa6c999ac3d0-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-khv5k\" (UID: \"91c3490d-d515-4cc2-bafb-aa6c999ac3d0\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-khv5k"
Sep 17 03:20:05.094457 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:05.094334    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/91c3490d-d515-4cc2-bafb-aa6c999ac3d0-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-khv5k\" (UID: \"91c3490d-d515-4cc2-bafb-aa6c999ac3d0\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-khv5k"
Sep 17 03:20:05.094457 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:05.094370    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wng2g\" (UniqueName: \"kubernetes.io/projected/0ef4d12b-4a1d-451c-996f-79c43615c307-kube-api-access-wng2g\") pod \"network-check-source-6f8b768455-kz6hm\" (UID: \"0ef4d12b-4a1d-451c-996f-79c43615c307\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-kz6hm"
Sep 17 03:20:05.094457 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:05.094397    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/88221296-e256-4d59-aba5-9465442a5727-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-qtch6\" (UID: \"88221296-e256-4d59-aba5-9465442a5727\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-qtch6"
Sep 17 03:20:05.094685 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:05.094490    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/4f8da691-30ee-4f3a-b816-d4d7b7a4b0a0-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-vtbq6\" (UID: \"4f8da691-30ee-4f3a-b816-d4d7b7a4b0a0\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-vtbq6"
Sep 17 03:20:05.099029 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:05.098997    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 17 03:20:05.110168 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:05.110099    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wng2g\" (UniqueName: \"kubernetes.io/projected/0ef4d12b-4a1d-451c-996f-79c43615c307-kube-api-access-wng2g\") pod \"network-check-source-6f8b768455-kz6hm\" (UID: \"0ef4d12b-4a1d-451c-996f-79c43615c307\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-kz6hm"
Sep 17 03:20:05.110168 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:05.110099    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lv7jb\" (UniqueName: \"kubernetes.io/projected/bf40e783-559e-4067-b28b-5016ab6fb8c8-kube-api-access-lv7jb\") pod \"volume-data-source-validator-86695b7cc7-srhm6\" (UID: \"bf40e783-559e-4067-b28b-5016ab6fb8c8\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-srhm6"
Sep 17 03:20:05.178389 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:05.178343    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-kz6hm"
Sep 17 03:20:05.194824 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:05.194788    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f68c2a78-217f-46fb-9003-c8ea7948e366-registry-certificates\") pod \"image-registry-56f7fb996f-x5tjj\" (UID: \"f68c2a78-217f-46fb-9003-c8ea7948e366\") " pod="openshift-image-registry/image-registry-56f7fb996f-x5tjj"
Sep 17 03:20:05.194915 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:05.194832    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0f65c6b8-41ff-4572-9e32-c14e524fa713-serving-cert\") pod \"insights-operator-5844b856db-68rbk\" (UID: \"0f65c6b8-41ff-4572-9e32-c14e524fa713\") " pod="openshift-insights/insights-operator-5844b856db-68rbk"
Sep 17 03:20:05.194915 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:05.194891    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/88221296-e256-4d59-aba5-9465442a5727-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-qtch6\" (UID: \"88221296-e256-4d59-aba5-9465442a5727\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-qtch6"
Sep 17 03:20:05.194915 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:05.194912    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0f65c6b8-41ff-4572-9e32-c14e524fa713-trusted-ca-bundle\") pod \"insights-operator-5844b856db-68rbk\" (UID: \"0f65c6b8-41ff-4572-9e32-c14e524fa713\") " pod="openshift-insights/insights-operator-5844b856db-68rbk"
Sep 17 03:20:05.195046 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:05.194937    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hr9f\" (UniqueName: \"kubernetes.io/projected/4f8da691-30ee-4f3a-b816-d4d7b7a4b0a0-kube-api-access-6hr9f\") pod \"cluster-monitoring-operator-6cd954b897-vtbq6\" (UID: \"4f8da691-30ee-4f3a-b816-d4d7b7a4b0a0\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-vtbq6"
Sep 17 03:20:05.195046 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:05.194972    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhmn5\" (UniqueName: \"kubernetes.io/projected/91c3490d-d515-4cc2-bafb-aa6c999ac3d0-kube-api-access-mhmn5\") pod \"kube-storage-version-migrator-operator-6c8f84f897-khv5k\" (UID: \"91c3490d-d515-4cc2-bafb-aa6c999ac3d0\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-khv5k"
Sep 17 03:20:05.195046 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:05.195003    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmdhm\" (UniqueName: \"kubernetes.io/projected/f68c2a78-217f-46fb-9003-c8ea7948e366-kube-api-access-bmdhm\") pod \"image-registry-56f7fb996f-x5tjj\" (UID: \"f68c2a78-217f-46fb-9003-c8ea7948e366\") " pod="openshift-image-registry/image-registry-56f7fb996f-x5tjj"
Sep 17 03:20:05.195185 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:05.195045    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f68c2a78-217f-46fb-9003-c8ea7948e366-image-registry-private-configuration\") pod \"image-registry-56f7fb996f-x5tjj\" (UID: \"f68c2a78-217f-46fb-9003-c8ea7948e366\") " pod="openshift-image-registry/image-registry-56f7fb996f-x5tjj"
Sep 17 03:20:05.195185 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:20:05.195061    2552 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 17 03:20:05.195185 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:05.195078    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/226fdca4-147c-4de3-acf2-7bf0b2e871ba-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-n77tm\" (UID: \"226fdca4-147c-4de3-acf2-7bf0b2e871ba\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-n77tm"
Sep 17 03:20:05.195185 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:05.195132    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0f65c6b8-41ff-4572-9e32-c14e524fa713-tmp\") pod \"insights-operator-5844b856db-68rbk\" (UID: \"0f65c6b8-41ff-4572-9e32-c14e524fa713\") " pod="openshift-insights/insights-operator-5844b856db-68rbk"
Sep 17 03:20:05.195185 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:20:05.195146    2552 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 17 03:20:05.195366 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:20:05.195149    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88221296-e256-4d59-aba5-9465442a5727-networking-console-plugin-cert podName:88221296-e256-4d59-aba5-9465442a5727 nodeName:}" failed. No retries permitted until 2026-09-17 03:20:05.695127062 +0000 UTC m=+41.161534587 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/88221296-e256-4d59-aba5-9465442a5727-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-qtch6" (UID: "88221296-e256-4d59-aba5-9465442a5727") : secret "networking-console-plugin-cert" not found
Sep 17 03:20:05.195366 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:20:05.195230    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/226fdca4-147c-4de3-acf2-7bf0b2e871ba-samples-operator-tls podName:226fdca4-147c-4de3-acf2-7bf0b2e871ba nodeName:}" failed. No retries permitted until 2026-09-17 03:20:05.69521304 +0000 UTC m=+41.161620548 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/226fdca4-147c-4de3-acf2-7bf0b2e871ba-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-n77tm" (UID: "226fdca4-147c-4de3-acf2-7bf0b2e871ba") : secret "samples-operator-tls" not found
Sep 17 03:20:05.195366 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:05.195260    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stkh8\" (UniqueName: \"kubernetes.io/projected/0f65c6b8-41ff-4572-9e32-c14e524fa713-kube-api-access-stkh8\") pod \"insights-operator-5844b856db-68rbk\" (UID: \"0f65c6b8-41ff-4572-9e32-c14e524fa713\") " pod="openshift-insights/insights-operator-5844b856db-68rbk"
Sep 17 03:20:05.195366 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:05.195308    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/4f8da691-30ee-4f3a-b816-d4d7b7a4b0a0-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-vtbq6\" (UID: \"4f8da691-30ee-4f3a-b816-d4d7b7a4b0a0\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-vtbq6"
Sep 17 03:20:05.195366 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:05.195344    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ths5\" (UniqueName: \"kubernetes.io/projected/226fdca4-147c-4de3-acf2-7bf0b2e871ba-kube-api-access-9ths5\") pod \"cluster-samples-operator-86499c89c4-n77tm\" (UID: \"226fdca4-147c-4de3-acf2-7bf0b2e871ba\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-n77tm"
Sep 17 03:20:05.195551 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:05.195427    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f68c2a78-217f-46fb-9003-c8ea7948e366-ca-trust-extracted\") pod \"image-registry-56f7fb996f-x5tjj\" (UID: \"f68c2a78-217f-46fb-9003-c8ea7948e366\") " pod="openshift-image-registry/image-registry-56f7fb996f-x5tjj"
Sep 17 03:20:05.195551 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:05.195448    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f68c2a78-217f-46fb-9003-c8ea7948e366-trusted-ca\") pod \"image-registry-56f7fb996f-x5tjj\" (UID: \"f68c2a78-217f-46fb-9003-c8ea7948e366\") " pod="openshift-image-registry/image-registry-56f7fb996f-x5tjj"
Sep 17 03:20:05.195551 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:05.195468    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/91c3490d-d515-4cc2-bafb-aa6c999ac3d0-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-khv5k\" (UID: \"91c3490d-d515-4cc2-bafb-aa6c999ac3d0\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-khv5k"
Sep 17 03:20:05.195551 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:05.195492    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/91c3490d-d515-4cc2-bafb-aa6c999ac3d0-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-khv5k\" (UID: \"91c3490d-d515-4cc2-bafb-aa6c999ac3d0\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-khv5k"
Sep 17 03:20:05.195551 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:05.195532    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f68c2a78-217f-46fb-9003-c8ea7948e366-registry-tls\") pod \"image-registry-56f7fb996f-x5tjj\" (UID: \"f68c2a78-217f-46fb-9003-c8ea7948e366\") " pod="openshift-image-registry/image-registry-56f7fb996f-x5tjj"
Sep 17 03:20:05.195821 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:05.195559    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/88221296-e256-4d59-aba5-9465442a5727-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-qtch6\" (UID: \"88221296-e256-4d59-aba5-9465442a5727\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-qtch6"
Sep 17 03:20:05.195821 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:05.195667    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f68c2a78-217f-46fb-9003-c8ea7948e366-installation-pull-secrets\") pod \"image-registry-56f7fb996f-x5tjj\" (UID: \"f68c2a78-217f-46fb-9003-c8ea7948e366\") " pod="openshift-image-registry/image-registry-56f7fb996f-x5tjj"
Sep 17 03:20:05.195821 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:05.195721    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f68c2a78-217f-46fb-9003-c8ea7948e366-bound-sa-token\") pod \"image-registry-56f7fb996f-x5tjj\" (UID: \"f68c2a78-217f-46fb-9003-c8ea7948e366\") " pod="openshift-image-registry/image-registry-56f7fb996f-x5tjj"
Sep 17 03:20:05.195821 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:05.195759    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/4f8da691-30ee-4f3a-b816-d4d7b7a4b0a0-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-vtbq6\" (UID: \"4f8da691-30ee-4f3a-b816-d4d7b7a4b0a0\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-vtbq6"
Sep 17 03:20:05.195821 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:05.195787    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/0f65c6b8-41ff-4572-9e32-c14e524fa713-snapshots\") pod \"insights-operator-5844b856db-68rbk\" (UID: \"0f65c6b8-41ff-4572-9e32-c14e524fa713\") " pod="openshift-insights/insights-operator-5844b856db-68rbk"
Sep 17 03:20:05.196016 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:20:05.195865    2552 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 17 03:20:05.196016 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:05.195900    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0f65c6b8-41ff-4572-9e32-c14e524fa713-service-ca-bundle\") pod \"insights-operator-5844b856db-68rbk\" (UID: \"0f65c6b8-41ff-4572-9e32-c14e524fa713\") " pod="openshift-insights/insights-operator-5844b856db-68rbk"
Sep 17 03:20:05.196016 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:20:05.195917    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f8da691-30ee-4f3a-b816-d4d7b7a4b0a0-cluster-monitoring-operator-tls podName:4f8da691-30ee-4f3a-b816-d4d7b7a4b0a0 nodeName:}" failed. No retries permitted until 2026-09-17 03:20:05.69590002 +0000 UTC m=+41.162307545 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/4f8da691-30ee-4f3a-b816-d4d7b7a4b0a0-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-vtbq6" (UID: "4f8da691-30ee-4f3a-b816-d4d7b7a4b0a0") : secret "cluster-monitoring-operator-tls" not found
Sep 17 03:20:05.196167 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:05.196136    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/91c3490d-d515-4cc2-bafb-aa6c999ac3d0-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-khv5k\" (UID: \"91c3490d-d515-4cc2-bafb-aa6c999ac3d0\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-khv5k"
Sep 17 03:20:05.196264 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:05.196246    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/88221296-e256-4d59-aba5-9465442a5727-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-qtch6\" (UID: \"88221296-e256-4d59-aba5-9465442a5727\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-qtch6"
Sep 17 03:20:05.198003 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:05.197973    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/91c3490d-d515-4cc2-bafb-aa6c999ac3d0-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-khv5k\" (UID: \"91c3490d-d515-4cc2-bafb-aa6c999ac3d0\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-khv5k"
Sep 17 03:20:05.202492 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:05.202470    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-srhm6"
Sep 17 03:20:05.203925 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:05.203900    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ths5\" (UniqueName: \"kubernetes.io/projected/226fdca4-147c-4de3-acf2-7bf0b2e871ba-kube-api-access-9ths5\") pod \"cluster-samples-operator-86499c89c4-n77tm\" (UID: \"226fdca4-147c-4de3-acf2-7bf0b2e871ba\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-n77tm"
Sep 17 03:20:05.204381 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:05.204351    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhmn5\" (UniqueName: \"kubernetes.io/projected/91c3490d-d515-4cc2-bafb-aa6c999ac3d0-kube-api-access-mhmn5\") pod \"kube-storage-version-migrator-operator-6c8f84f897-khv5k\" (UID: \"91c3490d-d515-4cc2-bafb-aa6c999ac3d0\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-khv5k"
Sep 17 03:20:05.204470 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:05.204430    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hr9f\" (UniqueName: \"kubernetes.io/projected/4f8da691-30ee-4f3a-b816-d4d7b7a4b0a0-kube-api-access-6hr9f\") pod \"cluster-monitoring-operator-6cd954b897-vtbq6\" (UID: \"4f8da691-30ee-4f3a-b816-d4d7b7a4b0a0\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-vtbq6"
Sep 17 03:20:05.208824 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:05.208784    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/4f8da691-30ee-4f3a-b816-d4d7b7a4b0a0-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-vtbq6\" (UID: \"4f8da691-30ee-4f3a-b816-d4d7b7a4b0a0\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-vtbq6"
Sep 17 03:20:05.289216 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:05.288484    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-tbtsz" event={"ID":"1a74da51-05c3-4067-b3a5-233734643f6e","Type":"ContainerStarted","Data":"68681ddb327fd0655bc9d1649c1ea99bc21a073f7978d7d887f9dd8d50a9af63"}
Sep 17 03:20:05.293517 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:05.293255    2552 generic.go:356] "Generic (PLEG): container finished" podID="0cc4aedf-1bb8-47c6-88f2-266bbbf99058" containerID="ae728833939312c3991f775adc673134da0d9c47c3a82ec0972c42e0122dd100" exitCode=0
Sep 17 03:20:05.293517 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:05.293316    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nkrxz" event={"ID":"0cc4aedf-1bb8-47c6-88f2-266bbbf99058","Type":"ContainerDied","Data":"ae728833939312c3991f775adc673134da0d9c47c3a82ec0972c42e0122dd100"}
Sep 17 03:20:05.296893 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:05.296866    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0f65c6b8-41ff-4572-9e32-c14e524fa713-tmp\") pod \"insights-operator-5844b856db-68rbk\" (UID: \"0f65c6b8-41ff-4572-9e32-c14e524fa713\") " pod="openshift-insights/insights-operator-5844b856db-68rbk"
Sep 17 03:20:05.297013 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:05.296901    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stkh8\" (UniqueName: \"kubernetes.io/projected/0f65c6b8-41ff-4572-9e32-c14e524fa713-kube-api-access-stkh8\") pod \"insights-operator-5844b856db-68rbk\" (UID: \"0f65c6b8-41ff-4572-9e32-c14e524fa713\") " pod="openshift-insights/insights-operator-5844b856db-68rbk"
Sep 17 03:20:05.297013 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:05.296936    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f68c2a78-217f-46fb-9003-c8ea7948e366-ca-trust-extracted\") pod \"image-registry-56f7fb996f-x5tjj\" (UID: \"f68c2a78-217f-46fb-9003-c8ea7948e366\") " pod="openshift-image-registry/image-registry-56f7fb996f-x5tjj"
Sep 17 03:20:05.297013 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:05.296953    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f68c2a78-217f-46fb-9003-c8ea7948e366-trusted-ca\") pod \"image-registry-56f7fb996f-x5tjj\" (UID: \"f68c2a78-217f-46fb-9003-c8ea7948e366\") " pod="openshift-image-registry/image-registry-56f7fb996f-x5tjj"
Sep 17 03:20:05.297013 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:05.296981    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f68c2a78-217f-46fb-9003-c8ea7948e366-registry-tls\") pod \"image-registry-56f7fb996f-x5tjj\" (UID: \"f68c2a78-217f-46fb-9003-c8ea7948e366\") " pod="openshift-image-registry/image-registry-56f7fb996f-x5tjj"
Sep 17 03:20:05.297013 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:05.296998    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f68c2a78-217f-46fb-9003-c8ea7948e366-installation-pull-secrets\") pod \"image-registry-56f7fb996f-x5tjj\" (UID: \"f68c2a78-217f-46fb-9003-c8ea7948e366\") " pod="openshift-image-registry/image-registry-56f7fb996f-x5tjj"
Sep 17 03:20:05.297208 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:05.297182    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f68c2a78-217f-46fb-9003-c8ea7948e366-bound-sa-token\") pod \"image-registry-56f7fb996f-x5tjj\" (UID: \"f68c2a78-217f-46fb-9003-c8ea7948e366\") " pod="openshift-image-registry/image-registry-56f7fb996f-x5tjj"
Sep 17 03:20:05.297208 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:20:05.297203    2552 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 03:20:05.297393 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:20:05.297216    2552 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-56f7fb996f-x5tjj: secret "image-registry-tls" not found
Sep 17 03:20:05.297393 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:05.297244    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/0f65c6b8-41ff-4572-9e32-c14e524fa713-snapshots\") pod \"insights-operator-5844b856db-68rbk\" (UID: \"0f65c6b8-41ff-4572-9e32-c14e524fa713\") " pod="openshift-insights/insights-operator-5844b856db-68rbk"
Sep 17 03:20:05.297393 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:05.297242    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0f65c6b8-41ff-4572-9e32-c14e524fa713-tmp\") pod \"insights-operator-5844b856db-68rbk\" (UID: \"0f65c6b8-41ff-4572-9e32-c14e524fa713\") " pod="openshift-insights/insights-operator-5844b856db-68rbk"
Sep 17 03:20:05.297393 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:20:05.297280    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f68c2a78-217f-46fb-9003-c8ea7948e366-registry-tls podName:f68c2a78-217f-46fb-9003-c8ea7948e366 nodeName:}" failed. No retries permitted until 2026-09-17 03:20:05.797261331 +0000 UTC m=+41.263668852 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f68c2a78-217f-46fb-9003-c8ea7948e366-registry-tls") pod "image-registry-56f7fb996f-x5tjj" (UID: "f68c2a78-217f-46fb-9003-c8ea7948e366") : secret "image-registry-tls" not found
Sep 17 03:20:05.297393 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:05.297297    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0f65c6b8-41ff-4572-9e32-c14e524fa713-service-ca-bundle\") pod \"insights-operator-5844b856db-68rbk\" (UID: \"0f65c6b8-41ff-4572-9e32-c14e524fa713\") " pod="openshift-insights/insights-operator-5844b856db-68rbk"
Sep 17 03:20:05.297393 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:05.297335    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f68c2a78-217f-46fb-9003-c8ea7948e366-registry-certificates\") pod \"image-registry-56f7fb996f-x5tjj\" (UID: \"f68c2a78-217f-46fb-9003-c8ea7948e366\") " pod="openshift-image-registry/image-registry-56f7fb996f-x5tjj"
Sep 17 03:20:05.297393 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:05.297355    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f68c2a78-217f-46fb-9003-c8ea7948e366-ca-trust-extracted\") pod \"image-registry-56f7fb996f-x5tjj\" (UID: \"f68c2a78-217f-46fb-9003-c8ea7948e366\") " pod="openshift-image-registry/image-registry-56f7fb996f-x5tjj"
Sep 17 03:20:05.297393 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:05.297370    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0f65c6b8-41ff-4572-9e32-c14e524fa713-serving-cert\") pod \"insights-operator-5844b856db-68rbk\" (UID: \"0f65c6b8-41ff-4572-9e32-c14e524fa713\") " pod="openshift-insights/insights-operator-5844b856db-68rbk"
Sep 17 03:20:05.297790 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:05.297449    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0f65c6b8-41ff-4572-9e32-c14e524fa713-trusted-ca-bundle\") pod \"insights-operator-5844b856db-68rbk\" (UID: \"0f65c6b8-41ff-4572-9e32-c14e524fa713\") " pod="openshift-insights/insights-operator-5844b856db-68rbk"
Sep 17 03:20:05.297790 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:05.297510    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmdhm\" (UniqueName: \"kubernetes.io/projected/f68c2a78-217f-46fb-9003-c8ea7948e366-kube-api-access-bmdhm\") pod \"image-registry-56f7fb996f-x5tjj\" (UID: \"f68c2a78-217f-46fb-9003-c8ea7948e366\") " pod="openshift-image-registry/image-registry-56f7fb996f-x5tjj"
Sep 17 03:20:05.297790 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:05.297581    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f68c2a78-217f-46fb-9003-c8ea7948e366-image-registry-private-configuration\") pod \"image-registry-56f7fb996f-x5tjj\" (UID: \"f68c2a78-217f-46fb-9003-c8ea7948e366\") " pod="openshift-image-registry/image-registry-56f7fb996f-x5tjj"
Sep 17 03:20:05.298005 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:05.297945    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/0f65c6b8-41ff-4572-9e32-c14e524fa713-snapshots\") pod \"insights-operator-5844b856db-68rbk\" (UID: \"0f65c6b8-41ff-4572-9e32-c14e524fa713\") " pod="openshift-insights/insights-operator-5844b856db-68rbk"
Sep 17 03:20:05.298005 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:05.297949    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0f65c6b8-41ff-4572-9e32-c14e524fa713-service-ca-bundle\") pod \"insights-operator-5844b856db-68rbk\" (UID: \"0f65c6b8-41ff-4572-9e32-c14e524fa713\") " pod="openshift-insights/insights-operator-5844b856db-68rbk"
Sep 17 03:20:05.298261 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:05.298237    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f68c2a78-217f-46fb-9003-c8ea7948e366-trusted-ca\") pod \"image-registry-56f7fb996f-x5tjj\" (UID: \"f68c2a78-217f-46fb-9003-c8ea7948e366\") " pod="openshift-image-registry/image-registry-56f7fb996f-x5tjj"
Sep 17 03:20:05.298341 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:05.298323    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0f65c6b8-41ff-4572-9e32-c14e524fa713-trusted-ca-bundle\") pod \"insights-operator-5844b856db-68rbk\" (UID: \"0f65c6b8-41ff-4572-9e32-c14e524fa713\") " pod="openshift-insights/insights-operator-5844b856db-68rbk"
Sep 17 03:20:05.298378 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:05.298331    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f68c2a78-217f-46fb-9003-c8ea7948e366-registry-certificates\") pod \"image-registry-56f7fb996f-x5tjj\" (UID: \"f68c2a78-217f-46fb-9003-c8ea7948e366\") " pod="openshift-image-registry/image-registry-56f7fb996f-x5tjj"
Sep 17 03:20:05.300178 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:05.300158    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0f65c6b8-41ff-4572-9e32-c14e524fa713-serving-cert\") pod \"insights-operator-5844b856db-68rbk\" (UID: \"0f65c6b8-41ff-4572-9e32-c14e524fa713\") " pod="openshift-insights/insights-operator-5844b856db-68rbk"
Sep 17 03:20:05.300551 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:05.300525    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f68c2a78-217f-46fb-9003-c8ea7948e366-installation-pull-secrets\") pod \"image-registry-56f7fb996f-x5tjj\" (UID: \"f68c2a78-217f-46fb-9003-c8ea7948e366\") " pod="openshift-image-registry/image-registry-56f7fb996f-x5tjj"
Sep 17 03:20:05.300676 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:05.300660    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f68c2a78-217f-46fb-9003-c8ea7948e366-image-registry-private-configuration\") pod \"image-registry-56f7fb996f-x5tjj\" (UID: \"f68c2a78-217f-46fb-9003-c8ea7948e366\") " pod="openshift-image-registry/image-registry-56f7fb996f-x5tjj"
Sep 17 03:20:05.304981 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:05.304936    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-tbtsz" podStartSLOduration=2.061781064 podStartE2EDuration="6.304920237s" totalImagesPullingTime="4.243139173s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 03:19:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:20:05.304292599 +0000 UTC m=+40.770700128" watchObservedRunningTime="2026-09-17 03:20:05.304920237 +0000 UTC m=+40.771327762"
Sep 17 03:20:05.307044 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:05.306878    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-khv5k"
Sep 17 03:20:05.309060 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:05.309033    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-stkh8\" (UniqueName: \"kubernetes.io/projected/0f65c6b8-41ff-4572-9e32-c14e524fa713-kube-api-access-stkh8\") pod \"insights-operator-5844b856db-68rbk\" (UID: \"0f65c6b8-41ff-4572-9e32-c14e524fa713\") " pod="openshift-insights/insights-operator-5844b856db-68rbk"
Sep 17 03:20:05.309230 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:05.309139    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmdhm\" (UniqueName: \"kubernetes.io/projected/f68c2a78-217f-46fb-9003-c8ea7948e366-kube-api-access-bmdhm\") pod \"image-registry-56f7fb996f-x5tjj\" (UID: \"f68c2a78-217f-46fb-9003-c8ea7948e366\") " pod="openshift-image-registry/image-registry-56f7fb996f-x5tjj"
Sep 17 03:20:05.309319 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:05.309293    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f68c2a78-217f-46fb-9003-c8ea7948e366-bound-sa-token\") pod \"image-registry-56f7fb996f-x5tjj\" (UID: \"f68c2a78-217f-46fb-9003-c8ea7948e366\") " pod="openshift-image-registry/image-registry-56f7fb996f-x5tjj"
Sep 17 03:20:05.378730 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:05.378698    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-68rbk"
Sep 17 03:20:05.472618 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:05.472574    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-kz6hm"]
Sep 17 03:20:05.475522 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:20:05.475492    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ef4d12b_4a1d_451c_996f_79c43615c307.slice/crio-e91a70fe61c71f4ce46939fea663ba6ccb6fd50338e6dfb744d31394cecae35b WatchSource:0}: Error finding container e91a70fe61c71f4ce46939fea663ba6ccb6fd50338e6dfb744d31394cecae35b: Status 404 returned error can't find the container with id e91a70fe61c71f4ce46939fea663ba6ccb6fd50338e6dfb744d31394cecae35b
Sep 17 03:20:05.501025 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:05.500990    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-srhm6"]
Sep 17 03:20:05.504187 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:20:05.504156    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf40e783_559e_4067_b28b_5016ab6fb8c8.slice/crio-59ac478c07cffbc9a27daf5fbf7a86e2ece7bf83e854b1ed18afcc5275a48da0 WatchSource:0}: Error finding container 59ac478c07cffbc9a27daf5fbf7a86e2ece7bf83e854b1ed18afcc5275a48da0: Status 404 returned error can't find the container with id 59ac478c07cffbc9a27daf5fbf7a86e2ece7bf83e854b1ed18afcc5275a48da0
Sep 17 03:20:05.606641 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:05.606609    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-khv5k"]
Sep 17 03:20:05.609387 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:20:05.609360    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91c3490d_d515_4cc2_bafb_aa6c999ac3d0.slice/crio-4b79e4b3121c9acabaf862d9a38b76d241e837b86a9dcc000c303ff2346781d3 WatchSource:0}: Error finding container 4b79e4b3121c9acabaf862d9a38b76d241e837b86a9dcc000c303ff2346781d3: Status 404 returned error can't find the container with id 4b79e4b3121c9acabaf862d9a38b76d241e837b86a9dcc000c303ff2346781d3
Sep 17 03:20:05.681562 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:05.681485    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-68rbk"]
Sep 17 03:20:05.684632 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:20:05.684585    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f65c6b8_41ff_4572_9e32_c14e524fa713.slice/crio-a6d9d71655afb726c2cb976cd10b4020064cef4cc2571055ac00c6aa94852b79 WatchSource:0}: Error finding container a6d9d71655afb726c2cb976cd10b4020064cef4cc2571055ac00c6aa94852b79: Status 404 returned error can't find the container with id a6d9d71655afb726c2cb976cd10b4020064cef4cc2571055ac00c6aa94852b79
Sep 17 03:20:05.701786 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:05.701752    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/4f8da691-30ee-4f3a-b816-d4d7b7a4b0a0-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-vtbq6\" (UID: \"4f8da691-30ee-4f3a-b816-d4d7b7a4b0a0\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-vtbq6"
Sep 17 03:20:05.701973 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:05.701810    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/88221296-e256-4d59-aba5-9465442a5727-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-qtch6\" (UID: \"88221296-e256-4d59-aba5-9465442a5727\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-qtch6"
Sep 17 03:20:05.701973 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:05.701850    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/226fdca4-147c-4de3-acf2-7bf0b2e871ba-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-n77tm\" (UID: \"226fdca4-147c-4de3-acf2-7bf0b2e871ba\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-n77tm"
Sep 17 03:20:05.701973 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:20:05.701927    2552 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 17 03:20:05.701973 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:20:05.701949    2552 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 17 03:20:05.702116 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:20:05.701950    2552 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 17 03:20:05.702116 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:20:05.702003    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f8da691-30ee-4f3a-b816-d4d7b7a4b0a0-cluster-monitoring-operator-tls podName:4f8da691-30ee-4f3a-b816-d4d7b7a4b0a0 nodeName:}" failed. No retries permitted until 2026-09-17 03:20:06.701982028 +0000 UTC m=+42.168389549 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/4f8da691-30ee-4f3a-b816-d4d7b7a4b0a0-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-vtbq6" (UID: "4f8da691-30ee-4f3a-b816-d4d7b7a4b0a0") : secret "cluster-monitoring-operator-tls" not found
Sep 17 03:20:05.702116 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:20:05.702019    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88221296-e256-4d59-aba5-9465442a5727-networking-console-plugin-cert podName:88221296-e256-4d59-aba5-9465442a5727 nodeName:}" failed. No retries permitted until 2026-09-17 03:20:06.702011439 +0000 UTC m=+42.168418946 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/88221296-e256-4d59-aba5-9465442a5727-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-qtch6" (UID: "88221296-e256-4d59-aba5-9465442a5727") : secret "networking-console-plugin-cert" not found
Sep 17 03:20:05.702116 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:20:05.702032    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/226fdca4-147c-4de3-acf2-7bf0b2e871ba-samples-operator-tls podName:226fdca4-147c-4de3-acf2-7bf0b2e871ba nodeName:}" failed. No retries permitted until 2026-09-17 03:20:06.702025343 +0000 UTC m=+42.168432847 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/226fdca4-147c-4de3-acf2-7bf0b2e871ba-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-n77tm" (UID: "226fdca4-147c-4de3-acf2-7bf0b2e871ba") : secret "samples-operator-tls" not found
Sep 17 03:20:05.803181 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:05.803143    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f68c2a78-217f-46fb-9003-c8ea7948e366-registry-tls\") pod \"image-registry-56f7fb996f-x5tjj\" (UID: \"f68c2a78-217f-46fb-9003-c8ea7948e366\") " pod="openshift-image-registry/image-registry-56f7fb996f-x5tjj"
Sep 17 03:20:05.803423 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:20:05.803318    2552 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 03:20:05.803423 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:20:05.803340    2552 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-56f7fb996f-x5tjj: secret "image-registry-tls" not found
Sep 17 03:20:05.803423 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:20:05.803407    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f68c2a78-217f-46fb-9003-c8ea7948e366-registry-tls podName:f68c2a78-217f-46fb-9003-c8ea7948e366 nodeName:}" failed. No retries permitted until 2026-09-17 03:20:06.80338856 +0000 UTC m=+42.269796087 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f68c2a78-217f-46fb-9003-c8ea7948e366-registry-tls") pod "image-registry-56f7fb996f-x5tjj" (UID: "f68c2a78-217f-46fb-9003-c8ea7948e366") : secret "image-registry-tls" not found
Sep 17 03:20:06.301283 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:06.301241    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nkrxz" event={"ID":"0cc4aedf-1bb8-47c6-88f2-266bbbf99058","Type":"ContainerStarted","Data":"5e4bfdf33f4a9a22e1055b002ae3151f99064a3c933519a89cd0f834ee6a0419"}
Sep 17 03:20:06.303195 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:06.303150    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-68rbk" event={"ID":"0f65c6b8-41ff-4572-9e32-c14e524fa713","Type":"ContainerStarted","Data":"a6d9d71655afb726c2cb976cd10b4020064cef4cc2571055ac00c6aa94852b79"}
Sep 17 03:20:06.304612 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:06.304572    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-khv5k" event={"ID":"91c3490d-d515-4cc2-bafb-aa6c999ac3d0","Type":"ContainerStarted","Data":"4b79e4b3121c9acabaf862d9a38b76d241e837b86a9dcc000c303ff2346781d3"}
Sep 17 03:20:06.307034 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:06.307002    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-kz6hm" event={"ID":"0ef4d12b-4a1d-451c-996f-79c43615c307","Type":"ContainerStarted","Data":"e91a70fe61c71f4ce46939fea663ba6ccb6fd50338e6dfb744d31394cecae35b"}
Sep 17 03:20:06.308353 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:06.308329    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-srhm6" event={"ID":"bf40e783-559e-4067-b28b-5016ab6fb8c8","Type":"ContainerStarted","Data":"59ac478c07cffbc9a27daf5fbf7a86e2ece7bf83e854b1ed18afcc5275a48da0"}
Sep 17 03:20:06.322999 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:06.322936    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-nkrxz" podStartSLOduration=10.408305157 podStartE2EDuration="41.322916183s" totalImagesPullingTime="30.914611026s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 03:19:25 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:20:06.320756731 +0000 UTC m=+41.787164260" watchObservedRunningTime="2026-09-17 03:20:06.322916183 +0000 UTC m=+41.789323742"
Sep 17 03:20:06.713338 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:06.713243    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/226fdca4-147c-4de3-acf2-7bf0b2e871ba-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-n77tm\" (UID: \"226fdca4-147c-4de3-acf2-7bf0b2e871ba\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-n77tm"
Sep 17 03:20:06.713494 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:06.713381    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/4f8da691-30ee-4f3a-b816-d4d7b7a4b0a0-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-vtbq6\" (UID: \"4f8da691-30ee-4f3a-b816-d4d7b7a4b0a0\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-vtbq6"
Sep 17 03:20:06.713494 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:20:06.713445    2552 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 17 03:20:06.713648 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:20:06.713531    2552 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 17 03:20:06.713648 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:20:06.713546    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/226fdca4-147c-4de3-acf2-7bf0b2e871ba-samples-operator-tls podName:226fdca4-147c-4de3-acf2-7bf0b2e871ba nodeName:}" failed. No retries permitted until 2026-09-17 03:20:08.713523122 +0000 UTC m=+44.179930633 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/226fdca4-147c-4de3-acf2-7bf0b2e871ba-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-n77tm" (UID: "226fdca4-147c-4de3-acf2-7bf0b2e871ba") : secret "samples-operator-tls" not found
Sep 17 03:20:06.713648 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:20:06.713552    2552 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 17 03:20:06.713648 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:06.713449    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/88221296-e256-4d59-aba5-9465442a5727-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-qtch6\" (UID: \"88221296-e256-4d59-aba5-9465442a5727\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-qtch6"
Sep 17 03:20:06.715609 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:20:06.715568    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88221296-e256-4d59-aba5-9465442a5727-networking-console-plugin-cert podName:88221296-e256-4d59-aba5-9465442a5727 nodeName:}" failed. No retries permitted until 2026-09-17 03:20:08.713578758 +0000 UTC m=+44.179986265 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/88221296-e256-4d59-aba5-9465442a5727-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-qtch6" (UID: "88221296-e256-4d59-aba5-9465442a5727") : secret "networking-console-plugin-cert" not found
Sep 17 03:20:06.715747 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:20:06.715641    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f8da691-30ee-4f3a-b816-d4d7b7a4b0a0-cluster-monitoring-operator-tls podName:4f8da691-30ee-4f3a-b816-d4d7b7a4b0a0 nodeName:}" failed. No retries permitted until 2026-09-17 03:20:08.71561929 +0000 UTC m=+44.182026807 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/4f8da691-30ee-4f3a-b816-d4d7b7a4b0a0-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-vtbq6" (UID: "4f8da691-30ee-4f3a-b816-d4d7b7a4b0a0") : secret "cluster-monitoring-operator-tls" not found
Sep 17 03:20:06.816463 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:06.816251    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f68c2a78-217f-46fb-9003-c8ea7948e366-registry-tls\") pod \"image-registry-56f7fb996f-x5tjj\" (UID: \"f68c2a78-217f-46fb-9003-c8ea7948e366\") " pod="openshift-image-registry/image-registry-56f7fb996f-x5tjj"
Sep 17 03:20:06.816679 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:20:06.816537    2552 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 03:20:06.816679 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:20:06.816556    2552 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-56f7fb996f-x5tjj: secret "image-registry-tls" not found
Sep 17 03:20:06.816679 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:20:06.816630    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f68c2a78-217f-46fb-9003-c8ea7948e366-registry-tls podName:f68c2a78-217f-46fb-9003-c8ea7948e366 nodeName:}" failed. No retries permitted until 2026-09-17 03:20:08.816611025 +0000 UTC m=+44.283018546 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f68c2a78-217f-46fb-9003-c8ea7948e366-registry-tls") pod "image-registry-56f7fb996f-x5tjj" (UID: "f68c2a78-217f-46fb-9003-c8ea7948e366") : secret "image-registry-tls" not found
Sep 17 03:20:07.220473 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:07.220425    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a7d922cf-74cc-4afb-b900-b21d63d4a225-metrics-tls\") pod \"dns-default-vb2fm\" (UID: \"a7d922cf-74cc-4afb-b900-b21d63d4a225\") " pod="openshift-dns/dns-default-vb2fm"
Sep 17 03:20:07.220745 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:20:07.220727    2552 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 03:20:07.220839 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:20:07.220795    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7d922cf-74cc-4afb-b900-b21d63d4a225-metrics-tls podName:a7d922cf-74cc-4afb-b900-b21d63d4a225 nodeName:}" failed. No retries permitted until 2026-09-17 03:20:15.220776178 +0000 UTC m=+50.687183686 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a7d922cf-74cc-4afb-b900-b21d63d4a225-metrics-tls") pod "dns-default-vb2fm" (UID: "a7d922cf-74cc-4afb-b900-b21d63d4a225") : secret "dns-default-metrics-tls" not found
Sep 17 03:20:08.735413 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:08.735366    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/4f8da691-30ee-4f3a-b816-d4d7b7a4b0a0-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-vtbq6\" (UID: \"4f8da691-30ee-4f3a-b816-d4d7b7a4b0a0\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-vtbq6"
Sep 17 03:20:08.735830 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:08.735432    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/88221296-e256-4d59-aba5-9465442a5727-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-qtch6\" (UID: \"88221296-e256-4d59-aba5-9465442a5727\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-qtch6"
Sep 17 03:20:08.735830 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:08.735475    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/226fdca4-147c-4de3-acf2-7bf0b2e871ba-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-n77tm\" (UID: \"226fdca4-147c-4de3-acf2-7bf0b2e871ba\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-n77tm"
Sep 17 03:20:08.735830 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:20:08.735514    2552 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 17 03:20:08.735830 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:20:08.735551    2552 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 17 03:20:08.735830 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:20:08.735580    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f8da691-30ee-4f3a-b816-d4d7b7a4b0a0-cluster-monitoring-operator-tls podName:4f8da691-30ee-4f3a-b816-d4d7b7a4b0a0 nodeName:}" failed. No retries permitted until 2026-09-17 03:20:12.735563716 +0000 UTC m=+48.201971251 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/4f8da691-30ee-4f3a-b816-d4d7b7a4b0a0-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-vtbq6" (UID: "4f8da691-30ee-4f3a-b816-d4d7b7a4b0a0") : secret "cluster-monitoring-operator-tls" not found
Sep 17 03:20:08.735830 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:20:08.735518    2552 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 17 03:20:08.735830 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:20:08.735619    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/226fdca4-147c-4de3-acf2-7bf0b2e871ba-samples-operator-tls podName:226fdca4-147c-4de3-acf2-7bf0b2e871ba nodeName:}" failed. No retries permitted until 2026-09-17 03:20:12.735587123 +0000 UTC m=+48.201994644 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/226fdca4-147c-4de3-acf2-7bf0b2e871ba-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-n77tm" (UID: "226fdca4-147c-4de3-acf2-7bf0b2e871ba") : secret "samples-operator-tls" not found
Sep 17 03:20:08.735830 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:20:08.735682    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88221296-e256-4d59-aba5-9465442a5727-networking-console-plugin-cert podName:88221296-e256-4d59-aba5-9465442a5727 nodeName:}" failed. No retries permitted until 2026-09-17 03:20:12.735667055 +0000 UTC m=+48.202074559 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/88221296-e256-4d59-aba5-9465442a5727-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-qtch6" (UID: "88221296-e256-4d59-aba5-9465442a5727") : secret "networking-console-plugin-cert" not found
Sep 17 03:20:08.836491 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:08.836452    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f68c2a78-217f-46fb-9003-c8ea7948e366-registry-tls\") pod \"image-registry-56f7fb996f-x5tjj\" (UID: \"f68c2a78-217f-46fb-9003-c8ea7948e366\") " pod="openshift-image-registry/image-registry-56f7fb996f-x5tjj"
Sep 17 03:20:08.836674 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:20:08.836629    2552 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 03:20:08.836674 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:20:08.836650    2552 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-56f7fb996f-x5tjj: secret "image-registry-tls" not found
Sep 17 03:20:08.836777 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:20:08.836705    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f68c2a78-217f-46fb-9003-c8ea7948e366-registry-tls podName:f68c2a78-217f-46fb-9003-c8ea7948e366 nodeName:}" failed. No retries permitted until 2026-09-17 03:20:12.836688591 +0000 UTC m=+48.303096113 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f68c2a78-217f-46fb-9003-c8ea7948e366-registry-tls") pod "image-registry-56f7fb996f-x5tjj" (UID: "f68c2a78-217f-46fb-9003-c8ea7948e366") : secret "image-registry-tls" not found
Sep 17 03:20:11.324256 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:11.324218    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-68rbk" event={"ID":"0f65c6b8-41ff-4572-9e32-c14e524fa713","Type":"ContainerStarted","Data":"e81ed406f4500d4df6439c75fec73dc2ac6f658ccf0077d8fd9e366aabeb396c"}
Sep 17 03:20:11.326061 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:11.326023    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-khv5k" event={"ID":"91c3490d-d515-4cc2-bafb-aa6c999ac3d0","Type":"ContainerStarted","Data":"a15ad1337839838a50f7c82ac0f3577b361b713ab585611eb5aac0752284c16e"}
Sep 17 03:20:11.327956 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:11.327913    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-kz6hm" event={"ID":"0ef4d12b-4a1d-451c-996f-79c43615c307","Type":"ContainerStarted","Data":"dc637691fe0a687f41dad74d20b5925bc8a5de6cbecbc8a43d4ef8d32d617123"}
Sep 17 03:20:11.330147 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:11.330097    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-srhm6" event={"ID":"bf40e783-559e-4067-b28b-5016ab6fb8c8","Type":"ContainerStarted","Data":"85e3017aba40619eb84eb211800eba7cd3d8c777ad9707fbf37a96c7f1043717"}
Sep 17 03:20:11.342925 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:11.342572    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5844b856db-68rbk" podStartSLOduration=1.5449383989999999 podStartE2EDuration="6.342549938s" totalImagesPullingTime="4.797611539s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 03:20:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:20:11.34045138 +0000 UTC m=+46.806858904" watchObservedRunningTime="2026-09-17 03:20:11.342549938 +0000 UTC m=+46.808957465"
Sep 17 03:20:11.356864 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:11.356804    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-khv5k" podStartSLOduration=2.4835001549999998 podStartE2EDuration="7.356781767s" totalImagesPullingTime="4.873281612s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 03:20:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:20:11.354820409 +0000 UTC m=+46.821227936" watchObservedRunningTime="2026-09-17 03:20:11.356781767 +0000 UTC m=+46.823189297"
Sep 17 03:20:11.372485 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:11.372417    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f8b768455-kz6hm" podStartSLOduration=2.3558059780000002 podStartE2EDuration="7.372392437s" totalImagesPullingTime="5.016586459s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 03:20:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:20:11.371289632 +0000 UTC m=+46.837697156" watchObservedRunningTime="2026-09-17 03:20:11.372392437 +0000 UTC m=+46.838799964"
Sep 17 03:20:11.389559 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:11.389482    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-srhm6" podStartSLOduration=2.433053611 podStartE2EDuration="7.38945422s" totalImagesPullingTime="4.956400609s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 03:20:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:20:11.388050337 +0000 UTC m=+46.854457864" watchObservedRunningTime="2026-09-17 03:20:11.38945422 +0000 UTC m=+46.855861748"
Sep 17 03:20:11.895364 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:11.895303    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-4dd7t"]
Sep 17 03:20:11.916404 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:11.916368    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-4dd7t"]
Sep 17 03:20:11.916572 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:11.916517    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-4dd7t"
Sep 17 03:20:11.919407 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:11.919377    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 17 03:20:11.919579 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:11.919385    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 17 03:20:11.919692 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:11.919619    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-52cf6\""
Sep 17 03:20:11.960684 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:11.960640    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8d497c2d-d126-4f70-a968-d7371e39793f-crio-socket\") pod \"insights-runtime-extractor-4dd7t\" (UID: \"8d497c2d-d126-4f70-a968-d7371e39793f\") " pod="openshift-insights/insights-runtime-extractor-4dd7t"
Sep 17 03:20:11.960870 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:11.960701    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggvjn\" (UniqueName: \"kubernetes.io/projected/8d497c2d-d126-4f70-a968-d7371e39793f-kube-api-access-ggvjn\") pod \"insights-runtime-extractor-4dd7t\" (UID: \"8d497c2d-d126-4f70-a968-d7371e39793f\") " pod="openshift-insights/insights-runtime-extractor-4dd7t"
Sep 17 03:20:11.960870 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:11.960736    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8d497c2d-d126-4f70-a968-d7371e39793f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4dd7t\" (UID: \"8d497c2d-d126-4f70-a968-d7371e39793f\") " pod="openshift-insights/insights-runtime-extractor-4dd7t"
Sep 17 03:20:11.960870 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:11.960778    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8d497c2d-d126-4f70-a968-d7371e39793f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4dd7t\" (UID: \"8d497c2d-d126-4f70-a968-d7371e39793f\") " pod="openshift-insights/insights-runtime-extractor-4dd7t"
Sep 17 03:20:11.960870 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:11.960866    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8d497c2d-d126-4f70-a968-d7371e39793f-data-volume\") pod \"insights-runtime-extractor-4dd7t\" (UID: \"8d497c2d-d126-4f70-a968-d7371e39793f\") " pod="openshift-insights/insights-runtime-extractor-4dd7t"
Sep 17 03:20:12.061924 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:12.061855    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8d497c2d-d126-4f70-a968-d7371e39793f-crio-socket\") pod \"insights-runtime-extractor-4dd7t\" (UID: \"8d497c2d-d126-4f70-a968-d7371e39793f\") " pod="openshift-insights/insights-runtime-extractor-4dd7t"
Sep 17 03:20:12.062109 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:12.061978    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggvjn\" (UniqueName: \"kubernetes.io/projected/8d497c2d-d126-4f70-a968-d7371e39793f-kube-api-access-ggvjn\") pod \"insights-runtime-extractor-4dd7t\" (UID: \"8d497c2d-d126-4f70-a968-d7371e39793f\") " pod="openshift-insights/insights-runtime-extractor-4dd7t"
Sep 17 03:20:12.062109 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:12.062010    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8d497c2d-d126-4f70-a968-d7371e39793f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4dd7t\" (UID: \"8d497c2d-d126-4f70-a968-d7371e39793f\") " pod="openshift-insights/insights-runtime-extractor-4dd7t"
Sep 17 03:20:12.062109 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:12.062034    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8d497c2d-d126-4f70-a968-d7371e39793f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4dd7t\" (UID: \"8d497c2d-d126-4f70-a968-d7371e39793f\") " pod="openshift-insights/insights-runtime-extractor-4dd7t"
Sep 17 03:20:12.062288 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:12.062123    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8d497c2d-d126-4f70-a968-d7371e39793f-crio-socket\") pod \"insights-runtime-extractor-4dd7t\" (UID: \"8d497c2d-d126-4f70-a968-d7371e39793f\") " pod="openshift-insights/insights-runtime-extractor-4dd7t"
Sep 17 03:20:12.062288 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:20:12.062162    2552 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 03:20:12.062288 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:20:12.062221    2552 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 03:20:12.062288 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:12.062254    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8d497c2d-d126-4f70-a968-d7371e39793f-data-volume\") pod \"insights-runtime-extractor-4dd7t\" (UID: \"8d497c2d-d126-4f70-a968-d7371e39793f\") " pod="openshift-insights/insights-runtime-extractor-4dd7t"
Sep 17 03:20:12.062288 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:20:12.062262    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8d497c2d-d126-4f70-a968-d7371e39793f-kube-rbac-proxy-cm podName:8d497c2d-d126-4f70-a968-d7371e39793f nodeName:}" failed. No retries permitted until 2026-09-17 03:20:12.562239304 +0000 UTC m=+48.028646813 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8d497c2d-d126-4f70-a968-d7371e39793f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4dd7t" (UID: "8d497c2d-d126-4f70-a968-d7371e39793f") : configmap "kube-rbac-proxy" not found
Sep 17 03:20:12.062514 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:20:12.062308    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d497c2d-d126-4f70-a968-d7371e39793f-insights-runtime-extractor-tls podName:8d497c2d-d126-4f70-a968-d7371e39793f nodeName:}" failed. No retries permitted until 2026-09-17 03:20:12.562285352 +0000 UTC m=+48.028692868 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8d497c2d-d126-4f70-a968-d7371e39793f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4dd7t" (UID: "8d497c2d-d126-4f70-a968-d7371e39793f") : secret "insights-runtime-extractor-tls" not found
Sep 17 03:20:12.062628 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:12.062592    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8d497c2d-d126-4f70-a968-d7371e39793f-data-volume\") pod \"insights-runtime-extractor-4dd7t\" (UID: \"8d497c2d-d126-4f70-a968-d7371e39793f\") " pod="openshift-insights/insights-runtime-extractor-4dd7t"
Sep 17 03:20:12.075252 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:12.075222    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggvjn\" (UniqueName: \"kubernetes.io/projected/8d497c2d-d126-4f70-a968-d7371e39793f-kube-api-access-ggvjn\") pod \"insights-runtime-extractor-4dd7t\" (UID: \"8d497c2d-d126-4f70-a968-d7371e39793f\") " pod="openshift-insights/insights-runtime-extractor-4dd7t"
Sep 17 03:20:12.567571 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:12.567530    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8d497c2d-d126-4f70-a968-d7371e39793f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4dd7t\" (UID: \"8d497c2d-d126-4f70-a968-d7371e39793f\") " pod="openshift-insights/insights-runtime-extractor-4dd7t"
Sep 17 03:20:12.567571 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:12.567573    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8d497c2d-d126-4f70-a968-d7371e39793f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4dd7t\" (UID: \"8d497c2d-d126-4f70-a968-d7371e39793f\") " pod="openshift-insights/insights-runtime-extractor-4dd7t"
Sep 17 03:20:12.568023 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:20:12.567679    2552 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 03:20:12.568023 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:20:12.567790    2552 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 03:20:12.568023 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:20:12.567804    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8d497c2d-d126-4f70-a968-d7371e39793f-kube-rbac-proxy-cm podName:8d497c2d-d126-4f70-a968-d7371e39793f nodeName:}" failed. No retries permitted until 2026-09-17 03:20:13.567784858 +0000 UTC m=+49.034192362 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8d497c2d-d126-4f70-a968-d7371e39793f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4dd7t" (UID: "8d497c2d-d126-4f70-a968-d7371e39793f") : configmap "kube-rbac-proxy" not found
Sep 17 03:20:12.568023 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:20:12.567843    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d497c2d-d126-4f70-a968-d7371e39793f-insights-runtime-extractor-tls podName:8d497c2d-d126-4f70-a968-d7371e39793f nodeName:}" failed. No retries permitted until 2026-09-17 03:20:13.567826678 +0000 UTC m=+49.034234192 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8d497c2d-d126-4f70-a968-d7371e39793f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4dd7t" (UID: "8d497c2d-d126-4f70-a968-d7371e39793f") : secret "insights-runtime-extractor-tls" not found
Sep 17 03:20:12.769190 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:12.769143    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/4f8da691-30ee-4f3a-b816-d4d7b7a4b0a0-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-vtbq6\" (UID: \"4f8da691-30ee-4f3a-b816-d4d7b7a4b0a0\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-vtbq6"
Sep 17 03:20:12.769375 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:12.769242    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/88221296-e256-4d59-aba5-9465442a5727-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-qtch6\" (UID: \"88221296-e256-4d59-aba5-9465442a5727\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-qtch6"
Sep 17 03:20:12.769375 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:12.769303    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/226fdca4-147c-4de3-acf2-7bf0b2e871ba-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-n77tm\" (UID: \"226fdca4-147c-4de3-acf2-7bf0b2e871ba\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-n77tm"
Sep 17 03:20:12.769375 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:20:12.769308    2552 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 17 03:20:12.769480 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:20:12.769376    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f8da691-30ee-4f3a-b816-d4d7b7a4b0a0-cluster-monitoring-operator-tls podName:4f8da691-30ee-4f3a-b816-d4d7b7a4b0a0 nodeName:}" failed. No retries permitted until 2026-09-17 03:20:20.769357189 +0000 UTC m=+56.235764713 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/4f8da691-30ee-4f3a-b816-d4d7b7a4b0a0-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-vtbq6" (UID: "4f8da691-30ee-4f3a-b816-d4d7b7a4b0a0") : secret "cluster-monitoring-operator-tls" not found
Sep 17 03:20:12.769480 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:20:12.769384    2552 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 17 03:20:12.769480 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:20:12.769447    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88221296-e256-4d59-aba5-9465442a5727-networking-console-plugin-cert podName:88221296-e256-4d59-aba5-9465442a5727 nodeName:}" failed. No retries permitted until 2026-09-17 03:20:20.769429558 +0000 UTC m=+56.235837062 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/88221296-e256-4d59-aba5-9465442a5727-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-qtch6" (UID: "88221296-e256-4d59-aba5-9465442a5727") : secret "networking-console-plugin-cert" not found
Sep 17 03:20:12.769480 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:20:12.769394    2552 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 17 03:20:12.769632 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:20:12.769492    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/226fdca4-147c-4de3-acf2-7bf0b2e871ba-samples-operator-tls podName:226fdca4-147c-4de3-acf2-7bf0b2e871ba nodeName:}" failed. No retries permitted until 2026-09-17 03:20:20.769481758 +0000 UTC m=+56.235889265 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/226fdca4-147c-4de3-acf2-7bf0b2e871ba-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-n77tm" (UID: "226fdca4-147c-4de3-acf2-7bf0b2e871ba") : secret "samples-operator-tls" not found
Sep 17 03:20:12.870755 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:12.870673    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f68c2a78-217f-46fb-9003-c8ea7948e366-registry-tls\") pod \"image-registry-56f7fb996f-x5tjj\" (UID: \"f68c2a78-217f-46fb-9003-c8ea7948e366\") " pod="openshift-image-registry/image-registry-56f7fb996f-x5tjj"
Sep 17 03:20:12.870906 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:20:12.870835    2552 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 03:20:12.870906 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:20:12.870863    2552 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-56f7fb996f-x5tjj: secret "image-registry-tls" not found
Sep 17 03:20:12.870968 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:20:12.870917    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f68c2a78-217f-46fb-9003-c8ea7948e366-registry-tls podName:f68c2a78-217f-46fb-9003-c8ea7948e366 nodeName:}" failed. No retries permitted until 2026-09-17 03:20:20.870902089 +0000 UTC m=+56.337309597 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f68c2a78-217f-46fb-9003-c8ea7948e366-registry-tls") pod "image-registry-56f7fb996f-x5tjj" (UID: "f68c2a78-217f-46fb-9003-c8ea7948e366") : secret "image-registry-tls" not found
Sep 17 03:20:13.577005 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:13.576960    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8d497c2d-d126-4f70-a968-d7371e39793f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4dd7t\" (UID: \"8d497c2d-d126-4f70-a968-d7371e39793f\") " pod="openshift-insights/insights-runtime-extractor-4dd7t"
Sep 17 03:20:13.577005 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:13.577003    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8d497c2d-d126-4f70-a968-d7371e39793f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4dd7t\" (UID: \"8d497c2d-d126-4f70-a968-d7371e39793f\") " pod="openshift-insights/insights-runtime-extractor-4dd7t"
Sep 17 03:20:13.577434 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:20:13.577085    2552 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 03:20:13.577434 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:20:13.577157    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8d497c2d-d126-4f70-a968-d7371e39793f-kube-rbac-proxy-cm podName:8d497c2d-d126-4f70-a968-d7371e39793f nodeName:}" failed. No retries permitted until 2026-09-17 03:20:15.577140841 +0000 UTC m=+51.043548345 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8d497c2d-d126-4f70-a968-d7371e39793f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4dd7t" (UID: "8d497c2d-d126-4f70-a968-d7371e39793f") : configmap "kube-rbac-proxy" not found
Sep 17 03:20:13.577434 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:20:13.577184    2552 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 03:20:13.577434 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:20:13.577241    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d497c2d-d126-4f70-a968-d7371e39793f-insights-runtime-extractor-tls podName:8d497c2d-d126-4f70-a968-d7371e39793f nodeName:}" failed. No retries permitted until 2026-09-17 03:20:15.577226197 +0000 UTC m=+51.043633705 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8d497c2d-d126-4f70-a968-d7371e39793f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4dd7t" (UID: "8d497c2d-d126-4f70-a968-d7371e39793f") : secret "insights-runtime-extractor-tls" not found
Sep 17 03:20:14.066626 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:14.066584    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-gxmbv_b3bac7f7-fbb1-4cec-8a26-046b702f4d70/dns-node-resolver/0.log"
Sep 17 03:20:14.866452 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:14.866426    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-rld2q_b7b79096-9bfc-4d82-9406-073a3f7961d0/node-ca/0.log"
Sep 17 03:20:15.293913 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:15.293859    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a7d922cf-74cc-4afb-b900-b21d63d4a225-metrics-tls\") pod \"dns-default-vb2fm\" (UID: \"a7d922cf-74cc-4afb-b900-b21d63d4a225\") " pod="openshift-dns/dns-default-vb2fm"
Sep 17 03:20:15.294075 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:20:15.294014    2552 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 03:20:15.294132 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:20:15.294090    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7d922cf-74cc-4afb-b900-b21d63d4a225-metrics-tls podName:a7d922cf-74cc-4afb-b900-b21d63d4a225 nodeName:}" failed. No retries permitted until 2026-09-17 03:20:31.294074803 +0000 UTC m=+66.760482307 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a7d922cf-74cc-4afb-b900-b21d63d4a225-metrics-tls") pod "dns-default-vb2fm" (UID: "a7d922cf-74cc-4afb-b900-b21d63d4a225") : secret "dns-default-metrics-tls" not found
Sep 17 03:20:15.596822 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:15.596721    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8d497c2d-d126-4f70-a968-d7371e39793f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4dd7t\" (UID: \"8d497c2d-d126-4f70-a968-d7371e39793f\") " pod="openshift-insights/insights-runtime-extractor-4dd7t"
Sep 17 03:20:15.596822 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:15.596770    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8d497c2d-d126-4f70-a968-d7371e39793f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4dd7t\" (UID: \"8d497c2d-d126-4f70-a968-d7371e39793f\") " pod="openshift-insights/insights-runtime-extractor-4dd7t"
Sep 17 03:20:15.597032 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:20:15.596860    2552 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 03:20:15.597032 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:20:15.596935    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8d497c2d-d126-4f70-a968-d7371e39793f-kube-rbac-proxy-cm podName:8d497c2d-d126-4f70-a968-d7371e39793f nodeName:}" failed. No retries permitted until 2026-09-17 03:20:19.59691606 +0000 UTC m=+55.063323581 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8d497c2d-d126-4f70-a968-d7371e39793f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4dd7t" (UID: "8d497c2d-d126-4f70-a968-d7371e39793f") : configmap "kube-rbac-proxy" not found
Sep 17 03:20:15.597032 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:20:15.596961    2552 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 03:20:15.597032 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:20:15.597033    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d497c2d-d126-4f70-a968-d7371e39793f-insights-runtime-extractor-tls podName:8d497c2d-d126-4f70-a968-d7371e39793f nodeName:}" failed. No retries permitted until 2026-09-17 03:20:19.597017517 +0000 UTC m=+55.063425021 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8d497c2d-d126-4f70-a968-d7371e39793f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4dd7t" (UID: "8d497c2d-d126-4f70-a968-d7371e39793f") : secret "insights-runtime-extractor-tls" not found
Sep 17 03:20:15.866323 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:15.866242    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-tbtsz_1a74da51-05c3-4067-b3a5-233734643f6e/serve-healthcheck-canary/0.log"
Sep 17 03:20:19.634164 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:19.634127    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8d497c2d-d126-4f70-a968-d7371e39793f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4dd7t\" (UID: \"8d497c2d-d126-4f70-a968-d7371e39793f\") " pod="openshift-insights/insights-runtime-extractor-4dd7t"
Sep 17 03:20:19.634164 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:19.634166    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8d497c2d-d126-4f70-a968-d7371e39793f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4dd7t\" (UID: \"8d497c2d-d126-4f70-a968-d7371e39793f\") " pod="openshift-insights/insights-runtime-extractor-4dd7t"
Sep 17 03:20:19.634766 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:20:19.634252    2552 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 03:20:19.634766 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:20:19.634325    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8d497c2d-d126-4f70-a968-d7371e39793f-kube-rbac-proxy-cm podName:8d497c2d-d126-4f70-a968-d7371e39793f nodeName:}" failed. No retries permitted until 2026-09-17 03:20:27.634305977 +0000 UTC m=+63.100713492 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8d497c2d-d126-4f70-a968-d7371e39793f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4dd7t" (UID: "8d497c2d-d126-4f70-a968-d7371e39793f") : configmap "kube-rbac-proxy" not found
Sep 17 03:20:19.634766 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:20:19.634324    2552 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 03:20:19.634766 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:20:19.634364    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d497c2d-d126-4f70-a968-d7371e39793f-insights-runtime-extractor-tls podName:8d497c2d-d126-4f70-a968-d7371e39793f nodeName:}" failed. No retries permitted until 2026-09-17 03:20:27.634354307 +0000 UTC m=+63.100761814 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8d497c2d-d126-4f70-a968-d7371e39793f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4dd7t" (UID: "8d497c2d-d126-4f70-a968-d7371e39793f") : secret "insights-runtime-extractor-tls" not found
Sep 17 03:20:20.844747 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:20.844700    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/226fdca4-147c-4de3-acf2-7bf0b2e871ba-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-n77tm\" (UID: \"226fdca4-147c-4de3-acf2-7bf0b2e871ba\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-n77tm"
Sep 17 03:20:20.845167 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:20.844903    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/4f8da691-30ee-4f3a-b816-d4d7b7a4b0a0-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-vtbq6\" (UID: \"4f8da691-30ee-4f3a-b816-d4d7b7a4b0a0\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-vtbq6"
Sep 17 03:20:20.845167 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:20.844977    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/88221296-e256-4d59-aba5-9465442a5727-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-qtch6\" (UID: \"88221296-e256-4d59-aba5-9465442a5727\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-qtch6"
Sep 17 03:20:20.845167 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:20:20.845029    2552 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 17 03:20:20.845167 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:20:20.845097    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f8da691-30ee-4f3a-b816-d4d7b7a4b0a0-cluster-monitoring-operator-tls podName:4f8da691-30ee-4f3a-b816-d4d7b7a4b0a0 nodeName:}" failed. No retries permitted until 2026-09-17 03:20:36.845077904 +0000 UTC m=+72.311485431 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/4f8da691-30ee-4f3a-b816-d4d7b7a4b0a0-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-vtbq6" (UID: "4f8da691-30ee-4f3a-b816-d4d7b7a4b0a0") : secret "cluster-monitoring-operator-tls" not found
Sep 17 03:20:20.845167 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:20:20.845138    2552 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 17 03:20:20.845364 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:20:20.845199    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88221296-e256-4d59-aba5-9465442a5727-networking-console-plugin-cert podName:88221296-e256-4d59-aba5-9465442a5727 nodeName:}" failed. No retries permitted until 2026-09-17 03:20:36.845185339 +0000 UTC m=+72.311592858 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/88221296-e256-4d59-aba5-9465442a5727-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-qtch6" (UID: "88221296-e256-4d59-aba5-9465442a5727") : secret "networking-console-plugin-cert" not found
Sep 17 03:20:20.847276 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:20.847244    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/226fdca4-147c-4de3-acf2-7bf0b2e871ba-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-n77tm\" (UID: \"226fdca4-147c-4de3-acf2-7bf0b2e871ba\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-n77tm"
Sep 17 03:20:20.851311 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:20.851288    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-n77tm"
Sep 17 03:20:20.945636 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:20.945574    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f68c2a78-217f-46fb-9003-c8ea7948e366-registry-tls\") pod \"image-registry-56f7fb996f-x5tjj\" (UID: \"f68c2a78-217f-46fb-9003-c8ea7948e366\") " pod="openshift-image-registry/image-registry-56f7fb996f-x5tjj"
Sep 17 03:20:20.948142 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:20.948119    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f68c2a78-217f-46fb-9003-c8ea7948e366-registry-tls\") pod \"image-registry-56f7fb996f-x5tjj\" (UID: \"f68c2a78-217f-46fb-9003-c8ea7948e366\") " pod="openshift-image-registry/image-registry-56f7fb996f-x5tjj"
Sep 17 03:20:20.998776 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:20.998734    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-56f7fb996f-x5tjj"
Sep 17 03:20:21.121693 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:21.121666    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-n77tm"]
Sep 17 03:20:21.271835 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:21.271799    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-56f7fb996f-x5tjj"]
Sep 17 03:20:21.274559 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:20:21.274524    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf68c2a78_217f_46fb_9003_c8ea7948e366.slice/crio-c4b31de317384f3f052756b5c7cab3887e14d32c872257a2258427ba2dc5897b WatchSource:0}: Error finding container c4b31de317384f3f052756b5c7cab3887e14d32c872257a2258427ba2dc5897b: Status 404 returned error can't find the container with id c4b31de317384f3f052756b5c7cab3887e14d32c872257a2258427ba2dc5897b
Sep 17 03:20:21.357656 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:21.357622    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-56f7fb996f-x5tjj" event={"ID":"f68c2a78-217f-46fb-9003-c8ea7948e366","Type":"ContainerStarted","Data":"dd2229e984f2d0d87a89c58a0cb4989d93c22d86efbc9d1e0ba127226e29d5e6"}
Sep 17 03:20:21.357832 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:21.357679    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-56f7fb996f-x5tjj" event={"ID":"f68c2a78-217f-46fb-9003-c8ea7948e366","Type":"ContainerStarted","Data":"c4b31de317384f3f052756b5c7cab3887e14d32c872257a2258427ba2dc5897b"}
Sep 17 03:20:21.357832 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:21.357704    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-56f7fb996f-x5tjj"
Sep 17 03:20:21.358942 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:21.358918    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-n77tm" event={"ID":"226fdca4-147c-4de3-acf2-7bf0b2e871ba","Type":"ContainerStarted","Data":"661768f99e4a9460bcebeb1400228f2a0c6ef60a4fdd9264f037efa93c35ccdc"}
Sep 17 03:20:21.376373 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:21.376259    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-56f7fb996f-x5tjj" podStartSLOduration=16.376239042 podStartE2EDuration="16.376239042s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 03:20:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:20:21.375064542 +0000 UTC m=+56.841472067" watchObservedRunningTime="2026-09-17 03:20:21.376239042 +0000 UTC m=+56.842646568"
Sep 17 03:20:23.368380 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:23.368345    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-n77tm" event={"ID":"226fdca4-147c-4de3-acf2-7bf0b2e871ba","Type":"ContainerStarted","Data":"38a41051f082370e49785980fd0d6a99756697528aa2cc997e679b302e2c915c"}
Sep 17 03:20:23.368822 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:23.368432    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-n77tm" event={"ID":"226fdca4-147c-4de3-acf2-7bf0b2e871ba","Type":"ContainerStarted","Data":"2f8829871682a1ce44d118f19d90f0c40fd2d0e914d84a60ecfaba676a3d78da"}
Sep 17 03:20:23.385941 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:23.385882    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-n77tm" podStartSLOduration=17.334726993 podStartE2EDuration="19.38586544s" totalImagesPullingTime="2.051138447s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 03:20:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:20:23.385555317 +0000 UTC m=+58.851962842" watchObservedRunningTime="2026-09-17 03:20:23.38586544 +0000 UTC m=+58.852272965"
Sep 17 03:20:26.265166 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:26.265135    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ldhl4"
Sep 17 03:20:27.709163 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:27.709116    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8d497c2d-d126-4f70-a968-d7371e39793f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4dd7t\" (UID: \"8d497c2d-d126-4f70-a968-d7371e39793f\") " pod="openshift-insights/insights-runtime-extractor-4dd7t"
Sep 17 03:20:27.709163 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:27.709162    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8d497c2d-d126-4f70-a968-d7371e39793f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4dd7t\" (UID: \"8d497c2d-d126-4f70-a968-d7371e39793f\") " pod="openshift-insights/insights-runtime-extractor-4dd7t"
Sep 17 03:20:27.709716 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:27.709692    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8d497c2d-d126-4f70-a968-d7371e39793f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4dd7t\" (UID: \"8d497c2d-d126-4f70-a968-d7371e39793f\") " pod="openshift-insights/insights-runtime-extractor-4dd7t"
Sep 17 03:20:27.711629 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:27.711588    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8d497c2d-d126-4f70-a968-d7371e39793f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4dd7t\" (UID: \"8d497c2d-d126-4f70-a968-d7371e39793f\") " pod="openshift-insights/insights-runtime-extractor-4dd7t"
Sep 17 03:20:27.830884 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:27.830848    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-52cf6\""
Sep 17 03:20:27.839109 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:27.839078    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-4dd7t"
Sep 17 03:20:28.109473 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:28.109441    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-4dd7t"]
Sep 17 03:20:28.113295 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:20:28.113269    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d497c2d_d126_4f70_a968_d7371e39793f.slice/crio-9e18e2e2f72345270434341aaeb782890c43f3ed853d19a5e9ce256ec7bd28f5 WatchSource:0}: Error finding container 9e18e2e2f72345270434341aaeb782890c43f3ed853d19a5e9ce256ec7bd28f5: Status 404 returned error can't find the container with id 9e18e2e2f72345270434341aaeb782890c43f3ed853d19a5e9ce256ec7bd28f5
Sep 17 03:20:28.387367 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:28.387289    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4dd7t" event={"ID":"8d497c2d-d126-4f70-a968-d7371e39793f","Type":"ContainerStarted","Data":"4cd33b13e36ca60384be9f47c7926158c23edf1be038984ad152c98cc6fc11d8"}
Sep 17 03:20:28.387367 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:28.387346    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4dd7t" event={"ID":"8d497c2d-d126-4f70-a968-d7371e39793f","Type":"ContainerStarted","Data":"9e18e2e2f72345270434341aaeb782890c43f3ed853d19a5e9ce256ec7bd28f5"}
Sep 17 03:20:29.392535 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:29.392502    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4dd7t" event={"ID":"8d497c2d-d126-4f70-a968-d7371e39793f","Type":"ContainerStarted","Data":"aee2f1842143227a9ddccb5533e77c60643ccf866a45c38b0bc2728493b6e649"}
Sep 17 03:20:30.733563 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:30.733476    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2758c0c8-e1ff-454a-9f0b-43cc8e1f8609-metrics-certs\") pod \"network-metrics-daemon-xl9wh\" (UID: \"2758c0c8-e1ff-454a-9f0b-43cc8e1f8609\") " pod="openshift-multus/network-metrics-daemon-xl9wh"
Sep 17 03:20:30.733977 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:30.733661    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmr6b\" (UniqueName: \"kubernetes.io/projected/9e5c90e7-d9db-4205-9bc6-c6790b31e53f-kube-api-access-nmr6b\") pod \"network-check-target-9d4m6\" (UID: \"9e5c90e7-d9db-4205-9bc6-c6790b31e53f\") " pod="openshift-network-diagnostics/network-check-target-9d4m6"
Sep 17 03:20:30.736271 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:30.736240    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 03:20:30.736406 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:30.736308    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmr6b\" (UniqueName: \"kubernetes.io/projected/9e5c90e7-d9db-4205-9bc6-c6790b31e53f-kube-api-access-nmr6b\") pod \"network-check-target-9d4m6\" (UID: \"9e5c90e7-d9db-4205-9bc6-c6790b31e53f\") " pod="openshift-network-diagnostics/network-check-target-9d4m6"
Sep 17 03:20:30.746454 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:30.746424    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2758c0c8-e1ff-454a-9f0b-43cc8e1f8609-metrics-certs\") pod \"network-metrics-daemon-xl9wh\" (UID: \"2758c0c8-e1ff-454a-9f0b-43cc8e1f8609\") " pod="openshift-multus/network-metrics-daemon-xl9wh"
Sep 17 03:20:30.834263 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:30.834201    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d1adf33b-0171-446e-aabe-f60fb74197e3-original-pull-secret\") pod \"global-pull-secret-syncer-4944l\" (UID: \"d1adf33b-0171-446e-aabe-f60fb74197e3\") " pod="kube-system/global-pull-secret-syncer-4944l"
Sep 17 03:20:30.837092 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:30.837066    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 03:20:30.847459 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:30.847428    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d1adf33b-0171-446e-aabe-f60fb74197e3-original-pull-secret\") pod \"global-pull-secret-syncer-4944l\" (UID: \"d1adf33b-0171-446e-aabe-f60fb74197e3\") " pod="kube-system/global-pull-secret-syncer-4944l"
Sep 17 03:20:30.895257 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:30.895221    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-d2k6b\""
Sep 17 03:20:30.899312 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:30.899288    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4944l"
Sep 17 03:20:30.903420 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:30.903398    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9d4m6"
Sep 17 03:20:30.994345 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:30.994274    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-tmrnq\""
Sep 17 03:20:31.000956 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:31.000916    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xl9wh"
Sep 17 03:20:31.169747 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:31.169713    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-4944l"]
Sep 17 03:20:31.173025 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:20:31.172996    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1adf33b_0171_446e_aabe_f60fb74197e3.slice/crio-6c10cea1429461546748b3b531b483e40974393a903c59313762428c7fd47350 WatchSource:0}: Error finding container 6c10cea1429461546748b3b531b483e40974393a903c59313762428c7fd47350: Status 404 returned error can't find the container with id 6c10cea1429461546748b3b531b483e40974393a903c59313762428c7fd47350
Sep 17 03:20:31.176330 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:31.176308    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-9d4m6"]
Sep 17 03:20:31.179174 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:20:31.179147    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e5c90e7_d9db_4205_9bc6_c6790b31e53f.slice/crio-5e742e84852679cfa8c806ea16d1fd6759dc943c8b4d18d396ce9496987cbffb WatchSource:0}: Error finding container 5e742e84852679cfa8c806ea16d1fd6759dc943c8b4d18d396ce9496987cbffb: Status 404 returned error can't find the container with id 5e742e84852679cfa8c806ea16d1fd6759dc943c8b4d18d396ce9496987cbffb
Sep 17 03:20:31.271945 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:31.271910    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-xl9wh"]
Sep 17 03:20:31.275006 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:20:31.274974    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2758c0c8_e1ff_454a_9f0b_43cc8e1f8609.slice/crio-353a1bc3ceac646ec5cbe4b5a35e96754660aa5ed7dc289863c1e6a93bc9ceef WatchSource:0}: Error finding container 353a1bc3ceac646ec5cbe4b5a35e96754660aa5ed7dc289863c1e6a93bc9ceef: Status 404 returned error can't find the container with id 353a1bc3ceac646ec5cbe4b5a35e96754660aa5ed7dc289863c1e6a93bc9ceef
Sep 17 03:20:31.338943 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:31.338903    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a7d922cf-74cc-4afb-b900-b21d63d4a225-metrics-tls\") pod \"dns-default-vb2fm\" (UID: \"a7d922cf-74cc-4afb-b900-b21d63d4a225\") " pod="openshift-dns/dns-default-vb2fm"
Sep 17 03:20:31.341280 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:31.341259    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a7d922cf-74cc-4afb-b900-b21d63d4a225-metrics-tls\") pod \"dns-default-vb2fm\" (UID: \"a7d922cf-74cc-4afb-b900-b21d63d4a225\") " pod="openshift-dns/dns-default-vb2fm"
Sep 17 03:20:31.401302 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:31.401273    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-9d4m6" event={"ID":"9e5c90e7-d9db-4205-9bc6-c6790b31e53f","Type":"ContainerStarted","Data":"16050da1b29b672075327eed34a6b0b63d3159d1c9a275ebcb7e381e9fcfaba1"}
Sep 17 03:20:31.401483 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:31.401318    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-9d4m6" event={"ID":"9e5c90e7-d9db-4205-9bc6-c6790b31e53f","Type":"ContainerStarted","Data":"5e742e84852679cfa8c806ea16d1fd6759dc943c8b4d18d396ce9496987cbffb"}
Sep 17 03:20:31.401483 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:31.401383    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-9d4m6"
Sep 17 03:20:31.402711 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:31.402688    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xl9wh" event={"ID":"2758c0c8-e1ff-454a-9f0b-43cc8e1f8609","Type":"ContainerStarted","Data":"353a1bc3ceac646ec5cbe4b5a35e96754660aa5ed7dc289863c1e6a93bc9ceef"}
Sep 17 03:20:31.403879 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:31.403859    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-4944l" event={"ID":"d1adf33b-0171-446e-aabe-f60fb74197e3","Type":"ContainerStarted","Data":"6c10cea1429461546748b3b531b483e40974393a903c59313762428c7fd47350"}
Sep 17 03:20:31.405941 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:31.405913    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4dd7t" event={"ID":"8d497c2d-d126-4f70-a968-d7371e39793f","Type":"ContainerStarted","Data":"c64de3642d25c1a5ed521778267d32ba39278741e8a0d128614080674e10a3cf"}
Sep 17 03:20:31.416806 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:31.416754    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-9d4m6" podStartSLOduration=66.416736697 podStartE2EDuration="1m6.416736697s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 03:19:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:20:31.41622898 +0000 UTC m=+66.882636510" watchObservedRunningTime="2026-09-17 03:20:31.416736697 +0000 UTC m=+66.883144217"
Sep 17 03:20:31.432516 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:31.432458    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-4dd7t" podStartSLOduration=18.226461647 podStartE2EDuration="20.432438248s" totalImagesPullingTime="2.205976601s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 03:20:11 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:20:31.431411379 +0000 UTC m=+66.897818916" watchObservedRunningTime="2026-09-17 03:20:31.432438248 +0000 UTC m=+66.898845774"
Sep 17 03:20:31.547168 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:31.547090    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-8xsvg\""
Sep 17 03:20:31.555303 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:31.555273    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-vb2fm"
Sep 17 03:20:31.825314 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:31.825225    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-vb2fm"]
Sep 17 03:20:31.828579 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:20:31.828540    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7d922cf_74cc_4afb_b900_b21d63d4a225.slice/crio-39c11a19722e81bac1df9b349a6b58f71a0dcc536d92c931fc38de5ba2c60769 WatchSource:0}: Error finding container 39c11a19722e81bac1df9b349a6b58f71a0dcc536d92c931fc38de5ba2c60769: Status 404 returned error can't find the container with id 39c11a19722e81bac1df9b349a6b58f71a0dcc536d92c931fc38de5ba2c60769
Sep 17 03:20:32.412352 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:32.412312    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vb2fm" event={"ID":"a7d922cf-74cc-4afb-b900-b21d63d4a225","Type":"ContainerStarted","Data":"39c11a19722e81bac1df9b349a6b58f71a0dcc536d92c931fc38de5ba2c60769"}
Sep 17 03:20:33.420993 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:33.420894    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xl9wh" event={"ID":"2758c0c8-e1ff-454a-9f0b-43cc8e1f8609","Type":"ContainerStarted","Data":"a913452bab9d3095f2a63540d1f7e68bd3513c6fe68a72892440c3a534b1fce1"}
Sep 17 03:20:33.420993 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:33.420992    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xl9wh" event={"ID":"2758c0c8-e1ff-454a-9f0b-43cc8e1f8609","Type":"ContainerStarted","Data":"d020c80fc5a1661d3c636560af6c956da907f49ec6418ab290bc2f15bf5a2b69"}
Sep 17 03:20:33.439254 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:33.439201    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-xl9wh" podStartSLOduration=66.864453598 podStartE2EDuration="1m8.439183643s" totalImagesPullingTime="1.574730045s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 03:19:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:20:33.438534516 +0000 UTC m=+68.904942043" watchObservedRunningTime="2026-09-17 03:20:33.439183643 +0000 UTC m=+68.905591168"
Sep 17 03:20:33.888781 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:33.888580    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6c4bff4958-z459s"]
Sep 17 03:20:33.908489 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:33.907622    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-56f7fb996f-x5tjj"]
Sep 17 03:20:33.908489 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:33.907678    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6c4bff4958-z459s"]
Sep 17 03:20:33.908489 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:33.907796    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6c4bff4958-z459s"
Sep 17 03:20:33.911794 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:33.911761    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 17 03:20:33.911937 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:33.911848    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 17 03:20:33.912112 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:33.912093    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 17 03:20:33.912338 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:33.912290    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 17 03:20:33.947514 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:33.947479    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7595f9b8-kknw8"]
Sep 17 03:20:33.963753 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:33.963721    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7595f9b8-kknw8"]
Sep 17 03:20:33.963940 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:33.963844    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7595f9b8-kknw8"
Sep 17 03:20:34.064922 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:34.064880    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hlj7\" (UniqueName: \"kubernetes.io/projected/070aa7f1-a578-454f-9714-06c6acb6dd20-kube-api-access-8hlj7\") pod \"klusterlet-addon-workmgr-6c4bff4958-z459s\" (UID: \"070aa7f1-a578-454f-9714-06c6acb6dd20\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6c4bff4958-z459s"
Sep 17 03:20:34.065096 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:34.064961    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/67b60bb3-13c4-4e20-a1b6-e6d766d8bcbe-registry-tls\") pod \"image-registry-7595f9b8-kknw8\" (UID: \"67b60bb3-13c4-4e20-a1b6-e6d766d8bcbe\") " pod="openshift-image-registry/image-registry-7595f9b8-kknw8"
Sep 17 03:20:34.065096 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:34.064994    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/67b60bb3-13c4-4e20-a1b6-e6d766d8bcbe-ca-trust-extracted\") pod \"image-registry-7595f9b8-kknw8\" (UID: \"67b60bb3-13c4-4e20-a1b6-e6d766d8bcbe\") " pod="openshift-image-registry/image-registry-7595f9b8-kknw8"
Sep 17 03:20:34.065096 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:34.065030    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/070aa7f1-a578-454f-9714-06c6acb6dd20-klusterlet-config\") pod \"klusterlet-addon-workmgr-6c4bff4958-z459s\" (UID: \"070aa7f1-a578-454f-9714-06c6acb6dd20\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6c4bff4958-z459s"
Sep 17 03:20:34.065096 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:34.065088    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/67b60bb3-13c4-4e20-a1b6-e6d766d8bcbe-trusted-ca\") pod \"image-registry-7595f9b8-kknw8\" (UID: \"67b60bb3-13c4-4e20-a1b6-e6d766d8bcbe\") " pod="openshift-image-registry/image-registry-7595f9b8-kknw8"
Sep 17 03:20:34.065329 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:34.065129    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6lct\" (UniqueName: \"kubernetes.io/projected/67b60bb3-13c4-4e20-a1b6-e6d766d8bcbe-kube-api-access-j6lct\") pod \"image-registry-7595f9b8-kknw8\" (UID: \"67b60bb3-13c4-4e20-a1b6-e6d766d8bcbe\") " pod="openshift-image-registry/image-registry-7595f9b8-kknw8"
Sep 17 03:20:34.065329 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:34.065171    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/070aa7f1-a578-454f-9714-06c6acb6dd20-tmp\") pod \"klusterlet-addon-workmgr-6c4bff4958-z459s\" (UID: \"070aa7f1-a578-454f-9714-06c6acb6dd20\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6c4bff4958-z459s"
Sep 17 03:20:34.065329 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:34.065218    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/67b60bb3-13c4-4e20-a1b6-e6d766d8bcbe-image-registry-private-configuration\") pod \"image-registry-7595f9b8-kknw8\" (UID: \"67b60bb3-13c4-4e20-a1b6-e6d766d8bcbe\") " pod="openshift-image-registry/image-registry-7595f9b8-kknw8"
Sep 17 03:20:34.065329 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:34.065247    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/67b60bb3-13c4-4e20-a1b6-e6d766d8bcbe-bound-sa-token\") pod \"image-registry-7595f9b8-kknw8\" (UID: \"67b60bb3-13c4-4e20-a1b6-e6d766d8bcbe\") " pod="openshift-image-registry/image-registry-7595f9b8-kknw8"
Sep 17 03:20:34.065329 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:34.065305    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/67b60bb3-13c4-4e20-a1b6-e6d766d8bcbe-installation-pull-secrets\") pod \"image-registry-7595f9b8-kknw8\" (UID: \"67b60bb3-13c4-4e20-a1b6-e6d766d8bcbe\") " pod="openshift-image-registry/image-registry-7595f9b8-kknw8"
Sep 17 03:20:34.065653 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:34.065355    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/67b60bb3-13c4-4e20-a1b6-e6d766d8bcbe-registry-certificates\") pod \"image-registry-7595f9b8-kknw8\" (UID: \"67b60bb3-13c4-4e20-a1b6-e6d766d8bcbe\") " pod="openshift-image-registry/image-registry-7595f9b8-kknw8"
Sep 17 03:20:34.166483 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:34.166389    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/070aa7f1-a578-454f-9714-06c6acb6dd20-klusterlet-config\") pod \"klusterlet-addon-workmgr-6c4bff4958-z459s\" (UID: \"070aa7f1-a578-454f-9714-06c6acb6dd20\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6c4bff4958-z459s"
Sep 17 03:20:34.166483 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:34.166442    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/67b60bb3-13c4-4e20-a1b6-e6d766d8bcbe-trusted-ca\") pod \"image-registry-7595f9b8-kknw8\" (UID: \"67b60bb3-13c4-4e20-a1b6-e6d766d8bcbe\") " pod="openshift-image-registry/image-registry-7595f9b8-kknw8"
Sep 17 03:20:34.166483 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:34.166474    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6lct\" (UniqueName: \"kubernetes.io/projected/67b60bb3-13c4-4e20-a1b6-e6d766d8bcbe-kube-api-access-j6lct\") pod \"image-registry-7595f9b8-kknw8\" (UID: \"67b60bb3-13c4-4e20-a1b6-e6d766d8bcbe\") " pod="openshift-image-registry/image-registry-7595f9b8-kknw8"
Sep 17 03:20:34.166770 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:34.166499    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/070aa7f1-a578-454f-9714-06c6acb6dd20-tmp\") pod \"klusterlet-addon-workmgr-6c4bff4958-z459s\" (UID: \"070aa7f1-a578-454f-9714-06c6acb6dd20\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6c4bff4958-z459s"
Sep 17 03:20:34.166770 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:34.166525    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/67b60bb3-13c4-4e20-a1b6-e6d766d8bcbe-image-registry-private-configuration\") pod \"image-registry-7595f9b8-kknw8\" (UID: \"67b60bb3-13c4-4e20-a1b6-e6d766d8bcbe\") " pod="openshift-image-registry/image-registry-7595f9b8-kknw8"
Sep 17 03:20:34.166770 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:34.166552    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/67b60bb3-13c4-4e20-a1b6-e6d766d8bcbe-bound-sa-token\") pod \"image-registry-7595f9b8-kknw8\" (UID: \"67b60bb3-13c4-4e20-a1b6-e6d766d8bcbe\") " pod="openshift-image-registry/image-registry-7595f9b8-kknw8"
Sep 17 03:20:34.166770 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:34.166578    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/67b60bb3-13c4-4e20-a1b6-e6d766d8bcbe-installation-pull-secrets\") pod \"image-registry-7595f9b8-kknw8\" (UID: \"67b60bb3-13c4-4e20-a1b6-e6d766d8bcbe\") " pod="openshift-image-registry/image-registry-7595f9b8-kknw8"
Sep 17 03:20:34.166770 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:34.166625    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/67b60bb3-13c4-4e20-a1b6-e6d766d8bcbe-registry-certificates\") pod \"image-registry-7595f9b8-kknw8\" (UID: \"67b60bb3-13c4-4e20-a1b6-e6d766d8bcbe\") " pod="openshift-image-registry/image-registry-7595f9b8-kknw8"
Sep 17 03:20:34.166770 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:34.166676    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hlj7\" (UniqueName: \"kubernetes.io/projected/070aa7f1-a578-454f-9714-06c6acb6dd20-kube-api-access-8hlj7\") pod \"klusterlet-addon-workmgr-6c4bff4958-z459s\" (UID: \"070aa7f1-a578-454f-9714-06c6acb6dd20\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6c4bff4958-z459s"
Sep 17 03:20:34.166770 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:34.166732    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/67b60bb3-13c4-4e20-a1b6-e6d766d8bcbe-registry-tls\") pod \"image-registry-7595f9b8-kknw8\" (UID: \"67b60bb3-13c4-4e20-a1b6-e6d766d8bcbe\") " pod="openshift-image-registry/image-registry-7595f9b8-kknw8"
Sep 17 03:20:34.166770 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:34.166758    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/67b60bb3-13c4-4e20-a1b6-e6d766d8bcbe-ca-trust-extracted\") pod \"image-registry-7595f9b8-kknw8\" (UID: \"67b60bb3-13c4-4e20-a1b6-e6d766d8bcbe\") " pod="openshift-image-registry/image-registry-7595f9b8-kknw8"
Sep 17 03:20:34.167273 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:34.167241    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/070aa7f1-a578-454f-9714-06c6acb6dd20-tmp\") pod \"klusterlet-addon-workmgr-6c4bff4958-z459s\" (UID: \"070aa7f1-a578-454f-9714-06c6acb6dd20\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6c4bff4958-z459s"
Sep 17 03:20:34.168026 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:34.167702    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/67b60bb3-13c4-4e20-a1b6-e6d766d8bcbe-registry-certificates\") pod \"image-registry-7595f9b8-kknw8\" (UID: \"67b60bb3-13c4-4e20-a1b6-e6d766d8bcbe\") " pod="openshift-image-registry/image-registry-7595f9b8-kknw8"
Sep 17 03:20:34.168026 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:34.167771    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/67b60bb3-13c4-4e20-a1b6-e6d766d8bcbe-trusted-ca\") pod \"image-registry-7595f9b8-kknw8\" (UID: \"67b60bb3-13c4-4e20-a1b6-e6d766d8bcbe\") " pod="openshift-image-registry/image-registry-7595f9b8-kknw8"
Sep 17 03:20:34.168026 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:34.167864    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/67b60bb3-13c4-4e20-a1b6-e6d766d8bcbe-ca-trust-extracted\") pod \"image-registry-7595f9b8-kknw8\" (UID: \"67b60bb3-13c4-4e20-a1b6-e6d766d8bcbe\") " pod="openshift-image-registry/image-registry-7595f9b8-kknw8"
Sep 17 03:20:34.170275 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:34.170249    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/070aa7f1-a578-454f-9714-06c6acb6dd20-klusterlet-config\") pod \"klusterlet-addon-workmgr-6c4bff4958-z459s\" (UID: \"070aa7f1-a578-454f-9714-06c6acb6dd20\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6c4bff4958-z459s"
Sep 17 03:20:34.170446 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:34.170419    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/67b60bb3-13c4-4e20-a1b6-e6d766d8bcbe-installation-pull-secrets\") pod \"image-registry-7595f9b8-kknw8\" (UID: \"67b60bb3-13c4-4e20-a1b6-e6d766d8bcbe\") " pod="openshift-image-registry/image-registry-7595f9b8-kknw8"
Sep 17 03:20:34.170724 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:34.170698    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/67b60bb3-13c4-4e20-a1b6-e6d766d8bcbe-image-registry-private-configuration\") pod \"image-registry-7595f9b8-kknw8\" (UID: \"67b60bb3-13c4-4e20-a1b6-e6d766d8bcbe\") " pod="openshift-image-registry/image-registry-7595f9b8-kknw8"
Sep 17 03:20:34.172263 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:34.172215    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/67b60bb3-13c4-4e20-a1b6-e6d766d8bcbe-registry-tls\") pod \"image-registry-7595f9b8-kknw8\" (UID: \"67b60bb3-13c4-4e20-a1b6-e6d766d8bcbe\") " pod="openshift-image-registry/image-registry-7595f9b8-kknw8"
Sep 17 03:20:34.176891 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:34.176861    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/67b60bb3-13c4-4e20-a1b6-e6d766d8bcbe-bound-sa-token\") pod \"image-registry-7595f9b8-kknw8\" (UID: \"67b60bb3-13c4-4e20-a1b6-e6d766d8bcbe\") " pod="openshift-image-registry/image-registry-7595f9b8-kknw8"
Sep 17 03:20:34.177192 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:34.177145    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hlj7\" (UniqueName: \"kubernetes.io/projected/070aa7f1-a578-454f-9714-06c6acb6dd20-kube-api-access-8hlj7\") pod \"klusterlet-addon-workmgr-6c4bff4958-z459s\" (UID: \"070aa7f1-a578-454f-9714-06c6acb6dd20\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6c4bff4958-z459s"
Sep 17 03:20:34.177192 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:34.177168    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6lct\" (UniqueName: \"kubernetes.io/projected/67b60bb3-13c4-4e20-a1b6-e6d766d8bcbe-kube-api-access-j6lct\") pod \"image-registry-7595f9b8-kknw8\" (UID: \"67b60bb3-13c4-4e20-a1b6-e6d766d8bcbe\") " pod="openshift-image-registry/image-registry-7595f9b8-kknw8"
Sep 17 03:20:34.222168 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:34.222126    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6c4bff4958-z459s"
Sep 17 03:20:34.275698 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:34.275662    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7595f9b8-kknw8"
Sep 17 03:20:34.724860 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:34.724789    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6c4bff4958-z459s"]
Sep 17 03:20:34.730507 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:34.730483    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7595f9b8-kknw8"]
Sep 17 03:20:36.223241 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:20:36.223191    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod070aa7f1_a578_454f_9714_06c6acb6dd20.slice/crio-58f06f477889fe0d57be0252a03dc0035bd07a7c44c948dc9f41f6b8b2f152a4 WatchSource:0}: Error finding container 58f06f477889fe0d57be0252a03dc0035bd07a7c44c948dc9f41f6b8b2f152a4: Status 404 returned error can't find the container with id 58f06f477889fe0d57be0252a03dc0035bd07a7c44c948dc9f41f6b8b2f152a4
Sep 17 03:20:36.224135 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:20:36.223778    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67b60bb3_13c4_4e20_a1b6_e6d766d8bcbe.slice/crio-166f07fdebb1df5da4cb1ba7bcabe2f2812e6d847c18399c4e6c418d00dcc8f6 WatchSource:0}: Error finding container 166f07fdebb1df5da4cb1ba7bcabe2f2812e6d847c18399c4e6c418d00dcc8f6: Status 404 returned error can't find the container with id 166f07fdebb1df5da4cb1ba7bcabe2f2812e6d847c18399c4e6c418d00dcc8f6
Sep 17 03:20:36.438055 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:36.438022    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-4944l" event={"ID":"d1adf33b-0171-446e-aabe-f60fb74197e3","Type":"ContainerStarted","Data":"ba67d49405ca8fbd192fa9fdc519fd201a9fe77327e0fa7ba6a3ad1945e59f27"}
Sep 17 03:20:36.441615 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:36.441161    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7595f9b8-kknw8" event={"ID":"67b60bb3-13c4-4e20-a1b6-e6d766d8bcbe","Type":"ContainerStarted","Data":"f1aac34875af4d048776ba3e26f5d31de12328f93432714fb5365aaf3b936952"}
Sep 17 03:20:36.441615 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:36.441199    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7595f9b8-kknw8" event={"ID":"67b60bb3-13c4-4e20-a1b6-e6d766d8bcbe","Type":"ContainerStarted","Data":"166f07fdebb1df5da4cb1ba7bcabe2f2812e6d847c18399c4e6c418d00dcc8f6"}
Sep 17 03:20:36.441615 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:36.441568    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7595f9b8-kknw8"
Sep 17 03:20:36.443983 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:36.443959    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vb2fm" event={"ID":"a7d922cf-74cc-4afb-b900-b21d63d4a225","Type":"ContainerStarted","Data":"c647c1d41f66e957fcc351954498c8d9123429bc380feba0e9f19eefd0de1d90"}
Sep 17 03:20:36.445250 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:36.445229    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6c4bff4958-z459s" event={"ID":"070aa7f1-a578-454f-9714-06c6acb6dd20","Type":"ContainerStarted","Data":"58f06f477889fe0d57be0252a03dc0035bd07a7c44c948dc9f41f6b8b2f152a4"}
Sep 17 03:20:36.452515 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:36.452452    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-4944l" podStartSLOduration=66.344929672 podStartE2EDuration="1m11.452437489s" totalImagesPullingTime="5.107507817s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 03:19:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:20:36.451745407 +0000 UTC m=+71.918152935" watchObservedRunningTime="2026-09-17 03:20:36.452437489 +0000 UTC m=+71.918845017"
Sep 17 03:20:36.470259 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:36.470202    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7595f9b8-kknw8" podStartSLOduration=3.470181764 podStartE2EDuration="3.470181764s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 03:20:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:20:36.468794882 +0000 UTC m=+71.935202409" watchObservedRunningTime="2026-09-17 03:20:36.470181764 +0000 UTC m=+71.936589291"
Sep 17 03:20:36.891592 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:36.891553    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/4f8da691-30ee-4f3a-b816-d4d7b7a4b0a0-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-vtbq6\" (UID: \"4f8da691-30ee-4f3a-b816-d4d7b7a4b0a0\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-vtbq6"
Sep 17 03:20:36.891787 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:36.891634    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/88221296-e256-4d59-aba5-9465442a5727-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-qtch6\" (UID: \"88221296-e256-4d59-aba5-9465442a5727\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-qtch6"
Sep 17 03:20:36.894108 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:36.894075    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/88221296-e256-4d59-aba5-9465442a5727-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-qtch6\" (UID: \"88221296-e256-4d59-aba5-9465442a5727\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-qtch6"
Sep 17 03:20:36.894216 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:36.894075    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/4f8da691-30ee-4f3a-b816-d4d7b7a4b0a0-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-vtbq6\" (UID: \"4f8da691-30ee-4f3a-b816-d4d7b7a4b0a0\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-vtbq6"
Sep 17 03:20:37.029500 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:37.029468    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-z2l2n\""
Sep 17 03:20:37.037467 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:37.037445    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-vtbq6"
Sep 17 03:20:37.078371 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:37.078341    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-sw29n\""
Sep 17 03:20:37.086545 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:37.086518    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-qtch6"
Sep 17 03:20:37.306119 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:37.306084    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-vtbq6"]
Sep 17 03:20:37.308832 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:20:37.308804    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f8da691_30ee_4f3a_b816_d4d7b7a4b0a0.slice/crio-29eab9cedc81ad33d39efc147012e84586fa4b7e28044402b4a206cb4fc76b67 WatchSource:0}: Error finding container 29eab9cedc81ad33d39efc147012e84586fa4b7e28044402b4a206cb4fc76b67: Status 404 returned error can't find the container with id 29eab9cedc81ad33d39efc147012e84586fa4b7e28044402b4a206cb4fc76b67
Sep 17 03:20:37.358834 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:37.358804    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-qtch6"]
Sep 17 03:20:37.364335 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:20:37.364303    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88221296_e256_4d59_aba5_9465442a5727.slice/crio-d719db8f2ffda6be3b140129a56dd72d90db096709083cda6bf706c27d4f42ac WatchSource:0}: Error finding container d719db8f2ffda6be3b140129a56dd72d90db096709083cda6bf706c27d4f42ac: Status 404 returned error can't find the container with id d719db8f2ffda6be3b140129a56dd72d90db096709083cda6bf706c27d4f42ac
Sep 17 03:20:37.450163 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:37.450128    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-qtch6" event={"ID":"88221296-e256-4d59-aba5-9465442a5727","Type":"ContainerStarted","Data":"d719db8f2ffda6be3b140129a56dd72d90db096709083cda6bf706c27d4f42ac"}
Sep 17 03:20:37.451895 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:37.451865    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vb2fm" event={"ID":"a7d922cf-74cc-4afb-b900-b21d63d4a225","Type":"ContainerStarted","Data":"56c926e341947e942438e66a5d43bac6dc4139594a5d8e9d768484b883a3f3c2"}
Sep 17 03:20:37.452032 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:37.451955    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-vb2fm"
Sep 17 03:20:37.453086 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:37.453067    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-vtbq6" event={"ID":"4f8da691-30ee-4f3a-b816-d4d7b7a4b0a0","Type":"ContainerStarted","Data":"29eab9cedc81ad33d39efc147012e84586fa4b7e28044402b4a206cb4fc76b67"}
Sep 17 03:20:37.468152 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:37.468114    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-vb2fm" podStartSLOduration=34.064482077 podStartE2EDuration="38.468101398s" totalImagesPullingTime="4.403619321s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 03:19:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:20:37.46723618 +0000 UTC m=+72.933643719" watchObservedRunningTime="2026-09-17 03:20:37.468101398 +0000 UTC m=+72.934508924"
Sep 17 03:20:39.463614 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:39.463566    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-qtch6" event={"ID":"88221296-e256-4d59-aba5-9465442a5727","Type":"ContainerStarted","Data":"afebae31cf2c3e47f119ce8a9043141cfa695702429dc6e17770c6249a22461f"}
Sep 17 03:20:39.465392 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:39.465359    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-vtbq6" event={"ID":"4f8da691-30ee-4f3a-b816-d4d7b7a4b0a0","Type":"ContainerStarted","Data":"27084ce4605f1ddcf6e9f18bdea479218731b3e3ee74bdd3c2c6c0ffecdf1005"}
Sep 17 03:20:39.479196 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:39.479138    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-qtch6" podStartSLOduration=34.207919363 podStartE2EDuration="35.479120358s" totalImagesPullingTime="1.271200995s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 03:20:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:20:39.478252424 +0000 UTC m=+74.944659952" watchObservedRunningTime="2026-09-17 03:20:39.479120358 +0000 UTC m=+74.945527884"
Sep 17 03:20:39.494187 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:39.494119    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-vtbq6" podStartSLOduration=33.489208121 podStartE2EDuration="35.49409675s" totalImagesPullingTime="2.004888629s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 03:20:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:20:39.49257549 +0000 UTC m=+74.958983031" watchObservedRunningTime="2026-09-17 03:20:39.49409675 +0000 UTC m=+74.960504277"
Sep 17 03:20:43.913329 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:43.913298    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-56f7fb996f-x5tjj"
Sep 17 03:20:47.200754 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:47.200625    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-vhrts"]
Sep 17 03:20:47.204982 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:47.204957    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-vhrts"
Sep 17 03:20:47.207718 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:47.207684    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 17 03:20:47.207922 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:47.207877    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-c2z72\""
Sep 17 03:20:47.209067 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:47.209041    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 17 03:20:47.209169 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:47.209043    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 17 03:20:47.209229 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:47.209166    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 17 03:20:47.375079 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:47.375045    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/849238c4-27b8-445a-85bb-e2c786c9dbcd-root\") pod \"node-exporter-vhrts\" (UID: \"849238c4-27b8-445a-85bb-e2c786c9dbcd\") " pod="openshift-monitoring/node-exporter-vhrts"
Sep 17 03:20:47.375079 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:47.375085    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/849238c4-27b8-445a-85bb-e2c786c9dbcd-node-exporter-wtmp\") pod \"node-exporter-vhrts\" (UID: \"849238c4-27b8-445a-85bb-e2c786c9dbcd\") " pod="openshift-monitoring/node-exporter-vhrts"
Sep 17 03:20:47.375295 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:47.375108    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/849238c4-27b8-445a-85bb-e2c786c9dbcd-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vhrts\" (UID: \"849238c4-27b8-445a-85bb-e2c786c9dbcd\") " pod="openshift-monitoring/node-exporter-vhrts"
Sep 17 03:20:47.375295 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:47.375154    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/849238c4-27b8-445a-85bb-e2c786c9dbcd-node-exporter-tls\") pod \"node-exporter-vhrts\" (UID: \"849238c4-27b8-445a-85bb-e2c786c9dbcd\") " pod="openshift-monitoring/node-exporter-vhrts"
Sep 17 03:20:47.375295 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:47.375178    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/849238c4-27b8-445a-85bb-e2c786c9dbcd-metrics-client-ca\") pod \"node-exporter-vhrts\" (UID: \"849238c4-27b8-445a-85bb-e2c786c9dbcd\") " pod="openshift-monitoring/node-exporter-vhrts"
Sep 17 03:20:47.375295 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:47.375203    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-979b2\" (UniqueName: \"kubernetes.io/projected/849238c4-27b8-445a-85bb-e2c786c9dbcd-kube-api-access-979b2\") pod \"node-exporter-vhrts\" (UID: \"849238c4-27b8-445a-85bb-e2c786c9dbcd\") " pod="openshift-monitoring/node-exporter-vhrts"
Sep 17 03:20:47.375295 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:47.375246    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/849238c4-27b8-445a-85bb-e2c786c9dbcd-node-exporter-textfile\") pod \"node-exporter-vhrts\" (UID: \"849238c4-27b8-445a-85bb-e2c786c9dbcd\") " pod="openshift-monitoring/node-exporter-vhrts"
Sep 17 03:20:47.375295 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:47.375292    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/849238c4-27b8-445a-85bb-e2c786c9dbcd-node-exporter-accelerators-collector-config\") pod \"node-exporter-vhrts\" (UID: \"849238c4-27b8-445a-85bb-e2c786c9dbcd\") " pod="openshift-monitoring/node-exporter-vhrts"
Sep 17 03:20:47.375477 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:47.375322    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/849238c4-27b8-445a-85bb-e2c786c9dbcd-sys\") pod \"node-exporter-vhrts\" (UID: \"849238c4-27b8-445a-85bb-e2c786c9dbcd\") " pod="openshift-monitoring/node-exporter-vhrts"
Sep 17 03:20:47.460051 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:47.459974    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-vb2fm"
Sep 17 03:20:47.476182 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:47.476137    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/849238c4-27b8-445a-85bb-e2c786c9dbcd-node-exporter-accelerators-collector-config\") pod \"node-exporter-vhrts\" (UID: \"849238c4-27b8-445a-85bb-e2c786c9dbcd\") " pod="openshift-monitoring/node-exporter-vhrts"
Sep 17 03:20:47.476329 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:47.476207    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/849238c4-27b8-445a-85bb-e2c786c9dbcd-sys\") pod \"node-exporter-vhrts\" (UID: \"849238c4-27b8-445a-85bb-e2c786c9dbcd\") " pod="openshift-monitoring/node-exporter-vhrts"
Sep 17 03:20:47.476329 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:47.476259    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/849238c4-27b8-445a-85bb-e2c786c9dbcd-root\") pod \"node-exporter-vhrts\" (UID: \"849238c4-27b8-445a-85bb-e2c786c9dbcd\") " pod="openshift-monitoring/node-exporter-vhrts"
Sep 17 03:20:47.476329 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:47.476289    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/849238c4-27b8-445a-85bb-e2c786c9dbcd-node-exporter-wtmp\") pod \"node-exporter-vhrts\" (UID: \"849238c4-27b8-445a-85bb-e2c786c9dbcd\") " pod="openshift-monitoring/node-exporter-vhrts"
Sep 17 03:20:47.476329 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:47.476315    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/849238c4-27b8-445a-85bb-e2c786c9dbcd-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vhrts\" (UID: \"849238c4-27b8-445a-85bb-e2c786c9dbcd\") " pod="openshift-monitoring/node-exporter-vhrts"
Sep 17 03:20:47.476555 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:47.476340    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/849238c4-27b8-445a-85bb-e2c786c9dbcd-node-exporter-tls\") pod \"node-exporter-vhrts\" (UID: \"849238c4-27b8-445a-85bb-e2c786c9dbcd\") " pod="openshift-monitoring/node-exporter-vhrts"
Sep 17 03:20:47.476555 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:47.476365    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/849238c4-27b8-445a-85bb-e2c786c9dbcd-metrics-client-ca\") pod \"node-exporter-vhrts\" (UID: \"849238c4-27b8-445a-85bb-e2c786c9dbcd\") " pod="openshift-monitoring/node-exporter-vhrts"
Sep 17 03:20:47.476555 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:47.476373    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/849238c4-27b8-445a-85bb-e2c786c9dbcd-root\") pod \"node-exporter-vhrts\" (UID: \"849238c4-27b8-445a-85bb-e2c786c9dbcd\") " pod="openshift-monitoring/node-exporter-vhrts"
Sep 17 03:20:47.476555 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:47.476388    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-979b2\" (UniqueName: \"kubernetes.io/projected/849238c4-27b8-445a-85bb-e2c786c9dbcd-kube-api-access-979b2\") pod \"node-exporter-vhrts\" (UID: \"849238c4-27b8-445a-85bb-e2c786c9dbcd\") " pod="openshift-monitoring/node-exporter-vhrts"
Sep 17 03:20:47.476555 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:20:47.476494    2552 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 17 03:20:47.476555 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:20:47.476552    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/849238c4-27b8-445a-85bb-e2c786c9dbcd-node-exporter-tls podName:849238c4-27b8-445a-85bb-e2c786c9dbcd nodeName:}" failed. No retries permitted until 2026-09-17 03:20:47.97653374 +0000 UTC m=+83.442941250 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/849238c4-27b8-445a-85bb-e2c786c9dbcd-node-exporter-tls") pod "node-exporter-vhrts" (UID: "849238c4-27b8-445a-85bb-e2c786c9dbcd") : secret "node-exporter-tls" not found
Sep 17 03:20:47.476895 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:47.476786    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/849238c4-27b8-445a-85bb-e2c786c9dbcd-node-exporter-textfile\") pod \"node-exporter-vhrts\" (UID: \"849238c4-27b8-445a-85bb-e2c786c9dbcd\") " pod="openshift-monitoring/node-exporter-vhrts"
Sep 17 03:20:47.476895 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:47.476798    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/849238c4-27b8-445a-85bb-e2c786c9dbcd-sys\") pod \"node-exporter-vhrts\" (UID: \"849238c4-27b8-445a-85bb-e2c786c9dbcd\") " pod="openshift-monitoring/node-exporter-vhrts"
Sep 17 03:20:47.476895 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:47.476802    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/849238c4-27b8-445a-85bb-e2c786c9dbcd-node-exporter-wtmp\") pod \"node-exporter-vhrts\" (UID: \"849238c4-27b8-445a-85bb-e2c786c9dbcd\") " pod="openshift-monitoring/node-exporter-vhrts"
Sep 17 03:20:47.476895 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:47.476841    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/849238c4-27b8-445a-85bb-e2c786c9dbcd-node-exporter-accelerators-collector-config\") pod \"node-exporter-vhrts\" (UID: \"849238c4-27b8-445a-85bb-e2c786c9dbcd\") " pod="openshift-monitoring/node-exporter-vhrts"
Sep 17 03:20:47.477087 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:47.477038    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/849238c4-27b8-445a-85bb-e2c786c9dbcd-node-exporter-textfile\") pod \"node-exporter-vhrts\" (UID: \"849238c4-27b8-445a-85bb-e2c786c9dbcd\") " pod="openshift-monitoring/node-exporter-vhrts"
Sep 17 03:20:47.477295 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:47.477273    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/849238c4-27b8-445a-85bb-e2c786c9dbcd-metrics-client-ca\") pod \"node-exporter-vhrts\" (UID: \"849238c4-27b8-445a-85bb-e2c786c9dbcd\") " pod="openshift-monitoring/node-exporter-vhrts"
Sep 17 03:20:47.479145 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:47.479125    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/849238c4-27b8-445a-85bb-e2c786c9dbcd-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vhrts\" (UID: \"849238c4-27b8-445a-85bb-e2c786c9dbcd\") " pod="openshift-monitoring/node-exporter-vhrts"
Sep 17 03:20:47.489417 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:47.489388    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-979b2\" (UniqueName: \"kubernetes.io/projected/849238c4-27b8-445a-85bb-e2c786c9dbcd-kube-api-access-979b2\") pod \"node-exporter-vhrts\" (UID: \"849238c4-27b8-445a-85bb-e2c786c9dbcd\") " pod="openshift-monitoring/node-exporter-vhrts"
Sep 17 03:20:47.981258 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:47.981209    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/849238c4-27b8-445a-85bb-e2c786c9dbcd-node-exporter-tls\") pod \"node-exporter-vhrts\" (UID: \"849238c4-27b8-445a-85bb-e2c786c9dbcd\") " pod="openshift-monitoring/node-exporter-vhrts"
Sep 17 03:20:47.981450 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:20:47.981376    2552 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 17 03:20:47.981450 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:20:47.981445    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/849238c4-27b8-445a-85bb-e2c786c9dbcd-node-exporter-tls podName:849238c4-27b8-445a-85bb-e2c786c9dbcd nodeName:}" failed. No retries permitted until 2026-09-17 03:20:48.981429428 +0000 UTC m=+84.447836932 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/849238c4-27b8-445a-85bb-e2c786c9dbcd-node-exporter-tls") pod "node-exporter-vhrts" (UID: "849238c4-27b8-445a-85bb-e2c786c9dbcd") : secret "node-exporter-tls" not found
Sep 17 03:20:48.286609 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:48.286561    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 03:20:48.291333 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:48.291311    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 17 03:20:48.294659 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:48.294629    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 17 03:20:48.294659 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:48.294654    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 17 03:20:48.294884 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:48.294639    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-szz6w\""
Sep 17 03:20:48.294884 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:48.294806    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 17 03:20:48.295001 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:48.294899    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 17 03:20:48.295001 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:48.294922    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 17 03:20:48.295100 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:48.295078    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 17 03:20:48.295191 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:48.295176    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 17 03:20:48.295292 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:48.295276    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 17 03:20:48.295510 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:48.295496    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 17 03:20:48.310206 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:48.310176    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 03:20:48.384650 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:48.384617    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9e36be1f-74b0-4dfa-8710-73a632a3f78a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"9e36be1f-74b0-4dfa-8710-73a632a3f78a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 03:20:48.384650 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:48.384655    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9e36be1f-74b0-4dfa-8710-73a632a3f78a-config-volume\") pod \"alertmanager-main-0\" (UID: \"9e36be1f-74b0-4dfa-8710-73a632a3f78a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 03:20:48.384873 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:48.384680    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9e36be1f-74b0-4dfa-8710-73a632a3f78a-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"9e36be1f-74b0-4dfa-8710-73a632a3f78a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 03:20:48.384873 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:48.384746    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9e36be1f-74b0-4dfa-8710-73a632a3f78a-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"9e36be1f-74b0-4dfa-8710-73a632a3f78a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 03:20:48.384873 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:48.384789    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9e36be1f-74b0-4dfa-8710-73a632a3f78a-web-config\") pod \"alertmanager-main-0\" (UID: \"9e36be1f-74b0-4dfa-8710-73a632a3f78a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 03:20:48.384873 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:48.384808    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9e36be1f-74b0-4dfa-8710-73a632a3f78a-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"9e36be1f-74b0-4dfa-8710-73a632a3f78a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 03:20:48.384873 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:48.384829    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9e36be1f-74b0-4dfa-8710-73a632a3f78a-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"9e36be1f-74b0-4dfa-8710-73a632a3f78a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 03:20:48.384873 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:48.384846    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9e36be1f-74b0-4dfa-8710-73a632a3f78a-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"9e36be1f-74b0-4dfa-8710-73a632a3f78a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 03:20:48.384873 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:48.384877    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfdhm\" (UniqueName: \"kubernetes.io/projected/9e36be1f-74b0-4dfa-8710-73a632a3f78a-kube-api-access-jfdhm\") pod \"alertmanager-main-0\" (UID: \"9e36be1f-74b0-4dfa-8710-73a632a3f78a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 03:20:48.385097 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:48.384896    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9e36be1f-74b0-4dfa-8710-73a632a3f78a-tls-assets\") pod \"alertmanager-main-0\" (UID: \"9e36be1f-74b0-4dfa-8710-73a632a3f78a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 03:20:48.385097 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:48.384932    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9e36be1f-74b0-4dfa-8710-73a632a3f78a-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"9e36be1f-74b0-4dfa-8710-73a632a3f78a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 03:20:48.385097 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:48.384951    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9e36be1f-74b0-4dfa-8710-73a632a3f78a-config-out\") pod \"alertmanager-main-0\" (UID: \"9e36be1f-74b0-4dfa-8710-73a632a3f78a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 03:20:48.385097 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:48.385021    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9e36be1f-74b0-4dfa-8710-73a632a3f78a-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"9e36be1f-74b0-4dfa-8710-73a632a3f78a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 03:20:48.486089 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:48.486047    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9e36be1f-74b0-4dfa-8710-73a632a3f78a-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"9e36be1f-74b0-4dfa-8710-73a632a3f78a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 03:20:48.486287 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:48.486108    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9e36be1f-74b0-4dfa-8710-73a632a3f78a-web-config\") pod \"alertmanager-main-0\" (UID: \"9e36be1f-74b0-4dfa-8710-73a632a3f78a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 03:20:48.486287 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:48.486139    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9e36be1f-74b0-4dfa-8710-73a632a3f78a-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"9e36be1f-74b0-4dfa-8710-73a632a3f78a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 03:20:48.486287 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:48.486173    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9e36be1f-74b0-4dfa-8710-73a632a3f78a-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"9e36be1f-74b0-4dfa-8710-73a632a3f78a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 03:20:48.486287 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:48.486197    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9e36be1f-74b0-4dfa-8710-73a632a3f78a-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"9e36be1f-74b0-4dfa-8710-73a632a3f78a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 03:20:48.486287 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:48.486224    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfdhm\" (UniqueName: \"kubernetes.io/projected/9e36be1f-74b0-4dfa-8710-73a632a3f78a-kube-api-access-jfdhm\") pod \"alertmanager-main-0\" (UID: \"9e36be1f-74b0-4dfa-8710-73a632a3f78a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 03:20:48.486287 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:48.486254    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9e36be1f-74b0-4dfa-8710-73a632a3f78a-tls-assets\") pod \"alertmanager-main-0\" (UID: \"9e36be1f-74b0-4dfa-8710-73a632a3f78a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 03:20:48.486287 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:48.486285    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9e36be1f-74b0-4dfa-8710-73a632a3f78a-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"9e36be1f-74b0-4dfa-8710-73a632a3f78a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 03:20:48.486671 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:48.486316    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9e36be1f-74b0-4dfa-8710-73a632a3f78a-config-out\") pod \"alertmanager-main-0\" (UID: \"9e36be1f-74b0-4dfa-8710-73a632a3f78a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 03:20:48.486671 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:48.486361    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9e36be1f-74b0-4dfa-8710-73a632a3f78a-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"9e36be1f-74b0-4dfa-8710-73a632a3f78a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 03:20:48.486671 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:48.486410    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9e36be1f-74b0-4dfa-8710-73a632a3f78a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"9e36be1f-74b0-4dfa-8710-73a632a3f78a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 03:20:48.486671 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:48.486438    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9e36be1f-74b0-4dfa-8710-73a632a3f78a-config-volume\") pod \"alertmanager-main-0\" (UID: \"9e36be1f-74b0-4dfa-8710-73a632a3f78a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 03:20:48.486671 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:48.486470    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9e36be1f-74b0-4dfa-8710-73a632a3f78a-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"9e36be1f-74b0-4dfa-8710-73a632a3f78a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 03:20:48.486671 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:20:48.486637    2552 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found
Sep 17 03:20:48.486976 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:20:48.486748    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9e36be1f-74b0-4dfa-8710-73a632a3f78a-secret-alertmanager-main-tls podName:9e36be1f-74b0-4dfa-8710-73a632a3f78a nodeName:}" failed. No retries permitted until 2026-09-17 03:20:48.986725256 +0000 UTC m=+84.453132777 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/9e36be1f-74b0-4dfa-8710-73a632a3f78a-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "9e36be1f-74b0-4dfa-8710-73a632a3f78a") : secret "alertmanager-main-tls" not found
Sep 17 03:20:48.488108 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:48.488034    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9e36be1f-74b0-4dfa-8710-73a632a3f78a-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"9e36be1f-74b0-4dfa-8710-73a632a3f78a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 03:20:48.488870 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:48.488778    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9e36be1f-74b0-4dfa-8710-73a632a3f78a-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"9e36be1f-74b0-4dfa-8710-73a632a3f78a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 03:20:48.489097 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:48.488939    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9e36be1f-74b0-4dfa-8710-73a632a3f78a-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"9e36be1f-74b0-4dfa-8710-73a632a3f78a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 03:20:48.489310 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:48.489289    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9e36be1f-74b0-4dfa-8710-73a632a3f78a-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"9e36be1f-74b0-4dfa-8710-73a632a3f78a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 03:20:48.489501 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:48.489477    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9e36be1f-74b0-4dfa-8710-73a632a3f78a-tls-assets\") pod \"alertmanager-main-0\" (UID: \"9e36be1f-74b0-4dfa-8710-73a632a3f78a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 03:20:48.489570 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:48.489488    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9e36be1f-74b0-4dfa-8710-73a632a3f78a-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"9e36be1f-74b0-4dfa-8710-73a632a3f78a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 03:20:48.490096 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:48.490064    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9e36be1f-74b0-4dfa-8710-73a632a3f78a-web-config\") pod \"alertmanager-main-0\" (UID: \"9e36be1f-74b0-4dfa-8710-73a632a3f78a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 03:20:48.491343 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:48.491316    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9e36be1f-74b0-4dfa-8710-73a632a3f78a-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"9e36be1f-74b0-4dfa-8710-73a632a3f78a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 03:20:48.491443 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:48.491316    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9e36be1f-74b0-4dfa-8710-73a632a3f78a-config-volume\") pod \"alertmanager-main-0\" (UID: \"9e36be1f-74b0-4dfa-8710-73a632a3f78a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 03:20:48.491504 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:48.491473    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9e36be1f-74b0-4dfa-8710-73a632a3f78a-config-out\") pod \"alertmanager-main-0\" (UID: \"9e36be1f-74b0-4dfa-8710-73a632a3f78a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 03:20:48.491875 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:48.491830    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9e36be1f-74b0-4dfa-8710-73a632a3f78a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"9e36be1f-74b0-4dfa-8710-73a632a3f78a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 03:20:48.495871 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:48.495850    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfdhm\" (UniqueName: \"kubernetes.io/projected/9e36be1f-74b0-4dfa-8710-73a632a3f78a-kube-api-access-jfdhm\") pod \"alertmanager-main-0\" (UID: \"9e36be1f-74b0-4dfa-8710-73a632a3f78a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 03:20:48.990503 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:48.990456    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9e36be1f-74b0-4dfa-8710-73a632a3f78a-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"9e36be1f-74b0-4dfa-8710-73a632a3f78a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 03:20:48.990730 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:48.990526    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/849238c4-27b8-445a-85bb-e2c786c9dbcd-node-exporter-tls\") pod \"node-exporter-vhrts\" (UID: \"849238c4-27b8-445a-85bb-e2c786c9dbcd\") " pod="openshift-monitoring/node-exporter-vhrts"
Sep 17 03:20:48.992909 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:48.992885    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/849238c4-27b8-445a-85bb-e2c786c9dbcd-node-exporter-tls\") pod \"node-exporter-vhrts\" (UID: \"849238c4-27b8-445a-85bb-e2c786c9dbcd\") " pod="openshift-monitoring/node-exporter-vhrts"
Sep 17 03:20:48.993151 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:48.993131    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9e36be1f-74b0-4dfa-8710-73a632a3f78a-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"9e36be1f-74b0-4dfa-8710-73a632a3f78a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 03:20:49.014839 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:49.014794    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-vhrts"
Sep 17 03:20:49.027043 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:20:49.027013    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod849238c4_27b8_445a_85bb_e2c786c9dbcd.slice/crio-fc3d1cd91e44c2f19e9e85c014c5ea939fb6a7d4827e7d489e1dd75451db830c WatchSource:0}: Error finding container fc3d1cd91e44c2f19e9e85c014c5ea939fb6a7d4827e7d489e1dd75451db830c: Status 404 returned error can't find the container with id fc3d1cd91e44c2f19e9e85c014c5ea939fb6a7d4827e7d489e1dd75451db830c
Sep 17 03:20:49.206496 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:49.206454    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 17 03:20:49.493579 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:49.493554    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 03:20:49.495178 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:20:49.495146    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e36be1f_74b0_4dfa_8710_73a632a3f78a.slice/crio-9e6459a7f80c1f1c18bfb3ec65c72020672b1141d813afb99d335e8a410a32f3 WatchSource:0}: Error finding container 9e6459a7f80c1f1c18bfb3ec65c72020672b1141d813afb99d335e8a410a32f3: Status 404 returned error can't find the container with id 9e6459a7f80c1f1c18bfb3ec65c72020672b1141d813afb99d335e8a410a32f3
Sep 17 03:20:49.504042 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:49.504008    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9e36be1f-74b0-4dfa-8710-73a632a3f78a","Type":"ContainerStarted","Data":"9e6459a7f80c1f1c18bfb3ec65c72020672b1141d813afb99d335e8a410a32f3"}
Sep 17 03:20:49.505332 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:49.505306    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vhrts" event={"ID":"849238c4-27b8-445a-85bb-e2c786c9dbcd","Type":"ContainerStarted","Data":"fc3d1cd91e44c2f19e9e85c014c5ea939fb6a7d4827e7d489e1dd75451db830c"}
Sep 17 03:20:50.180070 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:50.180031    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-574675c997-9jwg9"]
Sep 17 03:20:50.184715 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:50.184680    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-574675c997-9jwg9"
Sep 17 03:20:50.187655 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:50.187582    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-8psc37fjdapfi\""
Sep 17 03:20:50.187827 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:50.187583    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Sep 17 03:20:50.187827 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:50.187756    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Sep 17 03:20:50.187827 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:50.187764    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Sep 17 03:20:50.187827 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:50.187756    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-mzfp2\""
Sep 17 03:20:50.188059 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:50.187835    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Sep 17 03:20:50.188861 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:50.188659    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Sep 17 03:20:50.198288 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:50.198259    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-574675c997-9jwg9"]
Sep 17 03:20:50.302798 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:50.302693    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/ab812fe0-d6fb-45c0-a99e-41a4e6192740-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-574675c997-9jwg9\" (UID: \"ab812fe0-d6fb-45c0-a99e-41a4e6192740\") " pod="openshift-monitoring/thanos-querier-574675c997-9jwg9"
Sep 17 03:20:50.302798 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:50.302759    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twfbq\" (UniqueName: \"kubernetes.io/projected/ab812fe0-d6fb-45c0-a99e-41a4e6192740-kube-api-access-twfbq\") pod \"thanos-querier-574675c997-9jwg9\" (UID: \"ab812fe0-d6fb-45c0-a99e-41a4e6192740\") " pod="openshift-monitoring/thanos-querier-574675c997-9jwg9"
Sep 17 03:20:50.302798 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:50.302795    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ab812fe0-d6fb-45c0-a99e-41a4e6192740-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-574675c997-9jwg9\" (UID: \"ab812fe0-d6fb-45c0-a99e-41a4e6192740\") " pod="openshift-monitoring/thanos-querier-574675c997-9jwg9"
Sep 17 03:20:50.303209 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:50.302857    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/ab812fe0-d6fb-45c0-a99e-41a4e6192740-secret-thanos-querier-tls\") pod \"thanos-querier-574675c997-9jwg9\" (UID: \"ab812fe0-d6fb-45c0-a99e-41a4e6192740\") " pod="openshift-monitoring/thanos-querier-574675c997-9jwg9"
Sep 17 03:20:50.303209 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:50.302970    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/ab812fe0-d6fb-45c0-a99e-41a4e6192740-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-574675c997-9jwg9\" (UID: \"ab812fe0-d6fb-45c0-a99e-41a4e6192740\") " pod="openshift-monitoring/thanos-querier-574675c997-9jwg9"
Sep 17 03:20:50.303209 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:50.303017    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ab812fe0-d6fb-45c0-a99e-41a4e6192740-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-574675c997-9jwg9\" (UID: \"ab812fe0-d6fb-45c0-a99e-41a4e6192740\") " pod="openshift-monitoring/thanos-querier-574675c997-9jwg9"
Sep 17 03:20:50.303209 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:50.303045    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ab812fe0-d6fb-45c0-a99e-41a4e6192740-metrics-client-ca\") pod \"thanos-querier-574675c997-9jwg9\" (UID: \"ab812fe0-d6fb-45c0-a99e-41a4e6192740\") " pod="openshift-monitoring/thanos-querier-574675c997-9jwg9"
Sep 17 03:20:50.303209 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:50.303096    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ab812fe0-d6fb-45c0-a99e-41a4e6192740-secret-grpc-tls\") pod \"thanos-querier-574675c997-9jwg9\" (UID: \"ab812fe0-d6fb-45c0-a99e-41a4e6192740\") " pod="openshift-monitoring/thanos-querier-574675c997-9jwg9"
Sep 17 03:20:50.404316 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:50.404276    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/ab812fe0-d6fb-45c0-a99e-41a4e6192740-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-574675c997-9jwg9\" (UID: \"ab812fe0-d6fb-45c0-a99e-41a4e6192740\") " pod="openshift-monitoring/thanos-querier-574675c997-9jwg9"
Sep 17 03:20:50.404483 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:50.404336    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twfbq\" (UniqueName: \"kubernetes.io/projected/ab812fe0-d6fb-45c0-a99e-41a4e6192740-kube-api-access-twfbq\") pod \"thanos-querier-574675c997-9jwg9\" (UID: \"ab812fe0-d6fb-45c0-a99e-41a4e6192740\") " pod="openshift-monitoring/thanos-querier-574675c997-9jwg9"
Sep 17 03:20:50.404483 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:50.404375    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ab812fe0-d6fb-45c0-a99e-41a4e6192740-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-574675c997-9jwg9\" (UID: \"ab812fe0-d6fb-45c0-a99e-41a4e6192740\") " pod="openshift-monitoring/thanos-querier-574675c997-9jwg9"
Sep 17 03:20:50.404483 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:50.404421    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/ab812fe0-d6fb-45c0-a99e-41a4e6192740-secret-thanos-querier-tls\") pod \"thanos-querier-574675c997-9jwg9\" (UID: \"ab812fe0-d6fb-45c0-a99e-41a4e6192740\") " pod="openshift-monitoring/thanos-querier-574675c997-9jwg9"
Sep 17 03:20:50.404483 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:50.404456    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/ab812fe0-d6fb-45c0-a99e-41a4e6192740-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-574675c997-9jwg9\" (UID: \"ab812fe0-d6fb-45c0-a99e-41a4e6192740\") " pod="openshift-monitoring/thanos-querier-574675c997-9jwg9"
Sep 17 03:20:50.404737 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:50.404489    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ab812fe0-d6fb-45c0-a99e-41a4e6192740-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-574675c997-9jwg9\" (UID: \"ab812fe0-d6fb-45c0-a99e-41a4e6192740\") " pod="openshift-monitoring/thanos-querier-574675c997-9jwg9"
Sep 17 03:20:50.404737 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:50.404515    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ab812fe0-d6fb-45c0-a99e-41a4e6192740-metrics-client-ca\") pod \"thanos-querier-574675c997-9jwg9\" (UID: \"ab812fe0-d6fb-45c0-a99e-41a4e6192740\") " pod="openshift-monitoring/thanos-querier-574675c997-9jwg9"
Sep 17 03:20:50.404737 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:50.404561    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ab812fe0-d6fb-45c0-a99e-41a4e6192740-secret-grpc-tls\") pod \"thanos-querier-574675c997-9jwg9\" (UID: \"ab812fe0-d6fb-45c0-a99e-41a4e6192740\") " pod="openshift-monitoring/thanos-querier-574675c997-9jwg9"
Sep 17 03:20:50.405712 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:50.405657    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ab812fe0-d6fb-45c0-a99e-41a4e6192740-metrics-client-ca\") pod \"thanos-querier-574675c997-9jwg9\" (UID: \"ab812fe0-d6fb-45c0-a99e-41a4e6192740\") " pod="openshift-monitoring/thanos-querier-574675c997-9jwg9"
Sep 17 03:20:50.407433 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:50.407387    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ab812fe0-d6fb-45c0-a99e-41a4e6192740-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-574675c997-9jwg9\" (UID: \"ab812fe0-d6fb-45c0-a99e-41a4e6192740\") " pod="openshift-monitoring/thanos-querier-574675c997-9jwg9"
Sep 17 03:20:50.407708 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:50.407655    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ab812fe0-d6fb-45c0-a99e-41a4e6192740-secret-grpc-tls\") pod \"thanos-querier-574675c997-9jwg9\" (UID: \"ab812fe0-d6fb-45c0-a99e-41a4e6192740\") " pod="openshift-monitoring/thanos-querier-574675c997-9jwg9"
Sep 17 03:20:50.407825 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:50.407775    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/ab812fe0-d6fb-45c0-a99e-41a4e6192740-secret-thanos-querier-tls\") pod \"thanos-querier-574675c997-9jwg9\" (UID: \"ab812fe0-d6fb-45c0-a99e-41a4e6192740\") " pod="openshift-monitoring/thanos-querier-574675c997-9jwg9"
Sep 17 03:20:50.407920 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:50.407895    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ab812fe0-d6fb-45c0-a99e-41a4e6192740-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-574675c997-9jwg9\" (UID: \"ab812fe0-d6fb-45c0-a99e-41a4e6192740\") " pod="openshift-monitoring/thanos-querier-574675c997-9jwg9"
Sep 17 03:20:50.408040 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:50.408020    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/ab812fe0-d6fb-45c0-a99e-41a4e6192740-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-574675c997-9jwg9\" (UID: \"ab812fe0-d6fb-45c0-a99e-41a4e6192740\") " pod="openshift-monitoring/thanos-querier-574675c997-9jwg9"
Sep 17 03:20:50.408688 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:50.408665    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/ab812fe0-d6fb-45c0-a99e-41a4e6192740-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-574675c997-9jwg9\" (UID: \"ab812fe0-d6fb-45c0-a99e-41a4e6192740\") " pod="openshift-monitoring/thanos-querier-574675c997-9jwg9"
Sep 17 03:20:50.414812 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:50.414776    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-twfbq\" (UniqueName: \"kubernetes.io/projected/ab812fe0-d6fb-45c0-a99e-41a4e6192740-kube-api-access-twfbq\") pod \"thanos-querier-574675c997-9jwg9\" (UID: \"ab812fe0-d6fb-45c0-a99e-41a4e6192740\") " pod="openshift-monitoring/thanos-querier-574675c997-9jwg9"
Sep 17 03:20:50.511817 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:50.511785    2552 generic.go:356] "Generic (PLEG): container finished" podID="849238c4-27b8-445a-85bb-e2c786c9dbcd" containerID="43ef1210887e0d0c0c0fd6537157640236d4cdea295507e92cc8ab418fd0c430" exitCode=0
Sep 17 03:20:50.512194 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:50.511876    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vhrts" event={"ID":"849238c4-27b8-445a-85bb-e2c786c9dbcd","Type":"ContainerDied","Data":"43ef1210887e0d0c0c0fd6537157640236d4cdea295507e92cc8ab418fd0c430"}
Sep 17 03:20:50.553043 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:50.552936    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-574675c997-9jwg9"
Sep 17 03:20:50.940674 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:50.940634    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-574675c997-9jwg9"]
Sep 17 03:20:50.943640 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:20:50.943571    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab812fe0_d6fb_45c0_a99e_41a4e6192740.slice/crio-f621759d85a72afc481b0232a030a7f4f88d1421a0743f26545efe4fb5d84cfb WatchSource:0}: Error finding container f621759d85a72afc481b0232a030a7f4f88d1421a0743f26545efe4fb5d84cfb: Status 404 returned error can't find the container with id f621759d85a72afc481b0232a030a7f4f88d1421a0743f26545efe4fb5d84cfb
Sep 17 03:20:51.518525 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:51.518481    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vhrts" event={"ID":"849238c4-27b8-445a-85bb-e2c786c9dbcd","Type":"ContainerStarted","Data":"9124751bd6cc3c7ab5f7e533420d0cf1757967c0613e6f8c7ceb7f9d4f8d49e7"}
Sep 17 03:20:51.519030 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:51.518552    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vhrts" event={"ID":"849238c4-27b8-445a-85bb-e2c786c9dbcd","Type":"ContainerStarted","Data":"b53d43429006f330d02acb65a63b7c4b68e363c9a6bfc399c2116650664762ed"}
Sep 17 03:20:51.519820 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:51.519797    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-574675c997-9jwg9" event={"ID":"ab812fe0-d6fb-45c0-a99e-41a4e6192740","Type":"ContainerStarted","Data":"f621759d85a72afc481b0232a030a7f4f88d1421a0743f26545efe4fb5d84cfb"}
Sep 17 03:20:51.521418 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:51.521391    2552 generic.go:356] "Generic (PLEG): container finished" podID="9e36be1f-74b0-4dfa-8710-73a632a3f78a" containerID="6ab8a24c660d82178d6e63bf0d189f024056e2727580b28f32cc3ad78bdd16b7" exitCode=0
Sep 17 03:20:51.521541 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:51.521425    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9e36be1f-74b0-4dfa-8710-73a632a3f78a","Type":"ContainerDied","Data":"6ab8a24c660d82178d6e63bf0d189f024056e2727580b28f32cc3ad78bdd16b7"}
Sep 17 03:20:51.564299 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:51.564242    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-vhrts" podStartSLOduration=3.486984113 podStartE2EDuration="4.56422332s" totalImagesPullingTime="1.004922297s" totalInitContainerRuntime="72.31691ms" isStatefulPod=true podCreationTimestamp="2026-09-17 03:20:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:20:51.538516838 +0000 UTC m=+87.004924364" watchObservedRunningTime="2026-09-17 03:20:51.56422332 +0000 UTC m=+87.030630862"
Sep 17 03:20:52.545080 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:52.545039    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-7d84fddb95-j64fs"]
Sep 17 03:20:52.549288 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:52.549258    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7d84fddb95-j64fs"
Sep 17 03:20:52.552419 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:52.552385    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Sep 17 03:20:52.552614 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:52.552425    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Sep 17 03:20:52.552703 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:52.552688    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Sep 17 03:20:52.552753 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:52.552711    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-9knx6\""
Sep 17 03:20:52.552802 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:52.552767    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Sep 17 03:20:52.552869 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:52.552772    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Sep 17 03:20:52.559043 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:52.558503    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Sep 17 03:20:52.560235 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:52.560138    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7d84fddb95-j64fs"]
Sep 17 03:20:52.724693 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:52.724647    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/dc35de27-6f47-48d7-aaf1-0e40ecb04f5c-federate-client-tls\") pod \"telemeter-client-7d84fddb95-j64fs\" (UID: \"dc35de27-6f47-48d7-aaf1-0e40ecb04f5c\") " pod="openshift-monitoring/telemeter-client-7d84fddb95-j64fs"
Sep 17 03:20:52.724865 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:52.724707    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmz4b\" (UniqueName: \"kubernetes.io/projected/dc35de27-6f47-48d7-aaf1-0e40ecb04f5c-kube-api-access-vmz4b\") pod \"telemeter-client-7d84fddb95-j64fs\" (UID: \"dc35de27-6f47-48d7-aaf1-0e40ecb04f5c\") " pod="openshift-monitoring/telemeter-client-7d84fddb95-j64fs"
Sep 17 03:20:52.724865 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:52.724776    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dc35de27-6f47-48d7-aaf1-0e40ecb04f5c-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7d84fddb95-j64fs\" (UID: \"dc35de27-6f47-48d7-aaf1-0e40ecb04f5c\") " pod="openshift-monitoring/telemeter-client-7d84fddb95-j64fs"
Sep 17 03:20:52.724865 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:52.724853    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dc35de27-6f47-48d7-aaf1-0e40ecb04f5c-metrics-client-ca\") pod \"telemeter-client-7d84fddb95-j64fs\" (UID: \"dc35de27-6f47-48d7-aaf1-0e40ecb04f5c\") " pod="openshift-monitoring/telemeter-client-7d84fddb95-j64fs"
Sep 17 03:20:52.725032 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:52.724902    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dc35de27-6f47-48d7-aaf1-0e40ecb04f5c-serving-certs-ca-bundle\") pod \"telemeter-client-7d84fddb95-j64fs\" (UID: \"dc35de27-6f47-48d7-aaf1-0e40ecb04f5c\") " pod="openshift-monitoring/telemeter-client-7d84fddb95-j64fs"
Sep 17 03:20:52.725032 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:52.724943    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/dc35de27-6f47-48d7-aaf1-0e40ecb04f5c-telemeter-client-tls\") pod \"telemeter-client-7d84fddb95-j64fs\" (UID: \"dc35de27-6f47-48d7-aaf1-0e40ecb04f5c\") " pod="openshift-monitoring/telemeter-client-7d84fddb95-j64fs"
Sep 17 03:20:52.725032 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:52.724986    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dc35de27-6f47-48d7-aaf1-0e40ecb04f5c-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7d84fddb95-j64fs\" (UID: \"dc35de27-6f47-48d7-aaf1-0e40ecb04f5c\") " pod="openshift-monitoring/telemeter-client-7d84fddb95-j64fs"
Sep 17 03:20:52.725160 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:52.725045    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/dc35de27-6f47-48d7-aaf1-0e40ecb04f5c-secret-telemeter-client\") pod \"telemeter-client-7d84fddb95-j64fs\" (UID: \"dc35de27-6f47-48d7-aaf1-0e40ecb04f5c\") " pod="openshift-monitoring/telemeter-client-7d84fddb95-j64fs"
Sep 17 03:20:52.826552 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:52.826449    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/dc35de27-6f47-48d7-aaf1-0e40ecb04f5c-secret-telemeter-client\") pod \"telemeter-client-7d84fddb95-j64fs\" (UID: \"dc35de27-6f47-48d7-aaf1-0e40ecb04f5c\") " pod="openshift-monitoring/telemeter-client-7d84fddb95-j64fs"
Sep 17 03:20:52.826552 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:52.826518    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/dc35de27-6f47-48d7-aaf1-0e40ecb04f5c-federate-client-tls\") pod \"telemeter-client-7d84fddb95-j64fs\" (UID: \"dc35de27-6f47-48d7-aaf1-0e40ecb04f5c\") " pod="openshift-monitoring/telemeter-client-7d84fddb95-j64fs"
Sep 17 03:20:52.826552 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:52.826550    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmz4b\" (UniqueName: \"kubernetes.io/projected/dc35de27-6f47-48d7-aaf1-0e40ecb04f5c-kube-api-access-vmz4b\") pod \"telemeter-client-7d84fddb95-j64fs\" (UID: \"dc35de27-6f47-48d7-aaf1-0e40ecb04f5c\") " pod="openshift-monitoring/telemeter-client-7d84fddb95-j64fs"
Sep 17 03:20:52.826817 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:52.826586    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dc35de27-6f47-48d7-aaf1-0e40ecb04f5c-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7d84fddb95-j64fs\" (UID: \"dc35de27-6f47-48d7-aaf1-0e40ecb04f5c\") " pod="openshift-monitoring/telemeter-client-7d84fddb95-j64fs"
Sep 17 03:20:52.826817 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:52.826643    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dc35de27-6f47-48d7-aaf1-0e40ecb04f5c-metrics-client-ca\") pod \"telemeter-client-7d84fddb95-j64fs\" (UID: \"dc35de27-6f47-48d7-aaf1-0e40ecb04f5c\") " pod="openshift-monitoring/telemeter-client-7d84fddb95-j64fs"
Sep 17 03:20:52.826817 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:52.826674    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dc35de27-6f47-48d7-aaf1-0e40ecb04f5c-serving-certs-ca-bundle\") pod \"telemeter-client-7d84fddb95-j64fs\" (UID: \"dc35de27-6f47-48d7-aaf1-0e40ecb04f5c\") " pod="openshift-monitoring/telemeter-client-7d84fddb95-j64fs"
Sep 17 03:20:52.826817 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:52.826709    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/dc35de27-6f47-48d7-aaf1-0e40ecb04f5c-telemeter-client-tls\") pod \"telemeter-client-7d84fddb95-j64fs\" (UID: \"dc35de27-6f47-48d7-aaf1-0e40ecb04f5c\") " pod="openshift-monitoring/telemeter-client-7d84fddb95-j64fs"
Sep 17 03:20:52.826817 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:52.826730    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dc35de27-6f47-48d7-aaf1-0e40ecb04f5c-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7d84fddb95-j64fs\" (UID: \"dc35de27-6f47-48d7-aaf1-0e40ecb04f5c\") " pod="openshift-monitoring/telemeter-client-7d84fddb95-j64fs"
Sep 17 03:20:52.827629 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:52.827502    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dc35de27-6f47-48d7-aaf1-0e40ecb04f5c-metrics-client-ca\") pod \"telemeter-client-7d84fddb95-j64fs\" (UID: \"dc35de27-6f47-48d7-aaf1-0e40ecb04f5c\") " pod="openshift-monitoring/telemeter-client-7d84fddb95-j64fs"
Sep 17 03:20:52.827994 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:52.827943    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dc35de27-6f47-48d7-aaf1-0e40ecb04f5c-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7d84fddb95-j64fs\" (UID: \"dc35de27-6f47-48d7-aaf1-0e40ecb04f5c\") " pod="openshift-monitoring/telemeter-client-7d84fddb95-j64fs"
Sep 17 03:20:52.828713 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:52.828652    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dc35de27-6f47-48d7-aaf1-0e40ecb04f5c-serving-certs-ca-bundle\") pod \"telemeter-client-7d84fddb95-j64fs\" (UID: \"dc35de27-6f47-48d7-aaf1-0e40ecb04f5c\") " pod="openshift-monitoring/telemeter-client-7d84fddb95-j64fs"
Sep 17 03:20:52.829738 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:52.829716    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/dc35de27-6f47-48d7-aaf1-0e40ecb04f5c-federate-client-tls\") pod \"telemeter-client-7d84fddb95-j64fs\" (UID: \"dc35de27-6f47-48d7-aaf1-0e40ecb04f5c\") " pod="openshift-monitoring/telemeter-client-7d84fddb95-j64fs"
Sep 17 03:20:52.829954 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:52.829931    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/dc35de27-6f47-48d7-aaf1-0e40ecb04f5c-secret-telemeter-client\") pod \"telemeter-client-7d84fddb95-j64fs\" (UID: \"dc35de27-6f47-48d7-aaf1-0e40ecb04f5c\") " pod="openshift-monitoring/telemeter-client-7d84fddb95-j64fs"
Sep 17 03:20:52.830025 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:52.829951    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dc35de27-6f47-48d7-aaf1-0e40ecb04f5c-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7d84fddb95-j64fs\" (UID: \"dc35de27-6f47-48d7-aaf1-0e40ecb04f5c\") " pod="openshift-monitoring/telemeter-client-7d84fddb95-j64fs"
Sep 17 03:20:52.830471 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:52.830414    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/dc35de27-6f47-48d7-aaf1-0e40ecb04f5c-telemeter-client-tls\") pod \"telemeter-client-7d84fddb95-j64fs\" (UID: \"dc35de27-6f47-48d7-aaf1-0e40ecb04f5c\") " pod="openshift-monitoring/telemeter-client-7d84fddb95-j64fs"
Sep 17 03:20:52.835777 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:52.835552    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmz4b\" (UniqueName: \"kubernetes.io/projected/dc35de27-6f47-48d7-aaf1-0e40ecb04f5c-kube-api-access-vmz4b\") pod \"telemeter-client-7d84fddb95-j64fs\" (UID: \"dc35de27-6f47-48d7-aaf1-0e40ecb04f5c\") " pod="openshift-monitoring/telemeter-client-7d84fddb95-j64fs"
Sep 17 03:20:52.862498 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:52.862450    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7d84fddb95-j64fs"
Sep 17 03:20:53.534661 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:53.534492    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-574675c997-9jwg9" event={"ID":"ab812fe0-d6fb-45c0-a99e-41a4e6192740","Type":"ContainerStarted","Data":"2ae97e7b8364fb6e8c2a03dcb4dcbc68ebdd6b1109899265e5a7e62847c3f082"}
Sep 17 03:20:53.534661 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:53.534556    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-574675c997-9jwg9" event={"ID":"ab812fe0-d6fb-45c0-a99e-41a4e6192740","Type":"ContainerStarted","Data":"23dc9cd67993b26eba069b8a7e2d7b2fc4725135b6629383dc5c5afd564e3f25"}
Sep 17 03:20:53.534661 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:53.534582    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-574675c997-9jwg9" event={"ID":"ab812fe0-d6fb-45c0-a99e-41a4e6192740","Type":"ContainerStarted","Data":"07d27003f6654b70a51d67c444f8775e5dbfbc4bb0cc11498151b7b3edb1c896"}
Sep 17 03:20:53.537750 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:53.537709    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9e36be1f-74b0-4dfa-8710-73a632a3f78a","Type":"ContainerStarted","Data":"7cc2d86c41f99dc89fef8a42cf868eae6e3df93d026af7ab9bb7138b26379ad4"}
Sep 17 03:20:53.537908 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:53.537785    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9e36be1f-74b0-4dfa-8710-73a632a3f78a","Type":"ContainerStarted","Data":"5a066a74050dd98fd529a12d1e1424806acfa66aa835f271a87db7c44e7d6f50"}
Sep 17 03:20:53.537908 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:53.537823    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9e36be1f-74b0-4dfa-8710-73a632a3f78a","Type":"ContainerStarted","Data":"a4081ea222369421f80cc19c8e85b2e850016e59d0859f1db1f6e40c5c3abd18"}
Sep 17 03:20:53.568057 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:53.568027    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7d84fddb95-j64fs"]
Sep 17 03:20:53.571590 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:20:53.571558    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc35de27_6f47_48d7_aaf1_0e40ecb04f5c.slice/crio-b00aff0783a65a5260515a0a6df98739426d6a713ca24c6c9a72ee8ac280a9b4 WatchSource:0}: Error finding container b00aff0783a65a5260515a0a6df98739426d6a713ca24c6c9a72ee8ac280a9b4: Status 404 returned error can't find the container with id b00aff0783a65a5260515a0a6df98739426d6a713ca24c6c9a72ee8ac280a9b4
Sep 17 03:20:54.548561 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:54.548530    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9e36be1f-74b0-4dfa-8710-73a632a3f78a","Type":"ContainerStarted","Data":"471f5b125b4246c4cdc31c3cb40dab1afda0950c0d5da1984df955eaaccd10ff"}
Sep 17 03:20:54.548773 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:54.548626    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9e36be1f-74b0-4dfa-8710-73a632a3f78a","Type":"ContainerStarted","Data":"d4807b8c0662b63684961600121ba5e9f819e7eda9526fd56809d90f0e8e0e71"}
Sep 17 03:20:54.549970 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:54.549940    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7d84fddb95-j64fs" event={"ID":"dc35de27-6f47-48d7-aaf1-0e40ecb04f5c","Type":"ContainerStarted","Data":"b00aff0783a65a5260515a0a6df98739426d6a713ca24c6c9a72ee8ac280a9b4"}
Sep 17 03:20:55.558152 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:55.558049    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-574675c997-9jwg9" event={"ID":"ab812fe0-d6fb-45c0-a99e-41a4e6192740","Type":"ContainerStarted","Data":"43e6824d47ce3cf07529efc8a6e85148673a502c7f15fcfbcde18a2342345634"}
Sep 17 03:20:55.558152 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:55.558153    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-574675c997-9jwg9" event={"ID":"ab812fe0-d6fb-45c0-a99e-41a4e6192740","Type":"ContainerStarted","Data":"dfab5e670771075d3867b4aa3a03a9aa5a63efca4a9db4aa27ab05459d9c45eb"}
Sep 17 03:20:55.558713 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:55.558199    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-574675c997-9jwg9" event={"ID":"ab812fe0-d6fb-45c0-a99e-41a4e6192740","Type":"ContainerStarted","Data":"111303758de9df4ab1c2de1fd628e5aab3b19cb0c79701697274066432a7b843"}
Sep 17 03:20:55.558713 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:55.558228    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-574675c997-9jwg9"
Sep 17 03:20:55.562246 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:55.562209    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9e36be1f-74b0-4dfa-8710-73a632a3f78a","Type":"ContainerStarted","Data":"676fac31726644ea6bdf5f8ba1ef95baa1e4f695aae93a7baf1d1d3b056ca947"}
Sep 17 03:20:55.582867 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:55.582720    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-574675c997-9jwg9" podStartSLOduration=2.221843818 podStartE2EDuration="5.582697921s" totalImagesPullingTime="3.360854103s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 03:20:50 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:20:55.581019997 +0000 UTC m=+91.047427523" watchObservedRunningTime="2026-09-17 03:20:55.582697921 +0000 UTC m=+91.049105447"
Sep 17 03:20:55.611766 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:55.611700    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.714835452 podStartE2EDuration="7.611677969s" totalImagesPullingTime="3.878401297s" totalInitContainerRuntime="18.44122ms" isStatefulPod=false podCreationTimestamp="2026-09-17 03:20:48 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:20:55.610355716 +0000 UTC m=+91.076763240" watchObservedRunningTime="2026-09-17 03:20:55.611677969 +0000 UTC m=+91.078085497"
Sep 17 03:20:56.571292 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:56.571135    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7d84fddb95-j64fs" event={"ID":"dc35de27-6f47-48d7-aaf1-0e40ecb04f5c","Type":"ContainerStarted","Data":"105c0c78bde90574c41730168396df5e5c0a2d5c9586ce682a028e39e555eeea"}
Sep 17 03:20:56.571292 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:56.571213    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7d84fddb95-j64fs" event={"ID":"dc35de27-6f47-48d7-aaf1-0e40ecb04f5c","Type":"ContainerStarted","Data":"51a7296e4772f10f3825e4d8e645f1a85bdcc157590f73210a23486b8b079f75"}
Sep 17 03:20:56.571292 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:56.571249    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7d84fddb95-j64fs" event={"ID":"dc35de27-6f47-48d7-aaf1-0e40ecb04f5c","Type":"ContainerStarted","Data":"65d81cae275137421f85e9faa5ee2702d61659b44e376ec32e54bdd89de2333a"}
Sep 17 03:20:56.593934 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:56.593858    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-7d84fddb95-j64fs" podStartSLOduration=2.247878176 podStartE2EDuration="4.593832618s" totalImagesPullingTime="2.345954442s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 03:20:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:20:56.592147438 +0000 UTC m=+92.058555304" watchObservedRunningTime="2026-09-17 03:20:56.593832618 +0000 UTC m=+92.060240147"
Sep 17 03:20:58.462846 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:58.462813    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7595f9b8-kknw8"
Sep 17 03:20:58.933849 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:58.933784    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-56f7fb996f-x5tjj" podUID="f68c2a78-217f-46fb-9003-c8ea7948e366" containerName="registry" containerID="cri-o://dd2229e984f2d0d87a89c58a0cb4989d93c22d86efbc9d1e0ba127226e29d5e6" gracePeriod=30
Sep 17 03:20:59.187371 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:59.187295    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-56f7fb996f-x5tjj"
Sep 17 03:20:59.286948 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:59.286884    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f68c2a78-217f-46fb-9003-c8ea7948e366-trusted-ca\") pod \"f68c2a78-217f-46fb-9003-c8ea7948e366\" (UID: \"f68c2a78-217f-46fb-9003-c8ea7948e366\") "
Sep 17 03:20:59.286948 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:59.286963    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f68c2a78-217f-46fb-9003-c8ea7948e366-bound-sa-token\") pod \"f68c2a78-217f-46fb-9003-c8ea7948e366\" (UID: \"f68c2a78-217f-46fb-9003-c8ea7948e366\") "
Sep 17 03:20:59.287216 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:59.286986    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f68c2a78-217f-46fb-9003-c8ea7948e366-installation-pull-secrets\") pod \"f68c2a78-217f-46fb-9003-c8ea7948e366\" (UID: \"f68c2a78-217f-46fb-9003-c8ea7948e366\") "
Sep 17 03:20:59.287216 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:59.287008    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f68c2a78-217f-46fb-9003-c8ea7948e366-image-registry-private-configuration\") pod \"f68c2a78-217f-46fb-9003-c8ea7948e366\" (UID: \"f68c2a78-217f-46fb-9003-c8ea7948e366\") "
Sep 17 03:20:59.287216 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:59.287036    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f68c2a78-217f-46fb-9003-c8ea7948e366-registry-tls\") pod \"f68c2a78-217f-46fb-9003-c8ea7948e366\" (UID: \"f68c2a78-217f-46fb-9003-c8ea7948e366\") "
Sep 17 03:20:59.287216 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:59.287059    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f68c2a78-217f-46fb-9003-c8ea7948e366-ca-trust-extracted\") pod \"f68c2a78-217f-46fb-9003-c8ea7948e366\" (UID: \"f68c2a78-217f-46fb-9003-c8ea7948e366\") "
Sep 17 03:20:59.287216 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:59.287090    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bmdhm\" (UniqueName: \"kubernetes.io/projected/f68c2a78-217f-46fb-9003-c8ea7948e366-kube-api-access-bmdhm\") pod \"f68c2a78-217f-46fb-9003-c8ea7948e366\" (UID: \"f68c2a78-217f-46fb-9003-c8ea7948e366\") "
Sep 17 03:20:59.287216 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:59.287113    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f68c2a78-217f-46fb-9003-c8ea7948e366-registry-certificates\") pod \"f68c2a78-217f-46fb-9003-c8ea7948e366\" (UID: \"f68c2a78-217f-46fb-9003-c8ea7948e366\") "
Sep 17 03:20:59.287671 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:59.287640    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f68c2a78-217f-46fb-9003-c8ea7948e366-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "f68c2a78-217f-46fb-9003-c8ea7948e366" (UID: "f68c2a78-217f-46fb-9003-c8ea7948e366"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 03:20:59.287780 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:59.287743    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f68c2a78-217f-46fb-9003-c8ea7948e366-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "f68c2a78-217f-46fb-9003-c8ea7948e366" (UID: "f68c2a78-217f-46fb-9003-c8ea7948e366"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 03:20:59.289652 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:59.289607    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f68c2a78-217f-46fb-9003-c8ea7948e366-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "f68c2a78-217f-46fb-9003-c8ea7948e366" (UID: "f68c2a78-217f-46fb-9003-c8ea7948e366"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 03:20:59.289778 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:59.289760    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f68c2a78-217f-46fb-9003-c8ea7948e366-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "f68c2a78-217f-46fb-9003-c8ea7948e366" (UID: "f68c2a78-217f-46fb-9003-c8ea7948e366"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 03:20:59.289837 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:59.289808    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f68c2a78-217f-46fb-9003-c8ea7948e366-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "f68c2a78-217f-46fb-9003-c8ea7948e366" (UID: "f68c2a78-217f-46fb-9003-c8ea7948e366"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 03:20:59.289965 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:59.289943    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f68c2a78-217f-46fb-9003-c8ea7948e366-kube-api-access-bmdhm" (OuterVolumeSpecName: "kube-api-access-bmdhm") pod "f68c2a78-217f-46fb-9003-c8ea7948e366" (UID: "f68c2a78-217f-46fb-9003-c8ea7948e366"). InnerVolumeSpecName "kube-api-access-bmdhm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 03:20:59.290045 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:59.290024    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f68c2a78-217f-46fb-9003-c8ea7948e366-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "f68c2a78-217f-46fb-9003-c8ea7948e366" (UID: "f68c2a78-217f-46fb-9003-c8ea7948e366"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 03:20:59.296303 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:59.296262    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f68c2a78-217f-46fb-9003-c8ea7948e366-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "f68c2a78-217f-46fb-9003-c8ea7948e366" (UID: "f68c2a78-217f-46fb-9003-c8ea7948e366"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:20:59.388084 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:59.388034    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bmdhm\" (UniqueName: \"kubernetes.io/projected/f68c2a78-217f-46fb-9003-c8ea7948e366-kube-api-access-bmdhm\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:20:59.388084 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:59.388070    2552 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f68c2a78-217f-46fb-9003-c8ea7948e366-registry-certificates\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:20:59.388084 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:59.388082    2552 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f68c2a78-217f-46fb-9003-c8ea7948e366-trusted-ca\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:20:59.388084 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:59.388091    2552 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f68c2a78-217f-46fb-9003-c8ea7948e366-bound-sa-token\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:20:59.388341 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:59.388099    2552 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f68c2a78-217f-46fb-9003-c8ea7948e366-installation-pull-secrets\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:20:59.388341 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:59.388108    2552 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f68c2a78-217f-46fb-9003-c8ea7948e366-image-registry-private-configuration\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:20:59.388341 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:59.388117    2552 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f68c2a78-217f-46fb-9003-c8ea7948e366-registry-tls\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:20:59.388341 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:59.388125    2552 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f68c2a78-217f-46fb-9003-c8ea7948e366-ca-trust-extracted\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:20:59.586947 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:59.586897    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6c4bff4958-z459s" event={"ID":"070aa7f1-a578-454f-9714-06c6acb6dd20","Type":"ContainerStarted","Data":"5f164b7f6e07b608cbf6f8f9e0e768ebedca1cc4621e8c065c3baf0660a2ff4d"}
Sep 17 03:20:59.587629 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:59.587080    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6c4bff4958-z459s"
Sep 17 03:20:59.588806 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:59.588778    2552 generic.go:356] "Generic (PLEG): container finished" podID="f68c2a78-217f-46fb-9003-c8ea7948e366" containerID="dd2229e984f2d0d87a89c58a0cb4989d93c22d86efbc9d1e0ba127226e29d5e6" exitCode=0
Sep 17 03:20:59.588943 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:59.588821    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-56f7fb996f-x5tjj" event={"ID":"f68c2a78-217f-46fb-9003-c8ea7948e366","Type":"ContainerDied","Data":"dd2229e984f2d0d87a89c58a0cb4989d93c22d86efbc9d1e0ba127226e29d5e6"}
Sep 17 03:20:59.588943 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:59.588839    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-56f7fb996f-x5tjj"
Sep 17 03:20:59.588943 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:59.588861    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-56f7fb996f-x5tjj" event={"ID":"f68c2a78-217f-46fb-9003-c8ea7948e366","Type":"ContainerDied","Data":"c4b31de317384f3f052756b5c7cab3887e14d32c872257a2258427ba2dc5897b"}
Sep 17 03:20:59.588943 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:59.588888    2552 scope.go:117] "RemoveContainer" containerID="dd2229e984f2d0d87a89c58a0cb4989d93c22d86efbc9d1e0ba127226e29d5e6"
Sep 17 03:20:59.589157 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:59.589124    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6c4bff4958-z459s"
Sep 17 03:20:59.598986 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:59.598963    2552 scope.go:117] "RemoveContainer" containerID="dd2229e984f2d0d87a89c58a0cb4989d93c22d86efbc9d1e0ba127226e29d5e6"
Sep 17 03:20:59.599339 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:20:59.599320    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd2229e984f2d0d87a89c58a0cb4989d93c22d86efbc9d1e0ba127226e29d5e6\": container with ID starting with dd2229e984f2d0d87a89c58a0cb4989d93c22d86efbc9d1e0ba127226e29d5e6 not found: ID does not exist" containerID="dd2229e984f2d0d87a89c58a0cb4989d93c22d86efbc9d1e0ba127226e29d5e6"
Sep 17 03:20:59.599391 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:59.599348    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd2229e984f2d0d87a89c58a0cb4989d93c22d86efbc9d1e0ba127226e29d5e6"} err="failed to get container status \"dd2229e984f2d0d87a89c58a0cb4989d93c22d86efbc9d1e0ba127226e29d5e6\": rpc error: code = NotFound desc = could not find container \"dd2229e984f2d0d87a89c58a0cb4989d93c22d86efbc9d1e0ba127226e29d5e6\": container with ID starting with dd2229e984f2d0d87a89c58a0cb4989d93c22d86efbc9d1e0ba127226e29d5e6 not found: ID does not exist"
Sep 17 03:20:59.602420 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:59.602368    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6c4bff4958-z459s" podStartSLOduration=3.875885435 podStartE2EDuration="26.602350467s" totalImagesPullingTime="22.726465032s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 03:20:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:20:59.60152712 +0000 UTC m=+95.067934644" watchObservedRunningTime="2026-09-17 03:20:59.602350467 +0000 UTC m=+95.068757993"
Sep 17 03:20:59.628950 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:59.628916    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-56f7fb996f-x5tjj"]
Sep 17 03:20:59.634657 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:20:59.634583    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-56f7fb996f-x5tjj"]
Sep 17 03:21:00.775902 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:21:00.775865    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-76f99584f4-nqq7f"]
Sep 17 03:21:00.776382 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:21:00.776315    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f68c2a78-217f-46fb-9003-c8ea7948e366" containerName="registry"
Sep 17 03:21:00.776382 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:21:00.776334    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="f68c2a78-217f-46fb-9003-c8ea7948e366" containerName="registry"
Sep 17 03:21:00.776507 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:21:00.776433    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="f68c2a78-217f-46fb-9003-c8ea7948e366" containerName="registry"
Sep 17 03:21:00.782209 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:21:00.782180    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-nqq7f"
Sep 17 03:21:00.784762 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:21:00.784732    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 17 03:21:00.784924 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:21:00.784779    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-kmptw\""
Sep 17 03:21:00.784924 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:21:00.784837    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 17 03:21:00.790454 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:21:00.790415    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-nqq7f"]
Sep 17 03:21:00.802032 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:21:00.802003    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6thn\" (UniqueName: \"kubernetes.io/projected/90f3715a-13ff-4388-87e8-84c472dd4917-kube-api-access-h6thn\") pod \"downloads-76f99584f4-nqq7f\" (UID: \"90f3715a-13ff-4388-87e8-84c472dd4917\") " pod="openshift-console/downloads-76f99584f4-nqq7f"
Sep 17 03:21:00.903087 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:21:00.903047    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6thn\" (UniqueName: \"kubernetes.io/projected/90f3715a-13ff-4388-87e8-84c472dd4917-kube-api-access-h6thn\") pod \"downloads-76f99584f4-nqq7f\" (UID: \"90f3715a-13ff-4388-87e8-84c472dd4917\") " pod="openshift-console/downloads-76f99584f4-nqq7f"
Sep 17 03:21:00.911023 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:21:00.910991    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6thn\" (UniqueName: \"kubernetes.io/projected/90f3715a-13ff-4388-87e8-84c472dd4917-kube-api-access-h6thn\") pod \"downloads-76f99584f4-nqq7f\" (UID: \"90f3715a-13ff-4388-87e8-84c472dd4917\") " pod="openshift-console/downloads-76f99584f4-nqq7f"
Sep 17 03:21:00.985244 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:21:00.985206    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f68c2a78-217f-46fb-9003-c8ea7948e366" path="/var/lib/kubelet/pods/f68c2a78-217f-46fb-9003-c8ea7948e366/volumes"
Sep 17 03:21:01.099197 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:21:01.099101    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-nqq7f"
Sep 17 03:21:01.375611 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:21:01.375497    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-nqq7f"]
Sep 17 03:21:01.378350 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:21:01.378322    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90f3715a_13ff_4388_87e8_84c472dd4917.slice/crio-19002b11fd9bcde53713721d44f94516adca32798592c3740f668ec9fe5ad4f4 WatchSource:0}: Error finding container 19002b11fd9bcde53713721d44f94516adca32798592c3740f668ec9fe5ad4f4: Status 404 returned error can't find the container with id 19002b11fd9bcde53713721d44f94516adca32798592c3740f668ec9fe5ad4f4
Sep 17 03:21:01.578396 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:21:01.578361    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-574675c997-9jwg9"
Sep 17 03:21:01.599132 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:21:01.599099    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-nqq7f" event={"ID":"90f3715a-13ff-4388-87e8-84c472dd4917","Type":"ContainerStarted","Data":"19002b11fd9bcde53713721d44f94516adca32798592c3740f668ec9fe5ad4f4"}
Sep 17 03:21:02.416058 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:21:02.415899    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-9d4m6"
Sep 17 03:21:18.882495 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:21:18.882444    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-554ff6b6c4-mqp7s"]
Sep 17 03:21:18.885766 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:21:18.885736    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-554ff6b6c4-mqp7s"
Sep 17 03:21:18.888395 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:21:18.888367    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 17 03:21:18.889628 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:21:18.889534    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 17 03:21:18.889628 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:21:18.889562    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 17 03:21:18.889628 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:21:18.889562    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 17 03:21:18.889855 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:21:18.889647    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-5fg2h\""
Sep 17 03:21:18.889921 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:21:18.889886    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 17 03:21:18.893896 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:21:18.893869    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 17 03:21:18.896095 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:21:18.896071    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-554ff6b6c4-mqp7s"]
Sep 17 03:21:18.975822 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:21:18.975783    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/73bf9abb-6a04-458a-a5d2-75f6aeae7ab9-oauth-serving-cert\") pod \"console-554ff6b6c4-mqp7s\" (UID: \"73bf9abb-6a04-458a-a5d2-75f6aeae7ab9\") " pod="openshift-console/console-554ff6b6c4-mqp7s"
Sep 17 03:21:18.975991 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:21:18.975838    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/73bf9abb-6a04-458a-a5d2-75f6aeae7ab9-console-oauth-config\") pod \"console-554ff6b6c4-mqp7s\" (UID: \"73bf9abb-6a04-458a-a5d2-75f6aeae7ab9\") " pod="openshift-console/console-554ff6b6c4-mqp7s"
Sep 17 03:21:18.975991 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:21:18.975929    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/73bf9abb-6a04-458a-a5d2-75f6aeae7ab9-service-ca\") pod \"console-554ff6b6c4-mqp7s\" (UID: \"73bf9abb-6a04-458a-a5d2-75f6aeae7ab9\") " pod="openshift-console/console-554ff6b6c4-mqp7s"
Sep 17 03:21:18.976085 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:21:18.976049    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/73bf9abb-6a04-458a-a5d2-75f6aeae7ab9-console-serving-cert\") pod \"console-554ff6b6c4-mqp7s\" (UID: \"73bf9abb-6a04-458a-a5d2-75f6aeae7ab9\") " pod="openshift-console/console-554ff6b6c4-mqp7s"
Sep 17 03:21:18.976137 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:21:18.976089    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8hwx\" (UniqueName: \"kubernetes.io/projected/73bf9abb-6a04-458a-a5d2-75f6aeae7ab9-kube-api-access-r8hwx\") pod \"console-554ff6b6c4-mqp7s\" (UID: \"73bf9abb-6a04-458a-a5d2-75f6aeae7ab9\") " pod="openshift-console/console-554ff6b6c4-mqp7s"
Sep 17 03:21:18.976137 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:21:18.976128    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/73bf9abb-6a04-458a-a5d2-75f6aeae7ab9-console-config\") pod \"console-554ff6b6c4-mqp7s\" (UID: \"73bf9abb-6a04-458a-a5d2-75f6aeae7ab9\") " pod="openshift-console/console-554ff6b6c4-mqp7s"
Sep 17 03:21:18.976234 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:21:18.976155    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/73bf9abb-6a04-458a-a5d2-75f6aeae7ab9-trusted-ca-bundle\") pod \"console-554ff6b6c4-mqp7s\" (UID: \"73bf9abb-6a04-458a-a5d2-75f6aeae7ab9\") " pod="openshift-console/console-554ff6b6c4-mqp7s"
Sep 17 03:21:19.076960 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:21:19.076873    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/73bf9abb-6a04-458a-a5d2-75f6aeae7ab9-console-oauth-config\") pod \"console-554ff6b6c4-mqp7s\" (UID: \"73bf9abb-6a04-458a-a5d2-75f6aeae7ab9\") " pod="openshift-console/console-554ff6b6c4-mqp7s"
Sep 17 03:21:19.077141 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:21:19.077049    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/73bf9abb-6a04-458a-a5d2-75f6aeae7ab9-service-ca\") pod \"console-554ff6b6c4-mqp7s\" (UID: \"73bf9abb-6a04-458a-a5d2-75f6aeae7ab9\") " pod="openshift-console/console-554ff6b6c4-mqp7s"
Sep 17 03:21:19.077208 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:21:19.077161    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/73bf9abb-6a04-458a-a5d2-75f6aeae7ab9-console-serving-cert\") pod \"console-554ff6b6c4-mqp7s\" (UID: \"73bf9abb-6a04-458a-a5d2-75f6aeae7ab9\") " pod="openshift-console/console-554ff6b6c4-mqp7s"
Sep 17 03:21:19.077208 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:21:19.077193    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8hwx\" (UniqueName: \"kubernetes.io/projected/73bf9abb-6a04-458a-a5d2-75f6aeae7ab9-kube-api-access-r8hwx\") pod \"console-554ff6b6c4-mqp7s\" (UID: \"73bf9abb-6a04-458a-a5d2-75f6aeae7ab9\") " pod="openshift-console/console-554ff6b6c4-mqp7s"
Sep 17 03:21:19.077291 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:21:19.077233    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/73bf9abb-6a04-458a-a5d2-75f6aeae7ab9-console-config\") pod \"console-554ff6b6c4-mqp7s\" (UID: \"73bf9abb-6a04-458a-a5d2-75f6aeae7ab9\") " pod="openshift-console/console-554ff6b6c4-mqp7s"
Sep 17 03:21:19.077291 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:21:19.077272    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/73bf9abb-6a04-458a-a5d2-75f6aeae7ab9-trusted-ca-bundle\") pod \"console-554ff6b6c4-mqp7s\" (UID: \"73bf9abb-6a04-458a-a5d2-75f6aeae7ab9\") " pod="openshift-console/console-554ff6b6c4-mqp7s"
Sep 17 03:21:19.077394 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:21:19.077343    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/73bf9abb-6a04-458a-a5d2-75f6aeae7ab9-oauth-serving-cert\") pod \"console-554ff6b6c4-mqp7s\" (UID: \"73bf9abb-6a04-458a-a5d2-75f6aeae7ab9\") " pod="openshift-console/console-554ff6b6c4-mqp7s"
Sep 17 03:21:19.079477 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:21:19.078323    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/73bf9abb-6a04-458a-a5d2-75f6aeae7ab9-trusted-ca-bundle\") pod \"console-554ff6b6c4-mqp7s\" (UID: \"73bf9abb-6a04-458a-a5d2-75f6aeae7ab9\") " pod="openshift-console/console-554ff6b6c4-mqp7s"
Sep 17 03:21:19.079477 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:21:19.078879    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/73bf9abb-6a04-458a-a5d2-75f6aeae7ab9-console-config\") pod \"console-554ff6b6c4-mqp7s\" (UID: \"73bf9abb-6a04-458a-a5d2-75f6aeae7ab9\") " pod="openshift-console/console-554ff6b6c4-mqp7s"
Sep 17 03:21:19.079477 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:21:19.078903    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/73bf9abb-6a04-458a-a5d2-75f6aeae7ab9-oauth-serving-cert\") pod \"console-554ff6b6c4-mqp7s\" (UID: \"73bf9abb-6a04-458a-a5d2-75f6aeae7ab9\") " pod="openshift-console/console-554ff6b6c4-mqp7s"
Sep 17 03:21:19.079477 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:21:19.079432    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/73bf9abb-6a04-458a-a5d2-75f6aeae7ab9-service-ca\") pod \"console-554ff6b6c4-mqp7s\" (UID: \"73bf9abb-6a04-458a-a5d2-75f6aeae7ab9\") " pod="openshift-console/console-554ff6b6c4-mqp7s"
Sep 17 03:21:19.080518 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:21:19.080465    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/73bf9abb-6a04-458a-a5d2-75f6aeae7ab9-console-oauth-config\") pod \"console-554ff6b6c4-mqp7s\" (UID: \"73bf9abb-6a04-458a-a5d2-75f6aeae7ab9\") " pod="openshift-console/console-554ff6b6c4-mqp7s"
Sep 17 03:21:19.080665 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:21:19.080577    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/73bf9abb-6a04-458a-a5d2-75f6aeae7ab9-console-serving-cert\") pod \"console-554ff6b6c4-mqp7s\" (UID: \"73bf9abb-6a04-458a-a5d2-75f6aeae7ab9\") " pod="openshift-console/console-554ff6b6c4-mqp7s"
Sep 17 03:21:19.086464 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:21:19.086426    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8hwx\" (UniqueName: \"kubernetes.io/projected/73bf9abb-6a04-458a-a5d2-75f6aeae7ab9-kube-api-access-r8hwx\") pod \"console-554ff6b6c4-mqp7s\" (UID: \"73bf9abb-6a04-458a-a5d2-75f6aeae7ab9\") " pod="openshift-console/console-554ff6b6c4-mqp7s"
Sep 17 03:21:19.196931 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:21:19.196883    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-554ff6b6c4-mqp7s"
Sep 17 03:21:19.490041 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:21:19.489990    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-554ff6b6c4-mqp7s"]
Sep 17 03:21:19.492959 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:21:19.492922    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73bf9abb_6a04_458a_a5d2_75f6aeae7ab9.slice/crio-ebcc9873f5f0f7fe59a3621cbef4880c5cda2144a80eb021860874bc874b2a46 WatchSource:0}: Error finding container ebcc9873f5f0f7fe59a3621cbef4880c5cda2144a80eb021860874bc874b2a46: Status 404 returned error can't find the container with id ebcc9873f5f0f7fe59a3621cbef4880c5cda2144a80eb021860874bc874b2a46
Sep 17 03:21:19.682639 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:21:19.682576    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-554ff6b6c4-mqp7s" event={"ID":"73bf9abb-6a04-458a-a5d2-75f6aeae7ab9","Type":"ContainerStarted","Data":"ebcc9873f5f0f7fe59a3621cbef4880c5cda2144a80eb021860874bc874b2a46"}
Sep 17 03:21:19.684662 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:21:19.684619    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-nqq7f" event={"ID":"90f3715a-13ff-4388-87e8-84c472dd4917","Type":"ContainerStarted","Data":"9b6ad9948764609fe908be7e8716bb54b0f878ccafee473febfa7b451ea6218c"}
Sep 17 03:21:19.684883 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:21:19.684858    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-76f99584f4-nqq7f"
Sep 17 03:21:19.701580 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:21:19.701525    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-76f99584f4-nqq7f" podStartSLOduration=2.309703346 podStartE2EDuration="19.701503752s" totalImagesPullingTime="17.391800406s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 03:21:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:21:19.700907953 +0000 UTC m=+115.167315479" watchObservedRunningTime="2026-09-17 03:21:19.701503752 +0000 UTC m=+115.167911278"
Sep 17 03:21:19.703187 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:21:19.703157    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-76f99584f4-nqq7f"
Sep 17 03:21:22.702575 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:21:22.702529    2552 generic.go:356] "Generic (PLEG): container finished" podID="91c3490d-d515-4cc2-bafb-aa6c999ac3d0" containerID="a15ad1337839838a50f7c82ac0f3577b361b713ab585611eb5aac0752284c16e" exitCode=0
Sep 17 03:21:22.703074 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:21:22.702624    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-khv5k" event={"ID":"91c3490d-d515-4cc2-bafb-aa6c999ac3d0","Type":"ContainerDied","Data":"a15ad1337839838a50f7c82ac0f3577b361b713ab585611eb5aac0752284c16e"}
Sep 17 03:21:22.703146 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:21:22.703099    2552 scope.go:117] "RemoveContainer" containerID="a15ad1337839838a50f7c82ac0f3577b361b713ab585611eb5aac0752284c16e"
Sep 17 03:21:23.710017 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:21:23.709974    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-khv5k" event={"ID":"91c3490d-d515-4cc2-bafb-aa6c999ac3d0","Type":"ContainerStarted","Data":"d2deaf9d8fab829589d7ed708a08ef06716d1d2fc48799c76eafca47094778d2"}
Sep 17 03:21:23.712084 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:21:23.712042    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-554ff6b6c4-mqp7s" event={"ID":"73bf9abb-6a04-458a-a5d2-75f6aeae7ab9","Type":"ContainerStarted","Data":"d7006f13f054e09efe19704d3d10ac6ddbe82b0c9f9d9432162579e4b4db0831"}
Sep 17 03:21:23.744996 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:21:23.744754    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-554ff6b6c4-mqp7s" podStartSLOduration=2.1820151 podStartE2EDuration="5.744728418s" totalImagesPullingTime="3.562713318s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 03:21:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:21:23.742729101 +0000 UTC m=+119.209136630" watchObservedRunningTime="2026-09-17 03:21:23.744728418 +0000 UTC m=+119.211135948"
Sep 17 03:21:29.198026 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:21:29.197981    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-554ff6b6c4-mqp7s"
Sep 17 03:21:29.198500 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:21:29.198057    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-554ff6b6c4-mqp7s"
Sep 17 03:21:29.202750 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:21:29.202725    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-554ff6b6c4-mqp7s"
Sep 17 03:21:29.744302 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:21:29.744266    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-554ff6b6c4-mqp7s"
Sep 17 03:21:36.771899 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:21:36.771861    2552 generic.go:356] "Generic (PLEG): container finished" podID="0f65c6b8-41ff-4572-9e32-c14e524fa713" containerID="e81ed406f4500d4df6439c75fec73dc2ac6f658ccf0077d8fd9e366aabeb396c" exitCode=0
Sep 17 03:21:36.772319 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:21:36.771937    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-68rbk" event={"ID":"0f65c6b8-41ff-4572-9e32-c14e524fa713","Type":"ContainerDied","Data":"e81ed406f4500d4df6439c75fec73dc2ac6f658ccf0077d8fd9e366aabeb396c"}
Sep 17 03:21:36.772360 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:21:36.772319    2552 scope.go:117] "RemoveContainer" containerID="e81ed406f4500d4df6439c75fec73dc2ac6f658ccf0077d8fd9e366aabeb396c"
Sep 17 03:21:37.548653 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:21:37.548584    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-tbtsz_1a74da51-05c3-4067-b3a5-233734643f6e/serve-healthcheck-canary/0.log"
Sep 17 03:21:37.778643 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:21:37.778582    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-68rbk" event={"ID":"0f65c6b8-41ff-4572-9e32-c14e524fa713","Type":"ContainerStarted","Data":"925681de79cab15e4f125093d403597b9782fea1e8da129eb4a868a7b0c0fbeb"}
Sep 17 03:22:07.518621 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:07.518558    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 03:22:07.519176 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:07.518993    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="9e36be1f-74b0-4dfa-8710-73a632a3f78a" containerName="alertmanager" containerID="cri-o://a4081ea222369421f80cc19c8e85b2e850016e59d0859f1db1f6e40c5c3abd18" gracePeriod=120
Sep 17 03:22:07.519176 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:07.519026    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="9e36be1f-74b0-4dfa-8710-73a632a3f78a" containerName="kube-rbac-proxy-metric" containerID="cri-o://471f5b125b4246c4cdc31c3cb40dab1afda0950c0d5da1984df955eaaccd10ff" gracePeriod=120
Sep 17 03:22:07.519176 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:07.519057    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="9e36be1f-74b0-4dfa-8710-73a632a3f78a" containerName="kube-rbac-proxy-web" containerID="cri-o://7cc2d86c41f99dc89fef8a42cf868eae6e3df93d026af7ab9bb7138b26379ad4" gracePeriod=120
Sep 17 03:22:07.519176 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:07.519059    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="9e36be1f-74b0-4dfa-8710-73a632a3f78a" containerName="prom-label-proxy" containerID="cri-o://676fac31726644ea6bdf5f8ba1ef95baa1e4f695aae93a7baf1d1d3b056ca947" gracePeriod=120
Sep 17 03:22:07.519176 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:07.519090    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="9e36be1f-74b0-4dfa-8710-73a632a3f78a" containerName="config-reloader" containerID="cri-o://5a066a74050dd98fd529a12d1e1424806acfa66aa835f271a87db7c44e7d6f50" gracePeriod=120
Sep 17 03:22:07.519493 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:07.519253    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="9e36be1f-74b0-4dfa-8710-73a632a3f78a" containerName="kube-rbac-proxy" containerID="cri-o://d4807b8c0662b63684961600121ba5e9f819e7eda9526fd56809d90f0e8e0e71" gracePeriod=120
Sep 17 03:22:07.920014 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:07.919934    2552 generic.go:356] "Generic (PLEG): container finished" podID="9e36be1f-74b0-4dfa-8710-73a632a3f78a" containerID="676fac31726644ea6bdf5f8ba1ef95baa1e4f695aae93a7baf1d1d3b056ca947" exitCode=0
Sep 17 03:22:07.920014 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:07.919958    2552 generic.go:356] "Generic (PLEG): container finished" podID="9e36be1f-74b0-4dfa-8710-73a632a3f78a" containerID="d4807b8c0662b63684961600121ba5e9f819e7eda9526fd56809d90f0e8e0e71" exitCode=0
Sep 17 03:22:07.920014 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:07.919965    2552 generic.go:356] "Generic (PLEG): container finished" podID="9e36be1f-74b0-4dfa-8710-73a632a3f78a" containerID="5a066a74050dd98fd529a12d1e1424806acfa66aa835f271a87db7c44e7d6f50" exitCode=0
Sep 17 03:22:07.920014 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:07.919974    2552 generic.go:356] "Generic (PLEG): container finished" podID="9e36be1f-74b0-4dfa-8710-73a632a3f78a" containerID="a4081ea222369421f80cc19c8e85b2e850016e59d0859f1db1f6e40c5c3abd18" exitCode=0
Sep 17 03:22:07.920014 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:07.920005    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9e36be1f-74b0-4dfa-8710-73a632a3f78a","Type":"ContainerDied","Data":"676fac31726644ea6bdf5f8ba1ef95baa1e4f695aae93a7baf1d1d3b056ca947"}
Sep 17 03:22:07.920268 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:07.920067    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9e36be1f-74b0-4dfa-8710-73a632a3f78a","Type":"ContainerDied","Data":"d4807b8c0662b63684961600121ba5e9f819e7eda9526fd56809d90f0e8e0e71"}
Sep 17 03:22:07.920268 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:07.920093    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9e36be1f-74b0-4dfa-8710-73a632a3f78a","Type":"ContainerDied","Data":"5a066a74050dd98fd529a12d1e1424806acfa66aa835f271a87db7c44e7d6f50"}
Sep 17 03:22:07.920268 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:07.920118    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9e36be1f-74b0-4dfa-8710-73a632a3f78a","Type":"ContainerDied","Data":"a4081ea222369421f80cc19c8e85b2e850016e59d0859f1db1f6e40c5c3abd18"}
Sep 17 03:22:08.927927 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:08.927834    2552 generic.go:356] "Generic (PLEG): container finished" podID="9e36be1f-74b0-4dfa-8710-73a632a3f78a" containerID="471f5b125b4246c4cdc31c3cb40dab1afda0950c0d5da1984df955eaaccd10ff" exitCode=0
Sep 17 03:22:08.927927 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:08.927903    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9e36be1f-74b0-4dfa-8710-73a632a3f78a","Type":"ContainerDied","Data":"471f5b125b4246c4cdc31c3cb40dab1afda0950c0d5da1984df955eaaccd10ff"}
Sep 17 03:22:09.381529 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:09.381500    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 17 03:22:09.390672 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:09.390642    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9e36be1f-74b0-4dfa-8710-73a632a3f78a-tls-assets\") pod \"9e36be1f-74b0-4dfa-8710-73a632a3f78a\" (UID: \"9e36be1f-74b0-4dfa-8710-73a632a3f78a\") "
Sep 17 03:22:09.390863 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:09.390685    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9e36be1f-74b0-4dfa-8710-73a632a3f78a-alertmanager-main-db\") pod \"9e36be1f-74b0-4dfa-8710-73a632a3f78a\" (UID: \"9e36be1f-74b0-4dfa-8710-73a632a3f78a\") "
Sep 17 03:22:09.390863 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:09.390714    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9e36be1f-74b0-4dfa-8710-73a632a3f78a-config-out\") pod \"9e36be1f-74b0-4dfa-8710-73a632a3f78a\" (UID: \"9e36be1f-74b0-4dfa-8710-73a632a3f78a\") "
Sep 17 03:22:09.390863 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:09.390736    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9e36be1f-74b0-4dfa-8710-73a632a3f78a-cluster-tls-config\") pod \"9e36be1f-74b0-4dfa-8710-73a632a3f78a\" (UID: \"9e36be1f-74b0-4dfa-8710-73a632a3f78a\") "
Sep 17 03:22:09.390863 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:09.390772    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9e36be1f-74b0-4dfa-8710-73a632a3f78a-web-config\") pod \"9e36be1f-74b0-4dfa-8710-73a632a3f78a\" (UID: \"9e36be1f-74b0-4dfa-8710-73a632a3f78a\") "
Sep 17 03:22:09.390863 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:09.390797    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9e36be1f-74b0-4dfa-8710-73a632a3f78a-alertmanager-trusted-ca-bundle\") pod \"9e36be1f-74b0-4dfa-8710-73a632a3f78a\" (UID: \"9e36be1f-74b0-4dfa-8710-73a632a3f78a\") "
Sep 17 03:22:09.390863 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:09.390822    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9e36be1f-74b0-4dfa-8710-73a632a3f78a-config-volume\") pod \"9e36be1f-74b0-4dfa-8710-73a632a3f78a\" (UID: \"9e36be1f-74b0-4dfa-8710-73a632a3f78a\") "
Sep 17 03:22:09.390863 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:09.390859    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jfdhm\" (UniqueName: \"kubernetes.io/projected/9e36be1f-74b0-4dfa-8710-73a632a3f78a-kube-api-access-jfdhm\") pod \"9e36be1f-74b0-4dfa-8710-73a632a3f78a\" (UID: \"9e36be1f-74b0-4dfa-8710-73a632a3f78a\") "
Sep 17 03:22:09.391199 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:09.390912    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9e36be1f-74b0-4dfa-8710-73a632a3f78a-secret-alertmanager-main-tls\") pod \"9e36be1f-74b0-4dfa-8710-73a632a3f78a\" (UID: \"9e36be1f-74b0-4dfa-8710-73a632a3f78a\") "
Sep 17 03:22:09.391199 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:09.390941    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9e36be1f-74b0-4dfa-8710-73a632a3f78a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"9e36be1f-74b0-4dfa-8710-73a632a3f78a\" (UID: \"9e36be1f-74b0-4dfa-8710-73a632a3f78a\") "
Sep 17 03:22:09.391199 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:09.390969    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9e36be1f-74b0-4dfa-8710-73a632a3f78a-secret-alertmanager-kube-rbac-proxy\") pod \"9e36be1f-74b0-4dfa-8710-73a632a3f78a\" (UID: \"9e36be1f-74b0-4dfa-8710-73a632a3f78a\") "
Sep 17 03:22:09.391199 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:09.391014    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9e36be1f-74b0-4dfa-8710-73a632a3f78a-metrics-client-ca\") pod \"9e36be1f-74b0-4dfa-8710-73a632a3f78a\" (UID: \"9e36be1f-74b0-4dfa-8710-73a632a3f78a\") "
Sep 17 03:22:09.391199 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:09.391080    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9e36be1f-74b0-4dfa-8710-73a632a3f78a-secret-alertmanager-kube-rbac-proxy-web\") pod \"9e36be1f-74b0-4dfa-8710-73a632a3f78a\" (UID: \"9e36be1f-74b0-4dfa-8710-73a632a3f78a\") "
Sep 17 03:22:09.391435 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:09.391205    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e36be1f-74b0-4dfa-8710-73a632a3f78a-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "9e36be1f-74b0-4dfa-8710-73a632a3f78a" (UID: "9e36be1f-74b0-4dfa-8710-73a632a3f78a"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 03:22:09.391435 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:09.391350    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e36be1f-74b0-4dfa-8710-73a632a3f78a-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "9e36be1f-74b0-4dfa-8710-73a632a3f78a" (UID: "9e36be1f-74b0-4dfa-8710-73a632a3f78a"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:22:09.391435 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:09.391360    2552 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9e36be1f-74b0-4dfa-8710-73a632a3f78a-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:22:09.392132 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:09.392101    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e36be1f-74b0-4dfa-8710-73a632a3f78a-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "9e36be1f-74b0-4dfa-8710-73a632a3f78a" (UID: "9e36be1f-74b0-4dfa-8710-73a632a3f78a"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 03:22:09.394960 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:09.393985    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e36be1f-74b0-4dfa-8710-73a632a3f78a-config-out" (OuterVolumeSpecName: "config-out") pod "9e36be1f-74b0-4dfa-8710-73a632a3f78a" (UID: "9e36be1f-74b0-4dfa-8710-73a632a3f78a"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:22:09.394960 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:09.394280    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e36be1f-74b0-4dfa-8710-73a632a3f78a-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "9e36be1f-74b0-4dfa-8710-73a632a3f78a" (UID: "9e36be1f-74b0-4dfa-8710-73a632a3f78a"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 03:22:09.394960 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:09.394296    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e36be1f-74b0-4dfa-8710-73a632a3f78a-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "9e36be1f-74b0-4dfa-8710-73a632a3f78a" (UID: "9e36be1f-74b0-4dfa-8710-73a632a3f78a"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 03:22:09.394960 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:09.394491    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e36be1f-74b0-4dfa-8710-73a632a3f78a-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "9e36be1f-74b0-4dfa-8710-73a632a3f78a" (UID: "9e36be1f-74b0-4dfa-8710-73a632a3f78a"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 03:22:09.394960 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:09.394650    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e36be1f-74b0-4dfa-8710-73a632a3f78a-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "9e36be1f-74b0-4dfa-8710-73a632a3f78a" (UID: "9e36be1f-74b0-4dfa-8710-73a632a3f78a"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 03:22:09.395328 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:09.395303    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e36be1f-74b0-4dfa-8710-73a632a3f78a-config-volume" (OuterVolumeSpecName: "config-volume") pod "9e36be1f-74b0-4dfa-8710-73a632a3f78a" (UID: "9e36be1f-74b0-4dfa-8710-73a632a3f78a"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 03:22:09.395744 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:09.395719    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e36be1f-74b0-4dfa-8710-73a632a3f78a-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "9e36be1f-74b0-4dfa-8710-73a632a3f78a" (UID: "9e36be1f-74b0-4dfa-8710-73a632a3f78a"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 03:22:09.396162 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:09.396140    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e36be1f-74b0-4dfa-8710-73a632a3f78a-kube-api-access-jfdhm" (OuterVolumeSpecName: "kube-api-access-jfdhm") pod "9e36be1f-74b0-4dfa-8710-73a632a3f78a" (UID: "9e36be1f-74b0-4dfa-8710-73a632a3f78a"). InnerVolumeSpecName "kube-api-access-jfdhm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 03:22:09.399383 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:09.399354    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e36be1f-74b0-4dfa-8710-73a632a3f78a-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "9e36be1f-74b0-4dfa-8710-73a632a3f78a" (UID: "9e36be1f-74b0-4dfa-8710-73a632a3f78a"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 03:22:09.407592 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:09.407550    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e36be1f-74b0-4dfa-8710-73a632a3f78a-web-config" (OuterVolumeSpecName: "web-config") pod "9e36be1f-74b0-4dfa-8710-73a632a3f78a" (UID: "9e36be1f-74b0-4dfa-8710-73a632a3f78a"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 03:22:09.492558 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:09.492510    2552 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9e36be1f-74b0-4dfa-8710-73a632a3f78a-web-config\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:22:09.492558 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:09.492547    2552 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9e36be1f-74b0-4dfa-8710-73a632a3f78a-config-volume\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:22:09.492558 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:09.492557    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jfdhm\" (UniqueName: \"kubernetes.io/projected/9e36be1f-74b0-4dfa-8710-73a632a3f78a-kube-api-access-jfdhm\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:22:09.492558 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:09.492566    2552 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9e36be1f-74b0-4dfa-8710-73a632a3f78a-secret-alertmanager-main-tls\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:22:09.492852 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:09.492576    2552 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9e36be1f-74b0-4dfa-8710-73a632a3f78a-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:22:09.492852 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:09.492585    2552 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9e36be1f-74b0-4dfa-8710-73a632a3f78a-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:22:09.492852 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:09.492618    2552 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9e36be1f-74b0-4dfa-8710-73a632a3f78a-metrics-client-ca\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:22:09.492852 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:09.492630    2552 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9e36be1f-74b0-4dfa-8710-73a632a3f78a-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:22:09.492852 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:09.492638    2552 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9e36be1f-74b0-4dfa-8710-73a632a3f78a-tls-assets\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:22:09.492852 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:09.492646    2552 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9e36be1f-74b0-4dfa-8710-73a632a3f78a-alertmanager-main-db\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:22:09.492852 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:09.492654    2552 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9e36be1f-74b0-4dfa-8710-73a632a3f78a-config-out\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:22:09.492852 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:09.492663    2552 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9e36be1f-74b0-4dfa-8710-73a632a3f78a-cluster-tls-config\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:22:09.936011 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:09.935916    2552 generic.go:356] "Generic (PLEG): container finished" podID="9e36be1f-74b0-4dfa-8710-73a632a3f78a" containerID="7cc2d86c41f99dc89fef8a42cf868eae6e3df93d026af7ab9bb7138b26379ad4" exitCode=0
Sep 17 03:22:09.936011 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:09.935990    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9e36be1f-74b0-4dfa-8710-73a632a3f78a","Type":"ContainerDied","Data":"7cc2d86c41f99dc89fef8a42cf868eae6e3df93d026af7ab9bb7138b26379ad4"}
Sep 17 03:22:09.936011 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:09.936002    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 17 03:22:09.936693 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:09.936060    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9e36be1f-74b0-4dfa-8710-73a632a3f78a","Type":"ContainerDied","Data":"9e6459a7f80c1f1c18bfb3ec65c72020672b1141d813afb99d335e8a410a32f3"}
Sep 17 03:22:09.936693 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:09.936128    2552 scope.go:117] "RemoveContainer" containerID="676fac31726644ea6bdf5f8ba1ef95baa1e4f695aae93a7baf1d1d3b056ca947"
Sep 17 03:22:09.945226 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:09.945194    2552 scope.go:117] "RemoveContainer" containerID="471f5b125b4246c4cdc31c3cb40dab1afda0950c0d5da1984df955eaaccd10ff"
Sep 17 03:22:09.952885 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:09.952863    2552 scope.go:117] "RemoveContainer" containerID="d4807b8c0662b63684961600121ba5e9f819e7eda9526fd56809d90f0e8e0e71"
Sep 17 03:22:09.959682 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:09.959655    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 03:22:09.961209 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:09.961175    2552 scope.go:117] "RemoveContainer" containerID="7cc2d86c41f99dc89fef8a42cf868eae6e3df93d026af7ab9bb7138b26379ad4"
Sep 17 03:22:09.964385 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:09.964357    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 03:22:09.969572 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:09.969554    2552 scope.go:117] "RemoveContainer" containerID="5a066a74050dd98fd529a12d1e1424806acfa66aa835f271a87db7c44e7d6f50"
Sep 17 03:22:09.977392 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:09.977369    2552 scope.go:117] "RemoveContainer" containerID="a4081ea222369421f80cc19c8e85b2e850016e59d0859f1db1f6e40c5c3abd18"
Sep 17 03:22:09.985572 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:09.985511    2552 scope.go:117] "RemoveContainer" containerID="6ab8a24c660d82178d6e63bf0d189f024056e2727580b28f32cc3ad78bdd16b7"
Sep 17 03:22:09.987491 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:09.987468    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 03:22:09.988688 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:09.988415    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e36be1f-74b0-4dfa-8710-73a632a3f78a" containerName="kube-rbac-proxy-web"
Sep 17 03:22:09.988688 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:09.988442    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e36be1f-74b0-4dfa-8710-73a632a3f78a" containerName="kube-rbac-proxy-web"
Sep 17 03:22:09.988688 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:09.988466    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e36be1f-74b0-4dfa-8710-73a632a3f78a" containerName="alertmanager"
Sep 17 03:22:09.988688 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:09.988476    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e36be1f-74b0-4dfa-8710-73a632a3f78a" containerName="alertmanager"
Sep 17 03:22:09.988688 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:09.988487    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e36be1f-74b0-4dfa-8710-73a632a3f78a" containerName="init-config-reloader"
Sep 17 03:22:09.988688 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:09.988501    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e36be1f-74b0-4dfa-8710-73a632a3f78a" containerName="init-config-reloader"
Sep 17 03:22:09.988688 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:09.988512    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e36be1f-74b0-4dfa-8710-73a632a3f78a" containerName="kube-rbac-proxy-metric"
Sep 17 03:22:09.988688 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:09.988520    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e36be1f-74b0-4dfa-8710-73a632a3f78a" containerName="kube-rbac-proxy-metric"
Sep 17 03:22:09.988688 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:09.988552    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e36be1f-74b0-4dfa-8710-73a632a3f78a" containerName="config-reloader"
Sep 17 03:22:09.988688 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:09.988560    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e36be1f-74b0-4dfa-8710-73a632a3f78a" containerName="config-reloader"
Sep 17 03:22:09.988688 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:09.988576    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e36be1f-74b0-4dfa-8710-73a632a3f78a" containerName="kube-rbac-proxy"
Sep 17 03:22:09.988688 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:09.988584    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e36be1f-74b0-4dfa-8710-73a632a3f78a" containerName="kube-rbac-proxy"
Sep 17 03:22:09.988688 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:09.988624    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e36be1f-74b0-4dfa-8710-73a632a3f78a" containerName="prom-label-proxy"
Sep 17 03:22:09.988688 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:09.988633    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e36be1f-74b0-4dfa-8710-73a632a3f78a" containerName="prom-label-proxy"
Sep 17 03:22:09.989404 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:09.988773    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e36be1f-74b0-4dfa-8710-73a632a3f78a" containerName="kube-rbac-proxy"
Sep 17 03:22:09.989404 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:09.988785    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e36be1f-74b0-4dfa-8710-73a632a3f78a" containerName="alertmanager"
Sep 17 03:22:09.989404 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:09.988801    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e36be1f-74b0-4dfa-8710-73a632a3f78a" containerName="config-reloader"
Sep 17 03:22:09.989404 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:09.988809    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e36be1f-74b0-4dfa-8710-73a632a3f78a" containerName="kube-rbac-proxy-web"
Sep 17 03:22:09.989404 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:09.988828    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e36be1f-74b0-4dfa-8710-73a632a3f78a" containerName="kube-rbac-proxy-metric"
Sep 17 03:22:09.989404 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:09.988837    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e36be1f-74b0-4dfa-8710-73a632a3f78a" containerName="prom-label-proxy"
Sep 17 03:22:09.996213 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:09.996179    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 17 03:22:09.998552 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:09.998533    2552 scope.go:117] "RemoveContainer" containerID="676fac31726644ea6bdf5f8ba1ef95baa1e4f695aae93a7baf1d1d3b056ca947"
Sep 17 03:22:09.998940 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:22:09.998916    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"676fac31726644ea6bdf5f8ba1ef95baa1e4f695aae93a7baf1d1d3b056ca947\": container with ID starting with 676fac31726644ea6bdf5f8ba1ef95baa1e4f695aae93a7baf1d1d3b056ca947 not found: ID does not exist" containerID="676fac31726644ea6bdf5f8ba1ef95baa1e4f695aae93a7baf1d1d3b056ca947"
Sep 17 03:22:09.999124 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:09.998950    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"676fac31726644ea6bdf5f8ba1ef95baa1e4f695aae93a7baf1d1d3b056ca947"} err="failed to get container status \"676fac31726644ea6bdf5f8ba1ef95baa1e4f695aae93a7baf1d1d3b056ca947\": rpc error: code = NotFound desc = could not find container \"676fac31726644ea6bdf5f8ba1ef95baa1e4f695aae93a7baf1d1d3b056ca947\": container with ID starting with 676fac31726644ea6bdf5f8ba1ef95baa1e4f695aae93a7baf1d1d3b056ca947 not found: ID does not exist"
Sep 17 03:22:09.999124 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:09.998976    2552 scope.go:117] "RemoveContainer" containerID="471f5b125b4246c4cdc31c3cb40dab1afda0950c0d5da1984df955eaaccd10ff"
Sep 17 03:22:09.999124 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:09.998998    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 17 03:22:09.999124 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:09.999016    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 17 03:22:09.999124 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:09.999000    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-szz6w\""
Sep 17 03:22:09.999124 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:09.999047    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 17 03:22:09.999490 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:09.999127    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 17 03:22:09.999490 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:09.999180    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 17 03:22:09.999490 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:09.999402    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 17 03:22:09.999490 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:09.999443    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 17 03:22:09.999490 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:22:09.999446    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"471f5b125b4246c4cdc31c3cb40dab1afda0950c0d5da1984df955eaaccd10ff\": container with ID starting with 471f5b125b4246c4cdc31c3cb40dab1afda0950c0d5da1984df955eaaccd10ff not found: ID does not exist" containerID="471f5b125b4246c4cdc31c3cb40dab1afda0950c0d5da1984df955eaaccd10ff"
Sep 17 03:22:09.999490 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:09.999475    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 17 03:22:09.999490 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:09.999474    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"471f5b125b4246c4cdc31c3cb40dab1afda0950c0d5da1984df955eaaccd10ff"} err="failed to get container status \"471f5b125b4246c4cdc31c3cb40dab1afda0950c0d5da1984df955eaaccd10ff\": rpc error: code = NotFound desc = could not find container \"471f5b125b4246c4cdc31c3cb40dab1afda0950c0d5da1984df955eaaccd10ff\": container with ID starting with 471f5b125b4246c4cdc31c3cb40dab1afda0950c0d5da1984df955eaaccd10ff not found: ID does not exist"
Sep 17 03:22:09.999819 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:09.999496    2552 scope.go:117] "RemoveContainer" containerID="d4807b8c0662b63684961600121ba5e9f819e7eda9526fd56809d90f0e8e0e71"
Sep 17 03:22:09.999921 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:22:09.999904    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4807b8c0662b63684961600121ba5e9f819e7eda9526fd56809d90f0e8e0e71\": container with ID starting with d4807b8c0662b63684961600121ba5e9f819e7eda9526fd56809d90f0e8e0e71 not found: ID does not exist" containerID="d4807b8c0662b63684961600121ba5e9f819e7eda9526fd56809d90f0e8e0e71"
Sep 17 03:22:09.999971 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:09.999931    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4807b8c0662b63684961600121ba5e9f819e7eda9526fd56809d90f0e8e0e71"} err="failed to get container status \"d4807b8c0662b63684961600121ba5e9f819e7eda9526fd56809d90f0e8e0e71\": rpc error: code = NotFound desc = could not find container \"d4807b8c0662b63684961600121ba5e9f819e7eda9526fd56809d90f0e8e0e71\": container with ID starting with d4807b8c0662b63684961600121ba5e9f819e7eda9526fd56809d90f0e8e0e71 not found: ID does not exist"
Sep 17 03:22:09.999971 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:09.999947    2552 scope.go:117] "RemoveContainer" containerID="7cc2d86c41f99dc89fef8a42cf868eae6e3df93d026af7ab9bb7138b26379ad4"
Sep 17 03:22:10.000259 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:22:10.000238    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7cc2d86c41f99dc89fef8a42cf868eae6e3df93d026af7ab9bb7138b26379ad4\": container with ID starting with 7cc2d86c41f99dc89fef8a42cf868eae6e3df93d026af7ab9bb7138b26379ad4 not found: ID does not exist" containerID="7cc2d86c41f99dc89fef8a42cf868eae6e3df93d026af7ab9bb7138b26379ad4"
Sep 17 03:22:10.000336 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:10.000268    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7cc2d86c41f99dc89fef8a42cf868eae6e3df93d026af7ab9bb7138b26379ad4"} err="failed to get container status \"7cc2d86c41f99dc89fef8a42cf868eae6e3df93d026af7ab9bb7138b26379ad4\": rpc error: code = NotFound desc = could not find container \"7cc2d86c41f99dc89fef8a42cf868eae6e3df93d026af7ab9bb7138b26379ad4\": container with ID starting with 7cc2d86c41f99dc89fef8a42cf868eae6e3df93d026af7ab9bb7138b26379ad4 not found: ID does not exist"
Sep 17 03:22:10.000400 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:10.000343    2552 scope.go:117] "RemoveContainer" containerID="5a066a74050dd98fd529a12d1e1424806acfa66aa835f271a87db7c44e7d6f50"
Sep 17 03:22:10.000746 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:22:10.000719    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a066a74050dd98fd529a12d1e1424806acfa66aa835f271a87db7c44e7d6f50\": container with ID starting with 5a066a74050dd98fd529a12d1e1424806acfa66aa835f271a87db7c44e7d6f50 not found: ID does not exist" containerID="5a066a74050dd98fd529a12d1e1424806acfa66aa835f271a87db7c44e7d6f50"
Sep 17 03:22:10.000852 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:10.000749    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a066a74050dd98fd529a12d1e1424806acfa66aa835f271a87db7c44e7d6f50"} err="failed to get container status \"5a066a74050dd98fd529a12d1e1424806acfa66aa835f271a87db7c44e7d6f50\": rpc error: code = NotFound desc = could not find container \"5a066a74050dd98fd529a12d1e1424806acfa66aa835f271a87db7c44e7d6f50\": container with ID starting with 5a066a74050dd98fd529a12d1e1424806acfa66aa835f271a87db7c44e7d6f50 not found: ID does not exist"
Sep 17 03:22:10.000852 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:10.000765    2552 scope.go:117] "RemoveContainer" containerID="a4081ea222369421f80cc19c8e85b2e850016e59d0859f1db1f6e40c5c3abd18"
Sep 17 03:22:10.001034 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:22:10.001015    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4081ea222369421f80cc19c8e85b2e850016e59d0859f1db1f6e40c5c3abd18\": container with ID starting with a4081ea222369421f80cc19c8e85b2e850016e59d0859f1db1f6e40c5c3abd18 not found: ID does not exist" containerID="a4081ea222369421f80cc19c8e85b2e850016e59d0859f1db1f6e40c5c3abd18"
Sep 17 03:22:10.001214 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:10.001036    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4081ea222369421f80cc19c8e85b2e850016e59d0859f1db1f6e40c5c3abd18"} err="failed to get container status \"a4081ea222369421f80cc19c8e85b2e850016e59d0859f1db1f6e40c5c3abd18\": rpc error: code = NotFound desc = could not find container \"a4081ea222369421f80cc19c8e85b2e850016e59d0859f1db1f6e40c5c3abd18\": container with ID starting with a4081ea222369421f80cc19c8e85b2e850016e59d0859f1db1f6e40c5c3abd18 not found: ID does not exist"
Sep 17 03:22:10.001214 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:10.001055    2552 scope.go:117] "RemoveContainer" containerID="6ab8a24c660d82178d6e63bf0d189f024056e2727580b28f32cc3ad78bdd16b7"
Sep 17 03:22:10.001636 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:22:10.001366    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ab8a24c660d82178d6e63bf0d189f024056e2727580b28f32cc3ad78bdd16b7\": container with ID starting with 6ab8a24c660d82178d6e63bf0d189f024056e2727580b28f32cc3ad78bdd16b7 not found: ID does not exist" containerID="6ab8a24c660d82178d6e63bf0d189f024056e2727580b28f32cc3ad78bdd16b7"
Sep 17 03:22:10.001636 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:10.001415    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ab8a24c660d82178d6e63bf0d189f024056e2727580b28f32cc3ad78bdd16b7"} err="failed to get container status \"6ab8a24c660d82178d6e63bf0d189f024056e2727580b28f32cc3ad78bdd16b7\": rpc error: code = NotFound desc = could not find container \"6ab8a24c660d82178d6e63bf0d189f024056e2727580b28f32cc3ad78bdd16b7\": container with ID starting with 6ab8a24c660d82178d6e63bf0d189f024056e2727580b28f32cc3ad78bdd16b7 not found: ID does not exist"
Sep 17 03:22:10.004637 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:10.004613    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 17 03:22:10.004799 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:10.004664    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 03:22:10.097040 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:10.097000    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/2e583985-a9da-4679-9bd7-ae059f1ce9e9-config-volume\") pod \"alertmanager-main-0\" (UID: \"2e583985-a9da-4679-9bd7-ae059f1ce9e9\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 03:22:10.097040 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:10.097036    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2e583985-a9da-4679-9bd7-ae059f1ce9e9-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"2e583985-a9da-4679-9bd7-ae059f1ce9e9\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 03:22:10.097278 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:10.097055    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/2e583985-a9da-4679-9bd7-ae059f1ce9e9-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"2e583985-a9da-4679-9bd7-ae059f1ce9e9\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 03:22:10.097278 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:10.097077    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2wff\" (UniqueName: \"kubernetes.io/projected/2e583985-a9da-4679-9bd7-ae059f1ce9e9-kube-api-access-k2wff\") pod \"alertmanager-main-0\" (UID: \"2e583985-a9da-4679-9bd7-ae059f1ce9e9\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 03:22:10.097278 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:10.097129    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2e583985-a9da-4679-9bd7-ae059f1ce9e9-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"2e583985-a9da-4679-9bd7-ae059f1ce9e9\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 03:22:10.097278 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:10.097193    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/2e583985-a9da-4679-9bd7-ae059f1ce9e9-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"2e583985-a9da-4679-9bd7-ae059f1ce9e9\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 03:22:10.097278 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:10.097217    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2e583985-a9da-4679-9bd7-ae059f1ce9e9-tls-assets\") pod \"alertmanager-main-0\" (UID: \"2e583985-a9da-4679-9bd7-ae059f1ce9e9\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 03:22:10.097278 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:10.097232    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2e583985-a9da-4679-9bd7-ae059f1ce9e9-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"2e583985-a9da-4679-9bd7-ae059f1ce9e9\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 03:22:10.097278 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:10.097258    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/2e583985-a9da-4679-9bd7-ae059f1ce9e9-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"2e583985-a9da-4679-9bd7-ae059f1ce9e9\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 03:22:10.097557 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:10.097309    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2e583985-a9da-4679-9bd7-ae059f1ce9e9-config-out\") pod \"alertmanager-main-0\" (UID: \"2e583985-a9da-4679-9bd7-ae059f1ce9e9\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 03:22:10.097557 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:10.097350    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/2e583985-a9da-4679-9bd7-ae059f1ce9e9-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"2e583985-a9da-4679-9bd7-ae059f1ce9e9\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 03:22:10.097557 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:10.097388    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2e583985-a9da-4679-9bd7-ae059f1ce9e9-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"2e583985-a9da-4679-9bd7-ae059f1ce9e9\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 03:22:10.097557 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:10.097427    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2e583985-a9da-4679-9bd7-ae059f1ce9e9-web-config\") pod \"alertmanager-main-0\" (UID: \"2e583985-a9da-4679-9bd7-ae059f1ce9e9\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 03:22:10.197938 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:10.197850    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2e583985-a9da-4679-9bd7-ae059f1ce9e9-config-out\") pod \"alertmanager-main-0\" (UID: \"2e583985-a9da-4679-9bd7-ae059f1ce9e9\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 03:22:10.197938 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:10.197898    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/2e583985-a9da-4679-9bd7-ae059f1ce9e9-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"2e583985-a9da-4679-9bd7-ae059f1ce9e9\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 03:22:10.197938 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:10.197929    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2e583985-a9da-4679-9bd7-ae059f1ce9e9-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"2e583985-a9da-4679-9bd7-ae059f1ce9e9\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 03:22:10.198169 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:10.197950    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2e583985-a9da-4679-9bd7-ae059f1ce9e9-web-config\") pod \"alertmanager-main-0\" (UID: \"2e583985-a9da-4679-9bd7-ae059f1ce9e9\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 03:22:10.198169 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:10.197980    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/2e583985-a9da-4679-9bd7-ae059f1ce9e9-config-volume\") pod \"alertmanager-main-0\" (UID: \"2e583985-a9da-4679-9bd7-ae059f1ce9e9\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 03:22:10.198169 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:10.197998    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2e583985-a9da-4679-9bd7-ae059f1ce9e9-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"2e583985-a9da-4679-9bd7-ae059f1ce9e9\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 03:22:10.198169 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:10.198015    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/2e583985-a9da-4679-9bd7-ae059f1ce9e9-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"2e583985-a9da-4679-9bd7-ae059f1ce9e9\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 03:22:10.198169 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:10.198040    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2wff\" (UniqueName: \"kubernetes.io/projected/2e583985-a9da-4679-9bd7-ae059f1ce9e9-kube-api-access-k2wff\") pod \"alertmanager-main-0\" (UID: \"2e583985-a9da-4679-9bd7-ae059f1ce9e9\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 03:22:10.198169 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:10.198071    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2e583985-a9da-4679-9bd7-ae059f1ce9e9-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"2e583985-a9da-4679-9bd7-ae059f1ce9e9\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 03:22:10.198169 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:10.198100    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/2e583985-a9da-4679-9bd7-ae059f1ce9e9-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"2e583985-a9da-4679-9bd7-ae059f1ce9e9\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 03:22:10.198169 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:10.198125    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2e583985-a9da-4679-9bd7-ae059f1ce9e9-tls-assets\") pod \"alertmanager-main-0\" (UID: \"2e583985-a9da-4679-9bd7-ae059f1ce9e9\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 03:22:10.198169 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:10.198150    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2e583985-a9da-4679-9bd7-ae059f1ce9e9-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"2e583985-a9da-4679-9bd7-ae059f1ce9e9\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 03:22:10.198636 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:10.198179    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/2e583985-a9da-4679-9bd7-ae059f1ce9e9-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"2e583985-a9da-4679-9bd7-ae059f1ce9e9\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 03:22:10.198636 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:10.198453    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/2e583985-a9da-4679-9bd7-ae059f1ce9e9-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"2e583985-a9da-4679-9bd7-ae059f1ce9e9\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 03:22:10.199020 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:10.198992    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2e583985-a9da-4679-9bd7-ae059f1ce9e9-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"2e583985-a9da-4679-9bd7-ae059f1ce9e9\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 03:22:10.201210 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:10.201175    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/2e583985-a9da-4679-9bd7-ae059f1ce9e9-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"2e583985-a9da-4679-9bd7-ae059f1ce9e9\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 03:22:10.201210 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:10.201202    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2e583985-a9da-4679-9bd7-ae059f1ce9e9-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"2e583985-a9da-4679-9bd7-ae059f1ce9e9\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 03:22:10.201385 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:10.201235    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2e583985-a9da-4679-9bd7-ae059f1ce9e9-web-config\") pod \"alertmanager-main-0\" (UID: \"2e583985-a9da-4679-9bd7-ae059f1ce9e9\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 03:22:10.201585 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:10.201565    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2e583985-a9da-4679-9bd7-ae059f1ce9e9-config-out\") pod \"alertmanager-main-0\" (UID: \"2e583985-a9da-4679-9bd7-ae059f1ce9e9\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 03:22:10.201810 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:10.201788    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2e583985-a9da-4679-9bd7-ae059f1ce9e9-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"2e583985-a9da-4679-9bd7-ae059f1ce9e9\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 03:22:10.201897 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:10.201794    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2e583985-a9da-4679-9bd7-ae059f1ce9e9-tls-assets\") pod \"alertmanager-main-0\" (UID: \"2e583985-a9da-4679-9bd7-ae059f1ce9e9\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 03:22:10.201897 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:10.201837    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/2e583985-a9da-4679-9bd7-ae059f1ce9e9-config-volume\") pod \"alertmanager-main-0\" (UID: \"2e583985-a9da-4679-9bd7-ae059f1ce9e9\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 03:22:10.202275 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:10.202254    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/2e583985-a9da-4679-9bd7-ae059f1ce9e9-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"2e583985-a9da-4679-9bd7-ae059f1ce9e9\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 03:22:10.202356 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:10.202273    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2e583985-a9da-4679-9bd7-ae059f1ce9e9-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"2e583985-a9da-4679-9bd7-ae059f1ce9e9\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 03:22:10.202830 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:10.202811    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/2e583985-a9da-4679-9bd7-ae059f1ce9e9-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"2e583985-a9da-4679-9bd7-ae059f1ce9e9\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 03:22:10.211239 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:10.211211    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2wff\" (UniqueName: \"kubernetes.io/projected/2e583985-a9da-4679-9bd7-ae059f1ce9e9-kube-api-access-k2wff\") pod \"alertmanager-main-0\" (UID: \"2e583985-a9da-4679-9bd7-ae059f1ce9e9\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 03:22:10.308097 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:10.308061    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 17 03:22:10.590156 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:10.590127    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 03:22:10.592784 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:22:10.592747    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e583985_a9da_4679_9bd7_ae059f1ce9e9.slice/crio-0a25ff85fe28efee4ac06c3a16d1cf5a7662fba8b89b2b33a1ffae0a6522aed4 WatchSource:0}: Error finding container 0a25ff85fe28efee4ac06c3a16d1cf5a7662fba8b89b2b33a1ffae0a6522aed4: Status 404 returned error can't find the container with id 0a25ff85fe28efee4ac06c3a16d1cf5a7662fba8b89b2b33a1ffae0a6522aed4
Sep 17 03:22:10.942832 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:10.942739    2552 generic.go:356] "Generic (PLEG): container finished" podID="2e583985-a9da-4679-9bd7-ae059f1ce9e9" containerID="9255d70f06c55da67a84f6c33833a39e15cf39829302d3a37e9b9049694c3b46" exitCode=0
Sep 17 03:22:10.942832 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:10.942789    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2e583985-a9da-4679-9bd7-ae059f1ce9e9","Type":"ContainerDied","Data":"9255d70f06c55da67a84f6c33833a39e15cf39829302d3a37e9b9049694c3b46"}
Sep 17 03:22:10.943223 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:10.942841    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2e583985-a9da-4679-9bd7-ae059f1ce9e9","Type":"ContainerStarted","Data":"0a25ff85fe28efee4ac06c3a16d1cf5a7662fba8b89b2b33a1ffae0a6522aed4"}
Sep 17 03:22:10.984637 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:10.984590    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9e36be1f-74b0-4dfa-8710-73a632a3f78a" path="/var/lib/kubelet/pods/9e36be1f-74b0-4dfa-8710-73a632a3f78a/volumes"
Sep 17 03:22:11.950790 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:11.950757    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2e583985-a9da-4679-9bd7-ae059f1ce9e9","Type":"ContainerStarted","Data":"5eb58ed45ff97fc68c62662cac85843d0b88a0ec4f4dc8c474f84319ef1facd0"}
Sep 17 03:22:11.951201 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:11.950821    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2e583985-a9da-4679-9bd7-ae059f1ce9e9","Type":"ContainerStarted","Data":"ac0b98dff80f4516af2c79355dd030dd510141ad8ffef87f489371188d768f7f"}
Sep 17 03:22:11.951201 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:11.950851    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2e583985-a9da-4679-9bd7-ae059f1ce9e9","Type":"ContainerStarted","Data":"0bddc9d76ef8d5558e2d20c488c0344075df3dea1724b650d9ea4868d893360f"}
Sep 17 03:22:11.951201 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:11.950876    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2e583985-a9da-4679-9bd7-ae059f1ce9e9","Type":"ContainerStarted","Data":"46e270e4ea01658e630a4071364bcd22cc7b232d18bb067c4163be748c79a341"}
Sep 17 03:22:11.951201 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:11.950899    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2e583985-a9da-4679-9bd7-ae059f1ce9e9","Type":"ContainerStarted","Data":"879e58c8f2d24dd8e8dbda9907ba33c4150fd296e0435be7a12533428c999b71"}
Sep 17 03:22:11.951201 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:11.950925    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2e583985-a9da-4679-9bd7-ae059f1ce9e9","Type":"ContainerStarted","Data":"1b9eb8304332ab1ec6a1170d7a0a5339bb1baea085fdebcf6e8fb8ee25f605c5"}
Sep 17 03:22:11.976960 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:11.976903    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.959681979 podStartE2EDuration="2.976884697s" totalImagesPullingTime="0s" totalInitContainerRuntime="17.202718ms" isStatefulPod=false podCreationTimestamp="2026-09-17 03:22:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:22:11.975036178 +0000 UTC m=+167.441443704" watchObservedRunningTime="2026-09-17 03:22:11.976884697 +0000 UTC m=+167.443292223"
Sep 17 03:22:15.720009 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:15.719969    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-86d57d9c6c-xnrf4"]
Sep 17 03:22:15.723431 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:15.723405    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-86d57d9c6c-xnrf4"
Sep 17 03:22:15.734667 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:15.734638    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-86d57d9c6c-xnrf4"]
Sep 17 03:22:15.745878 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:15.745840    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7a64c037-e935-45f7-ba20-e0c2140c5920-trusted-ca-bundle\") pod \"console-86d57d9c6c-xnrf4\" (UID: \"7a64c037-e935-45f7-ba20-e0c2140c5920\") " pod="openshift-console/console-86d57d9c6c-xnrf4"
Sep 17 03:22:15.746061 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:15.745895    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrsqk\" (UniqueName: \"kubernetes.io/projected/7a64c037-e935-45f7-ba20-e0c2140c5920-kube-api-access-zrsqk\") pod \"console-86d57d9c6c-xnrf4\" (UID: \"7a64c037-e935-45f7-ba20-e0c2140c5920\") " pod="openshift-console/console-86d57d9c6c-xnrf4"
Sep 17 03:22:15.746061 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:15.745937    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7a64c037-e935-45f7-ba20-e0c2140c5920-service-ca\") pod \"console-86d57d9c6c-xnrf4\" (UID: \"7a64c037-e935-45f7-ba20-e0c2140c5920\") " pod="openshift-console/console-86d57d9c6c-xnrf4"
Sep 17 03:22:15.746061 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:15.746000    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7a64c037-e935-45f7-ba20-e0c2140c5920-console-config\") pod \"console-86d57d9c6c-xnrf4\" (UID: \"7a64c037-e935-45f7-ba20-e0c2140c5920\") " pod="openshift-console/console-86d57d9c6c-xnrf4"
Sep 17 03:22:15.746222 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:15.746064    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7a64c037-e935-45f7-ba20-e0c2140c5920-console-serving-cert\") pod \"console-86d57d9c6c-xnrf4\" (UID: \"7a64c037-e935-45f7-ba20-e0c2140c5920\") " pod="openshift-console/console-86d57d9c6c-xnrf4"
Sep 17 03:22:15.746222 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:15.746102    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7a64c037-e935-45f7-ba20-e0c2140c5920-console-oauth-config\") pod \"console-86d57d9c6c-xnrf4\" (UID: \"7a64c037-e935-45f7-ba20-e0c2140c5920\") " pod="openshift-console/console-86d57d9c6c-xnrf4"
Sep 17 03:22:15.746222 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:15.746141    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7a64c037-e935-45f7-ba20-e0c2140c5920-oauth-serving-cert\") pod \"console-86d57d9c6c-xnrf4\" (UID: \"7a64c037-e935-45f7-ba20-e0c2140c5920\") " pod="openshift-console/console-86d57d9c6c-xnrf4"
Sep 17 03:22:15.847174 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:15.847130    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7a64c037-e935-45f7-ba20-e0c2140c5920-service-ca\") pod \"console-86d57d9c6c-xnrf4\" (UID: \"7a64c037-e935-45f7-ba20-e0c2140c5920\") " pod="openshift-console/console-86d57d9c6c-xnrf4"
Sep 17 03:22:15.847174 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:15.847184    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7a64c037-e935-45f7-ba20-e0c2140c5920-console-config\") pod \"console-86d57d9c6c-xnrf4\" (UID: \"7a64c037-e935-45f7-ba20-e0c2140c5920\") " pod="openshift-console/console-86d57d9c6c-xnrf4"
Sep 17 03:22:15.847422 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:15.847219    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7a64c037-e935-45f7-ba20-e0c2140c5920-console-serving-cert\") pod \"console-86d57d9c6c-xnrf4\" (UID: \"7a64c037-e935-45f7-ba20-e0c2140c5920\") " pod="openshift-console/console-86d57d9c6c-xnrf4"
Sep 17 03:22:15.847422 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:15.847245    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7a64c037-e935-45f7-ba20-e0c2140c5920-console-oauth-config\") pod \"console-86d57d9c6c-xnrf4\" (UID: \"7a64c037-e935-45f7-ba20-e0c2140c5920\") " pod="openshift-console/console-86d57d9c6c-xnrf4"
Sep 17 03:22:15.847422 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:15.847270    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7a64c037-e935-45f7-ba20-e0c2140c5920-oauth-serving-cert\") pod \"console-86d57d9c6c-xnrf4\" (UID: \"7a64c037-e935-45f7-ba20-e0c2140c5920\") " pod="openshift-console/console-86d57d9c6c-xnrf4"
Sep 17 03:22:15.847422 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:15.847289    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7a64c037-e935-45f7-ba20-e0c2140c5920-trusted-ca-bundle\") pod \"console-86d57d9c6c-xnrf4\" (UID: \"7a64c037-e935-45f7-ba20-e0c2140c5920\") " pod="openshift-console/console-86d57d9c6c-xnrf4"
Sep 17 03:22:15.847422 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:15.847307    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrsqk\" (UniqueName: \"kubernetes.io/projected/7a64c037-e935-45f7-ba20-e0c2140c5920-kube-api-access-zrsqk\") pod \"console-86d57d9c6c-xnrf4\" (UID: \"7a64c037-e935-45f7-ba20-e0c2140c5920\") " pod="openshift-console/console-86d57d9c6c-xnrf4"
Sep 17 03:22:15.848058 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:15.848021    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7a64c037-e935-45f7-ba20-e0c2140c5920-service-ca\") pod \"console-86d57d9c6c-xnrf4\" (UID: \"7a64c037-e935-45f7-ba20-e0c2140c5920\") " pod="openshift-console/console-86d57d9c6c-xnrf4"
Sep 17 03:22:15.848187 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:15.848159    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7a64c037-e935-45f7-ba20-e0c2140c5920-console-config\") pod \"console-86d57d9c6c-xnrf4\" (UID: \"7a64c037-e935-45f7-ba20-e0c2140c5920\") " pod="openshift-console/console-86d57d9c6c-xnrf4"
Sep 17 03:22:15.848265 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:15.848184    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7a64c037-e935-45f7-ba20-e0c2140c5920-oauth-serving-cert\") pod \"console-86d57d9c6c-xnrf4\" (UID: \"7a64c037-e935-45f7-ba20-e0c2140c5920\") " pod="openshift-console/console-86d57d9c6c-xnrf4"
Sep 17 03:22:15.848425 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:15.848400    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7a64c037-e935-45f7-ba20-e0c2140c5920-trusted-ca-bundle\") pod \"console-86d57d9c6c-xnrf4\" (UID: \"7a64c037-e935-45f7-ba20-e0c2140c5920\") " pod="openshift-console/console-86d57d9c6c-xnrf4"
Sep 17 03:22:15.849744 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:15.849716    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7a64c037-e935-45f7-ba20-e0c2140c5920-console-oauth-config\") pod \"console-86d57d9c6c-xnrf4\" (UID: \"7a64c037-e935-45f7-ba20-e0c2140c5920\") " pod="openshift-console/console-86d57d9c6c-xnrf4"
Sep 17 03:22:15.849897 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:15.849877    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7a64c037-e935-45f7-ba20-e0c2140c5920-console-serving-cert\") pod \"console-86d57d9c6c-xnrf4\" (UID: \"7a64c037-e935-45f7-ba20-e0c2140c5920\") " pod="openshift-console/console-86d57d9c6c-xnrf4"
Sep 17 03:22:15.855110 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:15.855083    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrsqk\" (UniqueName: \"kubernetes.io/projected/7a64c037-e935-45f7-ba20-e0c2140c5920-kube-api-access-zrsqk\") pod \"console-86d57d9c6c-xnrf4\" (UID: \"7a64c037-e935-45f7-ba20-e0c2140c5920\") " pod="openshift-console/console-86d57d9c6c-xnrf4"
Sep 17 03:22:16.033952 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:16.033919    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-86d57d9c6c-xnrf4"
Sep 17 03:22:16.306673 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:16.306639    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-86d57d9c6c-xnrf4"]
Sep 17 03:22:16.975132 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:16.975097    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-86d57d9c6c-xnrf4" event={"ID":"7a64c037-e935-45f7-ba20-e0c2140c5920","Type":"ContainerStarted","Data":"7eeabb2d381626ee009ba9c46ec931dc1312931c867aee99e3225600117cde7c"}
Sep 17 03:22:16.975513 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:16.975152    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-86d57d9c6c-xnrf4" event={"ID":"7a64c037-e935-45f7-ba20-e0c2140c5920","Type":"ContainerStarted","Data":"4a6d4e4eead3a021429c7e2f1e363d78dc7a9770215a2ac38561e1c80ebb4db3"}
Sep 17 03:22:16.992389 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:16.992333    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-86d57d9c6c-xnrf4" podStartSLOduration=1.992314861 podStartE2EDuration="1.992314861s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 03:22:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:22:16.991328828 +0000 UTC m=+172.457736353" watchObservedRunningTime="2026-09-17 03:22:16.992314861 +0000 UTC m=+172.458722387"
Sep 17 03:22:26.034633 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:26.034570    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-86d57d9c6c-xnrf4"
Sep 17 03:22:26.035164 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:26.034663    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-86d57d9c6c-xnrf4"
Sep 17 03:22:26.039451 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:26.039423    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-86d57d9c6c-xnrf4"
Sep 17 03:22:27.023617 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:27.023565    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-86d57d9c6c-xnrf4"
Sep 17 03:22:27.069673 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:27.069630    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-554ff6b6c4-mqp7s"]
Sep 17 03:22:52.089723 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:52.089678    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-554ff6b6c4-mqp7s" podUID="73bf9abb-6a04-458a-a5d2-75f6aeae7ab9" containerName="console" containerID="cri-o://d7006f13f054e09efe19704d3d10ac6ddbe82b0c9f9d9432162579e4b4db0831" gracePeriod=15
Sep 17 03:22:52.331001 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:52.330966    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-554ff6b6c4-mqp7s"
Sep 17 03:22:52.468436 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:52.468331    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/73bf9abb-6a04-458a-a5d2-75f6aeae7ab9-console-config\") pod \"73bf9abb-6a04-458a-a5d2-75f6aeae7ab9\" (UID: \"73bf9abb-6a04-458a-a5d2-75f6aeae7ab9\") "
Sep 17 03:22:52.468436 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:52.468408    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/73bf9abb-6a04-458a-a5d2-75f6aeae7ab9-console-oauth-config\") pod \"73bf9abb-6a04-458a-a5d2-75f6aeae7ab9\" (UID: \"73bf9abb-6a04-458a-a5d2-75f6aeae7ab9\") "
Sep 17 03:22:52.468436 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:52.468440    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/73bf9abb-6a04-458a-a5d2-75f6aeae7ab9-service-ca\") pod \"73bf9abb-6a04-458a-a5d2-75f6aeae7ab9\" (UID: \"73bf9abb-6a04-458a-a5d2-75f6aeae7ab9\") "
Sep 17 03:22:52.468777 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:52.468457    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/73bf9abb-6a04-458a-a5d2-75f6aeae7ab9-trusted-ca-bundle\") pod \"73bf9abb-6a04-458a-a5d2-75f6aeae7ab9\" (UID: \"73bf9abb-6a04-458a-a5d2-75f6aeae7ab9\") "
Sep 17 03:22:52.468777 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:52.468503    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/73bf9abb-6a04-458a-a5d2-75f6aeae7ab9-oauth-serving-cert\") pod \"73bf9abb-6a04-458a-a5d2-75f6aeae7ab9\" (UID: \"73bf9abb-6a04-458a-a5d2-75f6aeae7ab9\") "
Sep 17 03:22:52.468777 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:52.468527    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/73bf9abb-6a04-458a-a5d2-75f6aeae7ab9-console-serving-cert\") pod \"73bf9abb-6a04-458a-a5d2-75f6aeae7ab9\" (UID: \"73bf9abb-6a04-458a-a5d2-75f6aeae7ab9\") "
Sep 17 03:22:52.468777 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:52.468546    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r8hwx\" (UniqueName: \"kubernetes.io/projected/73bf9abb-6a04-458a-a5d2-75f6aeae7ab9-kube-api-access-r8hwx\") pod \"73bf9abb-6a04-458a-a5d2-75f6aeae7ab9\" (UID: \"73bf9abb-6a04-458a-a5d2-75f6aeae7ab9\") "
Sep 17 03:22:52.468982 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:52.468842    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73bf9abb-6a04-458a-a5d2-75f6aeae7ab9-service-ca" (OuterVolumeSpecName: "service-ca") pod "73bf9abb-6a04-458a-a5d2-75f6aeae7ab9" (UID: "73bf9abb-6a04-458a-a5d2-75f6aeae7ab9"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 03:22:52.468982 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:52.468950    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73bf9abb-6a04-458a-a5d2-75f6aeae7ab9-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "73bf9abb-6a04-458a-a5d2-75f6aeae7ab9" (UID: "73bf9abb-6a04-458a-a5d2-75f6aeae7ab9"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 03:22:52.469089 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:52.468984    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73bf9abb-6a04-458a-a5d2-75f6aeae7ab9-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "73bf9abb-6a04-458a-a5d2-75f6aeae7ab9" (UID: "73bf9abb-6a04-458a-a5d2-75f6aeae7ab9"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 03:22:52.469227 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:52.469204    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73bf9abb-6a04-458a-a5d2-75f6aeae7ab9-console-config" (OuterVolumeSpecName: "console-config") pod "73bf9abb-6a04-458a-a5d2-75f6aeae7ab9" (UID: "73bf9abb-6a04-458a-a5d2-75f6aeae7ab9"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 03:22:52.470944 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:52.470917    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73bf9abb-6a04-458a-a5d2-75f6aeae7ab9-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "73bf9abb-6a04-458a-a5d2-75f6aeae7ab9" (UID: "73bf9abb-6a04-458a-a5d2-75f6aeae7ab9"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 03:22:52.471029 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:52.470978    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73bf9abb-6a04-458a-a5d2-75f6aeae7ab9-kube-api-access-r8hwx" (OuterVolumeSpecName: "kube-api-access-r8hwx") pod "73bf9abb-6a04-458a-a5d2-75f6aeae7ab9" (UID: "73bf9abb-6a04-458a-a5d2-75f6aeae7ab9"). InnerVolumeSpecName "kube-api-access-r8hwx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 03:22:52.471029 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:52.470980    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73bf9abb-6a04-458a-a5d2-75f6aeae7ab9-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "73bf9abb-6a04-458a-a5d2-75f6aeae7ab9" (UID: "73bf9abb-6a04-458a-a5d2-75f6aeae7ab9"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 03:22:52.569552 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:52.569512    2552 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/73bf9abb-6a04-458a-a5d2-75f6aeae7ab9-console-config\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:22:52.569552 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:52.569542    2552 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/73bf9abb-6a04-458a-a5d2-75f6aeae7ab9-console-oauth-config\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:22:52.569552 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:52.569553    2552 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/73bf9abb-6a04-458a-a5d2-75f6aeae7ab9-service-ca\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:22:52.569552 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:52.569562    2552 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/73bf9abb-6a04-458a-a5d2-75f6aeae7ab9-trusted-ca-bundle\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:22:52.569817 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:52.569570    2552 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/73bf9abb-6a04-458a-a5d2-75f6aeae7ab9-oauth-serving-cert\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:22:52.569817 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:52.569578    2552 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/73bf9abb-6a04-458a-a5d2-75f6aeae7ab9-console-serving-cert\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:22:52.569817 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:52.569586    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r8hwx\" (UniqueName: \"kubernetes.io/projected/73bf9abb-6a04-458a-a5d2-75f6aeae7ab9-kube-api-access-r8hwx\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:22:53.126337 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:53.126304    2552 generic.go:356] "Generic (PLEG): container finished" podID="73bf9abb-6a04-458a-a5d2-75f6aeae7ab9" containerID="d7006f13f054e09efe19704d3d10ac6ddbe82b0c9f9d9432162579e4b4db0831" exitCode=0
Sep 17 03:22:53.126774 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:53.126360    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-554ff6b6c4-mqp7s"
Sep 17 03:22:53.126774 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:53.126393    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-554ff6b6c4-mqp7s" event={"ID":"73bf9abb-6a04-458a-a5d2-75f6aeae7ab9","Type":"ContainerDied","Data":"d7006f13f054e09efe19704d3d10ac6ddbe82b0c9f9d9432162579e4b4db0831"}
Sep 17 03:22:53.126774 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:53.126447    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-554ff6b6c4-mqp7s" event={"ID":"73bf9abb-6a04-458a-a5d2-75f6aeae7ab9","Type":"ContainerDied","Data":"ebcc9873f5f0f7fe59a3621cbef4880c5cda2144a80eb021860874bc874b2a46"}
Sep 17 03:22:53.126774 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:53.126473    2552 scope.go:117] "RemoveContainer" containerID="d7006f13f054e09efe19704d3d10ac6ddbe82b0c9f9d9432162579e4b4db0831"
Sep 17 03:22:53.134964 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:53.134940    2552 scope.go:117] "RemoveContainer" containerID="d7006f13f054e09efe19704d3d10ac6ddbe82b0c9f9d9432162579e4b4db0831"
Sep 17 03:22:53.135296 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:22:53.135274    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7006f13f054e09efe19704d3d10ac6ddbe82b0c9f9d9432162579e4b4db0831\": container with ID starting with d7006f13f054e09efe19704d3d10ac6ddbe82b0c9f9d9432162579e4b4db0831 not found: ID does not exist" containerID="d7006f13f054e09efe19704d3d10ac6ddbe82b0c9f9d9432162579e4b4db0831"
Sep 17 03:22:53.135357 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:53.135306    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7006f13f054e09efe19704d3d10ac6ddbe82b0c9f9d9432162579e4b4db0831"} err="failed to get container status \"d7006f13f054e09efe19704d3d10ac6ddbe82b0c9f9d9432162579e4b4db0831\": rpc error: code = NotFound desc = could not find container \"d7006f13f054e09efe19704d3d10ac6ddbe82b0c9f9d9432162579e4b4db0831\": container with ID starting with d7006f13f054e09efe19704d3d10ac6ddbe82b0c9f9d9432162579e4b4db0831 not found: ID does not exist"
Sep 17 03:22:53.144225 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:53.144191    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-554ff6b6c4-mqp7s"]
Sep 17 03:22:53.147146 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:53.147121    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-554ff6b6c4-mqp7s"]
Sep 17 03:22:54.984840 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:22:54.984802    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="73bf9abb-6a04-458a-a5d2-75f6aeae7ab9" path="/var/lib/kubelet/pods/73bf9abb-6a04-458a-a5d2-75f6aeae7ab9/volumes"
Sep 17 03:23:48.069453 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:23:48.069372    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-q4sb6"]
Sep 17 03:23:48.069952 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:23:48.069703    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73bf9abb-6a04-458a-a5d2-75f6aeae7ab9" containerName="console"
Sep 17 03:23:48.069952 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:23:48.069714    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="73bf9abb-6a04-458a-a5d2-75f6aeae7ab9" containerName="console"
Sep 17 03:23:48.069952 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:23:48.069814    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="73bf9abb-6a04-458a-a5d2-75f6aeae7ab9" containerName="console"
Sep 17 03:23:48.072759 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:23:48.072741    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-q4sb6"
Sep 17 03:23:48.075355 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:23:48.075328    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 17 03:23:48.075355 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:23:48.075347    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 17 03:23:48.075527 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:23:48.075459    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-zll89\""
Sep 17 03:23:48.079930 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:23:48.079774    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-q4sb6"]
Sep 17 03:23:48.245676 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:23:48.245635    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1d1e4bc2-261f-42ba-9a77-57d1ed63db59-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-q4sb6\" (UID: \"1d1e4bc2-261f-42ba-9a77-57d1ed63db59\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-q4sb6"
Sep 17 03:23:48.245847 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:23:48.245692    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8x87\" (UniqueName: \"kubernetes.io/projected/1d1e4bc2-261f-42ba-9a77-57d1ed63db59-kube-api-access-f8x87\") pod \"cert-manager-cainjector-559cdd8c87-q4sb6\" (UID: \"1d1e4bc2-261f-42ba-9a77-57d1ed63db59\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-q4sb6"
Sep 17 03:23:48.346767 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:23:48.346668    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1d1e4bc2-261f-42ba-9a77-57d1ed63db59-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-q4sb6\" (UID: \"1d1e4bc2-261f-42ba-9a77-57d1ed63db59\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-q4sb6"
Sep 17 03:23:48.346767 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:23:48.346729    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8x87\" (UniqueName: \"kubernetes.io/projected/1d1e4bc2-261f-42ba-9a77-57d1ed63db59-kube-api-access-f8x87\") pod \"cert-manager-cainjector-559cdd8c87-q4sb6\" (UID: \"1d1e4bc2-261f-42ba-9a77-57d1ed63db59\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-q4sb6"
Sep 17 03:23:48.355431 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:23:48.355401    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1d1e4bc2-261f-42ba-9a77-57d1ed63db59-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-q4sb6\" (UID: \"1d1e4bc2-261f-42ba-9a77-57d1ed63db59\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-q4sb6"
Sep 17 03:23:48.355555 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:23:48.355401    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8x87\" (UniqueName: \"kubernetes.io/projected/1d1e4bc2-261f-42ba-9a77-57d1ed63db59-kube-api-access-f8x87\") pod \"cert-manager-cainjector-559cdd8c87-q4sb6\" (UID: \"1d1e4bc2-261f-42ba-9a77-57d1ed63db59\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-q4sb6"
Sep 17 03:23:48.394015 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:23:48.393977    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-q4sb6"
Sep 17 03:23:48.665702 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:23:48.665525    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-q4sb6"]
Sep 17 03:23:48.668501 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:23:48.668473    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1d1e4bc2_261f_42ba_9a77_57d1ed63db59.slice/crio-fef3708aab76c62622e59ebde27b411902ff2e607425b0e5a79f25ed2a658bbb WatchSource:0}: Error finding container fef3708aab76c62622e59ebde27b411902ff2e607425b0e5a79f25ed2a658bbb: Status 404 returned error can't find the container with id fef3708aab76c62622e59ebde27b411902ff2e607425b0e5a79f25ed2a658bbb
Sep 17 03:23:49.360454 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:23:49.360409    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-q4sb6" event={"ID":"1d1e4bc2-261f-42ba-9a77-57d1ed63db59","Type":"ContainerStarted","Data":"fef3708aab76c62622e59ebde27b411902ff2e607425b0e5a79f25ed2a658bbb"}
Sep 17 03:23:52.376786 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:23:52.376750    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-q4sb6" event={"ID":"1d1e4bc2-261f-42ba-9a77-57d1ed63db59","Type":"ContainerStarted","Data":"ac947c46b47dbd0753e2655063363e0a1302939a670c94857521f286869ba56e"}
Sep 17 03:23:52.394451 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:23:52.394398    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-q4sb6" podStartSLOduration=1.345447997 podStartE2EDuration="4.394380257s" totalImagesPullingTime="3.04893226s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 03:23:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:23:52.392921775 +0000 UTC m=+267.859329312" watchObservedRunningTime="2026-09-17 03:23:52.394380257 +0000 UTC m=+267.860787779"
Sep 17 03:24:19.746743 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:24:19.746701    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6977f7b7bc-css5r"]
Sep 17 03:24:19.751395 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:24:19.751370    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6977f7b7bc-css5r"
Sep 17 03:24:19.754472 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:24:19.754449    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 17 03:24:19.754708 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:24:19.754689    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 17 03:24:19.754810 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:24:19.754753    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 17 03:24:19.754810 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:24:19.754796    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-8vv95\""
Sep 17 03:24:19.754929 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:24:19.754913    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 17 03:24:19.760033 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:24:19.760011    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6977f7b7bc-css5r"]
Sep 17 03:24:19.820225 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:24:19.820187    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkshb\" (UniqueName: \"kubernetes.io/projected/f512d104-3bf3-4414-bed2-f7440c18ee3b-kube-api-access-dkshb\") pod \"opendatahub-operator-controller-manager-6977f7b7bc-css5r\" (UID: \"f512d104-3bf3-4414-bed2-f7440c18ee3b\") " pod="opendatahub/opendatahub-operator-controller-manager-6977f7b7bc-css5r"
Sep 17 03:24:19.820225 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:24:19.820226    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f512d104-3bf3-4414-bed2-f7440c18ee3b-manifests\") pod \"opendatahub-operator-controller-manager-6977f7b7bc-css5r\" (UID: \"f512d104-3bf3-4414-bed2-f7440c18ee3b\") " pod="opendatahub/opendatahub-operator-controller-manager-6977f7b7bc-css5r"
Sep 17 03:24:19.820440 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:24:19.820308    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f512d104-3bf3-4414-bed2-f7440c18ee3b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6977f7b7bc-css5r\" (UID: \"f512d104-3bf3-4414-bed2-f7440c18ee3b\") " pod="opendatahub/opendatahub-operator-controller-manager-6977f7b7bc-css5r"
Sep 17 03:24:19.820440 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:24:19.820342    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f512d104-3bf3-4414-bed2-f7440c18ee3b-webhook-cert\") pod \"opendatahub-operator-controller-manager-6977f7b7bc-css5r\" (UID: \"f512d104-3bf3-4414-bed2-f7440c18ee3b\") " pod="opendatahub/opendatahub-operator-controller-manager-6977f7b7bc-css5r"
Sep 17 03:24:19.820440 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:24:19.820366    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f512d104-3bf3-4414-bed2-f7440c18ee3b-tmp\") pod \"opendatahub-operator-controller-manager-6977f7b7bc-css5r\" (UID: \"f512d104-3bf3-4414-bed2-f7440c18ee3b\") " pod="opendatahub/opendatahub-operator-controller-manager-6977f7b7bc-css5r"
Sep 17 03:24:19.921530 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:24:19.921492    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f512d104-3bf3-4414-bed2-f7440c18ee3b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6977f7b7bc-css5r\" (UID: \"f512d104-3bf3-4414-bed2-f7440c18ee3b\") " pod="opendatahub/opendatahub-operator-controller-manager-6977f7b7bc-css5r"
Sep 17 03:24:19.921530 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:24:19.921530    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f512d104-3bf3-4414-bed2-f7440c18ee3b-webhook-cert\") pod \"opendatahub-operator-controller-manager-6977f7b7bc-css5r\" (UID: \"f512d104-3bf3-4414-bed2-f7440c18ee3b\") " pod="opendatahub/opendatahub-operator-controller-manager-6977f7b7bc-css5r"
Sep 17 03:24:19.921776 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:24:19.921553    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f512d104-3bf3-4414-bed2-f7440c18ee3b-tmp\") pod \"opendatahub-operator-controller-manager-6977f7b7bc-css5r\" (UID: \"f512d104-3bf3-4414-bed2-f7440c18ee3b\") " pod="opendatahub/opendatahub-operator-controller-manager-6977f7b7bc-css5r"
Sep 17 03:24:19.921776 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:24:19.921624    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkshb\" (UniqueName: \"kubernetes.io/projected/f512d104-3bf3-4414-bed2-f7440c18ee3b-kube-api-access-dkshb\") pod \"opendatahub-operator-controller-manager-6977f7b7bc-css5r\" (UID: \"f512d104-3bf3-4414-bed2-f7440c18ee3b\") " pod="opendatahub/opendatahub-operator-controller-manager-6977f7b7bc-css5r"
Sep 17 03:24:19.921776 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:24:19.921641    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f512d104-3bf3-4414-bed2-f7440c18ee3b-manifests\") pod \"opendatahub-operator-controller-manager-6977f7b7bc-css5r\" (UID: \"f512d104-3bf3-4414-bed2-f7440c18ee3b\") " pod="opendatahub/opendatahub-operator-controller-manager-6977f7b7bc-css5r"
Sep 17 03:24:19.921964 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:24:19.921947    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f512d104-3bf3-4414-bed2-f7440c18ee3b-manifests\") pod \"opendatahub-operator-controller-manager-6977f7b7bc-css5r\" (UID: \"f512d104-3bf3-4414-bed2-f7440c18ee3b\") " pod="opendatahub/opendatahub-operator-controller-manager-6977f7b7bc-css5r"
Sep 17 03:24:19.924057 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:24:19.924027    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f512d104-3bf3-4414-bed2-f7440c18ee3b-tmp\") pod \"opendatahub-operator-controller-manager-6977f7b7bc-css5r\" (UID: \"f512d104-3bf3-4414-bed2-f7440c18ee3b\") " pod="opendatahub/opendatahub-operator-controller-manager-6977f7b7bc-css5r"
Sep 17 03:24:19.924277 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:24:19.924258    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f512d104-3bf3-4414-bed2-f7440c18ee3b-webhook-cert\") pod \"opendatahub-operator-controller-manager-6977f7b7bc-css5r\" (UID: \"f512d104-3bf3-4414-bed2-f7440c18ee3b\") " pod="opendatahub/opendatahub-operator-controller-manager-6977f7b7bc-css5r"
Sep 17 03:24:19.924363 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:24:19.924342    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f512d104-3bf3-4414-bed2-f7440c18ee3b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6977f7b7bc-css5r\" (UID: \"f512d104-3bf3-4414-bed2-f7440c18ee3b\") " pod="opendatahub/opendatahub-operator-controller-manager-6977f7b7bc-css5r"
Sep 17 03:24:19.929830 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:24:19.929803    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkshb\" (UniqueName: \"kubernetes.io/projected/f512d104-3bf3-4414-bed2-f7440c18ee3b-kube-api-access-dkshb\") pod \"opendatahub-operator-controller-manager-6977f7b7bc-css5r\" (UID: \"f512d104-3bf3-4414-bed2-f7440c18ee3b\") " pod="opendatahub/opendatahub-operator-controller-manager-6977f7b7bc-css5r"
Sep 17 03:24:20.063026 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:24:20.062986    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6977f7b7bc-css5r"
Sep 17 03:24:20.348322 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:24:20.348299    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6977f7b7bc-css5r"]
Sep 17 03:24:20.350295 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:24:20.350270    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf512d104_3bf3_4414_bed2_f7440c18ee3b.slice/crio-05ab883b9b9ca8c0f61f40e9116f7fa1ea81d933d54d0a75eff2287d50207c5a WatchSource:0}: Error finding container 05ab883b9b9ca8c0f61f40e9116f7fa1ea81d933d54d0a75eff2287d50207c5a: Status 404 returned error can't find the container with id 05ab883b9b9ca8c0f61f40e9116f7fa1ea81d933d54d0a75eff2287d50207c5a
Sep 17 03:24:20.490267 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:24:20.490233    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6977f7b7bc-css5r" event={"ID":"f512d104-3bf3-4414-bed2-f7440c18ee3b","Type":"ContainerStarted","Data":"05ab883b9b9ca8c0f61f40e9116f7fa1ea81d933d54d0a75eff2287d50207c5a"}
Sep 17 03:24:23.506980 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:24:23.506949    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6977f7b7bc-css5r" event={"ID":"f512d104-3bf3-4414-bed2-f7440c18ee3b","Type":"ContainerStarted","Data":"7e5bf0a3a5009b2fa335f55a46917aa11b4dae9a5fec92aea819f55bcf849fb2"}
Sep 17 03:24:24.513253 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:24:24.513217    2552 generic.go:356] "Generic (PLEG): container finished" podID="f512d104-3bf3-4414-bed2-f7440c18ee3b" containerID="7e5bf0a3a5009b2fa335f55a46917aa11b4dae9a5fec92aea819f55bcf849fb2" exitCode=0
Sep 17 03:24:24.513664 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:24:24.513299    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6977f7b7bc-css5r" event={"ID":"f512d104-3bf3-4414-bed2-f7440c18ee3b","Type":"ContainerDied","Data":"7e5bf0a3a5009b2fa335f55a46917aa11b4dae9a5fec92aea819f55bcf849fb2"}
Sep 17 03:24:24.902424 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:24:24.902403    2552 kubelet.go:1648] "Image garbage collection succeeded"
Sep 17 03:24:25.520107 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:24:25.520067    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6977f7b7bc-css5r" event={"ID":"f512d104-3bf3-4414-bed2-f7440c18ee3b","Type":"ContainerStarted","Data":"c70e256d3efb80011e46c0b30949198c2b7283ecf02637622312cc548508c4b8"}
Sep 17 03:24:25.520523 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:24:25.520200    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-6977f7b7bc-css5r"
Sep 17 03:24:25.542294 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:24:25.542232    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-6977f7b7bc-css5r" podStartSLOduration=2.307224791 podStartE2EDuration="6.542211985s" totalImagesPullingTime="3.216070709s" totalInitContainerRuntime="1.018916485s" isStatefulPod=false podCreationTimestamp="2026-09-17 03:24:19 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:24:25.540073474 +0000 UTC m=+301.006481012" watchObservedRunningTime="2026-09-17 03:24:25.542211985 +0000 UTC m=+301.008619512"
Sep 17 03:24:36.527563 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:24:36.527528    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-6977f7b7bc-css5r"
Sep 17 03:24:39.094230 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:24:39.094191    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-5bc6dc7888-txj6c"]
Sep 17 03:24:39.102833 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:24:39.102809    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5bc6dc7888-txj6c"
Sep 17 03:24:39.105724 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:24:39.105700    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 17 03:24:39.106914 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:24:39.106891    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 17 03:24:39.107093 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:24:39.106950    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 17 03:24:39.107185 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:24:39.106971    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-fwmr2\""
Sep 17 03:24:39.107185 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:24:39.106984    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 17 03:24:39.110802 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:24:39.110781    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5bc6dc7888-txj6c"]
Sep 17 03:24:39.297211 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:24:39.297177    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/60cb769a-2eb5-433e-b97d-5ae489598831-tmp\") pod \"kube-auth-proxy-5bc6dc7888-txj6c\" (UID: \"60cb769a-2eb5-433e-b97d-5ae489598831\") " pod="openshift-ingress/kube-auth-proxy-5bc6dc7888-txj6c"
Sep 17 03:24:39.297393 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:24:39.297223    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60cb769a-2eb5-433e-b97d-5ae489598831-tls-certs\") pod \"kube-auth-proxy-5bc6dc7888-txj6c\" (UID: \"60cb769a-2eb5-433e-b97d-5ae489598831\") " pod="openshift-ingress/kube-auth-proxy-5bc6dc7888-txj6c"
Sep 17 03:24:39.297393 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:24:39.297260    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5g6bg\" (UniqueName: \"kubernetes.io/projected/60cb769a-2eb5-433e-b97d-5ae489598831-kube-api-access-5g6bg\") pod \"kube-auth-proxy-5bc6dc7888-txj6c\" (UID: \"60cb769a-2eb5-433e-b97d-5ae489598831\") " pod="openshift-ingress/kube-auth-proxy-5bc6dc7888-txj6c"
Sep 17 03:24:39.398014 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:24:39.397928    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/60cb769a-2eb5-433e-b97d-5ae489598831-tmp\") pod \"kube-auth-proxy-5bc6dc7888-txj6c\" (UID: \"60cb769a-2eb5-433e-b97d-5ae489598831\") " pod="openshift-ingress/kube-auth-proxy-5bc6dc7888-txj6c"
Sep 17 03:24:39.398014 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:24:39.397980    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60cb769a-2eb5-433e-b97d-5ae489598831-tls-certs\") pod \"kube-auth-proxy-5bc6dc7888-txj6c\" (UID: \"60cb769a-2eb5-433e-b97d-5ae489598831\") " pod="openshift-ingress/kube-auth-proxy-5bc6dc7888-txj6c"
Sep 17 03:24:39.398235 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:24:39.398022    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5g6bg\" (UniqueName: \"kubernetes.io/projected/60cb769a-2eb5-433e-b97d-5ae489598831-kube-api-access-5g6bg\") pod \"kube-auth-proxy-5bc6dc7888-txj6c\" (UID: \"60cb769a-2eb5-433e-b97d-5ae489598831\") " pod="openshift-ingress/kube-auth-proxy-5bc6dc7888-txj6c"
Sep 17 03:24:39.400283 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:24:39.400248    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/60cb769a-2eb5-433e-b97d-5ae489598831-tmp\") pod \"kube-auth-proxy-5bc6dc7888-txj6c\" (UID: \"60cb769a-2eb5-433e-b97d-5ae489598831\") " pod="openshift-ingress/kube-auth-proxy-5bc6dc7888-txj6c"
Sep 17 03:24:39.400428 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:24:39.400409    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60cb769a-2eb5-433e-b97d-5ae489598831-tls-certs\") pod \"kube-auth-proxy-5bc6dc7888-txj6c\" (UID: \"60cb769a-2eb5-433e-b97d-5ae489598831\") " pod="openshift-ingress/kube-auth-proxy-5bc6dc7888-txj6c"
Sep 17 03:24:39.408183 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:24:39.408154    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5g6bg\" (UniqueName: \"kubernetes.io/projected/60cb769a-2eb5-433e-b97d-5ae489598831-kube-api-access-5g6bg\") pod \"kube-auth-proxy-5bc6dc7888-txj6c\" (UID: \"60cb769a-2eb5-433e-b97d-5ae489598831\") " pod="openshift-ingress/kube-auth-proxy-5bc6dc7888-txj6c"
Sep 17 03:24:39.413793 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:24:39.413762    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5bc6dc7888-txj6c"
Sep 17 03:24:39.686931 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:24:39.686756    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5bc6dc7888-txj6c"]
Sep 17 03:24:39.689819 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:24:39.689791    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60cb769a_2eb5_433e_b97d_5ae489598831.slice/crio-27811c9fee8c75cf398b21d6c280dbbc92213fe9a84cd01ed7682471b98b0279 WatchSource:0}: Error finding container 27811c9fee8c75cf398b21d6c280dbbc92213fe9a84cd01ed7682471b98b0279: Status 404 returned error can't find the container with id 27811c9fee8c75cf398b21d6c280dbbc92213fe9a84cd01ed7682471b98b0279
Sep 17 03:24:39.691764 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:24:39.691742    2552 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 03:24:40.325551 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:24:40.325514    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-k2srk"]
Sep 17 03:24:40.329144 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:24:40.329117    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-k2srk"
Sep 17 03:24:40.331823 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:24:40.331797    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Sep 17 03:24:40.331968 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:24:40.331797    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Sep 17 03:24:40.331968 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:24:40.331803    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Sep 17 03:24:40.331968 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:24:40.331849    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-4czlc\""
Sep 17 03:24:40.332186 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:24:40.332168    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 17 03:24:40.342357 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:24:40.342334    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-k2srk"]
Sep 17 03:24:40.507270 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:24:40.507229    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f822eb82-abcd-445c-95bb-8fd01c0a3332-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-k2srk\" (UID: \"f822eb82-abcd-445c-95bb-8fd01c0a3332\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-k2srk"
Sep 17 03:24:40.507469 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:24:40.507283    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdg2g\" (UniqueName: \"kubernetes.io/projected/f822eb82-abcd-445c-95bb-8fd01c0a3332-kube-api-access-gdg2g\") pod \"istiod-openshift-gateway-7b7995d5cd-k2srk\" (UID: \"f822eb82-abcd-445c-95bb-8fd01c0a3332\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-k2srk"
Sep 17 03:24:40.507469 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:24:40.507318    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f822eb82-abcd-445c-95bb-8fd01c0a3332-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-k2srk\" (UID: \"f822eb82-abcd-445c-95bb-8fd01c0a3332\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-k2srk"
Sep 17 03:24:40.507469 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:24:40.507377    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f822eb82-abcd-445c-95bb-8fd01c0a3332-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-k2srk\" (UID: \"f822eb82-abcd-445c-95bb-8fd01c0a3332\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-k2srk"
Sep 17 03:24:40.507469 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:24:40.507435    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f822eb82-abcd-445c-95bb-8fd01c0a3332-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-k2srk\" (UID: \"f822eb82-abcd-445c-95bb-8fd01c0a3332\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-k2srk"
Sep 17 03:24:40.507469 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:24:40.507469    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f822eb82-abcd-445c-95bb-8fd01c0a3332-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-k2srk\" (UID: \"f822eb82-abcd-445c-95bb-8fd01c0a3332\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-k2srk"
Sep 17 03:24:40.507742 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:24:40.507543    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f822eb82-abcd-445c-95bb-8fd01c0a3332-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-k2srk\" (UID: \"f822eb82-abcd-445c-95bb-8fd01c0a3332\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-k2srk"
Sep 17 03:24:40.590135 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:24:40.590043    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5bc6dc7888-txj6c" event={"ID":"60cb769a-2eb5-433e-b97d-5ae489598831","Type":"ContainerStarted","Data":"27811c9fee8c75cf398b21d6c280dbbc92213fe9a84cd01ed7682471b98b0279"}
Sep 17 03:24:40.608326 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:24:40.608290    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f822eb82-abcd-445c-95bb-8fd01c0a3332-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-k2srk\" (UID: \"f822eb82-abcd-445c-95bb-8fd01c0a3332\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-k2srk"
Sep 17 03:24:40.608326 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:24:40.608336    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdg2g\" (UniqueName: \"kubernetes.io/projected/f822eb82-abcd-445c-95bb-8fd01c0a3332-kube-api-access-gdg2g\") pod \"istiod-openshift-gateway-7b7995d5cd-k2srk\" (UID: \"f822eb82-abcd-445c-95bb-8fd01c0a3332\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-k2srk"
Sep 17 03:24:40.608686 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:24:40.608658    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f822eb82-abcd-445c-95bb-8fd01c0a3332-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-k2srk\" (UID: \"f822eb82-abcd-445c-95bb-8fd01c0a3332\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-k2srk"
Sep 17 03:24:40.608814 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:24:40.608753    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f822eb82-abcd-445c-95bb-8fd01c0a3332-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-k2srk\" (UID: \"f822eb82-abcd-445c-95bb-8fd01c0a3332\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-k2srk"
Sep 17 03:24:40.608892 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:24:40.608818    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f822eb82-abcd-445c-95bb-8fd01c0a3332-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-k2srk\" (UID: \"f822eb82-abcd-445c-95bb-8fd01c0a3332\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-k2srk"
Sep 17 03:24:40.608892 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:24:40.608852    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f822eb82-abcd-445c-95bb-8fd01c0a3332-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-k2srk\" (UID: \"f822eb82-abcd-445c-95bb-8fd01c0a3332\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-k2srk"
Sep 17 03:24:40.608892 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:24:40.608887    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f822eb82-abcd-445c-95bb-8fd01c0a3332-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-k2srk\" (UID: \"f822eb82-abcd-445c-95bb-8fd01c0a3332\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-k2srk"
Sep 17 03:24:40.609092 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:24:40.609040    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f822eb82-abcd-445c-95bb-8fd01c0a3332-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-k2srk\" (UID: \"f822eb82-abcd-445c-95bb-8fd01c0a3332\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-k2srk"
Sep 17 03:24:40.611502 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:24:40.611475    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f822eb82-abcd-445c-95bb-8fd01c0a3332-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-k2srk\" (UID: \"f822eb82-abcd-445c-95bb-8fd01c0a3332\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-k2srk"
Sep 17 03:24:40.612193 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:24:40.611882    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f822eb82-abcd-445c-95bb-8fd01c0a3332-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-k2srk\" (UID: \"f822eb82-abcd-445c-95bb-8fd01c0a3332\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-k2srk"
Sep 17 03:24:40.612293 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:24:40.612199    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f822eb82-abcd-445c-95bb-8fd01c0a3332-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-k2srk\" (UID: \"f822eb82-abcd-445c-95bb-8fd01c0a3332\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-k2srk"
Sep 17 03:24:40.612541 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:24:40.612520    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f822eb82-abcd-445c-95bb-8fd01c0a3332-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-k2srk\" (UID: \"f822eb82-abcd-445c-95bb-8fd01c0a3332\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-k2srk"
Sep 17 03:24:40.616668 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:24:40.616611    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f822eb82-abcd-445c-95bb-8fd01c0a3332-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-k2srk\" (UID: \"f822eb82-abcd-445c-95bb-8fd01c0a3332\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-k2srk"
Sep 17 03:24:40.617031 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:24:40.617006    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdg2g\" (UniqueName: \"kubernetes.io/projected/f822eb82-abcd-445c-95bb-8fd01c0a3332-kube-api-access-gdg2g\") pod \"istiod-openshift-gateway-7b7995d5cd-k2srk\" (UID: \"f822eb82-abcd-445c-95bb-8fd01c0a3332\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-k2srk"
Sep 17 03:24:40.640510 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:24:40.640477    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-k2srk"
Sep 17 03:24:40.932566 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:24:40.932474    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-k2srk"]
Sep 17 03:24:40.936527 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:24:40.936494    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf822eb82_abcd_445c_95bb_8fd01c0a3332.slice/crio-46b40d72838b999062887bf062c490225576e256fba6723722236676f3f2d18e WatchSource:0}: Error finding container 46b40d72838b999062887bf062c490225576e256fba6723722236676f3f2d18e: Status 404 returned error can't find the container with id 46b40d72838b999062887bf062c490225576e256fba6723722236676f3f2d18e
Sep 17 03:24:41.596680 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:24:41.596644    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-k2srk" event={"ID":"f822eb82-abcd-445c-95bb-8fd01c0a3332","Type":"ContainerStarted","Data":"46b40d72838b999062887bf062c490225576e256fba6723722236676f3f2d18e"}
Sep 17 03:24:43.609133 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:24:43.609096    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5bc6dc7888-txj6c" event={"ID":"60cb769a-2eb5-433e-b97d-5ae489598831","Type":"ContainerStarted","Data":"f92ae09165311f15a14e132f97eae4f1f5c24b833941281f97bf0bddb7a3ce43"}
Sep 17 03:24:44.386929 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:24:44.386891    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 03:24:44.387006 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:24:44.386962    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 03:24:44.616658 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:24:44.616618    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-k2srk" event={"ID":"f822eb82-abcd-445c-95bb-8fd01c0a3332","Type":"ContainerStarted","Data":"c4ba72df2b09561f448a7bfdc1e1ad88e6ca78586f66872e15330ddf5fa0120e"}
Sep 17 03:24:44.637325 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:24:44.637234    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-5bc6dc7888-txj6c" podStartSLOduration=2.713482501 podStartE2EDuration="5.63721715s" totalImagesPullingTime="2.923734649s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 03:24:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:24:43.624057963 +0000 UTC m=+319.090465491" watchObservedRunningTime="2026-09-17 03:24:44.63721715 +0000 UTC m=+320.103624675"
Sep 17 03:24:44.637629 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:24:44.637592    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-k2srk" podStartSLOduration=1.190416301 podStartE2EDuration="4.637583466s" totalImagesPullingTime="3.447167165s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 03:24:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:24:44.635560063 +0000 UTC m=+320.101967589" watchObservedRunningTime="2026-09-17 03:24:44.637583466 +0000 UTC m=+320.103990993"
Sep 17 03:24:45.621677 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:24:45.621640    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-k2srk"
Sep 17 03:24:45.623493 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:24:45.623471    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-k2srk"
Sep 17 03:24:57.404710 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:24:57.404676    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-55d557bdbb-vbnn4"]
Sep 17 03:24:57.414057 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:24:57.414035    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-55d557bdbb-vbnn4"
Sep 17 03:24:57.417644 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:24:57.417619    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 17 03:24:57.418728 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:24:57.418706    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 17 03:24:57.418991 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:24:57.418968    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 17 03:24:57.419101 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:24:57.419023    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 17 03:24:57.419336 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:24:57.419309    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-gxnb5\""
Sep 17 03:24:57.419443 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:24:57.419331    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 17 03:24:57.421282 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:24:57.421069    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-55d557bdbb-vbnn4"]
Sep 17 03:24:57.462207 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:24:57.462168    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e13f17f0-340c-44a3-8974-330973d98f60-metrics-cert\") pod \"lws-controller-manager-55d557bdbb-vbnn4\" (UID: \"e13f17f0-340c-44a3-8974-330973d98f60\") " pod="openshift-lws-operator/lws-controller-manager-55d557bdbb-vbnn4"
Sep 17 03:24:57.462391 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:24:57.462264    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e13f17f0-340c-44a3-8974-330973d98f60-cert\") pod \"lws-controller-manager-55d557bdbb-vbnn4\" (UID: \"e13f17f0-340c-44a3-8974-330973d98f60\") " pod="openshift-lws-operator/lws-controller-manager-55d557bdbb-vbnn4"
Sep 17 03:24:57.462391 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:24:57.462316    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mf4xf\" (UniqueName: \"kubernetes.io/projected/e13f17f0-340c-44a3-8974-330973d98f60-kube-api-access-mf4xf\") pod \"lws-controller-manager-55d557bdbb-vbnn4\" (UID: \"e13f17f0-340c-44a3-8974-330973d98f60\") " pod="openshift-lws-operator/lws-controller-manager-55d557bdbb-vbnn4"
Sep 17 03:24:57.462391 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:24:57.462357    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/e13f17f0-340c-44a3-8974-330973d98f60-manager-config\") pod \"lws-controller-manager-55d557bdbb-vbnn4\" (UID: \"e13f17f0-340c-44a3-8974-330973d98f60\") " pod="openshift-lws-operator/lws-controller-manager-55d557bdbb-vbnn4"
Sep 17 03:24:57.563874 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:24:57.563829    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e13f17f0-340c-44a3-8974-330973d98f60-cert\") pod \"lws-controller-manager-55d557bdbb-vbnn4\" (UID: \"e13f17f0-340c-44a3-8974-330973d98f60\") " pod="openshift-lws-operator/lws-controller-manager-55d557bdbb-vbnn4"
Sep 17 03:24:57.564055 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:24:57.563894    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mf4xf\" (UniqueName: \"kubernetes.io/projected/e13f17f0-340c-44a3-8974-330973d98f60-kube-api-access-mf4xf\") pod \"lws-controller-manager-55d557bdbb-vbnn4\" (UID: \"e13f17f0-340c-44a3-8974-330973d98f60\") " pod="openshift-lws-operator/lws-controller-manager-55d557bdbb-vbnn4"
Sep 17 03:24:57.564055 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:24:57.563942    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/e13f17f0-340c-44a3-8974-330973d98f60-manager-config\") pod \"lws-controller-manager-55d557bdbb-vbnn4\" (UID: \"e13f17f0-340c-44a3-8974-330973d98f60\") " pod="openshift-lws-operator/lws-controller-manager-55d557bdbb-vbnn4"
Sep 17 03:24:57.564055 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:24:57.564008    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e13f17f0-340c-44a3-8974-330973d98f60-metrics-cert\") pod \"lws-controller-manager-55d557bdbb-vbnn4\" (UID: \"e13f17f0-340c-44a3-8974-330973d98f60\") " pod="openshift-lws-operator/lws-controller-manager-55d557bdbb-vbnn4"
Sep 17 03:24:57.564789 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:24:57.564763    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/e13f17f0-340c-44a3-8974-330973d98f60-manager-config\") pod \"lws-controller-manager-55d557bdbb-vbnn4\" (UID: \"e13f17f0-340c-44a3-8974-330973d98f60\") " pod="openshift-lws-operator/lws-controller-manager-55d557bdbb-vbnn4"
Sep 17 03:24:57.566488 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:24:57.566466    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e13f17f0-340c-44a3-8974-330973d98f60-metrics-cert\") pod \"lws-controller-manager-55d557bdbb-vbnn4\" (UID: \"e13f17f0-340c-44a3-8974-330973d98f60\") " pod="openshift-lws-operator/lws-controller-manager-55d557bdbb-vbnn4"
Sep 17 03:24:57.566488 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:24:57.566473    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e13f17f0-340c-44a3-8974-330973d98f60-cert\") pod \"lws-controller-manager-55d557bdbb-vbnn4\" (UID: \"e13f17f0-340c-44a3-8974-330973d98f60\") " pod="openshift-lws-operator/lws-controller-manager-55d557bdbb-vbnn4"
Sep 17 03:24:57.577238 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:24:57.577190    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mf4xf\" (UniqueName: \"kubernetes.io/projected/e13f17f0-340c-44a3-8974-330973d98f60-kube-api-access-mf4xf\") pod \"lws-controller-manager-55d557bdbb-vbnn4\" (UID: \"e13f17f0-340c-44a3-8974-330973d98f60\") " pod="openshift-lws-operator/lws-controller-manager-55d557bdbb-vbnn4"
Sep 17 03:24:57.724585 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:24:57.724487    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-55d557bdbb-vbnn4"
Sep 17 03:24:57.996407 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:24:57.996384    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-55d557bdbb-vbnn4"]
Sep 17 03:24:57.998611 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:24:57.998576    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode13f17f0_340c_44a3_8974_330973d98f60.slice/crio-b90f34a395c281664935753d23fedd0fa4a0c852456422449c098b67905ed39e WatchSource:0}: Error finding container b90f34a395c281664935753d23fedd0fa4a0c852456422449c098b67905ed39e: Status 404 returned error can't find the container with id b90f34a395c281664935753d23fedd0fa4a0c852456422449c098b67905ed39e
Sep 17 03:24:58.683811 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:24:58.683763    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-55d557bdbb-vbnn4" event={"ID":"e13f17f0-340c-44a3-8974-330973d98f60","Type":"ContainerStarted","Data":"b90f34a395c281664935753d23fedd0fa4a0c852456422449c098b67905ed39e"}
Sep 17 03:25:00.694686 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:25:00.694584    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-55d557bdbb-vbnn4" event={"ID":"e13f17f0-340c-44a3-8974-330973d98f60","Type":"ContainerStarted","Data":"f71d6604e9e4fb6f9268cf9b8fea0297f2cb89f7abc534ad0f5b94c223021f9a"}
Sep 17 03:25:00.694686 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:25:00.694674    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-55d557bdbb-vbnn4"
Sep 17 03:25:00.715221 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:25:00.715173    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-55d557bdbb-vbnn4" podStartSLOduration=1.278048395 podStartE2EDuration="3.715157001s" totalImagesPullingTime="2.437108606s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 03:24:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:25:00.713555857 +0000 UTC m=+336.179963383" watchObservedRunningTime="2026-09-17 03:25:00.715157001 +0000 UTC m=+336.181564526"
Sep 17 03:25:00.888801 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:25:00.888764    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-k2srk"]
Sep 17 03:25:00.889016 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:25:00.888992    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-k2srk" podUID="f822eb82-abcd-445c-95bb-8fd01c0a3332" containerName="discovery" containerID="cri-o://c4ba72df2b09561f448a7bfdc1e1ad88e6ca78586f66872e15330ddf5fa0120e" gracePeriod=30
Sep 17 03:25:01.125133 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:25:01.125107    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-k2srk"
Sep 17 03:25:01.199495 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:25:01.199457    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gdg2g\" (UniqueName: \"kubernetes.io/projected/f822eb82-abcd-445c-95bb-8fd01c0a3332-kube-api-access-gdg2g\") pod \"f822eb82-abcd-445c-95bb-8fd01c0a3332\" (UID: \"f822eb82-abcd-445c-95bb-8fd01c0a3332\") "
Sep 17 03:25:01.199495 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:25:01.199498    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f822eb82-abcd-445c-95bb-8fd01c0a3332-istio-csr-ca-configmap\") pod \"f822eb82-abcd-445c-95bb-8fd01c0a3332\" (UID: \"f822eb82-abcd-445c-95bb-8fd01c0a3332\") "
Sep 17 03:25:01.199763 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:25:01.199526    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f822eb82-abcd-445c-95bb-8fd01c0a3332-cacerts\") pod \"f822eb82-abcd-445c-95bb-8fd01c0a3332\" (UID: \"f822eb82-abcd-445c-95bb-8fd01c0a3332\") "
Sep 17 03:25:01.199763 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:25:01.199551    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f822eb82-abcd-445c-95bb-8fd01c0a3332-local-certs\") pod \"f822eb82-abcd-445c-95bb-8fd01c0a3332\" (UID: \"f822eb82-abcd-445c-95bb-8fd01c0a3332\") "
Sep 17 03:25:01.199763 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:25:01.199635    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f822eb82-abcd-445c-95bb-8fd01c0a3332-istio-kubeconfig\") pod \"f822eb82-abcd-445c-95bb-8fd01c0a3332\" (UID: \"f822eb82-abcd-445c-95bb-8fd01c0a3332\") "
Sep 17 03:25:01.199763 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:25:01.199721    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f822eb82-abcd-445c-95bb-8fd01c0a3332-istio-csr-dns-cert\") pod \"f822eb82-abcd-445c-95bb-8fd01c0a3332\" (UID: \"f822eb82-abcd-445c-95bb-8fd01c0a3332\") "
Sep 17 03:25:01.199763 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:25:01.199749    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f822eb82-abcd-445c-95bb-8fd01c0a3332-istio-token\") pod \"f822eb82-abcd-445c-95bb-8fd01c0a3332\" (UID: \"f822eb82-abcd-445c-95bb-8fd01c0a3332\") "
Sep 17 03:25:01.199994 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:25:01.199954    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f822eb82-abcd-445c-95bb-8fd01c0a3332-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "f822eb82-abcd-445c-95bb-8fd01c0a3332" (UID: "f822eb82-abcd-445c-95bb-8fd01c0a3332"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 03:25:01.200096 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:25:01.200076    2552 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f822eb82-abcd-445c-95bb-8fd01c0a3332-istio-csr-ca-configmap\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:25:01.201907 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:25:01.201880    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f822eb82-abcd-445c-95bb-8fd01c0a3332-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "f822eb82-abcd-445c-95bb-8fd01c0a3332" (UID: "f822eb82-abcd-445c-95bb-8fd01c0a3332"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 03:25:01.202027 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:25:01.201939    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f822eb82-abcd-445c-95bb-8fd01c0a3332-cacerts" (OuterVolumeSpecName: "cacerts") pod "f822eb82-abcd-445c-95bb-8fd01c0a3332" (UID: "f822eb82-abcd-445c-95bb-8fd01c0a3332"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 03:25:01.202188 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:25:01.202158    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f822eb82-abcd-445c-95bb-8fd01c0a3332-local-certs" (OuterVolumeSpecName: "local-certs") pod "f822eb82-abcd-445c-95bb-8fd01c0a3332" (UID: "f822eb82-abcd-445c-95bb-8fd01c0a3332"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:25:01.202188 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:25:01.202168    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f822eb82-abcd-445c-95bb-8fd01c0a3332-istio-token" (OuterVolumeSpecName: "istio-token") pod "f822eb82-abcd-445c-95bb-8fd01c0a3332" (UID: "f822eb82-abcd-445c-95bb-8fd01c0a3332"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 03:25:01.202318 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:25:01.202251    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f822eb82-abcd-445c-95bb-8fd01c0a3332-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "f822eb82-abcd-445c-95bb-8fd01c0a3332" (UID: "f822eb82-abcd-445c-95bb-8fd01c0a3332"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 03:25:01.202363 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:25:01.202351    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f822eb82-abcd-445c-95bb-8fd01c0a3332-kube-api-access-gdg2g" (OuterVolumeSpecName: "kube-api-access-gdg2g") pod "f822eb82-abcd-445c-95bb-8fd01c0a3332" (UID: "f822eb82-abcd-445c-95bb-8fd01c0a3332"). InnerVolumeSpecName "kube-api-access-gdg2g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 03:25:01.301048 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:25:01.300964    2552 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f822eb82-abcd-445c-95bb-8fd01c0a3332-istio-kubeconfig\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:25:01.301048 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:25:01.300994    2552 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f822eb82-abcd-445c-95bb-8fd01c0a3332-istio-csr-dns-cert\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:25:01.301048 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:25:01.301003    2552 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f822eb82-abcd-445c-95bb-8fd01c0a3332-istio-token\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:25:01.301048 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:25:01.301012    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gdg2g\" (UniqueName: \"kubernetes.io/projected/f822eb82-abcd-445c-95bb-8fd01c0a3332-kube-api-access-gdg2g\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:25:01.301048 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:25:01.301021    2552 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f822eb82-abcd-445c-95bb-8fd01c0a3332-cacerts\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:25:01.301048 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:25:01.301029    2552 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f822eb82-abcd-445c-95bb-8fd01c0a3332-local-certs\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:25:01.701542 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:25:01.701451    2552 generic.go:356] "Generic (PLEG): container finished" podID="f822eb82-abcd-445c-95bb-8fd01c0a3332" containerID="c4ba72df2b09561f448a7bfdc1e1ad88e6ca78586f66872e15330ddf5fa0120e" exitCode=0
Sep 17 03:25:01.701542 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:25:01.701496    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-k2srk"
Sep 17 03:25:01.702071 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:25:01.701544    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-k2srk" event={"ID":"f822eb82-abcd-445c-95bb-8fd01c0a3332","Type":"ContainerDied","Data":"c4ba72df2b09561f448a7bfdc1e1ad88e6ca78586f66872e15330ddf5fa0120e"}
Sep 17 03:25:01.702071 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:25:01.701649    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-k2srk" event={"ID":"f822eb82-abcd-445c-95bb-8fd01c0a3332","Type":"ContainerDied","Data":"46b40d72838b999062887bf062c490225576e256fba6723722236676f3f2d18e"}
Sep 17 03:25:01.702071 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:25:01.701676    2552 scope.go:117] "RemoveContainer" containerID="c4ba72df2b09561f448a7bfdc1e1ad88e6ca78586f66872e15330ddf5fa0120e"
Sep 17 03:25:01.711867 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:25:01.711843    2552 scope.go:117] "RemoveContainer" containerID="c4ba72df2b09561f448a7bfdc1e1ad88e6ca78586f66872e15330ddf5fa0120e"
Sep 17 03:25:01.712175 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:25:01.712150    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4ba72df2b09561f448a7bfdc1e1ad88e6ca78586f66872e15330ddf5fa0120e\": container with ID starting with c4ba72df2b09561f448a7bfdc1e1ad88e6ca78586f66872e15330ddf5fa0120e not found: ID does not exist" containerID="c4ba72df2b09561f448a7bfdc1e1ad88e6ca78586f66872e15330ddf5fa0120e"
Sep 17 03:25:01.712244 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:25:01.712181    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4ba72df2b09561f448a7bfdc1e1ad88e6ca78586f66872e15330ddf5fa0120e"} err="failed to get container status \"c4ba72df2b09561f448a7bfdc1e1ad88e6ca78586f66872e15330ddf5fa0120e\": rpc error: code = NotFound desc = could not find container \"c4ba72df2b09561f448a7bfdc1e1ad88e6ca78586f66872e15330ddf5fa0120e\": container with ID starting with c4ba72df2b09561f448a7bfdc1e1ad88e6ca78586f66872e15330ddf5fa0120e not found: ID does not exist"
Sep 17 03:25:01.731658 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:25:01.731626    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-k2srk"]
Sep 17 03:25:01.734682 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:25:01.734659    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-k2srk"]
Sep 17 03:25:02.985462 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:25:02.985427    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f822eb82-abcd-445c-95bb-8fd01c0a3332" path="/var/lib/kubelet/pods/f822eb82-abcd-445c-95bb-8fd01c0a3332/volumes"
Sep 17 03:25:11.703953 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:25:11.703924    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-55d557bdbb-vbnn4"
Sep 17 03:26:20.636516 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:26:20.636480    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-bffwq"]
Sep 17 03:26:20.637055 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:26:20.637030    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f822eb82-abcd-445c-95bb-8fd01c0a3332" containerName="discovery"
Sep 17 03:26:20.637055 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:26:20.637050    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="f822eb82-abcd-445c-95bb-8fd01c0a3332" containerName="discovery"
Sep 17 03:26:20.637164 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:26:20.637133    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="f822eb82-abcd-445c-95bb-8fd01c0a3332" containerName="discovery"
Sep 17 03:26:20.640037 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:26:20.640017    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-bffwq"
Sep 17 03:26:20.642897 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:26:20.642877    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Sep 17 03:26:20.642897 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:26:20.642891    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-f5jg8\""
Sep 17 03:26:20.643086 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:26:20.642927    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 17 03:26:20.645181 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:26:20.644591    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Sep 17 03:26:20.645181 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:26:20.644636    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 17 03:26:20.651691 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:26:20.651175    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-bffwq"]
Sep 17 03:26:20.680217 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:26:20.680179    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/c8cc769e-e0d1-40bd-ba4d-3ee56d0c48b4-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-bffwq\" (UID: \"c8cc769e-e0d1-40bd-ba4d-3ee56d0c48b4\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-bffwq"
Sep 17 03:26:20.680372 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:26:20.680236    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c8cc769e-e0d1-40bd-ba4d-3ee56d0c48b4-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-bffwq\" (UID: \"c8cc769e-e0d1-40bd-ba4d-3ee56d0c48b4\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-bffwq"
Sep 17 03:26:20.680372 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:26:20.680266    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6z4s\" (UniqueName: \"kubernetes.io/projected/c8cc769e-e0d1-40bd-ba4d-3ee56d0c48b4-kube-api-access-q6z4s\") pod \"kuadrant-console-plugin-fbdd4f445-bffwq\" (UID: \"c8cc769e-e0d1-40bd-ba4d-3ee56d0c48b4\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-bffwq"
Sep 17 03:26:20.781206 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:26:20.781168    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/c8cc769e-e0d1-40bd-ba4d-3ee56d0c48b4-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-bffwq\" (UID: \"c8cc769e-e0d1-40bd-ba4d-3ee56d0c48b4\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-bffwq"
Sep 17 03:26:20.781351 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:26:20.781228    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c8cc769e-e0d1-40bd-ba4d-3ee56d0c48b4-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-bffwq\" (UID: \"c8cc769e-e0d1-40bd-ba4d-3ee56d0c48b4\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-bffwq"
Sep 17 03:26:20.781351 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:26:20.781257    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6z4s\" (UniqueName: \"kubernetes.io/projected/c8cc769e-e0d1-40bd-ba4d-3ee56d0c48b4-kube-api-access-q6z4s\") pod \"kuadrant-console-plugin-fbdd4f445-bffwq\" (UID: \"c8cc769e-e0d1-40bd-ba4d-3ee56d0c48b4\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-bffwq"
Sep 17 03:26:20.781351 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:26:20.781326    2552 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found
Sep 17 03:26:20.781451 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:26:20.781413    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c8cc769e-e0d1-40bd-ba4d-3ee56d0c48b4-plugin-serving-cert podName:c8cc769e-e0d1-40bd-ba4d-3ee56d0c48b4 nodeName:}" failed. No retries permitted until 2026-09-17 03:26:21.281392211 +0000 UTC m=+416.747799718 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/c8cc769e-e0d1-40bd-ba4d-3ee56d0c48b4-plugin-serving-cert") pod "kuadrant-console-plugin-fbdd4f445-bffwq" (UID: "c8cc769e-e0d1-40bd-ba4d-3ee56d0c48b4") : secret "plugin-serving-cert" not found
Sep 17 03:26:20.781957 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:26:20.781936    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c8cc769e-e0d1-40bd-ba4d-3ee56d0c48b4-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-bffwq\" (UID: \"c8cc769e-e0d1-40bd-ba4d-3ee56d0c48b4\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-bffwq"
Sep 17 03:26:20.791326 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:26:20.791300    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6z4s\" (UniqueName: \"kubernetes.io/projected/c8cc769e-e0d1-40bd-ba4d-3ee56d0c48b4-kube-api-access-q6z4s\") pod \"kuadrant-console-plugin-fbdd4f445-bffwq\" (UID: \"c8cc769e-e0d1-40bd-ba4d-3ee56d0c48b4\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-bffwq"
Sep 17 03:26:21.285285 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:26:21.285242    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/c8cc769e-e0d1-40bd-ba4d-3ee56d0c48b4-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-bffwq\" (UID: \"c8cc769e-e0d1-40bd-ba4d-3ee56d0c48b4\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-bffwq"
Sep 17 03:26:21.287740 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:26:21.287707    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/c8cc769e-e0d1-40bd-ba4d-3ee56d0c48b4-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-bffwq\" (UID: \"c8cc769e-e0d1-40bd-ba4d-3ee56d0c48b4\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-bffwq"
Sep 17 03:26:21.553082 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:26:21.552995    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-bffwq"
Sep 17 03:26:21.827200 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:26:21.827175    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-bffwq"]
Sep 17 03:26:21.829319 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:26:21.829288    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc8cc769e_e0d1_40bd_ba4d_3ee56d0c48b4.slice/crio-4fca4e29af2c145855dfe43f666278c5cd67fd7c24638bc3104ccc2a105cb55c WatchSource:0}: Error finding container 4fca4e29af2c145855dfe43f666278c5cd67fd7c24638bc3104ccc2a105cb55c: Status 404 returned error can't find the container with id 4fca4e29af2c145855dfe43f666278c5cd67fd7c24638bc3104ccc2a105cb55c
Sep 17 03:26:22.064891 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:26:22.064852    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-bffwq" event={"ID":"c8cc769e-e0d1-40bd-ba4d-3ee56d0c48b4","Type":"ContainerStarted","Data":"4fca4e29af2c145855dfe43f666278c5cd67fd7c24638bc3104ccc2a105cb55c"}
Sep 17 03:26:22.382658 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:26:22.382625    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-86d57d9c6c-xnrf4"]
Sep 17 03:26:27.100475 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:26:27.100435    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-bffwq" event={"ID":"c8cc769e-e0d1-40bd-ba4d-3ee56d0c48b4","Type":"ContainerStarted","Data":"b39d3308f0e91667a5e08b074dd9b5f492de94f2d5f044f1617a1656969337db"}
Sep 17 03:26:27.118145 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:26:27.118099    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-bffwq" podStartSLOduration=2.587616435 podStartE2EDuration="7.118083803s" totalImagesPullingTime="4.530467368s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 03:26:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:26:27.116441774 +0000 UTC m=+422.582849299" watchObservedRunningTime="2026-09-17 03:26:27.118083803 +0000 UTC m=+422.584491336"
Sep 17 03:26:31.620186 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:26:31.620156    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-hlqc8"]
Sep 17 03:26:31.623925 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:26:31.623897    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-hlqc8"
Sep 17 03:26:31.627836 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:26:31.627813    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-wvpck\""
Sep 17 03:26:31.654820 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:26:31.654791    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-hlqc8"]
Sep 17 03:26:31.681699 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:26:31.681672    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvctb\" (UniqueName: \"kubernetes.io/projected/ffd803ff-57b5-4e73-ba88-a149336f8be5-kube-api-access-jvctb\") pod \"limitador-operator-controller-manager-74495c69f8-hlqc8\" (UID: \"ffd803ff-57b5-4e73-ba88-a149336f8be5\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-hlqc8"
Sep 17 03:26:31.783094 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:26:31.783053    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvctb\" (UniqueName: \"kubernetes.io/projected/ffd803ff-57b5-4e73-ba88-a149336f8be5-kube-api-access-jvctb\") pod \"limitador-operator-controller-manager-74495c69f8-hlqc8\" (UID: \"ffd803ff-57b5-4e73-ba88-a149336f8be5\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-hlqc8"
Sep 17 03:26:31.794789 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:26:31.794760    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvctb\" (UniqueName: \"kubernetes.io/projected/ffd803ff-57b5-4e73-ba88-a149336f8be5-kube-api-access-jvctb\") pod \"limitador-operator-controller-manager-74495c69f8-hlqc8\" (UID: \"ffd803ff-57b5-4e73-ba88-a149336f8be5\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-hlqc8"
Sep 17 03:26:31.934878 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:26:31.934788    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-hlqc8"
Sep 17 03:26:32.211215 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:26:32.211190    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-hlqc8"]
Sep 17 03:26:32.214232 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:26:32.214204    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffd803ff_57b5_4e73_ba88_a149336f8be5.slice/crio-1c6f1b06f1f6c5b089c1863f242570307e80ef9fc0dd49a125ae8f17027b5445 WatchSource:0}: Error finding container 1c6f1b06f1f6c5b089c1863f242570307e80ef9fc0dd49a125ae8f17027b5445: Status 404 returned error can't find the container with id 1c6f1b06f1f6c5b089c1863f242570307e80ef9fc0dd49a125ae8f17027b5445
Sep 17 03:26:33.127354 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:26:33.127322    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-hlqc8" event={"ID":"ffd803ff-57b5-4e73-ba88-a149336f8be5","Type":"ContainerStarted","Data":"1c6f1b06f1f6c5b089c1863f242570307e80ef9fc0dd49a125ae8f17027b5445"}
Sep 17 03:26:47.402039 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:26:47.401942    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-86d57d9c6c-xnrf4" podUID="7a64c037-e935-45f7-ba20-e0c2140c5920" containerName="console" containerID="cri-o://7eeabb2d381626ee009ba9c46ec931dc1312931c867aee99e3225600117cde7c" gracePeriod=15
Sep 17 03:26:48.200817 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:26:48.200776    2552 generic.go:356] "Generic (PLEG): container finished" podID="7a64c037-e935-45f7-ba20-e0c2140c5920" containerID="7eeabb2d381626ee009ba9c46ec931dc1312931c867aee99e3225600117cde7c" exitCode=0
Sep 17 03:26:48.201000 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:26:48.200855    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-86d57d9c6c-xnrf4" event={"ID":"7a64c037-e935-45f7-ba20-e0c2140c5920","Type":"ContainerDied","Data":"7eeabb2d381626ee009ba9c46ec931dc1312931c867aee99e3225600117cde7c"}
Sep 17 03:26:48.438667 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:26:48.438640    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-86d57d9c6c-xnrf4"
Sep 17 03:26:48.530549 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:26:48.530514    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7a64c037-e935-45f7-ba20-e0c2140c5920-console-oauth-config\") pod \"7a64c037-e935-45f7-ba20-e0c2140c5920\" (UID: \"7a64c037-e935-45f7-ba20-e0c2140c5920\") "
Sep 17 03:26:48.530736 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:26:48.530568    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zrsqk\" (UniqueName: \"kubernetes.io/projected/7a64c037-e935-45f7-ba20-e0c2140c5920-kube-api-access-zrsqk\") pod \"7a64c037-e935-45f7-ba20-e0c2140c5920\" (UID: \"7a64c037-e935-45f7-ba20-e0c2140c5920\") "
Sep 17 03:26:48.530736 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:26:48.530606    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7a64c037-e935-45f7-ba20-e0c2140c5920-trusted-ca-bundle\") pod \"7a64c037-e935-45f7-ba20-e0c2140c5920\" (UID: \"7a64c037-e935-45f7-ba20-e0c2140c5920\") "
Sep 17 03:26:48.530736 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:26:48.530725    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7a64c037-e935-45f7-ba20-e0c2140c5920-oauth-serving-cert\") pod \"7a64c037-e935-45f7-ba20-e0c2140c5920\" (UID: \"7a64c037-e935-45f7-ba20-e0c2140c5920\") "
Sep 17 03:26:48.530908 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:26:48.530767    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7a64c037-e935-45f7-ba20-e0c2140c5920-console-config\") pod \"7a64c037-e935-45f7-ba20-e0c2140c5920\" (UID: \"7a64c037-e935-45f7-ba20-e0c2140c5920\") "
Sep 17 03:26:48.530908 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:26:48.530835    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7a64c037-e935-45f7-ba20-e0c2140c5920-service-ca\") pod \"7a64c037-e935-45f7-ba20-e0c2140c5920\" (UID: \"7a64c037-e935-45f7-ba20-e0c2140c5920\") "
Sep 17 03:26:48.530908 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:26:48.530877    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7a64c037-e935-45f7-ba20-e0c2140c5920-console-serving-cert\") pod \"7a64c037-e935-45f7-ba20-e0c2140c5920\" (UID: \"7a64c037-e935-45f7-ba20-e0c2140c5920\") "
Sep 17 03:26:48.531223 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:26:48.531186    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a64c037-e935-45f7-ba20-e0c2140c5920-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "7a64c037-e935-45f7-ba20-e0c2140c5920" (UID: "7a64c037-e935-45f7-ba20-e0c2140c5920"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 03:26:48.531326 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:26:48.531297    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a64c037-e935-45f7-ba20-e0c2140c5920-service-ca" (OuterVolumeSpecName: "service-ca") pod "7a64c037-e935-45f7-ba20-e0c2140c5920" (UID: "7a64c037-e935-45f7-ba20-e0c2140c5920"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 03:26:48.531389 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:26:48.531342    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a64c037-e935-45f7-ba20-e0c2140c5920-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "7a64c037-e935-45f7-ba20-e0c2140c5920" (UID: "7a64c037-e935-45f7-ba20-e0c2140c5920"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 03:26:48.531437 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:26:48.531384    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a64c037-e935-45f7-ba20-e0c2140c5920-console-config" (OuterVolumeSpecName: "console-config") pod "7a64c037-e935-45f7-ba20-e0c2140c5920" (UID: "7a64c037-e935-45f7-ba20-e0c2140c5920"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 03:26:48.532736 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:26:48.532708    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a64c037-e935-45f7-ba20-e0c2140c5920-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "7a64c037-e935-45f7-ba20-e0c2140c5920" (UID: "7a64c037-e935-45f7-ba20-e0c2140c5920"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 03:26:48.532874 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:26:48.532790    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a64c037-e935-45f7-ba20-e0c2140c5920-kube-api-access-zrsqk" (OuterVolumeSpecName: "kube-api-access-zrsqk") pod "7a64c037-e935-45f7-ba20-e0c2140c5920" (UID: "7a64c037-e935-45f7-ba20-e0c2140c5920"). InnerVolumeSpecName "kube-api-access-zrsqk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 03:26:48.533376 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:26:48.533353    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a64c037-e935-45f7-ba20-e0c2140c5920-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "7a64c037-e935-45f7-ba20-e0c2140c5920" (UID: "7a64c037-e935-45f7-ba20-e0c2140c5920"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 03:26:48.631960 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:26:48.631928    2552 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7a64c037-e935-45f7-ba20-e0c2140c5920-service-ca\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:26:48.631960 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:26:48.631957    2552 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7a64c037-e935-45f7-ba20-e0c2140c5920-console-serving-cert\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:26:48.631960 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:26:48.631966    2552 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7a64c037-e935-45f7-ba20-e0c2140c5920-console-oauth-config\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:26:48.632173 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:26:48.631975    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zrsqk\" (UniqueName: \"kubernetes.io/projected/7a64c037-e935-45f7-ba20-e0c2140c5920-kube-api-access-zrsqk\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:26:48.632173 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:26:48.631983    2552 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7a64c037-e935-45f7-ba20-e0c2140c5920-trusted-ca-bundle\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:26:48.632173 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:26:48.631994    2552 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7a64c037-e935-45f7-ba20-e0c2140c5920-oauth-serving-cert\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:26:48.632173 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:26:48.632003    2552 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7a64c037-e935-45f7-ba20-e0c2140c5920-console-config\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:26:49.207758 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:26:49.207720    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-hlqc8" event={"ID":"ffd803ff-57b5-4e73-ba88-a149336f8be5","Type":"ContainerStarted","Data":"b9d0d4a5e735493283c265585cd34996c6ba560d5e39585c84f60065a5a989c2"}
Sep 17 03:26:49.207954 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:26:49.207839    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-hlqc8"
Sep 17 03:26:49.209079 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:26:49.209060    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-86d57d9c6c-xnrf4"
Sep 17 03:26:49.209189 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:26:49.209056    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-86d57d9c6c-xnrf4" event={"ID":"7a64c037-e935-45f7-ba20-e0c2140c5920","Type":"ContainerDied","Data":"4a6d4e4eead3a021429c7e2f1e363d78dc7a9770215a2ac38561e1c80ebb4db3"}
Sep 17 03:26:49.209249 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:26:49.209201    2552 scope.go:117] "RemoveContainer" containerID="7eeabb2d381626ee009ba9c46ec931dc1312931c867aee99e3225600117cde7c"
Sep 17 03:26:49.230376 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:26:49.230323    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-hlqc8" podStartSLOduration=2.096755834 podStartE2EDuration="18.23030857s" totalImagesPullingTime="16.133552736s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 03:26:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:26:49.228958706 +0000 UTC m=+444.695366233" watchObservedRunningTime="2026-09-17 03:26:49.23030857 +0000 UTC m=+444.696716096"
Sep 17 03:26:49.242809 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:26:49.242783    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-86d57d9c6c-xnrf4"]
Sep 17 03:26:49.246639 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:26:49.246578    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-86d57d9c6c-xnrf4"]
Sep 17 03:26:50.991296 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:26:50.991264    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7a64c037-e935-45f7-ba20-e0c2140c5920" path="/var/lib/kubelet/pods/7a64c037-e935-45f7-ba20-e0c2140c5920/volumes"
Sep 17 03:27:00.216767 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:27:00.216736    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-hlqc8"
Sep 17 03:27:17.848287 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:27:17.848246    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-wt7cw"]
Sep 17 03:27:17.848711 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:27:17.848613    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a64c037-e935-45f7-ba20-e0c2140c5920" containerName="console"
Sep 17 03:27:17.848711 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:27:17.848628    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a64c037-e935-45f7-ba20-e0c2140c5920" containerName="console"
Sep 17 03:27:17.848711 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:27:17.848690    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a64c037-e935-45f7-ba20-e0c2140c5920" containerName="console"
Sep 17 03:27:17.872278 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:27:17.872237    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-wt7cw"]
Sep 17 03:27:17.872278 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:27:17.872277    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-wt7cw"
Sep 17 03:27:17.874804 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:27:17.874778    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-g5wqw\""
Sep 17 03:27:17.881651 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:27:17.881611    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcjcn\" (UniqueName: \"kubernetes.io/projected/fab0cf60-e0ab-405f-a21e-b41aaa16eccd-kube-api-access-kcjcn\") pod \"authorino-7bcfd8f9cd-wt7cw\" (UID: \"fab0cf60-e0ab-405f-a21e-b41aaa16eccd\") " pod="kuadrant-system/authorino-7bcfd8f9cd-wt7cw"
Sep 17 03:27:17.982861 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:27:17.982822    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcjcn\" (UniqueName: \"kubernetes.io/projected/fab0cf60-e0ab-405f-a21e-b41aaa16eccd-kube-api-access-kcjcn\") pod \"authorino-7bcfd8f9cd-wt7cw\" (UID: \"fab0cf60-e0ab-405f-a21e-b41aaa16eccd\") " pod="kuadrant-system/authorino-7bcfd8f9cd-wt7cw"
Sep 17 03:27:17.994280 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:27:17.994251    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcjcn\" (UniqueName: \"kubernetes.io/projected/fab0cf60-e0ab-405f-a21e-b41aaa16eccd-kube-api-access-kcjcn\") pod \"authorino-7bcfd8f9cd-wt7cw\" (UID: \"fab0cf60-e0ab-405f-a21e-b41aaa16eccd\") " pod="kuadrant-system/authorino-7bcfd8f9cd-wt7cw"
Sep 17 03:27:18.181450 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:27:18.181359    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-wt7cw"
Sep 17 03:27:18.452779 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:27:18.452754    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-wt7cw"]
Sep 17 03:27:18.454898 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:27:18.454871    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfab0cf60_e0ab_405f_a21e_b41aaa16eccd.slice/crio-3c5464f1cd4e093ef9de623475abe653cb5771f63b23d025f42de3388e9a9ab5 WatchSource:0}: Error finding container 3c5464f1cd4e093ef9de623475abe653cb5771f63b23d025f42de3388e9a9ab5: Status 404 returned error can't find the container with id 3c5464f1cd4e093ef9de623475abe653cb5771f63b23d025f42de3388e9a9ab5
Sep 17 03:27:19.348306 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:27:19.348267    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-wt7cw" event={"ID":"fab0cf60-e0ab-405f-a21e-b41aaa16eccd","Type":"ContainerStarted","Data":"3c5464f1cd4e093ef9de623475abe653cb5771f63b23d025f42de3388e9a9ab5"}
Sep 17 03:27:20.355182 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:27:20.355131    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-wt7cw" event={"ID":"fab0cf60-e0ab-405f-a21e-b41aaa16eccd","Type":"ContainerStarted","Data":"509ba1d5d54695f130d79d8ac35eb743edf17b63fcf7a4aa46fb95b09b92aa67"}
Sep 17 03:27:20.372088 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:27:20.372035    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7bcfd8f9cd-wt7cw" podStartSLOduration=1.577163553 podStartE2EDuration="3.372016064s" totalImagesPullingTime="1.794852511s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 03:27:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:27:20.370394779 +0000 UTC m=+475.836802305" watchObservedRunningTime="2026-09-17 03:27:20.372016064 +0000 UTC m=+475.838423586"
Sep 17 03:27:21.221658 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:27:21.221627    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-wt7cw"]
Sep 17 03:27:22.364720 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:27:22.364680    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7bcfd8f9cd-wt7cw" podUID="fab0cf60-e0ab-405f-a21e-b41aaa16eccd" containerName="authorino" containerID="cri-o://509ba1d5d54695f130d79d8ac35eb743edf17b63fcf7a4aa46fb95b09b92aa67" gracePeriod=30
Sep 17 03:27:22.615093 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:27:22.615037    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-wt7cw"
Sep 17 03:27:22.723859 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:27:22.723825    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kcjcn\" (UniqueName: \"kubernetes.io/projected/fab0cf60-e0ab-405f-a21e-b41aaa16eccd-kube-api-access-kcjcn\") pod \"fab0cf60-e0ab-405f-a21e-b41aaa16eccd\" (UID: \"fab0cf60-e0ab-405f-a21e-b41aaa16eccd\") "
Sep 17 03:27:22.726029 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:27:22.726003    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fab0cf60-e0ab-405f-a21e-b41aaa16eccd-kube-api-access-kcjcn" (OuterVolumeSpecName: "kube-api-access-kcjcn") pod "fab0cf60-e0ab-405f-a21e-b41aaa16eccd" (UID: "fab0cf60-e0ab-405f-a21e-b41aaa16eccd"). InnerVolumeSpecName "kube-api-access-kcjcn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 03:27:22.824879 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:27:22.824844    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kcjcn\" (UniqueName: \"kubernetes.io/projected/fab0cf60-e0ab-405f-a21e-b41aaa16eccd-kube-api-access-kcjcn\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:27:23.370310 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:27:23.370274    2552 generic.go:356] "Generic (PLEG): container finished" podID="fab0cf60-e0ab-405f-a21e-b41aaa16eccd" containerID="509ba1d5d54695f130d79d8ac35eb743edf17b63fcf7a4aa46fb95b09b92aa67" exitCode=0
Sep 17 03:27:23.370774 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:27:23.370325    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-wt7cw"
Sep 17 03:27:23.370774 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:27:23.370327    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-wt7cw" event={"ID":"fab0cf60-e0ab-405f-a21e-b41aaa16eccd","Type":"ContainerDied","Data":"509ba1d5d54695f130d79d8ac35eb743edf17b63fcf7a4aa46fb95b09b92aa67"}
Sep 17 03:27:23.370774 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:27:23.370425    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-wt7cw" event={"ID":"fab0cf60-e0ab-405f-a21e-b41aaa16eccd","Type":"ContainerDied","Data":"3c5464f1cd4e093ef9de623475abe653cb5771f63b23d025f42de3388e9a9ab5"}
Sep 17 03:27:23.370774 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:27:23.370442    2552 scope.go:117] "RemoveContainer" containerID="509ba1d5d54695f130d79d8ac35eb743edf17b63fcf7a4aa46fb95b09b92aa67"
Sep 17 03:27:23.378710 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:27:23.378691    2552 scope.go:117] "RemoveContainer" containerID="509ba1d5d54695f130d79d8ac35eb743edf17b63fcf7a4aa46fb95b09b92aa67"
Sep 17 03:27:23.378971 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:27:23.378947    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"509ba1d5d54695f130d79d8ac35eb743edf17b63fcf7a4aa46fb95b09b92aa67\": container with ID starting with 509ba1d5d54695f130d79d8ac35eb743edf17b63fcf7a4aa46fb95b09b92aa67 not found: ID does not exist" containerID="509ba1d5d54695f130d79d8ac35eb743edf17b63fcf7a4aa46fb95b09b92aa67"
Sep 17 03:27:23.379066 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:27:23.378975    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"509ba1d5d54695f130d79d8ac35eb743edf17b63fcf7a4aa46fb95b09b92aa67"} err="failed to get container status \"509ba1d5d54695f130d79d8ac35eb743edf17b63fcf7a4aa46fb95b09b92aa67\": rpc error: code = NotFound desc = could not find container \"509ba1d5d54695f130d79d8ac35eb743edf17b63fcf7a4aa46fb95b09b92aa67\": container with ID starting with 509ba1d5d54695f130d79d8ac35eb743edf17b63fcf7a4aa46fb95b09b92aa67 not found: ID does not exist"
Sep 17 03:27:23.387634 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:27:23.387591    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-wt7cw"]
Sep 17 03:27:23.391123 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:27:23.391102    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-wt7cw"]
Sep 17 03:27:24.985499 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:27:24.985470    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fab0cf60-e0ab-405f-a21e-b41aaa16eccd" path="/var/lib/kubelet/pods/fab0cf60-e0ab-405f-a21e-b41aaa16eccd/volumes"
Sep 17 03:27:45.846225 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:27:45.846192    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-557f876f69-6vwmb"]
Sep 17 03:27:45.846616 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:27:45.846561    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fab0cf60-e0ab-405f-a21e-b41aaa16eccd" containerName="authorino"
Sep 17 03:27:45.846616 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:27:45.846573    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="fab0cf60-e0ab-405f-a21e-b41aaa16eccd" containerName="authorino"
Sep 17 03:27:45.846698 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:27:45.846640    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="fab0cf60-e0ab-405f-a21e-b41aaa16eccd" containerName="authorino"
Sep 17 03:27:45.851279 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:27:45.851257    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-6vwmb"
Sep 17 03:27:45.853764 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:27:45.853746    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-g5wqw\""
Sep 17 03:27:45.856811 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:27:45.856788    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-557f876f69-6vwmb"]
Sep 17 03:27:45.942619 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:27:45.942565    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6w96\" (UniqueName: \"kubernetes.io/projected/44cd2893-9600-4715-b627-41fbbea69bbd-kube-api-access-p6w96\") pod \"authorino-557f876f69-6vwmb\" (UID: \"44cd2893-9600-4715-b627-41fbbea69bbd\") " pod="kuadrant-system/authorino-557f876f69-6vwmb"
Sep 17 03:27:46.043949 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:27:46.043911    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6w96\" (UniqueName: \"kubernetes.io/projected/44cd2893-9600-4715-b627-41fbbea69bbd-kube-api-access-p6w96\") pod \"authorino-557f876f69-6vwmb\" (UID: \"44cd2893-9600-4715-b627-41fbbea69bbd\") " pod="kuadrant-system/authorino-557f876f69-6vwmb"
Sep 17 03:27:46.054542 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:27:46.054518    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6w96\" (UniqueName: \"kubernetes.io/projected/44cd2893-9600-4715-b627-41fbbea69bbd-kube-api-access-p6w96\") pod \"authorino-557f876f69-6vwmb\" (UID: \"44cd2893-9600-4715-b627-41fbbea69bbd\") " pod="kuadrant-system/authorino-557f876f69-6vwmb"
Sep 17 03:27:46.064303 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:27:46.064276    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-6vwmb"]
Sep 17 03:27:46.064527 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:27:46.064517    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-6vwmb"
Sep 17 03:27:46.093183 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:27:46.093156    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-2v5b7"]
Sep 17 03:27:46.096744 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:27:46.096697    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-2v5b7"
Sep 17 03:27:46.099447 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:27:46.099423    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 17 03:27:46.106400 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:27:46.106379    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-2v5b7"]
Sep 17 03:27:46.130550 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:27:46.130516    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-2v5b7"]
Sep 17 03:27:46.130888 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:27:46.130852    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-tf9gl tls-cert], unattached volumes=[], failed to process volumes=[kube-api-access-tf9gl tls-cert]: context canceled" pod="kuadrant-system/authorino-6d4dfc649f-2v5b7" podUID="4fc560b2-2638-474e-9785-a9d9c4a39c2f"
Sep 17 03:27:46.158892 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:27:46.158858    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-8644847cd9-hdpzh"]
Sep 17 03:27:46.162320 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:27:46.162303    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-8644847cd9-hdpzh"
Sep 17 03:27:46.171574 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:27:46.171546    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-8644847cd9-hdpzh"]
Sep 17 03:27:46.246006 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:27:46.245963    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4fc560b2-2638-474e-9785-a9d9c4a39c2f-tls-cert\") pod \"authorino-6d4dfc649f-2v5b7\" (UID: \"4fc560b2-2638-474e-9785-a9d9c4a39c2f\") " pod="kuadrant-system/authorino-6d4dfc649f-2v5b7"
Sep 17 03:27:46.246215 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:27:46.246014    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tf9gl\" (UniqueName: \"kubernetes.io/projected/4fc560b2-2638-474e-9785-a9d9c4a39c2f-kube-api-access-tf9gl\") pod \"authorino-6d4dfc649f-2v5b7\" (UID: \"4fc560b2-2638-474e-9785-a9d9c4a39c2f\") " pod="kuadrant-system/authorino-6d4dfc649f-2v5b7"
Sep 17 03:27:46.246215 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:27:46.246042    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrsn5\" (UniqueName: \"kubernetes.io/projected/8cc3d36e-8ae9-460e-993d-64e50412767e-kube-api-access-qrsn5\") pod \"authorino-8644847cd9-hdpzh\" (UID: \"8cc3d36e-8ae9-460e-993d-64e50412767e\") " pod="kuadrant-system/authorino-8644847cd9-hdpzh"
Sep 17 03:27:46.246215 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:27:46.246102    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/8cc3d36e-8ae9-460e-993d-64e50412767e-tls-cert\") pod \"authorino-8644847cd9-hdpzh\" (UID: \"8cc3d36e-8ae9-460e-993d-64e50412767e\") " pod="kuadrant-system/authorino-8644847cd9-hdpzh"
Sep 17 03:27:46.339959 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:27:46.339885    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-6vwmb"]
Sep 17 03:27:46.342354 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:27:46.342319    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod44cd2893_9600_4715_b627_41fbbea69bbd.slice/crio-26321ce9fc08895b09dfff6516ff890008b8607db032a759e680f7aa2fca2828 WatchSource:0}: Error finding container 26321ce9fc08895b09dfff6516ff890008b8607db032a759e680f7aa2fca2828: Status 404 returned error can't find the container with id 26321ce9fc08895b09dfff6516ff890008b8607db032a759e680f7aa2fca2828
Sep 17 03:27:46.347432 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:27:46.347411    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4fc560b2-2638-474e-9785-a9d9c4a39c2f-tls-cert\") pod \"authorino-6d4dfc649f-2v5b7\" (UID: \"4fc560b2-2638-474e-9785-a9d9c4a39c2f\") " pod="kuadrant-system/authorino-6d4dfc649f-2v5b7"
Sep 17 03:27:46.347495 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:27:46.347442    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tf9gl\" (UniqueName: \"kubernetes.io/projected/4fc560b2-2638-474e-9785-a9d9c4a39c2f-kube-api-access-tf9gl\") pod \"authorino-6d4dfc649f-2v5b7\" (UID: \"4fc560b2-2638-474e-9785-a9d9c4a39c2f\") " pod="kuadrant-system/authorino-6d4dfc649f-2v5b7"
Sep 17 03:27:46.347495 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:27:46.347475    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrsn5\" (UniqueName: \"kubernetes.io/projected/8cc3d36e-8ae9-460e-993d-64e50412767e-kube-api-access-qrsn5\") pod \"authorino-8644847cd9-hdpzh\" (UID: \"8cc3d36e-8ae9-460e-993d-64e50412767e\") " pod="kuadrant-system/authorino-8644847cd9-hdpzh"
Sep 17 03:27:46.347614 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:27:46.347545    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/8cc3d36e-8ae9-460e-993d-64e50412767e-tls-cert\") pod \"authorino-8644847cd9-hdpzh\" (UID: \"8cc3d36e-8ae9-460e-993d-64e50412767e\") " pod="kuadrant-system/authorino-8644847cd9-hdpzh"
Sep 17 03:27:46.350224 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:27:46.350202    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4fc560b2-2638-474e-9785-a9d9c4a39c2f-tls-cert\") pod \"authorino-6d4dfc649f-2v5b7\" (UID: \"4fc560b2-2638-474e-9785-a9d9c4a39c2f\") " pod="kuadrant-system/authorino-6d4dfc649f-2v5b7"
Sep 17 03:27:46.350315 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:27:46.350256    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/8cc3d36e-8ae9-460e-993d-64e50412767e-tls-cert\") pod \"authorino-8644847cd9-hdpzh\" (UID: \"8cc3d36e-8ae9-460e-993d-64e50412767e\") " pod="kuadrant-system/authorino-8644847cd9-hdpzh"
Sep 17 03:27:46.354951 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:27:46.354922    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrsn5\" (UniqueName: \"kubernetes.io/projected/8cc3d36e-8ae9-460e-993d-64e50412767e-kube-api-access-qrsn5\") pod \"authorino-8644847cd9-hdpzh\" (UID: \"8cc3d36e-8ae9-460e-993d-64e50412767e\") " pod="kuadrant-system/authorino-8644847cd9-hdpzh"
Sep 17 03:27:46.355034 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:27:46.354968    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tf9gl\" (UniqueName: \"kubernetes.io/projected/4fc560b2-2638-474e-9785-a9d9c4a39c2f-kube-api-access-tf9gl\") pod \"authorino-6d4dfc649f-2v5b7\" (UID: \"4fc560b2-2638-474e-9785-a9d9c4a39c2f\") " pod="kuadrant-system/authorino-6d4dfc649f-2v5b7"
Sep 17 03:27:46.471793 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:27:46.471755    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-8644847cd9-hdpzh"
Sep 17 03:27:46.473101 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:27:46.473077    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-2v5b7"
Sep 17 03:27:46.473278 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:27:46.473097    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-557f876f69-6vwmb" event={"ID":"44cd2893-9600-4715-b627-41fbbea69bbd","Type":"ContainerStarted","Data":"26321ce9fc08895b09dfff6516ff890008b8607db032a759e680f7aa2fca2828"}
Sep 17 03:27:46.480446 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:27:46.480426    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-2v5b7"
Sep 17 03:27:46.649955 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:27:46.649869    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4fc560b2-2638-474e-9785-a9d9c4a39c2f-tls-cert\") pod \"4fc560b2-2638-474e-9785-a9d9c4a39c2f\" (UID: \"4fc560b2-2638-474e-9785-a9d9c4a39c2f\") "
Sep 17 03:27:46.649955 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:27:46.649936    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tf9gl\" (UniqueName: \"kubernetes.io/projected/4fc560b2-2638-474e-9785-a9d9c4a39c2f-kube-api-access-tf9gl\") pod \"4fc560b2-2638-474e-9785-a9d9c4a39c2f\" (UID: \"4fc560b2-2638-474e-9785-a9d9c4a39c2f\") "
Sep 17 03:27:46.652076 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:27:46.652047    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4fc560b2-2638-474e-9785-a9d9c4a39c2f-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "4fc560b2-2638-474e-9785-a9d9c4a39c2f" (UID: "4fc560b2-2638-474e-9785-a9d9c4a39c2f"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 03:27:46.652169 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:27:46.652148    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fc560b2-2638-474e-9785-a9d9c4a39c2f-kube-api-access-tf9gl" (OuterVolumeSpecName: "kube-api-access-tf9gl") pod "4fc560b2-2638-474e-9785-a9d9c4a39c2f" (UID: "4fc560b2-2638-474e-9785-a9d9c4a39c2f"). InnerVolumeSpecName "kube-api-access-tf9gl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 03:27:46.744112 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:27:46.744080    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-8644847cd9-hdpzh"]
Sep 17 03:27:46.746060 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:27:46.746038    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8cc3d36e_8ae9_460e_993d_64e50412767e.slice/crio-71e071dfa84f2e4b7b3e08b442de2ae619b2cd05728dfa3ae75a3aac582ec839 WatchSource:0}: Error finding container 71e071dfa84f2e4b7b3e08b442de2ae619b2cd05728dfa3ae75a3aac582ec839: Status 404 returned error can't find the container with id 71e071dfa84f2e4b7b3e08b442de2ae619b2cd05728dfa3ae75a3aac582ec839
Sep 17 03:27:46.750979 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:27:46.750955    2552 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4fc560b2-2638-474e-9785-a9d9c4a39c2f-tls-cert\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:27:46.750979 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:27:46.750976    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tf9gl\" (UniqueName: \"kubernetes.io/projected/4fc560b2-2638-474e-9785-a9d9c4a39c2f-kube-api-access-tf9gl\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:27:47.479153 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:27:47.479114    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-8644847cd9-hdpzh" event={"ID":"8cc3d36e-8ae9-460e-993d-64e50412767e","Type":"ContainerStarted","Data":"1401bfba757d4264cdbe2ae01bd99ab72d04cd4c4bce2e24dc0c7ee7a09bda34"}
Sep 17 03:27:47.479583 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:27:47.479161    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-8644847cd9-hdpzh" event={"ID":"8cc3d36e-8ae9-460e-993d-64e50412767e","Type":"ContainerStarted","Data":"71e071dfa84f2e4b7b3e08b442de2ae619b2cd05728dfa3ae75a3aac582ec839"}
Sep 17 03:27:47.480562 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:27:47.480541    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-2v5b7"
Sep 17 03:27:47.480677 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:27:47.480565    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-557f876f69-6vwmb" podUID="44cd2893-9600-4715-b627-41fbbea69bbd" containerName="authorino" containerID="cri-o://93baefa9c8627e189e190a3577d09ede705085f7cba673b91a579e5e26aae1d7" gracePeriod=30
Sep 17 03:27:47.480677 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:27:47.480573    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-557f876f69-6vwmb" event={"ID":"44cd2893-9600-4715-b627-41fbbea69bbd","Type":"ContainerStarted","Data":"93baefa9c8627e189e190a3577d09ede705085f7cba673b91a579e5e26aae1d7"}
Sep 17 03:27:47.495517 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:27:47.495392    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-8644847cd9-hdpzh" podStartSLOduration=1.022547669 podStartE2EDuration="1.495377924s" totalImagesPullingTime="472.830255ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 03:27:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:27:47.493074659 +0000 UTC m=+502.959482187" watchObservedRunningTime="2026-09-17 03:27:47.495377924 +0000 UTC m=+502.961785449"
Sep 17 03:27:47.506715 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:27:47.506669    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-557f876f69-6vwmb" podStartSLOduration=2.105127371 podStartE2EDuration="2.506655399s" totalImagesPullingTime="401.528028ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 03:27:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:27:47.506388285 +0000 UTC m=+502.972795827" watchObservedRunningTime="2026-09-17 03:27:47.506655399 +0000 UTC m=+502.973062925"
Sep 17 03:27:47.534789 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:27:47.534752    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-2v5b7"]
Sep 17 03:27:47.540022 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:27:47.539988    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-2v5b7"]
Sep 17 03:27:47.711674 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:27:47.711648    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-6vwmb"
Sep 17 03:27:47.861930 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:27:47.861900    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p6w96\" (UniqueName: \"kubernetes.io/projected/44cd2893-9600-4715-b627-41fbbea69bbd-kube-api-access-p6w96\") pod \"44cd2893-9600-4715-b627-41fbbea69bbd\" (UID: \"44cd2893-9600-4715-b627-41fbbea69bbd\") "
Sep 17 03:27:47.863945 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:27:47.863924    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44cd2893-9600-4715-b627-41fbbea69bbd-kube-api-access-p6w96" (OuterVolumeSpecName: "kube-api-access-p6w96") pod "44cd2893-9600-4715-b627-41fbbea69bbd" (UID: "44cd2893-9600-4715-b627-41fbbea69bbd"). InnerVolumeSpecName "kube-api-access-p6w96". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 03:27:47.963230 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:27:47.963194    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6w96\" (UniqueName: \"kubernetes.io/projected/44cd2893-9600-4715-b627-41fbbea69bbd-kube-api-access-p6w96\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:27:48.487039 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:27:48.487004    2552 generic.go:356] "Generic (PLEG): container finished" podID="44cd2893-9600-4715-b627-41fbbea69bbd" containerID="93baefa9c8627e189e190a3577d09ede705085f7cba673b91a579e5e26aae1d7" exitCode=0
Sep 17 03:27:48.487429 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:27:48.487045    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-6vwmb"
Sep 17 03:27:48.487429 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:27:48.487084    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-557f876f69-6vwmb" event={"ID":"44cd2893-9600-4715-b627-41fbbea69bbd","Type":"ContainerDied","Data":"93baefa9c8627e189e190a3577d09ede705085f7cba673b91a579e5e26aae1d7"}
Sep 17 03:27:48.487429 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:27:48.487122    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-557f876f69-6vwmb" event={"ID":"44cd2893-9600-4715-b627-41fbbea69bbd","Type":"ContainerDied","Data":"26321ce9fc08895b09dfff6516ff890008b8607db032a759e680f7aa2fca2828"}
Sep 17 03:27:48.487429 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:27:48.487139    2552 scope.go:117] "RemoveContainer" containerID="93baefa9c8627e189e190a3577d09ede705085f7cba673b91a579e5e26aae1d7"
Sep 17 03:27:48.495947 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:27:48.495931    2552 scope.go:117] "RemoveContainer" containerID="93baefa9c8627e189e190a3577d09ede705085f7cba673b91a579e5e26aae1d7"
Sep 17 03:27:48.496220 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:27:48.496202    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93baefa9c8627e189e190a3577d09ede705085f7cba673b91a579e5e26aae1d7\": container with ID starting with 93baefa9c8627e189e190a3577d09ede705085f7cba673b91a579e5e26aae1d7 not found: ID does not exist" containerID="93baefa9c8627e189e190a3577d09ede705085f7cba673b91a579e5e26aae1d7"
Sep 17 03:27:48.496286 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:27:48.496233    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93baefa9c8627e189e190a3577d09ede705085f7cba673b91a579e5e26aae1d7"} err="failed to get container status \"93baefa9c8627e189e190a3577d09ede705085f7cba673b91a579e5e26aae1d7\": rpc error: code = NotFound desc = could not find container \"93baefa9c8627e189e190a3577d09ede705085f7cba673b91a579e5e26aae1d7\": container with ID starting with 93baefa9c8627e189e190a3577d09ede705085f7cba673b91a579e5e26aae1d7 not found: ID does not exist"
Sep 17 03:27:48.508288 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:27:48.508264    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-6vwmb"]
Sep 17 03:27:48.511670 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:27:48.511651    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-557f876f69-6vwmb"]
Sep 17 03:27:48.984986 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:27:48.984951    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="44cd2893-9600-4715-b627-41fbbea69bbd" path="/var/lib/kubelet/pods/44cd2893-9600-4715-b627-41fbbea69bbd/volumes"
Sep 17 03:27:48.985337 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:27:48.985325    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4fc560b2-2638-474e-9785-a9d9c4a39c2f" path="/var/lib/kubelet/pods/4fc560b2-2638-474e-9785-a9d9c4a39c2f/volumes"
Sep 17 03:29:14.986119 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:29:14.986087    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-wwmhr"]
Sep 17 03:29:14.986532 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:29:14.986404    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44cd2893-9600-4715-b627-41fbbea69bbd" containerName="authorino"
Sep 17 03:29:14.986532 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:29:14.986415    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="44cd2893-9600-4715-b627-41fbbea69bbd" containerName="authorino"
Sep 17 03:29:14.986532 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:29:14.986499    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="44cd2893-9600-4715-b627-41fbbea69bbd" containerName="authorino"
Sep 17 03:29:14.989435 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:29:14.989420    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-wwmhr"
Sep 17 03:29:14.992404 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:29:14.992378    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 17 03:29:14.992551 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:29:14.992423    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-g74zs\""
Sep 17 03:29:14.992551 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:29:14.992378    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 17 03:29:14.993648 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:29:14.993632    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\""
Sep 17 03:29:14.997830 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:29:14.997799    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-wwmhr"]
Sep 17 03:29:15.112228 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:29:15.112193    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxkwn\" (UniqueName: \"kubernetes.io/projected/53da0826-1bd9-440c-9e83-e437508931ea-kube-api-access-zxkwn\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-wwmhr\" (UID: \"53da0826-1bd9-440c-9e83-e437508931ea\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-wwmhr"
Sep 17 03:29:15.112228 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:29:15.112231    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53da0826-1bd9-440c-9e83-e437508931ea-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-wwmhr\" (UID: \"53da0826-1bd9-440c-9e83-e437508931ea\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-wwmhr"
Sep 17 03:29:15.112460 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:29:15.112254    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/53da0826-1bd9-440c-9e83-e437508931ea-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-wwmhr\" (UID: \"53da0826-1bd9-440c-9e83-e437508931ea\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-wwmhr"
Sep 17 03:29:15.112460 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:29:15.112332    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/53da0826-1bd9-440c-9e83-e437508931ea-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-wwmhr\" (UID: \"53da0826-1bd9-440c-9e83-e437508931ea\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-wwmhr"
Sep 17 03:29:15.112460 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:29:15.112383    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/53da0826-1bd9-440c-9e83-e437508931ea-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-wwmhr\" (UID: \"53da0826-1bd9-440c-9e83-e437508931ea\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-wwmhr"
Sep 17 03:29:15.112460 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:29:15.112419    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53da0826-1bd9-440c-9e83-e437508931ea-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-wwmhr\" (UID: \"53da0826-1bd9-440c-9e83-e437508931ea\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-wwmhr"
Sep 17 03:29:15.213550 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:29:15.213503    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/53da0826-1bd9-440c-9e83-e437508931ea-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-wwmhr\" (UID: \"53da0826-1bd9-440c-9e83-e437508931ea\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-wwmhr"
Sep 17 03:29:15.213550 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:29:15.213552    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53da0826-1bd9-440c-9e83-e437508931ea-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-wwmhr\" (UID: \"53da0826-1bd9-440c-9e83-e437508931ea\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-wwmhr"
Sep 17 03:29:15.213802 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:29:15.213587    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxkwn\" (UniqueName: \"kubernetes.io/projected/53da0826-1bd9-440c-9e83-e437508931ea-kube-api-access-zxkwn\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-wwmhr\" (UID: \"53da0826-1bd9-440c-9e83-e437508931ea\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-wwmhr"
Sep 17 03:29:15.213802 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:29:15.213628    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53da0826-1bd9-440c-9e83-e437508931ea-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-wwmhr\" (UID: \"53da0826-1bd9-440c-9e83-e437508931ea\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-wwmhr"
Sep 17 03:29:15.213802 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:29:15.213678    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/53da0826-1bd9-440c-9e83-e437508931ea-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-wwmhr\" (UID: \"53da0826-1bd9-440c-9e83-e437508931ea\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-wwmhr"
Sep 17 03:29:15.213802 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:29:15.213731    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/53da0826-1bd9-440c-9e83-e437508931ea-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-wwmhr\" (UID: \"53da0826-1bd9-440c-9e83-e437508931ea\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-wwmhr"
Sep 17 03:29:15.214018 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:29:15.213904    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/53da0826-1bd9-440c-9e83-e437508931ea-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-wwmhr\" (UID: \"53da0826-1bd9-440c-9e83-e437508931ea\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-wwmhr"
Sep 17 03:29:15.214018 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:29:15.213957    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53da0826-1bd9-440c-9e83-e437508931ea-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-wwmhr\" (UID: \"53da0826-1bd9-440c-9e83-e437508931ea\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-wwmhr"
Sep 17 03:29:15.214130 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:29:15.214025    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/53da0826-1bd9-440c-9e83-e437508931ea-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-wwmhr\" (UID: \"53da0826-1bd9-440c-9e83-e437508931ea\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-wwmhr"
Sep 17 03:29:15.215970 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:29:15.215951    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/53da0826-1bd9-440c-9e83-e437508931ea-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-wwmhr\" (UID: \"53da0826-1bd9-440c-9e83-e437508931ea\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-wwmhr"
Sep 17 03:29:15.216225 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:29:15.216207    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53da0826-1bd9-440c-9e83-e437508931ea-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-wwmhr\" (UID: \"53da0826-1bd9-440c-9e83-e437508931ea\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-wwmhr"
Sep 17 03:29:15.223373 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:29:15.223352    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxkwn\" (UniqueName: \"kubernetes.io/projected/53da0826-1bd9-440c-9e83-e437508931ea-kube-api-access-zxkwn\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-wwmhr\" (UID: \"53da0826-1bd9-440c-9e83-e437508931ea\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-wwmhr"
Sep 17 03:29:15.300216 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:29:15.300182    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-wwmhr"
Sep 17 03:29:15.574552 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:29:15.574490    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-wwmhr"]
Sep 17 03:29:15.577094 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:29:15.577057    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53da0826_1bd9_440c_9e83_e437508931ea.slice/crio-a6f1953b91abc3bfdbd07d35bf6cd4092b795a7abac0c7d8c35aa4fa99d77d7e WatchSource:0}: Error finding container a6f1953b91abc3bfdbd07d35bf6cd4092b795a7abac0c7d8c35aa4fa99d77d7e: Status 404 returned error can't find the container with id a6f1953b91abc3bfdbd07d35bf6cd4092b795a7abac0c7d8c35aa4fa99d77d7e
Sep 17 03:29:15.895511 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:29:15.895427    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-wwmhr" event={"ID":"53da0826-1bd9-440c-9e83-e437508931ea","Type":"ContainerStarted","Data":"a6f1953b91abc3bfdbd07d35bf6cd4092b795a7abac0c7d8c35aa4fa99d77d7e"}
Sep 17 03:29:16.902009 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:29:16.901923    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-wwmhr" event={"ID":"53da0826-1bd9-440c-9e83-e437508931ea","Type":"ContainerStarted","Data":"283ae9a78d81c95c4d5fcaced67ae1494d8d5d103509ed8e0e4882776f692457"}
Sep 17 03:29:16.926736 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:29:16.926681    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-wwmhr" podStartSLOduration=1.97054935 podStartE2EDuration="2.926662327s" totalImagesPullingTime="956.112977ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 03:29:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:29:16.925870223 +0000 UTC m=+592.392277750" watchObservedRunningTime="2026-09-17 03:29:16.926662327 +0000 UTC m=+592.393069847"
Sep 17 03:29:25.300542 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:29:25.300493    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-wwmhr"
Sep 17 03:29:25.302919 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:29:25.300558    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-wwmhr"
Sep 17 03:29:25.313245 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:29:25.313220    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-wwmhr"
Sep 17 03:29:25.953837 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:29:25.953804    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-wwmhr"
Sep 17 03:29:48.380711 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:29:48.380621    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-4hdgk"]
Sep 17 03:29:48.385536 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:29:48.385508    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4hdgk"
Sep 17 03:29:48.387926 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:29:48.387898    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Sep 17 03:29:48.397093 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:29:48.397073    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84edcbbb-cf1d-4a6e-b7ef-d48f9a776cf7-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-4hdgk\" (UID: \"84edcbbb-cf1d-4a6e-b7ef-d48f9a776cf7\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4hdgk"
Sep 17 03:29:48.397190 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:29:48.397104    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/84edcbbb-cf1d-4a6e-b7ef-d48f9a776cf7-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-4hdgk\" (UID: \"84edcbbb-cf1d-4a6e-b7ef-d48f9a776cf7\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4hdgk"
Sep 17 03:29:48.397190 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:29:48.397120    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84edcbbb-cf1d-4a6e-b7ef-d48f9a776cf7-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-4hdgk\" (UID: \"84edcbbb-cf1d-4a6e-b7ef-d48f9a776cf7\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4hdgk"
Sep 17 03:29:48.397190 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:29:48.397178    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/84edcbbb-cf1d-4a6e-b7ef-d48f9a776cf7-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-4hdgk\" (UID: \"84edcbbb-cf1d-4a6e-b7ef-d48f9a776cf7\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4hdgk"
Sep 17 03:29:48.397311 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:29:48.397215    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n22lp\" (UniqueName: \"kubernetes.io/projected/84edcbbb-cf1d-4a6e-b7ef-d48f9a776cf7-kube-api-access-n22lp\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-4hdgk\" (UID: \"84edcbbb-cf1d-4a6e-b7ef-d48f9a776cf7\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4hdgk"
Sep 17 03:29:48.397311 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:29:48.397262    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/84edcbbb-cf1d-4a6e-b7ef-d48f9a776cf7-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-4hdgk\" (UID: \"84edcbbb-cf1d-4a6e-b7ef-d48f9a776cf7\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4hdgk"
Sep 17 03:29:48.400716 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:29:48.400694    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-4hdgk"]
Sep 17 03:29:48.497830 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:29:48.497799    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/84edcbbb-cf1d-4a6e-b7ef-d48f9a776cf7-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-4hdgk\" (UID: \"84edcbbb-cf1d-4a6e-b7ef-d48f9a776cf7\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4hdgk"
Sep 17 03:29:48.497991 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:29:48.497843    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n22lp\" (UniqueName: \"kubernetes.io/projected/84edcbbb-cf1d-4a6e-b7ef-d48f9a776cf7-kube-api-access-n22lp\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-4hdgk\" (UID: \"84edcbbb-cf1d-4a6e-b7ef-d48f9a776cf7\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4hdgk"
Sep 17 03:29:48.497991 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:29:48.497963    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/84edcbbb-cf1d-4a6e-b7ef-d48f9a776cf7-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-4hdgk\" (UID: \"84edcbbb-cf1d-4a6e-b7ef-d48f9a776cf7\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4hdgk"
Sep 17 03:29:48.498069 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:29:48.498043    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84edcbbb-cf1d-4a6e-b7ef-d48f9a776cf7-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-4hdgk\" (UID: \"84edcbbb-cf1d-4a6e-b7ef-d48f9a776cf7\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4hdgk"
Sep 17 03:29:48.498104 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:29:48.498080    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/84edcbbb-cf1d-4a6e-b7ef-d48f9a776cf7-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-4hdgk\" (UID: \"84edcbbb-cf1d-4a6e-b7ef-d48f9a776cf7\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4hdgk"
Sep 17 03:29:48.498156 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:29:48.498107    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84edcbbb-cf1d-4a6e-b7ef-d48f9a776cf7-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-4hdgk\" (UID: \"84edcbbb-cf1d-4a6e-b7ef-d48f9a776cf7\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4hdgk"
Sep 17 03:29:48.498406 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:29:48.498387    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/84edcbbb-cf1d-4a6e-b7ef-d48f9a776cf7-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-4hdgk\" (UID: \"84edcbbb-cf1d-4a6e-b7ef-d48f9a776cf7\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4hdgk"
Sep 17 03:29:48.498516 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:29:48.498433    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/84edcbbb-cf1d-4a6e-b7ef-d48f9a776cf7-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-4hdgk\" (UID: \"84edcbbb-cf1d-4a6e-b7ef-d48f9a776cf7\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4hdgk"
Sep 17 03:29:48.498516 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:29:48.498482    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84edcbbb-cf1d-4a6e-b7ef-d48f9a776cf7-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-4hdgk\" (UID: \"84edcbbb-cf1d-4a6e-b7ef-d48f9a776cf7\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4hdgk"
Sep 17 03:29:48.500166 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:29:48.500146    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/84edcbbb-cf1d-4a6e-b7ef-d48f9a776cf7-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-4hdgk\" (UID: \"84edcbbb-cf1d-4a6e-b7ef-d48f9a776cf7\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4hdgk"
Sep 17 03:29:48.500455 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:29:48.500437    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84edcbbb-cf1d-4a6e-b7ef-d48f9a776cf7-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-4hdgk\" (UID: \"84edcbbb-cf1d-4a6e-b7ef-d48f9a776cf7\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4hdgk"
Sep 17 03:29:48.510343 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:29:48.510318    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n22lp\" (UniqueName: \"kubernetes.io/projected/84edcbbb-cf1d-4a6e-b7ef-d48f9a776cf7-kube-api-access-n22lp\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-4hdgk\" (UID: \"84edcbbb-cf1d-4a6e-b7ef-d48f9a776cf7\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4hdgk"
Sep 17 03:29:48.694978 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:29:48.694878    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4hdgk"
Sep 17 03:29:48.967330 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:29:48.967304    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-4hdgk"]
Sep 17 03:29:48.969710 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:29:48.969683    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod84edcbbb_cf1d_4a6e_b7ef_d48f9a776cf7.slice/crio-b398147401500d15d0ba3a20a1ec9007634a18b13014b62742185a657e6c557c WatchSource:0}: Error finding container b398147401500d15d0ba3a20a1ec9007634a18b13014b62742185a657e6c557c: Status 404 returned error can't find the container with id b398147401500d15d0ba3a20a1ec9007634a18b13014b62742185a657e6c557c
Sep 17 03:29:48.971092 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:29:48.971073    2552 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 03:29:49.045196 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:29:49.045165    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4hdgk" event={"ID":"84edcbbb-cf1d-4a6e-b7ef-d48f9a776cf7","Type":"ContainerStarted","Data":"b398147401500d15d0ba3a20a1ec9007634a18b13014b62742185a657e6c557c"}
Sep 17 03:29:50.050811 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:29:50.050768    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4hdgk" event={"ID":"84edcbbb-cf1d-4a6e-b7ef-d48f9a776cf7","Type":"ContainerStarted","Data":"4f8c78554ee67c5cf9459c0eba5618e041d6876935a9dede472133d16a163533"}
Sep 17 03:29:50.069787 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:29:50.069740    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4hdgk" podStartSLOduration=1.884138267 podStartE2EDuration="2.06972571s" totalImagesPullingTime="185.587443ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 03:29:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:29:50.068781878 +0000 UTC m=+625.535189398" watchObservedRunningTime="2026-09-17 03:29:50.06972571 +0000 UTC m=+625.536133236"
Sep 17 03:29:58.695938 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:29:58.695902    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4hdgk"
Sep 17 03:29:58.696303 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:29:58.695962    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4hdgk"
Sep 17 03:29:58.707793 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:29:58.707763    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4hdgk"
Sep 17 03:29:59.102589 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:29:59.102562    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4hdgk"
Sep 17 03:30:00.140400 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:30:00.140369    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29826930-lfrx5"]
Sep 17 03:30:00.143753 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:30:00.143737    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826930-lfrx5"
Sep 17 03:30:00.146350 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:30:00.146330    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8vwnw\""
Sep 17 03:30:00.151716 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:30:00.151556    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29826930-lfrx5"]
Sep 17 03:30:00.155146 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:30:00.155124    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826930-lfrx5"
Sep 17 03:30:00.424388 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:30:00.424364    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29826930-lfrx5"]
Sep 17 03:30:00.426630 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:30:00.426590    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b326935_9539_47d9_903b_9f45e404c65a.slice/crio-c0356f852a8de53ae16babcd5dd1098044a1bf0d97398e7a036bc3badd44be5e WatchSource:0}: Error finding container c0356f852a8de53ae16babcd5dd1098044a1bf0d97398e7a036bc3badd44be5e: Status 404 returned error can't find the container with id c0356f852a8de53ae16babcd5dd1098044a1bf0d97398e7a036bc3badd44be5e
Sep 17 03:30:01.101975 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:30:01.101944    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826930-lfrx5" event={"ID":"3b326935-9539-47d9-903b-9f45e404c65a","Type":"ContainerStarted","Data":"c0356f852a8de53ae16babcd5dd1098044a1bf0d97398e7a036bc3badd44be5e"}
Sep 17 03:30:03.114219 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:30:03.114178    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826930-lfrx5" event={"ID":"3b326935-9539-47d9-903b-9f45e404c65a","Type":"ContainerStarted","Data":"c0b3f243c10c77639d5eac9ad6f6a3283cbb369f9433b59ad3f562334934fe4d"}
Sep 17 03:30:03.134610 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:30:03.134544    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826930-lfrx5" podStartSLOduration=0.978858866 podStartE2EDuration="3.134529598s" totalImagesPullingTime="2.155670732s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 03:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:30:03.13278992 +0000 UTC m=+638.599197446" watchObservedRunningTime="2026-09-17 03:30:03.134529598 +0000 UTC m=+638.600937124"
Sep 17 03:30:04.120731 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:30:04.120690    2552 generic.go:356] "Generic (PLEG): container finished" podID="3b326935-9539-47d9-903b-9f45e404c65a" containerID="c0b3f243c10c77639d5eac9ad6f6a3283cbb369f9433b59ad3f562334934fe4d" exitCode=0
Sep 17 03:30:04.121105 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:30:04.120781    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826930-lfrx5" event={"ID":"3b326935-9539-47d9-903b-9f45e404c65a","Type":"ContainerDied","Data":"c0b3f243c10c77639d5eac9ad6f6a3283cbb369f9433b59ad3f562334934fe4d"}
Sep 17 03:30:05.251939 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:30:05.251912    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826930-lfrx5"
Sep 17 03:30:06.130495 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:30:06.130466    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826930-lfrx5"
Sep 17 03:30:06.130495 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:30:06.130483    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826930-lfrx5" event={"ID":"3b326935-9539-47d9-903b-9f45e404c65a","Type":"ContainerDied","Data":"c0356f852a8de53ae16babcd5dd1098044a1bf0d97398e7a036bc3badd44be5e"}
Sep 17 03:30:06.130719 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:30:06.130531    2552 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c0356f852a8de53ae16babcd5dd1098044a1bf0d97398e7a036bc3badd44be5e"
Sep 17 03:31:21.816443 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:21.816406    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5999746d68-vdmx6"]
Sep 17 03:31:21.816916 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:21.816769    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b326935-9539-47d9-903b-9f45e404c65a" containerName="cleanup"
Sep 17 03:31:21.816916 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:21.816782    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b326935-9539-47d9-903b-9f45e404c65a" containerName="cleanup"
Sep 17 03:31:21.816916 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:21.816853    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b326935-9539-47d9-903b-9f45e404c65a" containerName="cleanup"
Sep 17 03:31:21.819794 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:21.819777    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5999746d68-vdmx6"
Sep 17 03:31:21.827756 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:21.827730    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5999746d68-vdmx6"]
Sep 17 03:31:21.930098 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:21.930063    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbmvd\" (UniqueName: \"kubernetes.io/projected/b968d89b-da89-4601-9590-3a48d48e7c0c-kube-api-access-lbmvd\") pod \"authorino-5999746d68-vdmx6\" (UID: \"b968d89b-da89-4601-9590-3a48d48e7c0c\") " pod="kuadrant-system/authorino-5999746d68-vdmx6"
Sep 17 03:31:21.930228 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:21.930196    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b968d89b-da89-4601-9590-3a48d48e7c0c-tls-cert\") pod \"authorino-5999746d68-vdmx6\" (UID: \"b968d89b-da89-4601-9590-3a48d48e7c0c\") " pod="kuadrant-system/authorino-5999746d68-vdmx6"
Sep 17 03:31:22.031094 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:22.031062    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b968d89b-da89-4601-9590-3a48d48e7c0c-tls-cert\") pod \"authorino-5999746d68-vdmx6\" (UID: \"b968d89b-da89-4601-9590-3a48d48e7c0c\") " pod="kuadrant-system/authorino-5999746d68-vdmx6"
Sep 17 03:31:22.031260 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:22.031102    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbmvd\" (UniqueName: \"kubernetes.io/projected/b968d89b-da89-4601-9590-3a48d48e7c0c-kube-api-access-lbmvd\") pod \"authorino-5999746d68-vdmx6\" (UID: \"b968d89b-da89-4601-9590-3a48d48e7c0c\") " pod="kuadrant-system/authorino-5999746d68-vdmx6"
Sep 17 03:31:22.033472 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:22.033436    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b968d89b-da89-4601-9590-3a48d48e7c0c-tls-cert\") pod \"authorino-5999746d68-vdmx6\" (UID: \"b968d89b-da89-4601-9590-3a48d48e7c0c\") " pod="kuadrant-system/authorino-5999746d68-vdmx6"
Sep 17 03:31:22.040280 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:22.040258    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbmvd\" (UniqueName: \"kubernetes.io/projected/b968d89b-da89-4601-9590-3a48d48e7c0c-kube-api-access-lbmvd\") pod \"authorino-5999746d68-vdmx6\" (UID: \"b968d89b-da89-4601-9590-3a48d48e7c0c\") " pod="kuadrant-system/authorino-5999746d68-vdmx6"
Sep 17 03:31:22.129234 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:22.129177    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5999746d68-vdmx6"
Sep 17 03:31:22.399451 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:22.399428    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5999746d68-vdmx6"]
Sep 17 03:31:22.402143 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:31:22.402112    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb968d89b_da89_4601_9590_3a48d48e7c0c.slice/crio-31f60fcc67803c2cfac1e5dbfef85b90d2c6ef425a63af3a313e7cf655c1050d WatchSource:0}: Error finding container 31f60fcc67803c2cfac1e5dbfef85b90d2c6ef425a63af3a313e7cf655c1050d: Status 404 returned error can't find the container with id 31f60fcc67803c2cfac1e5dbfef85b90d2c6ef425a63af3a313e7cf655c1050d
Sep 17 03:31:22.485101 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:22.485072    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5999746d68-vdmx6" event={"ID":"b968d89b-da89-4601-9590-3a48d48e7c0c","Type":"ContainerStarted","Data":"31f60fcc67803c2cfac1e5dbfef85b90d2c6ef425a63af3a313e7cf655c1050d"}
Sep 17 03:31:23.492997 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:23.492909    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5999746d68-vdmx6" event={"ID":"b968d89b-da89-4601-9590-3a48d48e7c0c","Type":"ContainerStarted","Data":"f54f84ea04ff05ff711cff5c4d7abe01724f4275c51a3d31ca1e57f76ab23ebb"}
Sep 17 03:31:23.514970 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:23.514914    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5999746d68-vdmx6" podStartSLOduration=1.79382149 podStartE2EDuration="2.514897254s" totalImagesPullingTime="721.075764ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 03:31:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:31:23.512849149 +0000 UTC m=+718.979256676" watchObservedRunningTime="2026-09-17 03:31:23.514897254 +0000 UTC m=+718.981304780"
Sep 17 03:31:23.552913 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:23.552877    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-8644847cd9-hdpzh"]
Sep 17 03:31:23.553351 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:23.553291    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-8644847cd9-hdpzh" podUID="8cc3d36e-8ae9-460e-993d-64e50412767e" containerName="authorino" containerID="cri-o://1401bfba757d4264cdbe2ae01bd99ab72d04cd4c4bce2e24dc0c7ee7a09bda34" gracePeriod=30
Sep 17 03:31:23.816558 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:23.816530    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-8644847cd9-hdpzh"
Sep 17 03:31:23.948561 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:23.948526    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/8cc3d36e-8ae9-460e-993d-64e50412767e-tls-cert\") pod \"8cc3d36e-8ae9-460e-993d-64e50412767e\" (UID: \"8cc3d36e-8ae9-460e-993d-64e50412767e\") "
Sep 17 03:31:23.948769 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:23.948668    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qrsn5\" (UniqueName: \"kubernetes.io/projected/8cc3d36e-8ae9-460e-993d-64e50412767e-kube-api-access-qrsn5\") pod \"8cc3d36e-8ae9-460e-993d-64e50412767e\" (UID: \"8cc3d36e-8ae9-460e-993d-64e50412767e\") "
Sep 17 03:31:23.951289 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:23.951261    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cc3d36e-8ae9-460e-993d-64e50412767e-kube-api-access-qrsn5" (OuterVolumeSpecName: "kube-api-access-qrsn5") pod "8cc3d36e-8ae9-460e-993d-64e50412767e" (UID: "8cc3d36e-8ae9-460e-993d-64e50412767e"). InnerVolumeSpecName "kube-api-access-qrsn5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 03:31:23.958625 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:23.958589    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8cc3d36e-8ae9-460e-993d-64e50412767e-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "8cc3d36e-8ae9-460e-993d-64e50412767e" (UID: "8cc3d36e-8ae9-460e-993d-64e50412767e"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 03:31:24.049801 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:24.049747    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrsn5\" (UniqueName: \"kubernetes.io/projected/8cc3d36e-8ae9-460e-993d-64e50412767e-kube-api-access-qrsn5\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:31:24.049801 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:24.049796    2552 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/8cc3d36e-8ae9-460e-993d-64e50412767e-tls-cert\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:31:24.500263 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:24.500231    2552 generic.go:356] "Generic (PLEG): container finished" podID="8cc3d36e-8ae9-460e-993d-64e50412767e" containerID="1401bfba757d4264cdbe2ae01bd99ab72d04cd4c4bce2e24dc0c7ee7a09bda34" exitCode=0
Sep 17 03:31:24.500710 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:24.500271    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-8644847cd9-hdpzh"
Sep 17 03:31:24.500710 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:24.500321    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-8644847cd9-hdpzh" event={"ID":"8cc3d36e-8ae9-460e-993d-64e50412767e","Type":"ContainerDied","Data":"1401bfba757d4264cdbe2ae01bd99ab72d04cd4c4bce2e24dc0c7ee7a09bda34"}
Sep 17 03:31:24.500710 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:24.500358    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-8644847cd9-hdpzh" event={"ID":"8cc3d36e-8ae9-460e-993d-64e50412767e","Type":"ContainerDied","Data":"71e071dfa84f2e4b7b3e08b442de2ae619b2cd05728dfa3ae75a3aac582ec839"}
Sep 17 03:31:24.500710 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:24.500376    2552 scope.go:117] "RemoveContainer" containerID="1401bfba757d4264cdbe2ae01bd99ab72d04cd4c4bce2e24dc0c7ee7a09bda34"
Sep 17 03:31:24.510258 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:24.510243    2552 scope.go:117] "RemoveContainer" containerID="1401bfba757d4264cdbe2ae01bd99ab72d04cd4c4bce2e24dc0c7ee7a09bda34"
Sep 17 03:31:24.510634 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:31:24.510614    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1401bfba757d4264cdbe2ae01bd99ab72d04cd4c4bce2e24dc0c7ee7a09bda34\": container with ID starting with 1401bfba757d4264cdbe2ae01bd99ab72d04cd4c4bce2e24dc0c7ee7a09bda34 not found: ID does not exist" containerID="1401bfba757d4264cdbe2ae01bd99ab72d04cd4c4bce2e24dc0c7ee7a09bda34"
Sep 17 03:31:24.510714 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:24.510657    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1401bfba757d4264cdbe2ae01bd99ab72d04cd4c4bce2e24dc0c7ee7a09bda34"} err="failed to get container status \"1401bfba757d4264cdbe2ae01bd99ab72d04cd4c4bce2e24dc0c7ee7a09bda34\": rpc error: code = NotFound desc = could not find container \"1401bfba757d4264cdbe2ae01bd99ab72d04cd4c4bce2e24dc0c7ee7a09bda34\": container with ID starting with 1401bfba757d4264cdbe2ae01bd99ab72d04cd4c4bce2e24dc0c7ee7a09bda34 not found: ID does not exist"
Sep 17 03:31:24.524106 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:24.524083    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-8644847cd9-hdpzh"]
Sep 17 03:31:24.528025 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:24.528008    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-8644847cd9-hdpzh"]
Sep 17 03:31:24.985578 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:24.985496    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8cc3d36e-8ae9-460e-993d-64e50412767e" path="/var/lib/kubelet/pods/8cc3d36e-8ae9-460e-993d-64e50412767e/volumes"
Sep 17 03:31:43.348040 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:43.348006    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7755-eb9b37"]
Sep 17 03:31:43.348497 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:43.348385    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8cc3d36e-8ae9-460e-993d-64e50412767e" containerName="authorino"
Sep 17 03:31:43.348497 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:43.348399    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="8cc3d36e-8ae9-460e-993d-64e50412767e" containerName="authorino"
Sep 17 03:31:43.348497 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:43.348480    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="8cc3d36e-8ae9-460e-993d-64e50412767e" containerName="authorino"
Sep 17 03:31:43.353006 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:43.352989    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7755-eb9b37"
Sep 17 03:31:43.355991 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:43.355968    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-s27lz\""
Sep 17 03:31:43.357137 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:43.357114    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7755-eb9b37"]
Sep 17 03:31:43.524313 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:43.524283    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wr4th\" (UniqueName: \"kubernetes.io/projected/f4eb1ba7-e426-4dc0-8cd2-43a4cf6aac05-kube-api-access-wr4th\") pod \"test-curl-7755-eb9b37\" (UID: \"f4eb1ba7-e426-4dc0-8cd2-43a4cf6aac05\") " pod="opendatahub/test-curl-7755-eb9b37"
Sep 17 03:31:43.625033 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:43.624960    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wr4th\" (UniqueName: \"kubernetes.io/projected/f4eb1ba7-e426-4dc0-8cd2-43a4cf6aac05-kube-api-access-wr4th\") pod \"test-curl-7755-eb9b37\" (UID: \"f4eb1ba7-e426-4dc0-8cd2-43a4cf6aac05\") " pod="opendatahub/test-curl-7755-eb9b37"
Sep 17 03:31:43.633569 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:43.633542    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wr4th\" (UniqueName: \"kubernetes.io/projected/f4eb1ba7-e426-4dc0-8cd2-43a4cf6aac05-kube-api-access-wr4th\") pod \"test-curl-7755-eb9b37\" (UID: \"f4eb1ba7-e426-4dc0-8cd2-43a4cf6aac05\") " pod="opendatahub/test-curl-7755-eb9b37"
Sep 17 03:31:43.662563 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:43.662535    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7755-eb9b37"
Sep 17 03:31:43.934276 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:43.934255    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7755-eb9b37"]
Sep 17 03:31:43.936546 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:31:43.936518    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf4eb1ba7_e426_4dc0_8cd2_43a4cf6aac05.slice/crio-411b6b7006081b1657f51c70855db98d0368371d131c4b1429d3ab96a402ca05 WatchSource:0}: Error finding container 411b6b7006081b1657f51c70855db98d0368371d131c4b1429d3ab96a402ca05: Status 404 returned error can't find the container with id 411b6b7006081b1657f51c70855db98d0368371d131c4b1429d3ab96a402ca05
Sep 17 03:31:44.596320 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:44.596280    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7755-eb9b37" event={"ID":"f4eb1ba7-e426-4dc0-8cd2-43a4cf6aac05","Type":"ContainerStarted","Data":"411b6b7006081b1657f51c70855db98d0368371d131c4b1429d3ab96a402ca05"}
Sep 17 03:31:47.043573 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:47.043542    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-297324-67cdb564cf-cxrsw"]
Sep 17 03:31:47.047069 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:47.047047    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-297324-67cdb564cf-cxrsw"
Sep 17 03:31:47.051347 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:47.051325    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-297324\""
Sep 17 03:31:47.051758 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:47.051739    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-297324-dockercfg-g6zhd\""
Sep 17 03:31:47.060283 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:47.060261    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-297324-67cdb564cf-cxrsw"]
Sep 17 03:31:47.157999 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:47.157971    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9h92d\" (UniqueName: \"kubernetes.io/projected/eb8c83fc-9471-4e90-88b4-b0e0de8d4803-kube-api-access-9h92d\") pod \"payload-processing-e2e-inf-a-297324-67cdb564cf-cxrsw\" (UID: \"eb8c83fc-9471-4e90-88b4-b0e0de8d4803\") " pod="openshift-ingress/payload-processing-e2e-inf-a-297324-67cdb564cf-cxrsw"
Sep 17 03:31:47.158123 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:47.158027    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb8c83fc-9471-4e90-88b4-b0e0de8d4803-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-297324-67cdb564cf-cxrsw\" (UID: \"eb8c83fc-9471-4e90-88b4-b0e0de8d4803\") " pod="openshift-ingress/payload-processing-e2e-inf-a-297324-67cdb564cf-cxrsw"
Sep 17 03:31:47.258979 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:47.258934    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9h92d\" (UniqueName: \"kubernetes.io/projected/eb8c83fc-9471-4e90-88b4-b0e0de8d4803-kube-api-access-9h92d\") pod \"payload-processing-e2e-inf-a-297324-67cdb564cf-cxrsw\" (UID: \"eb8c83fc-9471-4e90-88b4-b0e0de8d4803\") " pod="openshift-ingress/payload-processing-e2e-inf-a-297324-67cdb564cf-cxrsw"
Sep 17 03:31:47.259157 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:47.258991    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb8c83fc-9471-4e90-88b4-b0e0de8d4803-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-297324-67cdb564cf-cxrsw\" (UID: \"eb8c83fc-9471-4e90-88b4-b0e0de8d4803\") " pod="openshift-ingress/payload-processing-e2e-inf-a-297324-67cdb564cf-cxrsw"
Sep 17 03:31:47.259744 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:47.259726    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb8c83fc-9471-4e90-88b4-b0e0de8d4803-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-297324-67cdb564cf-cxrsw\" (UID: \"eb8c83fc-9471-4e90-88b4-b0e0de8d4803\") " pod="openshift-ingress/payload-processing-e2e-inf-a-297324-67cdb564cf-cxrsw"
Sep 17 03:31:47.267552 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:47.267528    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9h92d\" (UniqueName: \"kubernetes.io/projected/eb8c83fc-9471-4e90-88b4-b0e0de8d4803-kube-api-access-9h92d\") pod \"payload-processing-e2e-inf-a-297324-67cdb564cf-cxrsw\" (UID: \"eb8c83fc-9471-4e90-88b4-b0e0de8d4803\") " pod="openshift-ingress/payload-processing-e2e-inf-a-297324-67cdb564cf-cxrsw"
Sep 17 03:31:47.358028 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:47.357947    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-297324-67cdb564cf-cxrsw"
Sep 17 03:31:47.613903 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:47.613828    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7755-eb9b37" event={"ID":"f4eb1ba7-e426-4dc0-8cd2-43a4cf6aac05","Type":"ContainerStarted","Data":"623e685332affac8e8534a35c24e4dd0cd7ff1c9bc9a56996f4b7766e6dea966"}
Sep 17 03:31:47.629151 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:47.629121    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-297324-67cdb564cf-cxrsw"]
Sep 17 03:31:47.632503 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:47.632453    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7755-eb9b37" podStartSLOduration=1.484691902 podStartE2EDuration="4.632435177s" totalImagesPullingTime="3.147743275s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 03:31:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:31:47.629991835 +0000 UTC m=+743.096399360" watchObservedRunningTime="2026-09-17 03:31:47.632435177 +0000 UTC m=+743.098842705"
Sep 17 03:31:47.632651 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:31:47.632531    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb8c83fc_9471_4e90_88b4_b0e0de8d4803.slice/crio-98b127155ecc226d44f4b17d0a53fddec9b632adaf3592037ae7f5e1c350a9e3 WatchSource:0}: Error finding container 98b127155ecc226d44f4b17d0a53fddec9b632adaf3592037ae7f5e1c350a9e3: Status 404 returned error can't find the container with id 98b127155ecc226d44f4b17d0a53fddec9b632adaf3592037ae7f5e1c350a9e3
Sep 17 03:31:48.268806 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:48.267912    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7755-eb9b37"]
Sep 17 03:31:48.275475 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:48.275443    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7755-eb9b37"]
Sep 17 03:31:48.333653 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:48.333620    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-214fea-57f567459d-hq9js"]
Sep 17 03:31:48.334050 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:48.334034    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4eb1ba7-e426-4dc0-8cd2-43a4cf6aac05" containerName="test-curl-7755-eb9b37"
Sep 17 03:31:48.334115 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:48.334052    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4eb1ba7-e426-4dc0-8cd2-43a4cf6aac05" containerName="test-curl-7755-eb9b37"
Sep 17 03:31:48.334154 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:48.334144    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="f4eb1ba7-e426-4dc0-8cd2-43a4cf6aac05" containerName="test-curl-7755-eb9b37"
Sep 17 03:31:48.337350 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:48.337316    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-214fea-57f567459d-hq9js"
Sep 17 03:31:48.340259 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:48.340235    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-214fea\""
Sep 17 03:31:48.340361 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:48.340289    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-214fea-dockercfg-vll9t\""
Sep 17 03:31:48.346037 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:48.345998    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-214fea-57f567459d-hq9js"]
Sep 17 03:31:48.391337 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:48.391304    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7755-11c4d1"]
Sep 17 03:31:48.395006 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:48.394981    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7755-11c4d1"
Sep 17 03:31:48.401633 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:48.401563    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7755-11c4d1"]
Sep 17 03:31:48.470253 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:48.470217    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9a9f472-6a40-4fe0-b9ab-5cfd765aae46-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-214fea-57f567459d-hq9js\" (UID: \"e9a9f472-6a40-4fe0-b9ab-5cfd765aae46\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-214fea-57f567459d-hq9js"
Sep 17 03:31:48.470445 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:48.470288    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxdvm\" (UniqueName: \"kubernetes.io/projected/e9a9f472-6a40-4fe0-b9ab-5cfd765aae46-kube-api-access-qxdvm\") pod \"payload-pre-processing-e2e-worker-w4-214fea-57f567459d-hq9js\" (UID: \"e9a9f472-6a40-4fe0-b9ab-5cfd765aae46\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-214fea-57f567459d-hq9js"
Sep 17 03:31:48.571416 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:48.571333    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22lhj\" (UniqueName: \"kubernetes.io/projected/a8c82789-5bdc-4f78-8091-72f7951df835-kube-api-access-22lhj\") pod \"test-curl-7755-11c4d1\" (UID: \"a8c82789-5bdc-4f78-8091-72f7951df835\") " pod="opendatahub/test-curl-7755-11c4d1"
Sep 17 03:31:48.571416 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:48.571404    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9a9f472-6a40-4fe0-b9ab-5cfd765aae46-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-214fea-57f567459d-hq9js\" (UID: \"e9a9f472-6a40-4fe0-b9ab-5cfd765aae46\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-214fea-57f567459d-hq9js"
Sep 17 03:31:48.571669 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:48.571460    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxdvm\" (UniqueName: \"kubernetes.io/projected/e9a9f472-6a40-4fe0-b9ab-5cfd765aae46-kube-api-access-qxdvm\") pod \"payload-pre-processing-e2e-worker-w4-214fea-57f567459d-hq9js\" (UID: \"e9a9f472-6a40-4fe0-b9ab-5cfd765aae46\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-214fea-57f567459d-hq9js"
Sep 17 03:31:48.572402 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:48.572350    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9a9f472-6a40-4fe0-b9ab-5cfd765aae46-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-214fea-57f567459d-hq9js\" (UID: \"e9a9f472-6a40-4fe0-b9ab-5cfd765aae46\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-214fea-57f567459d-hq9js"
Sep 17 03:31:48.584239 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:48.584217    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxdvm\" (UniqueName: \"kubernetes.io/projected/e9a9f472-6a40-4fe0-b9ab-5cfd765aae46-kube-api-access-qxdvm\") pod \"payload-pre-processing-e2e-worker-w4-214fea-57f567459d-hq9js\" (UID: \"e9a9f472-6a40-4fe0-b9ab-5cfd765aae46\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-214fea-57f567459d-hq9js"
Sep 17 03:31:48.588160 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:48.588132    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-1afd34-664ddf4d68-7pnj6"]
Sep 17 03:31:48.593004 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:48.592977    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1afd34-664ddf4d68-7pnj6"
Sep 17 03:31:48.598180 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:48.598162    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-1afd34-serving-cert\""
Sep 17 03:31:48.598288 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:48.598256    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-trmz9\""
Sep 17 03:31:48.598363 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:48.598194    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 17 03:31:48.598420 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:48.598217    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 17 03:31:48.598475 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:48.598237    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 17 03:31:48.609544 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:48.609520    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-1afd34-664ddf4d68-7pnj6"]
Sep 17 03:31:48.623272 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:48.623077    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-297324-67cdb564cf-cxrsw" event={"ID":"eb8c83fc-9471-4e90-88b4-b0e0de8d4803","Type":"ContainerStarted","Data":"98b127155ecc226d44f4b17d0a53fddec9b632adaf3592037ae7f5e1c350a9e3"}
Sep 17 03:31:48.623386 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:31:48.623350    2552 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7755-eb9b37" container="test-curl-7755-eb9b37"
Sep 17 03:31:48.653338 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:48.653313    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-214fea-57f567459d-hq9js"
Sep 17 03:31:48.672577 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:48.672552    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22lhj\" (UniqueName: \"kubernetes.io/projected/a8c82789-5bdc-4f78-8091-72f7951df835-kube-api-access-22lhj\") pod \"test-curl-7755-11c4d1\" (UID: \"a8c82789-5bdc-4f78-8091-72f7951df835\") " pod="opendatahub/test-curl-7755-11c4d1"
Sep 17 03:31:48.689164 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:48.689145    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-22lhj\" (UniqueName: \"kubernetes.io/projected/a8c82789-5bdc-4f78-8091-72f7951df835-kube-api-access-22lhj\") pod \"test-curl-7755-11c4d1\" (UID: \"a8c82789-5bdc-4f78-8091-72f7951df835\") " pod="opendatahub/test-curl-7755-11c4d1"
Sep 17 03:31:48.707256 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:48.707237    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7755-11c4d1"
Sep 17 03:31:48.773517 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:48.773481    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c76c1a2-3cb7-43ae-be88-fe95001caf30-maas-api-tls\") pod \"maas-api-e2e-worker-w0-1afd34-664ddf4d68-7pnj6\" (UID: \"6c76c1a2-3cb7-43ae-be88-fe95001caf30\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1afd34-664ddf4d68-7pnj6"
Sep 17 03:31:48.773690 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:48.773531    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c76c1a2-3cb7-43ae-be88-fe95001caf30-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-1afd34-664ddf4d68-7pnj6\" (UID: \"6c76c1a2-3cb7-43ae-be88-fe95001caf30\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1afd34-664ddf4d68-7pnj6"
Sep 17 03:31:48.773690 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:48.773558    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wpbzv\" (UniqueName: \"kubernetes.io/projected/6c76c1a2-3cb7-43ae-be88-fe95001caf30-kube-api-access-wpbzv\") pod \"maas-api-e2e-worker-w0-1afd34-664ddf4d68-7pnj6\" (UID: \"6c76c1a2-3cb7-43ae-be88-fe95001caf30\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1afd34-664ddf4d68-7pnj6"
Sep 17 03:31:48.773796 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:48.773683    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c76c1a2-3cb7-43ae-be88-fe95001caf30-metrics-certs\") pod \"maas-api-e2e-worker-w0-1afd34-664ddf4d68-7pnj6\" (UID: \"6c76c1a2-3cb7-43ae-be88-fe95001caf30\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1afd34-664ddf4d68-7pnj6"
Sep 17 03:31:48.875799 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:48.875461    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c76c1a2-3cb7-43ae-be88-fe95001caf30-metrics-certs\") pod \"maas-api-e2e-worker-w0-1afd34-664ddf4d68-7pnj6\" (UID: \"6c76c1a2-3cb7-43ae-be88-fe95001caf30\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1afd34-664ddf4d68-7pnj6"
Sep 17 03:31:48.875799 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:48.875583    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c76c1a2-3cb7-43ae-be88-fe95001caf30-maas-api-tls\") pod \"maas-api-e2e-worker-w0-1afd34-664ddf4d68-7pnj6\" (UID: \"6c76c1a2-3cb7-43ae-be88-fe95001caf30\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1afd34-664ddf4d68-7pnj6"
Sep 17 03:31:48.875799 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:48.875646    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c76c1a2-3cb7-43ae-be88-fe95001caf30-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-1afd34-664ddf4d68-7pnj6\" (UID: \"6c76c1a2-3cb7-43ae-be88-fe95001caf30\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1afd34-664ddf4d68-7pnj6"
Sep 17 03:31:48.875799 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:48.875673    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpbzv\" (UniqueName: \"kubernetes.io/projected/6c76c1a2-3cb7-43ae-be88-fe95001caf30-kube-api-access-wpbzv\") pod \"maas-api-e2e-worker-w0-1afd34-664ddf4d68-7pnj6\" (UID: \"6c76c1a2-3cb7-43ae-be88-fe95001caf30\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1afd34-664ddf4d68-7pnj6"
Sep 17 03:31:48.877191 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:48.876534    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c76c1a2-3cb7-43ae-be88-fe95001caf30-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-1afd34-664ddf4d68-7pnj6\" (UID: \"6c76c1a2-3cb7-43ae-be88-fe95001caf30\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1afd34-664ddf4d68-7pnj6"
Sep 17 03:31:48.879331 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:48.879306    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c76c1a2-3cb7-43ae-be88-fe95001caf30-maas-api-tls\") pod \"maas-api-e2e-worker-w0-1afd34-664ddf4d68-7pnj6\" (UID: \"6c76c1a2-3cb7-43ae-be88-fe95001caf30\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1afd34-664ddf4d68-7pnj6"
Sep 17 03:31:48.879911 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:48.879887    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c76c1a2-3cb7-43ae-be88-fe95001caf30-metrics-certs\") pod \"maas-api-e2e-worker-w0-1afd34-664ddf4d68-7pnj6\" (UID: \"6c76c1a2-3cb7-43ae-be88-fe95001caf30\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1afd34-664ddf4d68-7pnj6"
Sep 17 03:31:48.890182 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:48.890136    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpbzv\" (UniqueName: \"kubernetes.io/projected/6c76c1a2-3cb7-43ae-be88-fe95001caf30-kube-api-access-wpbzv\") pod \"maas-api-e2e-worker-w0-1afd34-664ddf4d68-7pnj6\" (UID: \"6c76c1a2-3cb7-43ae-be88-fe95001caf30\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1afd34-664ddf4d68-7pnj6"
Sep 17 03:31:48.908281 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:48.908186    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1afd34-664ddf4d68-7pnj6"
Sep 17 03:31:49.116178 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:49.116126    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7755-11c4d1"]
Sep 17 03:31:49.118518 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:49.118482    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-214fea-57f567459d-hq9js"]
Sep 17 03:31:49.118845 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:31:49.118824    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda8c82789_5bdc_4f78_8091_72f7951df835.slice/crio-60337d102dc828a91d936cd2d0ea193d88ded97cde55a82ac595526c1b301197 WatchSource:0}: Error finding container 60337d102dc828a91d936cd2d0ea193d88ded97cde55a82ac595526c1b301197: Status 404 returned error can't find the container with id 60337d102dc828a91d936cd2d0ea193d88ded97cde55a82ac595526c1b301197
Sep 17 03:31:49.119458 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:31:49.119439    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9a9f472_6a40_4fe0_b9ab_5cfd765aae46.slice/crio-dd5fd123632484c968725d7d168c1efe32a7fb8ed71e90ceecc8b9386360eec5 WatchSource:0}: Error finding container dd5fd123632484c968725d7d168c1efe32a7fb8ed71e90ceecc8b9386360eec5: Status 404 returned error can't find the container with id dd5fd123632484c968725d7d168c1efe32a7fb8ed71e90ceecc8b9386360eec5
Sep 17 03:31:49.189052 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:49.189029    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-1afd34-664ddf4d68-7pnj6"]
Sep 17 03:31:49.191505 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:31:49.191481    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c76c1a2_3cb7_43ae_be88_fe95001caf30.slice/crio-1889e98fcbca3036159993af0d911e0efd3cdc27aad16d9c3968f050dacd79d3 WatchSource:0}: Error finding container 1889e98fcbca3036159993af0d911e0efd3cdc27aad16d9c3968f050dacd79d3: Status 404 returned error can't find the container with id 1889e98fcbca3036159993af0d911e0efd3cdc27aad16d9c3968f050dacd79d3
Sep 17 03:31:49.634550 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:49.634406    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-214fea-57f567459d-hq9js" event={"ID":"e9a9f472-6a40-4fe0-b9ab-5cfd765aae46","Type":"ContainerStarted","Data":"66eaea4bcd4bd06ce0622b2f69a8fd0c42156cb8ac72cd5b45fd30dc58c26464"}
Sep 17 03:31:49.635019 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:49.634712    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-214fea-57f567459d-hq9js" event={"ID":"e9a9f472-6a40-4fe0-b9ab-5cfd765aae46","Type":"ContainerStarted","Data":"dd5fd123632484c968725d7d168c1efe32a7fb8ed71e90ceecc8b9386360eec5"}
Sep 17 03:31:49.636772 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:49.636395    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-214fea-57f567459d-hq9js"
Sep 17 03:31:49.640980 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:49.640947    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-297324-67cdb564cf-cxrsw" event={"ID":"eb8c83fc-9471-4e90-88b4-b0e0de8d4803","Type":"ContainerStarted","Data":"aa1f6b20abd15406d4e168b02d70bfec851f3c470106afbebca888fb5255119b"}
Sep 17 03:31:49.641534 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:49.641512    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-297324-67cdb564cf-cxrsw"
Sep 17 03:31:49.644052 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:49.643995    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7755-11c4d1" event={"ID":"a8c82789-5bdc-4f78-8091-72f7951df835","Type":"ContainerStarted","Data":"76a39837f5ab207e7ace1d4c2777dce85c6d5e457e9a0ec88edba79a0c633333"}
Sep 17 03:31:49.644052 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:49.644025    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7755-11c4d1" event={"ID":"a8c82789-5bdc-4f78-8091-72f7951df835","Type":"ContainerStarted","Data":"60337d102dc828a91d936cd2d0ea193d88ded97cde55a82ac595526c1b301197"}
Sep 17 03:31:49.646093 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:49.646064    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7755-eb9b37" podUID="f4eb1ba7-e426-4dc0-8cd2-43a4cf6aac05" containerName="test-curl-7755-eb9b37" containerID="cri-o://623e685332affac8e8534a35c24e4dd0cd7ff1c9bc9a56996f4b7766e6dea966" gracePeriod=2
Sep 17 03:31:49.646358 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:49.646333    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1afd34-664ddf4d68-7pnj6" event={"ID":"6c76c1a2-3cb7-43ae-be88-fe95001caf30","Type":"ContainerStarted","Data":"1889e98fcbca3036159993af0d911e0efd3cdc27aad16d9c3968f050dacd79d3"}
Sep 17 03:31:49.657118 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:49.657071    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-214fea-57f567459d-hq9js" podStartSLOduration=1.657053782 podStartE2EDuration="1.657053782s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 03:31:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:31:49.653430383 +0000 UTC m=+745.119837904" watchObservedRunningTime="2026-09-17 03:31:49.657053782 +0000 UTC m=+745.123461309"
Sep 17 03:31:49.672845 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:49.672800    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-297324-67cdb564cf-cxrsw" podStartSLOduration=1.4141947080000001 podStartE2EDuration="2.672784537s" totalImagesPullingTime="1.258589829s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 03:31:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:31:49.670556146 +0000 UTC m=+745.136963682" watchObservedRunningTime="2026-09-17 03:31:49.672784537 +0000 UTC m=+745.139192062"
Sep 17 03:31:49.692544 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:49.691910    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7755-11c4d1" podStartSLOduration=1.6918928229999999 podStartE2EDuration="1.691892823s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 03:31:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:31:49.689838121 +0000 UTC m=+745.156245648" watchObservedRunningTime="2026-09-17 03:31:49.691892823 +0000 UTC m=+745.158300350"
Sep 17 03:31:50.314947 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:50.314916    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7755-11c4d1"]
Sep 17 03:31:50.319314 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:50.319283    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7755-11c4d1"]
Sep 17 03:31:50.651553 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:31:50.651518    2552 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7755-11c4d1" container="test-curl-7755-11c4d1"
Sep 17 03:31:50.759647 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:50.759608    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-c0b7e751-openshift-default-5c659ddbd8-sbrd4"]
Sep 17 03:31:50.761543 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:50.761508    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8c82789-5bdc-4f78-8091-72f7951df835" containerName="test-curl-7755-11c4d1"
Sep 17 03:31:50.761543 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:50.761545    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8c82789-5bdc-4f78-8091-72f7951df835" containerName="test-curl-7755-11c4d1"
Sep 17 03:31:50.764529 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:50.761830    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8c82789-5bdc-4f78-8091-72f7951df835" containerName="test-curl-7755-11c4d1"
Sep 17 03:31:50.778184 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:50.778155    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c0b7e751-openshift-default-5c659ddbd8-sbrd4"
Sep 17 03:31:50.779311 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:50.779289    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-c0b7e751-openshift-default-5c659ddbd8-sbrd4"]
Sep 17 03:31:50.781027 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:50.780801    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 17 03:31:50.781027 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:50.780830    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-c0b7e751-openshift-default-dockercfg-x2hhg\""
Sep 17 03:31:50.894080 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:50.894048    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5adcfc6-e1c3-4e82-b719-51e7fa179a5c-credential-socket\") pod \"e2e-ait-c0b7e751-openshift-default-5c659ddbd8-sbrd4\" (UID: \"e5adcfc6-e1c3-4e82-b719-51e7fa179a5c\") " pod="openshift-ingress/e2e-ait-c0b7e751-openshift-default-5c659ddbd8-sbrd4"
Sep 17 03:31:50.894416 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:50.894089    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5adcfc6-e1c3-4e82-b719-51e7fa179a5c-workload-socket\") pod \"e2e-ait-c0b7e751-openshift-default-5c659ddbd8-sbrd4\" (UID: \"e5adcfc6-e1c3-4e82-b719-51e7fa179a5c\") " pod="openshift-ingress/e2e-ait-c0b7e751-openshift-default-5c659ddbd8-sbrd4"
Sep 17 03:31:50.894416 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:50.894211    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpdtd\" (UniqueName: \"kubernetes.io/projected/e5adcfc6-e1c3-4e82-b719-51e7fa179a5c-kube-api-access-jpdtd\") pod \"e2e-ait-c0b7e751-openshift-default-5c659ddbd8-sbrd4\" (UID: \"e5adcfc6-e1c3-4e82-b719-51e7fa179a5c\") " pod="openshift-ingress/e2e-ait-c0b7e751-openshift-default-5c659ddbd8-sbrd4"
Sep 17 03:31:50.894416 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:50.894266    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5adcfc6-e1c3-4e82-b719-51e7fa179a5c-workload-certs\") pod \"e2e-ait-c0b7e751-openshift-default-5c659ddbd8-sbrd4\" (UID: \"e5adcfc6-e1c3-4e82-b719-51e7fa179a5c\") " pod="openshift-ingress/e2e-ait-c0b7e751-openshift-default-5c659ddbd8-sbrd4"
Sep 17 03:31:50.894416 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:50.894294    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5adcfc6-e1c3-4e82-b719-51e7fa179a5c-istio-envoy\") pod \"e2e-ait-c0b7e751-openshift-default-5c659ddbd8-sbrd4\" (UID: \"e5adcfc6-e1c3-4e82-b719-51e7fa179a5c\") " pod="openshift-ingress/e2e-ait-c0b7e751-openshift-default-5c659ddbd8-sbrd4"
Sep 17 03:31:50.894416 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:50.894387    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5adcfc6-e1c3-4e82-b719-51e7fa179a5c-istio-podinfo\") pod \"e2e-ait-c0b7e751-openshift-default-5c659ddbd8-sbrd4\" (UID: \"e5adcfc6-e1c3-4e82-b719-51e7fa179a5c\") " pod="openshift-ingress/e2e-ait-c0b7e751-openshift-default-5c659ddbd8-sbrd4"
Sep 17 03:31:50.894766 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:50.894503    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5adcfc6-e1c3-4e82-b719-51e7fa179a5c-istio-token\") pod \"e2e-ait-c0b7e751-openshift-default-5c659ddbd8-sbrd4\" (UID: \"e5adcfc6-e1c3-4e82-b719-51e7fa179a5c\") " pod="openshift-ingress/e2e-ait-c0b7e751-openshift-default-5c659ddbd8-sbrd4"
Sep 17 03:31:50.894766 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:50.894539    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5adcfc6-e1c3-4e82-b719-51e7fa179a5c-istiod-ca-cert\") pod \"e2e-ait-c0b7e751-openshift-default-5c659ddbd8-sbrd4\" (UID: \"e5adcfc6-e1c3-4e82-b719-51e7fa179a5c\") " pod="openshift-ingress/e2e-ait-c0b7e751-openshift-default-5c659ddbd8-sbrd4"
Sep 17 03:31:50.894766 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:50.894626    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5adcfc6-e1c3-4e82-b719-51e7fa179a5c-istio-data\") pod \"e2e-ait-c0b7e751-openshift-default-5c659ddbd8-sbrd4\" (UID: \"e5adcfc6-e1c3-4e82-b719-51e7fa179a5c\") " pod="openshift-ingress/e2e-ait-c0b7e751-openshift-default-5c659ddbd8-sbrd4"
Sep 17 03:31:50.995338 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:50.995264    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5adcfc6-e1c3-4e82-b719-51e7fa179a5c-istio-token\") pod \"e2e-ait-c0b7e751-openshift-default-5c659ddbd8-sbrd4\" (UID: \"e5adcfc6-e1c3-4e82-b719-51e7fa179a5c\") " pod="openshift-ingress/e2e-ait-c0b7e751-openshift-default-5c659ddbd8-sbrd4"
Sep 17 03:31:50.995338 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:50.995307    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5adcfc6-e1c3-4e82-b719-51e7fa179a5c-istiod-ca-cert\") pod \"e2e-ait-c0b7e751-openshift-default-5c659ddbd8-sbrd4\" (UID: \"e5adcfc6-e1c3-4e82-b719-51e7fa179a5c\") " pod="openshift-ingress/e2e-ait-c0b7e751-openshift-default-5c659ddbd8-sbrd4"
Sep 17 03:31:50.995338 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:50.995336    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5adcfc6-e1c3-4e82-b719-51e7fa179a5c-istio-data\") pod \"e2e-ait-c0b7e751-openshift-default-5c659ddbd8-sbrd4\" (UID: \"e5adcfc6-e1c3-4e82-b719-51e7fa179a5c\") " pod="openshift-ingress/e2e-ait-c0b7e751-openshift-default-5c659ddbd8-sbrd4"
Sep 17 03:31:50.995638 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:50.995374    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5adcfc6-e1c3-4e82-b719-51e7fa179a5c-credential-socket\") pod \"e2e-ait-c0b7e751-openshift-default-5c659ddbd8-sbrd4\" (UID: \"e5adcfc6-e1c3-4e82-b719-51e7fa179a5c\") " pod="openshift-ingress/e2e-ait-c0b7e751-openshift-default-5c659ddbd8-sbrd4"
Sep 17 03:31:50.995638 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:50.995393    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5adcfc6-e1c3-4e82-b719-51e7fa179a5c-workload-socket\") pod \"e2e-ait-c0b7e751-openshift-default-5c659ddbd8-sbrd4\" (UID: \"e5adcfc6-e1c3-4e82-b719-51e7fa179a5c\") " pod="openshift-ingress/e2e-ait-c0b7e751-openshift-default-5c659ddbd8-sbrd4"
Sep 17 03:31:50.995638 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:50.995426    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpdtd\" (UniqueName: \"kubernetes.io/projected/e5adcfc6-e1c3-4e82-b719-51e7fa179a5c-kube-api-access-jpdtd\") pod \"e2e-ait-c0b7e751-openshift-default-5c659ddbd8-sbrd4\" (UID: \"e5adcfc6-e1c3-4e82-b719-51e7fa179a5c\") " pod="openshift-ingress/e2e-ait-c0b7e751-openshift-default-5c659ddbd8-sbrd4"
Sep 17 03:31:50.995638 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:50.995462    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5adcfc6-e1c3-4e82-b719-51e7fa179a5c-workload-certs\") pod \"e2e-ait-c0b7e751-openshift-default-5c659ddbd8-sbrd4\" (UID: \"e5adcfc6-e1c3-4e82-b719-51e7fa179a5c\") " pod="openshift-ingress/e2e-ait-c0b7e751-openshift-default-5c659ddbd8-sbrd4"
Sep 17 03:31:50.995638 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:50.995489    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5adcfc6-e1c3-4e82-b719-51e7fa179a5c-istio-envoy\") pod \"e2e-ait-c0b7e751-openshift-default-5c659ddbd8-sbrd4\" (UID: \"e5adcfc6-e1c3-4e82-b719-51e7fa179a5c\") " pod="openshift-ingress/e2e-ait-c0b7e751-openshift-default-5c659ddbd8-sbrd4"
Sep 17 03:31:50.995638 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:50.995512    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5adcfc6-e1c3-4e82-b719-51e7fa179a5c-istio-podinfo\") pod \"e2e-ait-c0b7e751-openshift-default-5c659ddbd8-sbrd4\" (UID: \"e5adcfc6-e1c3-4e82-b719-51e7fa179a5c\") " pod="openshift-ingress/e2e-ait-c0b7e751-openshift-default-5c659ddbd8-sbrd4"
Sep 17 03:31:50.995988 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:50.995966    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5adcfc6-e1c3-4e82-b719-51e7fa179a5c-workload-socket\") pod \"e2e-ait-c0b7e751-openshift-default-5c659ddbd8-sbrd4\" (UID: \"e5adcfc6-e1c3-4e82-b719-51e7fa179a5c\") " pod="openshift-ingress/e2e-ait-c0b7e751-openshift-default-5c659ddbd8-sbrd4"
Sep 17 03:31:50.996053 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:50.996002    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5adcfc6-e1c3-4e82-b719-51e7fa179a5c-workload-certs\") pod \"e2e-ait-c0b7e751-openshift-default-5c659ddbd8-sbrd4\" (UID: \"e5adcfc6-e1c3-4e82-b719-51e7fa179a5c\") " pod="openshift-ingress/e2e-ait-c0b7e751-openshift-default-5c659ddbd8-sbrd4"
Sep 17 03:31:50.996109 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:50.996056    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5adcfc6-e1c3-4e82-b719-51e7fa179a5c-istio-data\") pod \"e2e-ait-c0b7e751-openshift-default-5c659ddbd8-sbrd4\" (UID: \"e5adcfc6-e1c3-4e82-b719-51e7fa179a5c\") " pod="openshift-ingress/e2e-ait-c0b7e751-openshift-default-5c659ddbd8-sbrd4"
Sep 17 03:31:50.996159 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:50.996149    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5adcfc6-e1c3-4e82-b719-51e7fa179a5c-istiod-ca-cert\") pod \"e2e-ait-c0b7e751-openshift-default-5c659ddbd8-sbrd4\" (UID: \"e5adcfc6-e1c3-4e82-b719-51e7fa179a5c\") " pod="openshift-ingress/e2e-ait-c0b7e751-openshift-default-5c659ddbd8-sbrd4"
Sep 17 03:31:50.996209 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:50.996189    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5adcfc6-e1c3-4e82-b719-51e7fa179a5c-credential-socket\") pod \"e2e-ait-c0b7e751-openshift-default-5c659ddbd8-sbrd4\" (UID: \"e5adcfc6-e1c3-4e82-b719-51e7fa179a5c\") " pod="openshift-ingress/e2e-ait-c0b7e751-openshift-default-5c659ddbd8-sbrd4"
Sep 17 03:31:50.998360 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:50.998330    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5adcfc6-e1c3-4e82-b719-51e7fa179a5c-istio-envoy\") pod \"e2e-ait-c0b7e751-openshift-default-5c659ddbd8-sbrd4\" (UID: \"e5adcfc6-e1c3-4e82-b719-51e7fa179a5c\") " pod="openshift-ingress/e2e-ait-c0b7e751-openshift-default-5c659ddbd8-sbrd4"
Sep 17 03:31:50.998622 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:50.998584    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5adcfc6-e1c3-4e82-b719-51e7fa179a5c-istio-podinfo\") pod \"e2e-ait-c0b7e751-openshift-default-5c659ddbd8-sbrd4\" (UID: \"e5adcfc6-e1c3-4e82-b719-51e7fa179a5c\") " pod="openshift-ingress/e2e-ait-c0b7e751-openshift-default-5c659ddbd8-sbrd4"
Sep 17 03:31:51.005356 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:51.005327    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpdtd\" (UniqueName: \"kubernetes.io/projected/e5adcfc6-e1c3-4e82-b719-51e7fa179a5c-kube-api-access-jpdtd\") pod \"e2e-ait-c0b7e751-openshift-default-5c659ddbd8-sbrd4\" (UID: \"e5adcfc6-e1c3-4e82-b719-51e7fa179a5c\") " pod="openshift-ingress/e2e-ait-c0b7e751-openshift-default-5c659ddbd8-sbrd4"
Sep 17 03:31:51.005531 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:51.005512    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5adcfc6-e1c3-4e82-b719-51e7fa179a5c-istio-token\") pod \"e2e-ait-c0b7e751-openshift-default-5c659ddbd8-sbrd4\" (UID: \"e5adcfc6-e1c3-4e82-b719-51e7fa179a5c\") " pod="openshift-ingress/e2e-ait-c0b7e751-openshift-default-5c659ddbd8-sbrd4"
Sep 17 03:31:51.093469 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:51.093438    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c0b7e751-openshift-default-5c659ddbd8-sbrd4"
Sep 17 03:31:51.659222 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:51.659190    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1afd34-664ddf4d68-7pnj6" event={"ID":"6c76c1a2-3cb7-43ae-be88-fe95001caf30","Type":"ContainerStarted","Data":"ef100d227e639097f69c90bfb47ed63c548d6253085e5172446f9bc4db772d8a"}
Sep 17 03:31:51.659587 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:51.659493    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1afd34-664ddf4d68-7pnj6"
Sep 17 03:31:51.659587 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:51.659512    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7755-11c4d1" podUID="a8c82789-5bdc-4f78-8091-72f7951df835" containerName="test-curl-7755-11c4d1" containerID="cri-o://76a39837f5ab207e7ace1d4c2777dce85c6d5e457e9a0ec88edba79a0c633333" gracePeriod=2
Sep 17 03:31:51.679122 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:51.679039    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1afd34-664ddf4d68-7pnj6" podStartSLOduration=1.387690002 podStartE2EDuration="3.679023796s" totalImagesPullingTime="2.291333794s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 03:31:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:31:51.676919011 +0000 UTC m=+747.143326538" watchObservedRunningTime="2026-09-17 03:31:51.679023796 +0000 UTC m=+747.145431371"
Sep 17 03:31:51.691246 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:51.691222    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-c0b7e751-openshift-default-5c659ddbd8-sbrd4"]
Sep 17 03:31:51.697670 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:31:51.697636    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5adcfc6_e1c3_4e82_b719_51e7fa179a5c.slice/crio-161c0cea8b705f59dbfbacf0e0940849edec8a803fe1696aab09ed1aa13d54a7 WatchSource:0}: Error finding container 161c0cea8b705f59dbfbacf0e0940849edec8a803fe1696aab09ed1aa13d54a7: Status 404 returned error can't find the container with id 161c0cea8b705f59dbfbacf0e0940849edec8a803fe1696aab09ed1aa13d54a7
Sep 17 03:31:51.780024 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:51.780000    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7755-eb9b37"
Sep 17 03:31:51.802889 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:51.802868    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wr4th\" (UniqueName: \"kubernetes.io/projected/f4eb1ba7-e426-4dc0-8cd2-43a4cf6aac05-kube-api-access-wr4th\") pod \"f4eb1ba7-e426-4dc0-8cd2-43a4cf6aac05\" (UID: \"f4eb1ba7-e426-4dc0-8cd2-43a4cf6aac05\") "
Sep 17 03:31:51.804913 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:51.804889    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4eb1ba7-e426-4dc0-8cd2-43a4cf6aac05-kube-api-access-wr4th" (OuterVolumeSpecName: "kube-api-access-wr4th") pod "f4eb1ba7-e426-4dc0-8cd2-43a4cf6aac05" (UID: "f4eb1ba7-e426-4dc0-8cd2-43a4cf6aac05"). InnerVolumeSpecName "kube-api-access-wr4th". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 03:31:51.904294 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:51.904266    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wr4th\" (UniqueName: \"kubernetes.io/projected/f4eb1ba7-e426-4dc0-8cd2-43a4cf6aac05-kube-api-access-wr4th\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:31:52.667128 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:52.667089    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c0b7e751-openshift-default-5c659ddbd8-sbrd4" event={"ID":"e5adcfc6-e1c3-4e82-b719-51e7fa179a5c","Type":"ContainerStarted","Data":"161c0cea8b705f59dbfbacf0e0940849edec8a803fe1696aab09ed1aa13d54a7"}
Sep 17 03:31:52.668429 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:52.668397    2552 generic.go:356] "Generic (PLEG): container finished" podID="f4eb1ba7-e426-4dc0-8cd2-43a4cf6aac05" containerID="623e685332affac8e8534a35c24e4dd0cd7ff1c9bc9a56996f4b7766e6dea966" exitCode=137
Sep 17 03:31:52.668555 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:52.668443    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7755-eb9b37"
Sep 17 03:31:52.668555 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:52.668514    2552 scope.go:117] "RemoveContainer" containerID="623e685332affac8e8534a35c24e4dd0cd7ff1c9bc9a56996f4b7766e6dea966"
Sep 17 03:31:52.679682 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:52.679617    2552 scope.go:117] "RemoveContainer" containerID="623e685332affac8e8534a35c24e4dd0cd7ff1c9bc9a56996f4b7766e6dea966"
Sep 17 03:31:52.679925 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:31:52.679900    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"623e685332affac8e8534a35c24e4dd0cd7ff1c9bc9a56996f4b7766e6dea966\": container with ID starting with 623e685332affac8e8534a35c24e4dd0cd7ff1c9bc9a56996f4b7766e6dea966 not found: ID does not exist" containerID="623e685332affac8e8534a35c24e4dd0cd7ff1c9bc9a56996f4b7766e6dea966"
Sep 17 03:31:52.680022 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:52.679935    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"623e685332affac8e8534a35c24e4dd0cd7ff1c9bc9a56996f4b7766e6dea966"} err="failed to get container status \"623e685332affac8e8534a35c24e4dd0cd7ff1c9bc9a56996f4b7766e6dea966\": rpc error: code = NotFound desc = could not find container \"623e685332affac8e8534a35c24e4dd0cd7ff1c9bc9a56996f4b7766e6dea966\": container with ID starting with 623e685332affac8e8534a35c24e4dd0cd7ff1c9bc9a56996f4b7766e6dea966 not found: ID does not exist"
Sep 17 03:31:52.680871 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:31:52.680843    2552 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7755-eb9b37\" is forbidden: User \"system:node:ip-10-0-139-53.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-53.ec2.internal' and this object" podUID="f4eb1ba7-e426-4dc0-8cd2-43a4cf6aac05" pod="opendatahub/test-curl-7755-eb9b37"
Sep 17 03:31:52.987885 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:52.987802    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f4eb1ba7-e426-4dc0-8cd2-43a4cf6aac05" path="/var/lib/kubelet/pods/f4eb1ba7-e426-4dc0-8cd2-43a4cf6aac05/volumes"
Sep 17 03:31:54.392208 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:54.392157    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 03:31:54.392531 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:54.392264    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 03:31:54.392531 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:54.392308    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 03:31:54.463268 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:54.463248    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7755-11c4d1"
Sep 17 03:31:54.534557 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:54.534531    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-22lhj\" (UniqueName: \"kubernetes.io/projected/a8c82789-5bdc-4f78-8091-72f7951df835-kube-api-access-22lhj\") pod \"a8c82789-5bdc-4f78-8091-72f7951df835\" (UID: \"a8c82789-5bdc-4f78-8091-72f7951df835\") "
Sep 17 03:31:54.536648 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:54.536620    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8c82789-5bdc-4f78-8091-72f7951df835-kube-api-access-22lhj" (OuterVolumeSpecName: "kube-api-access-22lhj") pod "a8c82789-5bdc-4f78-8091-72f7951df835" (UID: "a8c82789-5bdc-4f78-8091-72f7951df835"). InnerVolumeSpecName "kube-api-access-22lhj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 03:31:54.636174 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:54.636142    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-22lhj\" (UniqueName: \"kubernetes.io/projected/a8c82789-5bdc-4f78-8091-72f7951df835-kube-api-access-22lhj\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:31:54.682956 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:54.682920    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c0b7e751-openshift-default-5c659ddbd8-sbrd4" event={"ID":"e5adcfc6-e1c3-4e82-b719-51e7fa179a5c","Type":"ContainerStarted","Data":"582a4472c6ed4fc6062d650e536e32e48d9f0efbee667ee66457448fadcb2f26"}
Sep 17 03:31:54.684231 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:54.684206    2552 generic.go:356] "Generic (PLEG): container finished" podID="a8c82789-5bdc-4f78-8091-72f7951df835" containerID="76a39837f5ab207e7ace1d4c2777dce85c6d5e457e9a0ec88edba79a0c633333" exitCode=137
Sep 17 03:31:54.684337 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:54.684255    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7755-11c4d1"
Sep 17 03:31:54.684337 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:54.684267    2552 scope.go:117] "RemoveContainer" containerID="76a39837f5ab207e7ace1d4c2777dce85c6d5e457e9a0ec88edba79a0c633333"
Sep 17 03:31:54.693283 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:54.693102    2552 scope.go:117] "RemoveContainer" containerID="76a39837f5ab207e7ace1d4c2777dce85c6d5e457e9a0ec88edba79a0c633333"
Sep 17 03:31:54.693428 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:31:54.693404    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76a39837f5ab207e7ace1d4c2777dce85c6d5e457e9a0ec88edba79a0c633333\": container with ID starting with 76a39837f5ab207e7ace1d4c2777dce85c6d5e457e9a0ec88edba79a0c633333 not found: ID does not exist" containerID="76a39837f5ab207e7ace1d4c2777dce85c6d5e457e9a0ec88edba79a0c633333"
Sep 17 03:31:54.693519 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:54.693433    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76a39837f5ab207e7ace1d4c2777dce85c6d5e457e9a0ec88edba79a0c633333"} err="failed to get container status \"76a39837f5ab207e7ace1d4c2777dce85c6d5e457e9a0ec88edba79a0c633333\": rpc error: code = NotFound desc = could not find container \"76a39837f5ab207e7ace1d4c2777dce85c6d5e457e9a0ec88edba79a0c633333\": container with ID starting with 76a39837f5ab207e7ace1d4c2777dce85c6d5e457e9a0ec88edba79a0c633333 not found: ID does not exist"
Sep 17 03:31:54.706097 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:54.706024    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-c0b7e751-openshift-default-5c659ddbd8-sbrd4" podStartSLOduration=2.014130563 podStartE2EDuration="4.705986537s" totalImagesPullingTime="2.691855974s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 03:31:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:31:54.704895812 +0000 UTC m=+750.171303339" watchObservedRunningTime="2026-09-17 03:31:54.705986537 +0000 UTC m=+750.172394063"
Sep 17 03:31:54.706996 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:31:54.706971    2552 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7755-11c4d1\" is forbidden: User \"system:node:ip-10-0-139-53.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-53.ec2.internal' and this object" podUID="a8c82789-5bdc-4f78-8091-72f7951df835" pod="opendatahub/test-curl-7755-11c4d1"
Sep 17 03:31:54.983790 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:31:54.983699    2552 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7755-11c4d1\" is forbidden: User \"system:node:ip-10-0-139-53.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-53.ec2.internal' and this object" podUID="a8c82789-5bdc-4f78-8091-72f7951df835" pod="opendatahub/test-curl-7755-11c4d1"
Sep 17 03:31:54.985167 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:54.985147    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a8c82789-5bdc-4f78-8091-72f7951df835" path="/var/lib/kubelet/pods/a8c82789-5bdc-4f78-8091-72f7951df835/volumes"
Sep 17 03:31:55.094148 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:55.094126    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-c0b7e751-openshift-default-5c659ddbd8-sbrd4"
Sep 17 03:31:55.099315 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:55.099284    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-c0b7e751-openshift-default-5c659ddbd8-sbrd4"
Sep 17 03:31:55.690724 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:55.690696    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-c0b7e751-openshift-default-5c659ddbd8-sbrd4"
Sep 17 03:31:55.691584 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:55.691562    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-c0b7e751-openshift-default-5c659ddbd8-sbrd4"
Sep 17 03:31:57.674544 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:31:57.674517    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1afd34-664ddf4d68-7pnj6"
Sep 17 03:32:00.135273 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:00.135241    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-f6a774-64f58fc685-4x5ng"]
Sep 17 03:32:00.140038 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:00.140022    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-f6a774-64f58fc685-4x5ng"
Sep 17 03:32:00.142641 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:00.142575    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-f6a774-serving-cert\""
Sep 17 03:32:00.153626 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:00.153587    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-f6a774-64f58fc685-4x5ng"]
Sep 17 03:32:00.184874 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:00.184845    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/145c1a8c-a6e6-4910-98e5-58ee2b6753d3-maas-api-tls\") pod \"maas-api-e2e-autores-f6a774-64f58fc685-4x5ng\" (UID: \"145c1a8c-a6e6-4910-98e5-58ee2b6753d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-f6a774-64f58fc685-4x5ng"
Sep 17 03:32:00.184874 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:00.184876    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/145c1a8c-a6e6-4910-98e5-58ee2b6753d3-openshift-service-ca\") pod \"maas-api-e2e-autores-f6a774-64f58fc685-4x5ng\" (UID: \"145c1a8c-a6e6-4910-98e5-58ee2b6753d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-f6a774-64f58fc685-4x5ng"
Sep 17 03:32:00.185041 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:00.184987    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/145c1a8c-a6e6-4910-98e5-58ee2b6753d3-metrics-certs\") pod \"maas-api-e2e-autores-f6a774-64f58fc685-4x5ng\" (UID: \"145c1a8c-a6e6-4910-98e5-58ee2b6753d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-f6a774-64f58fc685-4x5ng"
Sep 17 03:32:00.185088 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:00.185041    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2p9jg\" (UniqueName: \"kubernetes.io/projected/145c1a8c-a6e6-4910-98e5-58ee2b6753d3-kube-api-access-2p9jg\") pod \"maas-api-e2e-autores-f6a774-64f58fc685-4x5ng\" (UID: \"145c1a8c-a6e6-4910-98e5-58ee2b6753d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-f6a774-64f58fc685-4x5ng"
Sep 17 03:32:00.285765 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:00.285727    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/145c1a8c-a6e6-4910-98e5-58ee2b6753d3-metrics-certs\") pod \"maas-api-e2e-autores-f6a774-64f58fc685-4x5ng\" (UID: \"145c1a8c-a6e6-4910-98e5-58ee2b6753d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-f6a774-64f58fc685-4x5ng"
Sep 17 03:32:00.285942 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:00.285794    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2p9jg\" (UniqueName: \"kubernetes.io/projected/145c1a8c-a6e6-4910-98e5-58ee2b6753d3-kube-api-access-2p9jg\") pod \"maas-api-e2e-autores-f6a774-64f58fc685-4x5ng\" (UID: \"145c1a8c-a6e6-4910-98e5-58ee2b6753d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-f6a774-64f58fc685-4x5ng"
Sep 17 03:32:00.285942 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:00.285845    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/145c1a8c-a6e6-4910-98e5-58ee2b6753d3-maas-api-tls\") pod \"maas-api-e2e-autores-f6a774-64f58fc685-4x5ng\" (UID: \"145c1a8c-a6e6-4910-98e5-58ee2b6753d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-f6a774-64f58fc685-4x5ng"
Sep 17 03:32:00.285942 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:00.285870    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/145c1a8c-a6e6-4910-98e5-58ee2b6753d3-openshift-service-ca\") pod \"maas-api-e2e-autores-f6a774-64f58fc685-4x5ng\" (UID: \"145c1a8c-a6e6-4910-98e5-58ee2b6753d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-f6a774-64f58fc685-4x5ng"
Sep 17 03:32:00.286114 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:32:00.285993    2552 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-autores-f6a774-serving-cert: secret "maas-api-e2e-autores-f6a774-serving-cert" not found
Sep 17 03:32:00.286114 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:32:00.286070    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/145c1a8c-a6e6-4910-98e5-58ee2b6753d3-maas-api-tls podName:145c1a8c-a6e6-4910-98e5-58ee2b6753d3 nodeName:}" failed. No retries permitted until 2026-09-17 03:32:00.786050051 +0000 UTC m=+756.252457557 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/145c1a8c-a6e6-4910-98e5-58ee2b6753d3-maas-api-tls") pod "maas-api-e2e-autores-f6a774-64f58fc685-4x5ng" (UID: "145c1a8c-a6e6-4910-98e5-58ee2b6753d3") : secret "maas-api-e2e-autores-f6a774-serving-cert" not found
Sep 17 03:32:00.286580 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:00.286558    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/145c1a8c-a6e6-4910-98e5-58ee2b6753d3-openshift-service-ca\") pod \"maas-api-e2e-autores-f6a774-64f58fc685-4x5ng\" (UID: \"145c1a8c-a6e6-4910-98e5-58ee2b6753d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-f6a774-64f58fc685-4x5ng"
Sep 17 03:32:00.288321 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:00.288298    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/145c1a8c-a6e6-4910-98e5-58ee2b6753d3-metrics-certs\") pod \"maas-api-e2e-autores-f6a774-64f58fc685-4x5ng\" (UID: \"145c1a8c-a6e6-4910-98e5-58ee2b6753d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-f6a774-64f58fc685-4x5ng"
Sep 17 03:32:00.302207 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:00.302186    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2p9jg\" (UniqueName: \"kubernetes.io/projected/145c1a8c-a6e6-4910-98e5-58ee2b6753d3-kube-api-access-2p9jg\") pod \"maas-api-e2e-autores-f6a774-64f58fc685-4x5ng\" (UID: \"145c1a8c-a6e6-4910-98e5-58ee2b6753d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-f6a774-64f58fc685-4x5ng"
Sep 17 03:32:00.653249 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:00.653218    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-214fea-57f567459d-hq9js"
Sep 17 03:32:00.790907 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:00.790874    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/145c1a8c-a6e6-4910-98e5-58ee2b6753d3-maas-api-tls\") pod \"maas-api-e2e-autores-f6a774-64f58fc685-4x5ng\" (UID: \"145c1a8c-a6e6-4910-98e5-58ee2b6753d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-f6a774-64f58fc685-4x5ng"
Sep 17 03:32:00.793234 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:00.793215    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/145c1a8c-a6e6-4910-98e5-58ee2b6753d3-maas-api-tls\") pod \"maas-api-e2e-autores-f6a774-64f58fc685-4x5ng\" (UID: \"145c1a8c-a6e6-4910-98e5-58ee2b6753d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-f6a774-64f58fc685-4x5ng"
Sep 17 03:32:01.050650 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:01.050582    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-f6a774-64f58fc685-4x5ng"
Sep 17 03:32:01.329181 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:01.329157    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-f6a774-64f58fc685-4x5ng"]
Sep 17 03:32:01.331390 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:32:01.331364    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod145c1a8c_a6e6_4910_98e5_58ee2b6753d3.slice/crio-83eee38a215003031949ed8013339dd7e03dbe71301bad6b8f4f5d32effd227b WatchSource:0}: Error finding container 83eee38a215003031949ed8013339dd7e03dbe71301bad6b8f4f5d32effd227b: Status 404 returned error can't find the container with id 83eee38a215003031949ed8013339dd7e03dbe71301bad6b8f4f5d32effd227b
Sep 17 03:32:01.394685 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:01.394654    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c0b7e751-7d6c94f4f5-bvj9m"]
Sep 17 03:32:01.399686 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:01.399662    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c0b7e751-7d6c94f4f5-bvj9m"
Sep 17 03:32:01.402539 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:01.402518    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-c0b7e751-serving-cert\""
Sep 17 03:32:01.411414 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:01.411354    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c0b7e751-7d6c94f4f5-bvj9m"]
Sep 17 03:32:01.497249 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:01.497219    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef222c1b-f104-4575-aded-35dd5711cd91-metrics-certs\") pod \"maas-api-e2e-ait-c0b7e751-7d6c94f4f5-bvj9m\" (UID: \"ef222c1b-f104-4575-aded-35dd5711cd91\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c0b7e751-7d6c94f4f5-bvj9m"
Sep 17 03:32:01.497432 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:01.497255    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef222c1b-f104-4575-aded-35dd5711cd91-maas-api-tls\") pod \"maas-api-e2e-ait-c0b7e751-7d6c94f4f5-bvj9m\" (UID: \"ef222c1b-f104-4575-aded-35dd5711cd91\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c0b7e751-7d6c94f4f5-bvj9m"
Sep 17 03:32:01.497432 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:01.497275    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef222c1b-f104-4575-aded-35dd5711cd91-openshift-service-ca\") pod \"maas-api-e2e-ait-c0b7e751-7d6c94f4f5-bvj9m\" (UID: \"ef222c1b-f104-4575-aded-35dd5711cd91\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c0b7e751-7d6c94f4f5-bvj9m"
Sep 17 03:32:01.497432 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:01.497383    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxz9v\" (UniqueName: \"kubernetes.io/projected/ef222c1b-f104-4575-aded-35dd5711cd91-kube-api-access-dxz9v\") pod \"maas-api-e2e-ait-c0b7e751-7d6c94f4f5-bvj9m\" (UID: \"ef222c1b-f104-4575-aded-35dd5711cd91\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c0b7e751-7d6c94f4f5-bvj9m"
Sep 17 03:32:01.598409 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:01.598312    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef222c1b-f104-4575-aded-35dd5711cd91-metrics-certs\") pod \"maas-api-e2e-ait-c0b7e751-7d6c94f4f5-bvj9m\" (UID: \"ef222c1b-f104-4575-aded-35dd5711cd91\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c0b7e751-7d6c94f4f5-bvj9m"
Sep 17 03:32:01.598409 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:01.598364    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef222c1b-f104-4575-aded-35dd5711cd91-maas-api-tls\") pod \"maas-api-e2e-ait-c0b7e751-7d6c94f4f5-bvj9m\" (UID: \"ef222c1b-f104-4575-aded-35dd5711cd91\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c0b7e751-7d6c94f4f5-bvj9m"
Sep 17 03:32:01.598409 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:01.598391    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef222c1b-f104-4575-aded-35dd5711cd91-openshift-service-ca\") pod \"maas-api-e2e-ait-c0b7e751-7d6c94f4f5-bvj9m\" (UID: \"ef222c1b-f104-4575-aded-35dd5711cd91\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c0b7e751-7d6c94f4f5-bvj9m"
Sep 17 03:32:01.598740 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:01.598456    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxz9v\" (UniqueName: \"kubernetes.io/projected/ef222c1b-f104-4575-aded-35dd5711cd91-kube-api-access-dxz9v\") pod \"maas-api-e2e-ait-c0b7e751-7d6c94f4f5-bvj9m\" (UID: \"ef222c1b-f104-4575-aded-35dd5711cd91\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c0b7e751-7d6c94f4f5-bvj9m"
Sep 17 03:32:01.598740 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:32:01.598511    2552 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-c0b7e751-serving-cert: secret "maas-api-e2e-ait-c0b7e751-serving-cert" not found
Sep 17 03:32:01.598740 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:32:01.598574    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef222c1b-f104-4575-aded-35dd5711cd91-maas-api-tls podName:ef222c1b-f104-4575-aded-35dd5711cd91 nodeName:}" failed. No retries permitted until 2026-09-17 03:32:02.098558253 +0000 UTC m=+757.564965757 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ef222c1b-f104-4575-aded-35dd5711cd91-maas-api-tls") pod "maas-api-e2e-ait-c0b7e751-7d6c94f4f5-bvj9m" (UID: "ef222c1b-f104-4575-aded-35dd5711cd91") : secret "maas-api-e2e-ait-c0b7e751-serving-cert" not found
Sep 17 03:32:01.599108 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:01.599089    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef222c1b-f104-4575-aded-35dd5711cd91-openshift-service-ca\") pod \"maas-api-e2e-ait-c0b7e751-7d6c94f4f5-bvj9m\" (UID: \"ef222c1b-f104-4575-aded-35dd5711cd91\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c0b7e751-7d6c94f4f5-bvj9m"
Sep 17 03:32:01.600804 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:01.600780    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef222c1b-f104-4575-aded-35dd5711cd91-metrics-certs\") pod \"maas-api-e2e-ait-c0b7e751-7d6c94f4f5-bvj9m\" (UID: \"ef222c1b-f104-4575-aded-35dd5711cd91\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c0b7e751-7d6c94f4f5-bvj9m"
Sep 17 03:32:01.620327 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:01.620302    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxz9v\" (UniqueName: \"kubernetes.io/projected/ef222c1b-f104-4575-aded-35dd5711cd91-kube-api-access-dxz9v\") pod \"maas-api-e2e-ait-c0b7e751-7d6c94f4f5-bvj9m\" (UID: \"ef222c1b-f104-4575-aded-35dd5711cd91\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c0b7e751-7d6c94f4f5-bvj9m"
Sep 17 03:32:01.661054 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:01.661027    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-297324-67cdb564cf-cxrsw"
Sep 17 03:32:01.723610 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:01.723564    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-f6a774-64f58fc685-4x5ng" event={"ID":"145c1a8c-a6e6-4910-98e5-58ee2b6753d3","Type":"ContainerStarted","Data":"83eee38a215003031949ed8013339dd7e03dbe71301bad6b8f4f5d32effd227b"}
Sep 17 03:32:02.104339 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:02.104301    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef222c1b-f104-4575-aded-35dd5711cd91-maas-api-tls\") pod \"maas-api-e2e-ait-c0b7e751-7d6c94f4f5-bvj9m\" (UID: \"ef222c1b-f104-4575-aded-35dd5711cd91\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c0b7e751-7d6c94f4f5-bvj9m"
Sep 17 03:32:02.104502 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:32:02.104444    2552 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-c0b7e751-serving-cert: secret "maas-api-e2e-ait-c0b7e751-serving-cert" not found
Sep 17 03:32:02.104546 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:32:02.104509    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef222c1b-f104-4575-aded-35dd5711cd91-maas-api-tls podName:ef222c1b-f104-4575-aded-35dd5711cd91 nodeName:}" failed. No retries permitted until 2026-09-17 03:32:03.104493148 +0000 UTC m=+758.570900652 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ef222c1b-f104-4575-aded-35dd5711cd91-maas-api-tls") pod "maas-api-e2e-ait-c0b7e751-7d6c94f4f5-bvj9m" (UID: "ef222c1b-f104-4575-aded-35dd5711cd91") : secret "maas-api-e2e-ait-c0b7e751-serving-cert" not found
Sep 17 03:32:02.731287 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:02.731253    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-f6a774-64f58fc685-4x5ng" event={"ID":"145c1a8c-a6e6-4910-98e5-58ee2b6753d3","Type":"ContainerStarted","Data":"8e05b4f624230081f71f947face151188a8c3e37cab72334cd359c4331a2f926"}
Sep 17 03:32:02.731682 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:02.731368    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-f6a774-64f58fc685-4x5ng"
Sep 17 03:32:02.748947 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:02.748902    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-f6a774-64f58fc685-4x5ng" podStartSLOduration=2.374698924 podStartE2EDuration="2.748887004s" totalImagesPullingTime="374.18808ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 03:32:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:32:02.748005774 +0000 UTC m=+758.214413302" watchObservedRunningTime="2026-09-17 03:32:02.748887004 +0000 UTC m=+758.215294529"
Sep 17 03:32:03.114374 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:03.114330    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef222c1b-f104-4575-aded-35dd5711cd91-maas-api-tls\") pod \"maas-api-e2e-ait-c0b7e751-7d6c94f4f5-bvj9m\" (UID: \"ef222c1b-f104-4575-aded-35dd5711cd91\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c0b7e751-7d6c94f4f5-bvj9m"
Sep 17 03:32:03.116829 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:03.116804    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef222c1b-f104-4575-aded-35dd5711cd91-maas-api-tls\") pod \"maas-api-e2e-ait-c0b7e751-7d6c94f4f5-bvj9m\" (UID: \"ef222c1b-f104-4575-aded-35dd5711cd91\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c0b7e751-7d6c94f4f5-bvj9m"
Sep 17 03:32:03.213176 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:03.213146    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c0b7e751-7d6c94f4f5-bvj9m"
Sep 17 03:32:03.488905 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:03.488880    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c0b7e751-7d6c94f4f5-bvj9m"]
Sep 17 03:32:03.490193 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:32:03.490169    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef222c1b_f104_4575_aded_35dd5711cd91.slice/crio-2d5861af3fd0ebd5c4d3c331e4f832e7ac90817912f75596f106a8b2a14dcb0a WatchSource:0}: Error finding container 2d5861af3fd0ebd5c4d3c331e4f832e7ac90817912f75596f106a8b2a14dcb0a: Status 404 returned error can't find the container with id 2d5861af3fd0ebd5c4d3c331e4f832e7ac90817912f75596f106a8b2a14dcb0a
Sep 17 03:32:03.738614 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:03.738513    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c0b7e751-7d6c94f4f5-bvj9m" event={"ID":"ef222c1b-f104-4575-aded-35dd5711cd91","Type":"ContainerStarted","Data":"2d5861af3fd0ebd5c4d3c331e4f832e7ac90817912f75596f106a8b2a14dcb0a"}
Sep 17 03:32:04.751073 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:04.751038    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c0b7e751-7d6c94f4f5-bvj9m" event={"ID":"ef222c1b-f104-4575-aded-35dd5711cd91","Type":"ContainerStarted","Data":"008293a57db3f783699ef9bc7aa4f4ec43c88c6364667f2fea54a2ed2d4f1b8a"}
Sep 17 03:32:04.751501 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:04.751178    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c0b7e751-7d6c94f4f5-bvj9m"
Sep 17 03:32:04.776556 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:04.776506    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c0b7e751-7d6c94f4f5-bvj9m" podStartSLOduration=3.374155299 podStartE2EDuration="3.776490332s" totalImagesPullingTime="402.335033ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 03:32:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:32:04.774735758 +0000 UTC m=+760.241143286" watchObservedRunningTime="2026-09-17 03:32:04.776490332 +0000 UTC m=+760.242897858"
Sep 17 03:32:08.744101 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:08.744074    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-f6a774-64f58fc685-4x5ng"
Sep 17 03:32:10.360657 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:10.360622    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-3d0c3c-openshift-default-7bb86f7b56-zbt2v"]
Sep 17 03:32:10.364573 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:10.364548    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-3d0c3c-openshift-default-7bb86f7b56-zbt2v"
Sep 17 03:32:10.368260 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:10.368240    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-3d0c3c-openshift-default-dockercfg-rdwpn\""
Sep 17 03:32:10.380732 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:10.380691    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-3d0c3c-openshift-default-7bb86f7b56-zbt2v"]
Sep 17 03:32:10.482534 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:10.482499    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b5b7160-05b5-4218-ab7b-0e7cdf8383a2-workload-socket\") pod \"e2e-inf-b-3d0c3c-openshift-default-7bb86f7b56-zbt2v\" (UID: \"8b5b7160-05b5-4218-ab7b-0e7cdf8383a2\") " pod="openshift-ingress/e2e-inf-b-3d0c3c-openshift-default-7bb86f7b56-zbt2v"
Sep 17 03:32:10.482534 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:10.482538    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b5b7160-05b5-4218-ab7b-0e7cdf8383a2-istio-data\") pod \"e2e-inf-b-3d0c3c-openshift-default-7bb86f7b56-zbt2v\" (UID: \"8b5b7160-05b5-4218-ab7b-0e7cdf8383a2\") " pod="openshift-ingress/e2e-inf-b-3d0c3c-openshift-default-7bb86f7b56-zbt2v"
Sep 17 03:32:10.482763 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:10.482662    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b5b7160-05b5-4218-ab7b-0e7cdf8383a2-credential-socket\") pod \"e2e-inf-b-3d0c3c-openshift-default-7bb86f7b56-zbt2v\" (UID: \"8b5b7160-05b5-4218-ab7b-0e7cdf8383a2\") " pod="openshift-ingress/e2e-inf-b-3d0c3c-openshift-default-7bb86f7b56-zbt2v"
Sep 17 03:32:10.482763 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:10.482692    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b5b7160-05b5-4218-ab7b-0e7cdf8383a2-istio-envoy\") pod \"e2e-inf-b-3d0c3c-openshift-default-7bb86f7b56-zbt2v\" (UID: \"8b5b7160-05b5-4218-ab7b-0e7cdf8383a2\") " pod="openshift-ingress/e2e-inf-b-3d0c3c-openshift-default-7bb86f7b56-zbt2v"
Sep 17 03:32:10.482763 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:10.482718    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b5b7160-05b5-4218-ab7b-0e7cdf8383a2-istiod-ca-cert\") pod \"e2e-inf-b-3d0c3c-openshift-default-7bb86f7b56-zbt2v\" (UID: \"8b5b7160-05b5-4218-ab7b-0e7cdf8383a2\") " pod="openshift-ingress/e2e-inf-b-3d0c3c-openshift-default-7bb86f7b56-zbt2v"
Sep 17 03:32:10.482869 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:10.482799    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scppn\" (UniqueName: \"kubernetes.io/projected/8b5b7160-05b5-4218-ab7b-0e7cdf8383a2-kube-api-access-scppn\") pod \"e2e-inf-b-3d0c3c-openshift-default-7bb86f7b56-zbt2v\" (UID: \"8b5b7160-05b5-4218-ab7b-0e7cdf8383a2\") " pod="openshift-ingress/e2e-inf-b-3d0c3c-openshift-default-7bb86f7b56-zbt2v"
Sep 17 03:32:10.482869 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:10.482832    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b5b7160-05b5-4218-ab7b-0e7cdf8383a2-istio-podinfo\") pod \"e2e-inf-b-3d0c3c-openshift-default-7bb86f7b56-zbt2v\" (UID: \"8b5b7160-05b5-4218-ab7b-0e7cdf8383a2\") " pod="openshift-ingress/e2e-inf-b-3d0c3c-openshift-default-7bb86f7b56-zbt2v"
Sep 17 03:32:10.482869 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:10.482866    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b5b7160-05b5-4218-ab7b-0e7cdf8383a2-workload-certs\") pod \"e2e-inf-b-3d0c3c-openshift-default-7bb86f7b56-zbt2v\" (UID: \"8b5b7160-05b5-4218-ab7b-0e7cdf8383a2\") " pod="openshift-ingress/e2e-inf-b-3d0c3c-openshift-default-7bb86f7b56-zbt2v"
Sep 17 03:32:10.482959 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:10.482907    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b5b7160-05b5-4218-ab7b-0e7cdf8383a2-istio-token\") pod \"e2e-inf-b-3d0c3c-openshift-default-7bb86f7b56-zbt2v\" (UID: \"8b5b7160-05b5-4218-ab7b-0e7cdf8383a2\") " pod="openshift-ingress/e2e-inf-b-3d0c3c-openshift-default-7bb86f7b56-zbt2v"
Sep 17 03:32:10.584248 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:10.584215    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b5b7160-05b5-4218-ab7b-0e7cdf8383a2-credential-socket\") pod \"e2e-inf-b-3d0c3c-openshift-default-7bb86f7b56-zbt2v\" (UID: \"8b5b7160-05b5-4218-ab7b-0e7cdf8383a2\") " pod="openshift-ingress/e2e-inf-b-3d0c3c-openshift-default-7bb86f7b56-zbt2v"
Sep 17 03:32:10.584248 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:10.584253    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b5b7160-05b5-4218-ab7b-0e7cdf8383a2-istio-envoy\") pod \"e2e-inf-b-3d0c3c-openshift-default-7bb86f7b56-zbt2v\" (UID: \"8b5b7160-05b5-4218-ab7b-0e7cdf8383a2\") " pod="openshift-ingress/e2e-inf-b-3d0c3c-openshift-default-7bb86f7b56-zbt2v"
Sep 17 03:32:10.584461 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:10.584275    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b5b7160-05b5-4218-ab7b-0e7cdf8383a2-istiod-ca-cert\") pod \"e2e-inf-b-3d0c3c-openshift-default-7bb86f7b56-zbt2v\" (UID: \"8b5b7160-05b5-4218-ab7b-0e7cdf8383a2\") " pod="openshift-ingress/e2e-inf-b-3d0c3c-openshift-default-7bb86f7b56-zbt2v"
Sep 17 03:32:10.584461 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:10.584395    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scppn\" (UniqueName: \"kubernetes.io/projected/8b5b7160-05b5-4218-ab7b-0e7cdf8383a2-kube-api-access-scppn\") pod \"e2e-inf-b-3d0c3c-openshift-default-7bb86f7b56-zbt2v\" (UID: \"8b5b7160-05b5-4218-ab7b-0e7cdf8383a2\") " pod="openshift-ingress/e2e-inf-b-3d0c3c-openshift-default-7bb86f7b56-zbt2v"
Sep 17 03:32:10.584546 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:10.584465    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b5b7160-05b5-4218-ab7b-0e7cdf8383a2-istio-podinfo\") pod \"e2e-inf-b-3d0c3c-openshift-default-7bb86f7b56-zbt2v\" (UID: \"8b5b7160-05b5-4218-ab7b-0e7cdf8383a2\") " pod="openshift-ingress/e2e-inf-b-3d0c3c-openshift-default-7bb86f7b56-zbt2v"
Sep 17 03:32:10.584546 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:10.584526    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b5b7160-05b5-4218-ab7b-0e7cdf8383a2-workload-certs\") pod \"e2e-inf-b-3d0c3c-openshift-default-7bb86f7b56-zbt2v\" (UID: \"8b5b7160-05b5-4218-ab7b-0e7cdf8383a2\") " pod="openshift-ingress/e2e-inf-b-3d0c3c-openshift-default-7bb86f7b56-zbt2v"
Sep 17 03:32:10.584666 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:10.584570    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b5b7160-05b5-4218-ab7b-0e7cdf8383a2-istio-token\") pod \"e2e-inf-b-3d0c3c-openshift-default-7bb86f7b56-zbt2v\" (UID: \"8b5b7160-05b5-4218-ab7b-0e7cdf8383a2\") " pod="openshift-ingress/e2e-inf-b-3d0c3c-openshift-default-7bb86f7b56-zbt2v"
Sep 17 03:32:10.584666 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:10.584629    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b5b7160-05b5-4218-ab7b-0e7cdf8383a2-workload-socket\") pod \"e2e-inf-b-3d0c3c-openshift-default-7bb86f7b56-zbt2v\" (UID: \"8b5b7160-05b5-4218-ab7b-0e7cdf8383a2\") " pod="openshift-ingress/e2e-inf-b-3d0c3c-openshift-default-7bb86f7b56-zbt2v"
Sep 17 03:32:10.584666 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:10.584659    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b5b7160-05b5-4218-ab7b-0e7cdf8383a2-istio-data\") pod \"e2e-inf-b-3d0c3c-openshift-default-7bb86f7b56-zbt2v\" (UID: \"8b5b7160-05b5-4218-ab7b-0e7cdf8383a2\") " pod="openshift-ingress/e2e-inf-b-3d0c3c-openshift-default-7bb86f7b56-zbt2v"
Sep 17 03:32:10.584831 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:10.584674    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b5b7160-05b5-4218-ab7b-0e7cdf8383a2-credential-socket\") pod \"e2e-inf-b-3d0c3c-openshift-default-7bb86f7b56-zbt2v\" (UID: \"8b5b7160-05b5-4218-ab7b-0e7cdf8383a2\") " pod="openshift-ingress/e2e-inf-b-3d0c3c-openshift-default-7bb86f7b56-zbt2v"
Sep 17 03:32:10.584951 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:10.584900    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b5b7160-05b5-4218-ab7b-0e7cdf8383a2-workload-certs\") pod \"e2e-inf-b-3d0c3c-openshift-default-7bb86f7b56-zbt2v\" (UID: \"8b5b7160-05b5-4218-ab7b-0e7cdf8383a2\") " pod="openshift-ingress/e2e-inf-b-3d0c3c-openshift-default-7bb86f7b56-zbt2v"
Sep 17 03:32:10.585035 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:10.584973    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b5b7160-05b5-4218-ab7b-0e7cdf8383a2-istio-data\") pod \"e2e-inf-b-3d0c3c-openshift-default-7bb86f7b56-zbt2v\" (UID: \"8b5b7160-05b5-4218-ab7b-0e7cdf8383a2\") " pod="openshift-ingress/e2e-inf-b-3d0c3c-openshift-default-7bb86f7b56-zbt2v"
Sep 17 03:32:10.585097 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:10.585039    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b5b7160-05b5-4218-ab7b-0e7cdf8383a2-istiod-ca-cert\") pod \"e2e-inf-b-3d0c3c-openshift-default-7bb86f7b56-zbt2v\" (UID: \"8b5b7160-05b5-4218-ab7b-0e7cdf8383a2\") " pod="openshift-ingress/e2e-inf-b-3d0c3c-openshift-default-7bb86f7b56-zbt2v"
Sep 17 03:32:10.585097 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:10.585049    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b5b7160-05b5-4218-ab7b-0e7cdf8383a2-workload-socket\") pod \"e2e-inf-b-3d0c3c-openshift-default-7bb86f7b56-zbt2v\" (UID: \"8b5b7160-05b5-4218-ab7b-0e7cdf8383a2\") " pod="openshift-ingress/e2e-inf-b-3d0c3c-openshift-default-7bb86f7b56-zbt2v"
Sep 17 03:32:10.586642 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:10.586621    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b5b7160-05b5-4218-ab7b-0e7cdf8383a2-istio-envoy\") pod \"e2e-inf-b-3d0c3c-openshift-default-7bb86f7b56-zbt2v\" (UID: \"8b5b7160-05b5-4218-ab7b-0e7cdf8383a2\") " pod="openshift-ingress/e2e-inf-b-3d0c3c-openshift-default-7bb86f7b56-zbt2v"
Sep 17 03:32:10.586949 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:10.586931    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b5b7160-05b5-4218-ab7b-0e7cdf8383a2-istio-podinfo\") pod \"e2e-inf-b-3d0c3c-openshift-default-7bb86f7b56-zbt2v\" (UID: \"8b5b7160-05b5-4218-ab7b-0e7cdf8383a2\") " pod="openshift-ingress/e2e-inf-b-3d0c3c-openshift-default-7bb86f7b56-zbt2v"
Sep 17 03:32:10.593689 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:10.593660    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b5b7160-05b5-4218-ab7b-0e7cdf8383a2-istio-token\") pod \"e2e-inf-b-3d0c3c-openshift-default-7bb86f7b56-zbt2v\" (UID: \"8b5b7160-05b5-4218-ab7b-0e7cdf8383a2\") " pod="openshift-ingress/e2e-inf-b-3d0c3c-openshift-default-7bb86f7b56-zbt2v"
Sep 17 03:32:10.593942 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:10.593922    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-scppn\" (UniqueName: \"kubernetes.io/projected/8b5b7160-05b5-4218-ab7b-0e7cdf8383a2-kube-api-access-scppn\") pod \"e2e-inf-b-3d0c3c-openshift-default-7bb86f7b56-zbt2v\" (UID: \"8b5b7160-05b5-4218-ab7b-0e7cdf8383a2\") " pod="openshift-ingress/e2e-inf-b-3d0c3c-openshift-default-7bb86f7b56-zbt2v"
Sep 17 03:32:10.677060 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:10.676985    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-3d0c3c-openshift-default-7bb86f7b56-zbt2v"
Sep 17 03:32:10.774411 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:10.773701    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c0b7e751-7d6c94f4f5-bvj9m"
Sep 17 03:32:10.975357 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:10.975332    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-3d0c3c-openshift-default-7bb86f7b56-zbt2v"]
Sep 17 03:32:10.977303 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:32:10.977271    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b5b7160_05b5_4218_ab7b_0e7cdf8383a2.slice/crio-16aa2262f9f573dda9994e330948dfccbd7c815aac35dce92cea346119604ff6 WatchSource:0}: Error finding container 16aa2262f9f573dda9994e330948dfccbd7c815aac35dce92cea346119604ff6: Status 404 returned error can't find the container with id 16aa2262f9f573dda9994e330948dfccbd7c815aac35dce92cea346119604ff6
Sep 17 03:32:10.979705 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:10.979666    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 03:32:10.979806 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:10.979737    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 03:32:10.979806 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:10.979772    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 03:32:11.787076 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:11.787045    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-3d0c3c-openshift-default-7bb86f7b56-zbt2v" event={"ID":"8b5b7160-05b5-4218-ab7b-0e7cdf8383a2","Type":"ContainerStarted","Data":"1d6e97980462193c02b6a8371658c294ac8ab6b69706403ad3c2f021f8908caf"}
Sep 17 03:32:11.787483 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:11.787099    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-3d0c3c-openshift-default-7bb86f7b56-zbt2v" event={"ID":"8b5b7160-05b5-4218-ab7b-0e7cdf8383a2","Type":"ContainerStarted","Data":"16aa2262f9f573dda9994e330948dfccbd7c815aac35dce92cea346119604ff6"}
Sep 17 03:32:11.810984 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:11.810937    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-3d0c3c-openshift-default-7bb86f7b56-zbt2v" podStartSLOduration=1.810921035 podStartE2EDuration="1.810921035s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 03:32:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:32:11.809147372 +0000 UTC m=+767.275554897" watchObservedRunningTime="2026-09-17 03:32:11.810921035 +0000 UTC m=+767.277328560"
Sep 17 03:32:12.678040 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:12.678008    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-3d0c3c-openshift-default-7bb86f7b56-zbt2v"
Sep 17 03:32:12.683190 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:12.683162    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-3d0c3c-openshift-default-7bb86f7b56-zbt2v"
Sep 17 03:32:12.793076 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:12.793043    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-3d0c3c-openshift-default-7bb86f7b56-zbt2v"
Sep 17 03:32:12.794338 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:12.794321    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-3d0c3c-openshift-default-7bb86f7b56-zbt2v"
Sep 17 03:32:17.821247 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:17.821220    2552 generic.go:356] "Generic (PLEG): container finished" podID="e5adcfc6-e1c3-4e82-b719-51e7fa179a5c" containerID="582a4472c6ed4fc6062d650e536e32e48d9f0efbee667ee66457448fadcb2f26" exitCode=0
Sep 17 03:32:17.821563 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:17.821280    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c0b7e751-openshift-default-5c659ddbd8-sbrd4" event={"ID":"e5adcfc6-e1c3-4e82-b719-51e7fa179a5c","Type":"ContainerDied","Data":"582a4472c6ed4fc6062d650e536e32e48d9f0efbee667ee66457448fadcb2f26"}
Sep 17 03:32:17.821718 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:17.821699    2552 scope.go:117] "RemoveContainer" containerID="582a4472c6ed4fc6062d650e536e32e48d9f0efbee667ee66457448fadcb2f26"
Sep 17 03:32:17.822766 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:17.822736    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 03:32:17.822827 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:17.822806    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 03:32:17.822865 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:17.822836    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 03:32:18.830472 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:18.829475    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c0b7e751-openshift-default-5c659ddbd8-sbrd4" event={"ID":"e5adcfc6-e1c3-4e82-b719-51e7fa179a5c","Type":"ContainerStarted","Data":"4d08ff2537519c790e3a31f93c8d9029ae671c361c9ce44bcfd7dbbf36d2caa8"}
Sep 17 03:32:19.093722 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:19.093634    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-c0b7e751-openshift-default-5c659ddbd8-sbrd4"
Sep 17 03:32:19.093878 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:19.093763    2552 patch_prober.go:28] interesting pod/e2e-ait-c0b7e751-openshift-default-5c659ddbd8-sbrd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Sep 17 03:32:19.093878 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:19.093815    2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c0b7e751-openshift-default-5c659ddbd8-sbrd4" podUID="e5adcfc6-e1c3-4e82-b719-51e7fa179a5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Sep 17 03:32:20.094019 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:20.093985    2552 patch_prober.go:28] interesting pod/e2e-ait-c0b7e751-openshift-default-5c659ddbd8-sbrd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Sep 17 03:32:20.094440 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:20.094039    2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c0b7e751-openshift-default-5c659ddbd8-sbrd4" podUID="e5adcfc6-e1c3-4e82-b719-51e7fa179a5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Sep 17 03:32:21.094683 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:21.094651    2552 patch_prober.go:28] interesting pod/e2e-ait-c0b7e751-openshift-default-5c659ddbd8-sbrd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Sep 17 03:32:21.095049 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:21.094703    2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c0b7e751-openshift-default-5c659ddbd8-sbrd4" podUID="e5adcfc6-e1c3-4e82-b719-51e7fa179a5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Sep 17 03:32:22.094038 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:22.093997    2552 patch_prober.go:28] interesting pod/e2e-ait-c0b7e751-openshift-default-5c659ddbd8-sbrd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Sep 17 03:32:22.094216 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:22.094061    2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c0b7e751-openshift-default-5c659ddbd8-sbrd4" podUID="e5adcfc6-e1c3-4e82-b719-51e7fa179a5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Sep 17 03:32:23.093979 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:23.093940    2552 patch_prober.go:28] interesting pod/e2e-ait-c0b7e751-openshift-default-5c659ddbd8-sbrd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Sep 17 03:32:23.094393 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:23.093996    2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c0b7e751-openshift-default-5c659ddbd8-sbrd4" podUID="e5adcfc6-e1c3-4e82-b719-51e7fa179a5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Sep 17 03:32:24.098081 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:24.098052    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-c0b7e751-openshift-default-5c659ddbd8-sbrd4"
Sep 17 03:32:24.098473 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:24.098339    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-c0b7e751-openshift-default-5c659ddbd8-sbrd4"
Sep 17 03:32:24.099105 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:24.099090    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-c0b7e751-openshift-default-5c659ddbd8-sbrd4"
Sep 17 03:32:33.922392 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:33.922286    2552 generic.go:356] "Generic (PLEG): container finished" podID="8b5b7160-05b5-4218-ab7b-0e7cdf8383a2" containerID="1d6e97980462193c02b6a8371658c294ac8ab6b69706403ad3c2f021f8908caf" exitCode=0
Sep 17 03:32:33.922392 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:33.922362    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-3d0c3c-openshift-default-7bb86f7b56-zbt2v" event={"ID":"8b5b7160-05b5-4218-ab7b-0e7cdf8383a2","Type":"ContainerDied","Data":"1d6e97980462193c02b6a8371658c294ac8ab6b69706403ad3c2f021f8908caf"}
Sep 17 03:32:33.922833 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:33.922765    2552 scope.go:117] "RemoveContainer" containerID="1d6e97980462193c02b6a8371658c294ac8ab6b69706403ad3c2f021f8908caf"
Sep 17 03:32:33.923982 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:33.923957    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 03:32:33.924046 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:33.924025    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 03:32:33.924084 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:33.924055    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 03:32:34.931875 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:34.930878    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-3d0c3c-openshift-default-7bb86f7b56-zbt2v" event={"ID":"8b5b7160-05b5-4218-ab7b-0e7cdf8383a2","Type":"ContainerStarted","Data":"09cb90acbcd3bcf39f778c365bde47570dcefbd34599ff8643894e330642d123"}
Sep 17 03:32:35.677445 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:35.677404    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-3d0c3c-openshift-default-7bb86f7b56-zbt2v"
Sep 17 03:32:35.677684 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:35.677649    2552 patch_prober.go:28] interesting pod/e2e-inf-b-3d0c3c-openshift-default-7bb86f7b56-zbt2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Sep 17 03:32:35.677768 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:35.677707    2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3d0c3c-openshift-default-7bb86f7b56-zbt2v" podUID="8b5b7160-05b5-4218-ab7b-0e7cdf8383a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Sep 17 03:32:36.677512 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:36.677476    2552 patch_prober.go:28] interesting pod/e2e-inf-b-3d0c3c-openshift-default-7bb86f7b56-zbt2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Sep 17 03:32:36.677916 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:36.677528    2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3d0c3c-openshift-default-7bb86f7b56-zbt2v" podUID="8b5b7160-05b5-4218-ab7b-0e7cdf8383a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Sep 17 03:32:37.677706 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:37.677671    2552 patch_prober.go:28] interesting pod/e2e-inf-b-3d0c3c-openshift-default-7bb86f7b56-zbt2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Sep 17 03:32:37.678072 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:37.677726    2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3d0c3c-openshift-default-7bb86f7b56-zbt2v" podUID="8b5b7160-05b5-4218-ab7b-0e7cdf8383a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Sep 17 03:32:38.480865 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:38.480832    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c0b7e751-7d6c94f4f5-bvj9m"]
Sep 17 03:32:38.481084 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:38.481054    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c0b7e751-7d6c94f4f5-bvj9m" podUID="ef222c1b-f104-4575-aded-35dd5711cd91" containerName="maas-api" containerID="cri-o://008293a57db3f783699ef9bc7aa4f4ec43c88c6364667f2fea54a2ed2d4f1b8a" gracePeriod=30
Sep 17 03:32:38.678126 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:38.678095    2552 patch_prober.go:28] interesting pod/e2e-inf-b-3d0c3c-openshift-default-7bb86f7b56-zbt2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Sep 17 03:32:38.678476 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:38.678176    2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3d0c3c-openshift-default-7bb86f7b56-zbt2v" podUID="8b5b7160-05b5-4218-ab7b-0e7cdf8383a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Sep 17 03:32:38.764467 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:38.764441    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c0b7e751-7d6c94f4f5-bvj9m"
Sep 17 03:32:38.802329 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:38.802299    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-c0b7e751-openshift-default-5c659ddbd8-sbrd4"]
Sep 17 03:32:38.802699 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:38.802644    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-c0b7e751-openshift-default-5c659ddbd8-sbrd4" podUID="e5adcfc6-e1c3-4e82-b719-51e7fa179a5c" containerName="istio-proxy" containerID="cri-o://4d08ff2537519c790e3a31f93c8d9029ae671c361c9ce44bcfd7dbbf36d2caa8" gracePeriod=30
Sep 17 03:32:38.882965 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:38.882931    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef222c1b-f104-4575-aded-35dd5711cd91-metrics-certs\") pod \"ef222c1b-f104-4575-aded-35dd5711cd91\" (UID: \"ef222c1b-f104-4575-aded-35dd5711cd91\") "
Sep 17 03:32:38.883142 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:38.883023    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dxz9v\" (UniqueName: \"kubernetes.io/projected/ef222c1b-f104-4575-aded-35dd5711cd91-kube-api-access-dxz9v\") pod \"ef222c1b-f104-4575-aded-35dd5711cd91\" (UID: \"ef222c1b-f104-4575-aded-35dd5711cd91\") "
Sep 17 03:32:38.883142 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:38.883074    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef222c1b-f104-4575-aded-35dd5711cd91-openshift-service-ca\") pod \"ef222c1b-f104-4575-aded-35dd5711cd91\" (UID: \"ef222c1b-f104-4575-aded-35dd5711cd91\") "
Sep 17 03:32:38.883223 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:38.883160    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef222c1b-f104-4575-aded-35dd5711cd91-maas-api-tls\") pod \"ef222c1b-f104-4575-aded-35dd5711cd91\" (UID: \"ef222c1b-f104-4575-aded-35dd5711cd91\") "
Sep 17 03:32:38.883588 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:38.883538    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef222c1b-f104-4575-aded-35dd5711cd91-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ef222c1b-f104-4575-aded-35dd5711cd91" (UID: "ef222c1b-f104-4575-aded-35dd5711cd91"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 03:32:38.885852 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:38.885795    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef222c1b-f104-4575-aded-35dd5711cd91-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "ef222c1b-f104-4575-aded-35dd5711cd91" (UID: "ef222c1b-f104-4575-aded-35dd5711cd91"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 03:32:38.885852 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:38.885805    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef222c1b-f104-4575-aded-35dd5711cd91-kube-api-access-dxz9v" (OuterVolumeSpecName: "kube-api-access-dxz9v") pod "ef222c1b-f104-4575-aded-35dd5711cd91" (UID: "ef222c1b-f104-4575-aded-35dd5711cd91"). InnerVolumeSpecName "kube-api-access-dxz9v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 03:32:38.885852 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:38.885832    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef222c1b-f104-4575-aded-35dd5711cd91-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ef222c1b-f104-4575-aded-35dd5711cd91" (UID: "ef222c1b-f104-4575-aded-35dd5711cd91"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 03:32:38.966519 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:38.966448    2552 generic.go:356] "Generic (PLEG): container finished" podID="ef222c1b-f104-4575-aded-35dd5711cd91" containerID="008293a57db3f783699ef9bc7aa4f4ec43c88c6364667f2fea54a2ed2d4f1b8a" exitCode=0
Sep 17 03:32:38.966751 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:38.966525    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c0b7e751-7d6c94f4f5-bvj9m"
Sep 17 03:32:38.966751 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:38.966549    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c0b7e751-7d6c94f4f5-bvj9m" event={"ID":"ef222c1b-f104-4575-aded-35dd5711cd91","Type":"ContainerDied","Data":"008293a57db3f783699ef9bc7aa4f4ec43c88c6364667f2fea54a2ed2d4f1b8a"}
Sep 17 03:32:38.966751 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:38.966587    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c0b7e751-7d6c94f4f5-bvj9m" event={"ID":"ef222c1b-f104-4575-aded-35dd5711cd91","Type":"ContainerDied","Data":"2d5861af3fd0ebd5c4d3c331e4f832e7ac90817912f75596f106a8b2a14dcb0a"}
Sep 17 03:32:38.966751 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:38.966630    2552 scope.go:117] "RemoveContainer" containerID="008293a57db3f783699ef9bc7aa4f4ec43c88c6364667f2fea54a2ed2d4f1b8a"
Sep 17 03:32:38.977420 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:38.977404    2552 scope.go:117] "RemoveContainer" containerID="008293a57db3f783699ef9bc7aa4f4ec43c88c6364667f2fea54a2ed2d4f1b8a"
Sep 17 03:32:38.977731 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:32:38.977713    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"008293a57db3f783699ef9bc7aa4f4ec43c88c6364667f2fea54a2ed2d4f1b8a\": container with ID starting with 008293a57db3f783699ef9bc7aa4f4ec43c88c6364667f2fea54a2ed2d4f1b8a not found: ID does not exist" containerID="008293a57db3f783699ef9bc7aa4f4ec43c88c6364667f2fea54a2ed2d4f1b8a"
Sep 17 03:32:38.977788 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:38.977738    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"008293a57db3f783699ef9bc7aa4f4ec43c88c6364667f2fea54a2ed2d4f1b8a"} err="failed to get container status \"008293a57db3f783699ef9bc7aa4f4ec43c88c6364667f2fea54a2ed2d4f1b8a\": rpc error: code = NotFound desc = could not find container \"008293a57db3f783699ef9bc7aa4f4ec43c88c6364667f2fea54a2ed2d4f1b8a\": container with ID starting with 008293a57db3f783699ef9bc7aa4f4ec43c88c6364667f2fea54a2ed2d4f1b8a not found: ID does not exist"
Sep 17 03:32:38.984158 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:38.984141    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dxz9v\" (UniqueName: \"kubernetes.io/projected/ef222c1b-f104-4575-aded-35dd5711cd91-kube-api-access-dxz9v\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:32:38.984158 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:38.984159    2552 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef222c1b-f104-4575-aded-35dd5711cd91-openshift-service-ca\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:32:38.984281 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:38.984169    2552 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef222c1b-f104-4575-aded-35dd5711cd91-maas-api-tls\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:32:38.984281 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:38.984177    2552 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef222c1b-f104-4575-aded-35dd5711cd91-metrics-certs\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:32:38.988298 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:38.988274    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c0b7e751-7d6c94f4f5-bvj9m"]
Sep 17 03:32:38.990152 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:38.990135    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c0b7e751-7d6c94f4f5-bvj9m"]
Sep 17 03:32:39.101555 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:39.101475    2552 patch_prober.go:28] interesting pod/e2e-ait-c0b7e751-openshift-default-5c659ddbd8-sbrd4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 17 03:32:39.101555 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:39.101530    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-c0b7e751-openshift-default-5c659ddbd8-sbrd4" podUID="e5adcfc6-e1c3-4e82-b719-51e7fa179a5c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 03:32:39.678005 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:39.677965    2552 patch_prober.go:28] interesting pod/e2e-inf-b-3d0c3c-openshift-default-7bb86f7b56-zbt2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Sep 17 03:32:39.678188 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:39.678033    2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3d0c3c-openshift-default-7bb86f7b56-zbt2v" podUID="8b5b7160-05b5-4218-ab7b-0e7cdf8383a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Sep 17 03:32:39.761276 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:39.761245    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-0ae566a4-gw-openshift-default-7ff59f8f76-9zf22"]
Sep 17 03:32:39.761724 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:39.761700    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef222c1b-f104-4575-aded-35dd5711cd91" containerName="maas-api"
Sep 17 03:32:39.761724 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:39.761726    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef222c1b-f104-4575-aded-35dd5711cd91" containerName="maas-api"
Sep 17 03:32:39.761831 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:39.761821    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef222c1b-f104-4575-aded-35dd5711cd91" containerName="maas-api"
Sep 17 03:32:39.766167 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:39.766149    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-0ae566a4-gw-openshift-default-7ff59f8f76-9zf22"
Sep 17 03:32:39.769100 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:39.769070    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-0ae566a4-gw-openshift-default-dockercfg-c2kjz\""
Sep 17 03:32:39.783990 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:39.783962    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-0ae566a4-gw-openshift-default-7ff59f8f76-9zf22"]
Sep 17 03:32:39.792088 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:39.792059    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c1e74de-3dc7-463d-acde-e9ab14a22970-istiod-ca-cert\") pod \"e2e-migrate-0ae566a4-gw-openshift-default-7ff59f8f76-9zf22\" (UID: \"0c1e74de-3dc7-463d-acde-e9ab14a22970\") " pod="openshift-ingress/e2e-migrate-0ae566a4-gw-openshift-default-7ff59f8f76-9zf22"
Sep 17 03:32:39.792181 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:39.792100    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c1e74de-3dc7-463d-acde-e9ab14a22970-istio-token\") pod \"e2e-migrate-0ae566a4-gw-openshift-default-7ff59f8f76-9zf22\" (UID: \"0c1e74de-3dc7-463d-acde-e9ab14a22970\") " pod="openshift-ingress/e2e-migrate-0ae566a4-gw-openshift-default-7ff59f8f76-9zf22"
Sep 17 03:32:39.792181 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:39.792132    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c1e74de-3dc7-463d-acde-e9ab14a22970-istio-podinfo\") pod \"e2e-migrate-0ae566a4-gw-openshift-default-7ff59f8f76-9zf22\" (UID: \"0c1e74de-3dc7-463d-acde-e9ab14a22970\") " pod="openshift-ingress/e2e-migrate-0ae566a4-gw-openshift-default-7ff59f8f76-9zf22"
Sep 17 03:32:39.792181 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:39.792170    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c1e74de-3dc7-463d-acde-e9ab14a22970-istio-envoy\") pod \"e2e-migrate-0ae566a4-gw-openshift-default-7ff59f8f76-9zf22\" (UID: \"0c1e74de-3dc7-463d-acde-e9ab14a22970\") " pod="openshift-ingress/e2e-migrate-0ae566a4-gw-openshift-default-7ff59f8f76-9zf22"
Sep 17 03:32:39.792312 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:39.792189    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c1e74de-3dc7-463d-acde-e9ab14a22970-istio-data\") pod \"e2e-migrate-0ae566a4-gw-openshift-default-7ff59f8f76-9zf22\" (UID: \"0c1e74de-3dc7-463d-acde-e9ab14a22970\") " pod="openshift-ingress/e2e-migrate-0ae566a4-gw-openshift-default-7ff59f8f76-9zf22"
Sep 17 03:32:39.792312 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:39.792257    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c1e74de-3dc7-463d-acde-e9ab14a22970-workload-certs\") pod \"e2e-migrate-0ae566a4-gw-openshift-default-7ff59f8f76-9zf22\" (UID: \"0c1e74de-3dc7-463d-acde-e9ab14a22970\") " pod="openshift-ingress/e2e-migrate-0ae566a4-gw-openshift-default-7ff59f8f76-9zf22"
Sep 17 03:32:39.792312 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:39.792304    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nk4s\" (UniqueName: \"kubernetes.io/projected/0c1e74de-3dc7-463d-acde-e9ab14a22970-kube-api-access-2nk4s\") pod \"e2e-migrate-0ae566a4-gw-openshift-default-7ff59f8f76-9zf22\" (UID: \"0c1e74de-3dc7-463d-acde-e9ab14a22970\") " pod="openshift-ingress/e2e-migrate-0ae566a4-gw-openshift-default-7ff59f8f76-9zf22"
Sep 17 03:32:39.792435 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:39.792333    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c1e74de-3dc7-463d-acde-e9ab14a22970-credential-socket\") pod \"e2e-migrate-0ae566a4-gw-openshift-default-7ff59f8f76-9zf22\" (UID: \"0c1e74de-3dc7-463d-acde-e9ab14a22970\") " pod="openshift-ingress/e2e-migrate-0ae566a4-gw-openshift-default-7ff59f8f76-9zf22"
Sep 17 03:32:39.792435 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:39.792361    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c1e74de-3dc7-463d-acde-e9ab14a22970-workload-socket\") pod \"e2e-migrate-0ae566a4-gw-openshift-default-7ff59f8f76-9zf22\" (UID: \"0c1e74de-3dc7-463d-acde-e9ab14a22970\") " pod="openshift-ingress/e2e-migrate-0ae566a4-gw-openshift-default-7ff59f8f76-9zf22"
Sep 17 03:32:39.893589 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:39.893556    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c1e74de-3dc7-463d-acde-e9ab14a22970-istio-envoy\") pod \"e2e-migrate-0ae566a4-gw-openshift-default-7ff59f8f76-9zf22\" (UID: \"0c1e74de-3dc7-463d-acde-e9ab14a22970\") " pod="openshift-ingress/e2e-migrate-0ae566a4-gw-openshift-default-7ff59f8f76-9zf22"
Sep 17 03:32:39.893589 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:39.893590    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c1e74de-3dc7-463d-acde-e9ab14a22970-istio-data\") pod \"e2e-migrate-0ae566a4-gw-openshift-default-7ff59f8f76-9zf22\" (UID: \"0c1e74de-3dc7-463d-acde-e9ab14a22970\") " pod="openshift-ingress/e2e-migrate-0ae566a4-gw-openshift-default-7ff59f8f76-9zf22"
Sep 17 03:32:39.894000 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:39.893966    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c1e74de-3dc7-463d-acde-e9ab14a22970-workload-certs\") pod \"e2e-migrate-0ae566a4-gw-openshift-default-7ff59f8f76-9zf22\" (UID: \"0c1e74de-3dc7-463d-acde-e9ab14a22970\") " pod="openshift-ingress/e2e-migrate-0ae566a4-gw-openshift-default-7ff59f8f76-9zf22"
Sep 17 03:32:39.894134 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:39.894016    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nk4s\" (UniqueName: \"kubernetes.io/projected/0c1e74de-3dc7-463d-acde-e9ab14a22970-kube-api-access-2nk4s\") pod \"e2e-migrate-0ae566a4-gw-openshift-default-7ff59f8f76-9zf22\" (UID: \"0c1e74de-3dc7-463d-acde-e9ab14a22970\") " pod="openshift-ingress/e2e-migrate-0ae566a4-gw-openshift-default-7ff59f8f76-9zf22"
Sep 17 03:32:39.894134 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:39.894060    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c1e74de-3dc7-463d-acde-e9ab14a22970-credential-socket\") pod \"e2e-migrate-0ae566a4-gw-openshift-default-7ff59f8f76-9zf22\" (UID: \"0c1e74de-3dc7-463d-acde-e9ab14a22970\") " pod="openshift-ingress/e2e-migrate-0ae566a4-gw-openshift-default-7ff59f8f76-9zf22"
Sep 17 03:32:39.894134 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:39.893901    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c1e74de-3dc7-463d-acde-e9ab14a22970-istio-data\") pod \"e2e-migrate-0ae566a4-gw-openshift-default-7ff59f8f76-9zf22\" (UID: \"0c1e74de-3dc7-463d-acde-e9ab14a22970\") " pod="openshift-ingress/e2e-migrate-0ae566a4-gw-openshift-default-7ff59f8f76-9zf22"
Sep 17 03:32:39.894309 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:39.894150    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c1e74de-3dc7-463d-acde-e9ab14a22970-workload-socket\") pod \"e2e-migrate-0ae566a4-gw-openshift-default-7ff59f8f76-9zf22\" (UID: \"0c1e74de-3dc7-463d-acde-e9ab14a22970\") " pod="openshift-ingress/e2e-migrate-0ae566a4-gw-openshift-default-7ff59f8f76-9zf22"
Sep 17 03:32:39.894309 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:39.894236    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c1e74de-3dc7-463d-acde-e9ab14a22970-workload-certs\") pod \"e2e-migrate-0ae566a4-gw-openshift-default-7ff59f8f76-9zf22\" (UID: \"0c1e74de-3dc7-463d-acde-e9ab14a22970\") " pod="openshift-ingress/e2e-migrate-0ae566a4-gw-openshift-default-7ff59f8f76-9zf22"
Sep 17 03:32:39.894400 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:39.894316    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c1e74de-3dc7-463d-acde-e9ab14a22970-credential-socket\") pod \"e2e-migrate-0ae566a4-gw-openshift-default-7ff59f8f76-9zf22\" (UID: \"0c1e74de-3dc7-463d-acde-e9ab14a22970\") " pod="openshift-ingress/e2e-migrate-0ae566a4-gw-openshift-default-7ff59f8f76-9zf22"
Sep 17 03:32:39.894469 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:39.894456    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c1e74de-3dc7-463d-acde-e9ab14a22970-istiod-ca-cert\") pod \"e2e-migrate-0ae566a4-gw-openshift-default-7ff59f8f76-9zf22\" (UID: \"0c1e74de-3dc7-463d-acde-e9ab14a22970\") " pod="openshift-ingress/e2e-migrate-0ae566a4-gw-openshift-default-7ff59f8f76-9zf22"
Sep 17 03:32:39.894516 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:39.894491    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c1e74de-3dc7-463d-acde-e9ab14a22970-istio-token\") pod \"e2e-migrate-0ae566a4-gw-openshift-default-7ff59f8f76-9zf22\" (UID: \"0c1e74de-3dc7-463d-acde-e9ab14a22970\") " pod="openshift-ingress/e2e-migrate-0ae566a4-gw-openshift-default-7ff59f8f76-9zf22"
Sep 17 03:32:39.894580 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:39.894517    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c1e74de-3dc7-463d-acde-e9ab14a22970-istio-podinfo\") pod \"e2e-migrate-0ae566a4-gw-openshift-default-7ff59f8f76-9zf22\" (UID: \"0c1e74de-3dc7-463d-acde-e9ab14a22970\") " pod="openshift-ingress/e2e-migrate-0ae566a4-gw-openshift-default-7ff59f8f76-9zf22"
Sep 17 03:32:39.894857 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:39.894834    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c1e74de-3dc7-463d-acde-e9ab14a22970-workload-socket\") pod \"e2e-migrate-0ae566a4-gw-openshift-default-7ff59f8f76-9zf22\" (UID: \"0c1e74de-3dc7-463d-acde-e9ab14a22970\") " pod="openshift-ingress/e2e-migrate-0ae566a4-gw-openshift-default-7ff59f8f76-9zf22"
Sep 17 03:32:39.895134 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:39.895114    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c1e74de-3dc7-463d-acde-e9ab14a22970-istiod-ca-cert\") pod \"e2e-migrate-0ae566a4-gw-openshift-default-7ff59f8f76-9zf22\" (UID: \"0c1e74de-3dc7-463d-acde-e9ab14a22970\") " pod="openshift-ingress/e2e-migrate-0ae566a4-gw-openshift-default-7ff59f8f76-9zf22"
Sep 17 03:32:39.895826 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:39.895790    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c1e74de-3dc7-463d-acde-e9ab14a22970-istio-envoy\") pod \"e2e-migrate-0ae566a4-gw-openshift-default-7ff59f8f76-9zf22\" (UID: \"0c1e74de-3dc7-463d-acde-e9ab14a22970\") " pod="openshift-ingress/e2e-migrate-0ae566a4-gw-openshift-default-7ff59f8f76-9zf22"
Sep 17 03:32:39.896989 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:39.896974    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c1e74de-3dc7-463d-acde-e9ab14a22970-istio-podinfo\") pod \"e2e-migrate-0ae566a4-gw-openshift-default-7ff59f8f76-9zf22\" (UID: \"0c1e74de-3dc7-463d-acde-e9ab14a22970\") " pod="openshift-ingress/e2e-migrate-0ae566a4-gw-openshift-default-7ff59f8f76-9zf22"
Sep 17 03:32:39.901846 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:39.901824    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nk4s\" (UniqueName: \"kubernetes.io/projected/0c1e74de-3dc7-463d-acde-e9ab14a22970-kube-api-access-2nk4s\") pod \"e2e-migrate-0ae566a4-gw-openshift-default-7ff59f8f76-9zf22\" (UID: \"0c1e74de-3dc7-463d-acde-e9ab14a22970\") " pod="openshift-ingress/e2e-migrate-0ae566a4-gw-openshift-default-7ff59f8f76-9zf22"
Sep 17 03:32:39.902233 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:39.902218    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c1e74de-3dc7-463d-acde-e9ab14a22970-istio-token\") pod \"e2e-migrate-0ae566a4-gw-openshift-default-7ff59f8f76-9zf22\" (UID: \"0c1e74de-3dc7-463d-acde-e9ab14a22970\") " pod="openshift-ingress/e2e-migrate-0ae566a4-gw-openshift-default-7ff59f8f76-9zf22"
Sep 17 03:32:40.077080 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:40.077047    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-0ae566a4-gw-openshift-default-7ff59f8f76-9zf22"
Sep 17 03:32:40.376655 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:40.376497    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-0ae566a4-gw-openshift-default-7ff59f8f76-9zf22"]
Sep 17 03:32:40.379668 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:32:40.379637    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c1e74de_3dc7_463d_acde_e9ab14a22970.slice/crio-e81fc9dd886a34c9bcfa86dcea03474ecf5b4ba12d13ea8106eb73555b28ca1d WatchSource:0}: Error finding container e81fc9dd886a34c9bcfa86dcea03474ecf5b4ba12d13ea8106eb73555b28ca1d: Status 404 returned error can't find the container with id e81fc9dd886a34c9bcfa86dcea03474ecf5b4ba12d13ea8106eb73555b28ca1d
Sep 17 03:32:40.382483 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:40.382444    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 03:32:40.382588 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:40.382534    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 03:32:40.382588 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:40.382566    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 03:32:40.681722 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:40.681650    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-3d0c3c-openshift-default-7bb86f7b56-zbt2v"
Sep 17 03:32:40.682109 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:40.681942    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-3d0c3c-openshift-default-7bb86f7b56-zbt2v"
Sep 17 03:32:40.682840 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:40.682823    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-3d0c3c-openshift-default-7bb86f7b56-zbt2v"
Sep 17 03:32:40.989823 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:40.989749    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ef222c1b-f104-4575-aded-35dd5711cd91" path="/var/lib/kubelet/pods/ef222c1b-f104-4575-aded-35dd5711cd91/volumes"
Sep 17 03:32:40.990017 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:40.990002    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-0ae566a4-gw-openshift-default-7ff59f8f76-9zf22" event={"ID":"0c1e74de-3dc7-463d-acde-e9ab14a22970","Type":"ContainerStarted","Data":"5679717a5818448203f9acd08d33c0af4b322b48ff873661702c4caab81bd67b"}
Sep 17 03:32:40.990063 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:40.990039    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-0ae566a4-gw-openshift-default-7ff59f8f76-9zf22" event={"ID":"0c1e74de-3dc7-463d-acde-e9ab14a22970","Type":"ContainerStarted","Data":"e81fc9dd886a34c9bcfa86dcea03474ecf5b4ba12d13ea8106eb73555b28ca1d"}
Sep 17 03:32:41.011677 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:41.011627    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-0ae566a4-gw-openshift-default-7ff59f8f76-9zf22" podStartSLOduration=2.011587134 podStartE2EDuration="2.011587134s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 03:32:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:32:41.010448865 +0000 UTC m=+796.476856391" watchObservedRunningTime="2026-09-17 03:32:41.011587134 +0000 UTC m=+796.477994659"
Sep 17 03:32:41.077796 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:41.077763    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-0ae566a4-gw-openshift-default-7ff59f8f76-9zf22"
Sep 17 03:32:41.082588 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:41.082566    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-0ae566a4-gw-openshift-default-7ff59f8f76-9zf22"
Sep 17 03:32:41.770618 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:41.770560    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-214fea/simulator-w4-214fea-kserve-5965bff56b-mkmt9"]
Sep 17 03:32:41.774249 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:41.774232    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-214fea/simulator-w4-214fea-kserve-5965bff56b-mkmt9"
Sep 17 03:32:41.777057 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:41.777032    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-214fea\"/\"openshift-service-ca.crt\""
Sep 17 03:32:41.777183 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:41.777165    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-214fea\"/\"simulator-w4-214fea-kserve-self-signed-certs\""
Sep 17 03:32:41.777251 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:41.777236    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-214fea\"/\"default-dockercfg-g7kkn\""
Sep 17 03:32:41.777307 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:41.777289    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-214fea\"/\"kube-root-ca.crt\""
Sep 17 03:32:41.783377 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:41.783355    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-214fea/simulator-w4-214fea-kserve-5965bff56b-mkmt9"]
Sep 17 03:32:41.811521 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:41.811485    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jpz8\" (UniqueName: \"kubernetes.io/projected/f29ebfeb-d96e-41de-b4e5-c5f1d6fbe785-kube-api-access-2jpz8\") pod \"simulator-w4-214fea-kserve-5965bff56b-mkmt9\" (UID: \"f29ebfeb-d96e-41de-b4e5-c5f1d6fbe785\") " pod="e2e-models-e2e-worker-w4-214fea/simulator-w4-214fea-kserve-5965bff56b-mkmt9"
Sep 17 03:32:41.811696 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:41.811533    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f29ebfeb-d96e-41de-b4e5-c5f1d6fbe785-model-cache\") pod \"simulator-w4-214fea-kserve-5965bff56b-mkmt9\" (UID: \"f29ebfeb-d96e-41de-b4e5-c5f1d6fbe785\") " pod="e2e-models-e2e-worker-w4-214fea/simulator-w4-214fea-kserve-5965bff56b-mkmt9"
Sep 17 03:32:41.811696 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:41.811573    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f29ebfeb-d96e-41de-b4e5-c5f1d6fbe785-home\") pod \"simulator-w4-214fea-kserve-5965bff56b-mkmt9\" (UID: \"f29ebfeb-d96e-41de-b4e5-c5f1d6fbe785\") " pod="e2e-models-e2e-worker-w4-214fea/simulator-w4-214fea-kserve-5965bff56b-mkmt9"
Sep 17 03:32:41.811696 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:41.811626    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f29ebfeb-d96e-41de-b4e5-c5f1d6fbe785-dshm\") pod \"simulator-w4-214fea-kserve-5965bff56b-mkmt9\" (UID: \"f29ebfeb-d96e-41de-b4e5-c5f1d6fbe785\") " pod="e2e-models-e2e-worker-w4-214fea/simulator-w4-214fea-kserve-5965bff56b-mkmt9"
Sep 17 03:32:41.811810 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:41.811686    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f29ebfeb-d96e-41de-b4e5-c5f1d6fbe785-tls-certs\") pod \"simulator-w4-214fea-kserve-5965bff56b-mkmt9\" (UID: \"f29ebfeb-d96e-41de-b4e5-c5f1d6fbe785\") " pod="e2e-models-e2e-worker-w4-214fea/simulator-w4-214fea-kserve-5965bff56b-mkmt9"
Sep 17 03:32:41.811810 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:41.811748    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f29ebfeb-d96e-41de-b4e5-c5f1d6fbe785-tmp-dir\") pod \"simulator-w4-214fea-kserve-5965bff56b-mkmt9\" (UID: \"f29ebfeb-d96e-41de-b4e5-c5f1d6fbe785\") " pod="e2e-models-e2e-worker-w4-214fea/simulator-w4-214fea-kserve-5965bff56b-mkmt9"
Sep 17 03:32:41.913053 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:41.913021    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f29ebfeb-d96e-41de-b4e5-c5f1d6fbe785-tmp-dir\") pod \"simulator-w4-214fea-kserve-5965bff56b-mkmt9\" (UID: \"f29ebfeb-d96e-41de-b4e5-c5f1d6fbe785\") " pod="e2e-models-e2e-worker-w4-214fea/simulator-w4-214fea-kserve-5965bff56b-mkmt9"
Sep 17 03:32:41.913220 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:41.913087    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jpz8\" (UniqueName: \"kubernetes.io/projected/f29ebfeb-d96e-41de-b4e5-c5f1d6fbe785-kube-api-access-2jpz8\") pod \"simulator-w4-214fea-kserve-5965bff56b-mkmt9\" (UID: \"f29ebfeb-d96e-41de-b4e5-c5f1d6fbe785\") " pod="e2e-models-e2e-worker-w4-214fea/simulator-w4-214fea-kserve-5965bff56b-mkmt9"
Sep 17 03:32:41.913220 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:41.913111    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f29ebfeb-d96e-41de-b4e5-c5f1d6fbe785-model-cache\") pod \"simulator-w4-214fea-kserve-5965bff56b-mkmt9\" (UID: \"f29ebfeb-d96e-41de-b4e5-c5f1d6fbe785\") " pod="e2e-models-e2e-worker-w4-214fea/simulator-w4-214fea-kserve-5965bff56b-mkmt9"
Sep 17 03:32:41.913220 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:41.913135    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f29ebfeb-d96e-41de-b4e5-c5f1d6fbe785-home\") pod \"simulator-w4-214fea-kserve-5965bff56b-mkmt9\" (UID: \"f29ebfeb-d96e-41de-b4e5-c5f1d6fbe785\") " pod="e2e-models-e2e-worker-w4-214fea/simulator-w4-214fea-kserve-5965bff56b-mkmt9"
Sep 17 03:32:41.913220 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:41.913163    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f29ebfeb-d96e-41de-b4e5-c5f1d6fbe785-dshm\") pod \"simulator-w4-214fea-kserve-5965bff56b-mkmt9\" (UID: \"f29ebfeb-d96e-41de-b4e5-c5f1d6fbe785\") " pod="e2e-models-e2e-worker-w4-214fea/simulator-w4-214fea-kserve-5965bff56b-mkmt9"
Sep 17 03:32:41.913407 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:41.913307    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f29ebfeb-d96e-41de-b4e5-c5f1d6fbe785-tls-certs\") pod \"simulator-w4-214fea-kserve-5965bff56b-mkmt9\" (UID: \"f29ebfeb-d96e-41de-b4e5-c5f1d6fbe785\") " pod="e2e-models-e2e-worker-w4-214fea/simulator-w4-214fea-kserve-5965bff56b-mkmt9"
Sep 17 03:32:41.913510 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:41.913487    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f29ebfeb-d96e-41de-b4e5-c5f1d6fbe785-tmp-dir\") pod \"simulator-w4-214fea-kserve-5965bff56b-mkmt9\" (UID: \"f29ebfeb-d96e-41de-b4e5-c5f1d6fbe785\") " pod="e2e-models-e2e-worker-w4-214fea/simulator-w4-214fea-kserve-5965bff56b-mkmt9"
Sep 17 03:32:41.913554 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:41.913525    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f29ebfeb-d96e-41de-b4e5-c5f1d6fbe785-model-cache\") pod \"simulator-w4-214fea-kserve-5965bff56b-mkmt9\" (UID: \"f29ebfeb-d96e-41de-b4e5-c5f1d6fbe785\") " pod="e2e-models-e2e-worker-w4-214fea/simulator-w4-214fea-kserve-5965bff56b-mkmt9"
Sep 17 03:32:41.913554 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:41.913545    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f29ebfeb-d96e-41de-b4e5-c5f1d6fbe785-home\") pod \"simulator-w4-214fea-kserve-5965bff56b-mkmt9\" (UID: \"f29ebfeb-d96e-41de-b4e5-c5f1d6fbe785\") " pod="e2e-models-e2e-worker-w4-214fea/simulator-w4-214fea-kserve-5965bff56b-mkmt9"
Sep 17 03:32:41.915396 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:41.915377    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f29ebfeb-d96e-41de-b4e5-c5f1d6fbe785-dshm\") pod \"simulator-w4-214fea-kserve-5965bff56b-mkmt9\" (UID: \"f29ebfeb-d96e-41de-b4e5-c5f1d6fbe785\") " pod="e2e-models-e2e-worker-w4-214fea/simulator-w4-214fea-kserve-5965bff56b-mkmt9"
Sep 17 03:32:41.915906 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:41.915885    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f29ebfeb-d96e-41de-b4e5-c5f1d6fbe785-tls-certs\") pod \"simulator-w4-214fea-kserve-5965bff56b-mkmt9\" (UID: \"f29ebfeb-d96e-41de-b4e5-c5f1d6fbe785\") " pod="e2e-models-e2e-worker-w4-214fea/simulator-w4-214fea-kserve-5965bff56b-mkmt9"
Sep 17 03:32:41.929209 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:41.929184    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jpz8\" (UniqueName: \"kubernetes.io/projected/f29ebfeb-d96e-41de-b4e5-c5f1d6fbe785-kube-api-access-2jpz8\") pod \"simulator-w4-214fea-kserve-5965bff56b-mkmt9\" (UID: \"f29ebfeb-d96e-41de-b4e5-c5f1d6fbe785\") " pod="e2e-models-e2e-worker-w4-214fea/simulator-w4-214fea-kserve-5965bff56b-mkmt9"
Sep 17 03:32:41.992152 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:41.992129    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-0ae566a4-gw-openshift-default-7ff59f8f76-9zf22"
Sep 17 03:32:41.993523 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:41.993504    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-0ae566a4-gw-openshift-default-7ff59f8f76-9zf22"
Sep 17 03:32:42.085250 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:42.085154    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-214fea/simulator-w4-214fea-kserve-5965bff56b-mkmt9"
Sep 17 03:32:42.356906 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:42.356882    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-214fea/simulator-w4-214fea-kserve-5965bff56b-mkmt9"]
Sep 17 03:32:42.358812 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:32:42.358783    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf29ebfeb_d96e_41de_b4e5_c5f1d6fbe785.slice/crio-c0cb5fe07f472fbba59c2bd5816dbcfafb7db4c2f1ef786d5a099183c11922e4 WatchSource:0}: Error finding container c0cb5fe07f472fbba59c2bd5816dbcfafb7db4c2f1ef786d5a099183c11922e4: Status 404 returned error can't find the container with id c0cb5fe07f472fbba59c2bd5816dbcfafb7db4c2f1ef786d5a099183c11922e4
Sep 17 03:32:42.998659 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:42.998621    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-214fea/simulator-w4-214fea-kserve-5965bff56b-mkmt9" event={"ID":"f29ebfeb-d96e-41de-b4e5-c5f1d6fbe785","Type":"ContainerStarted","Data":"8fb54ab60850e74ec523ec3ada66a1c496063a36c6c973a6605df2a9b8f3e852"}
Sep 17 03:32:42.998659 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:42.998665    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-214fea/simulator-w4-214fea-kserve-5965bff56b-mkmt9" event={"ID":"f29ebfeb-d96e-41de-b4e5-c5f1d6fbe785","Type":"ContainerStarted","Data":"c0cb5fe07f472fbba59c2bd5816dbcfafb7db4c2f1ef786d5a099183c11922e4"}
Sep 17 03:32:43.016152 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:43.016107    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-214fea/simulator-w4-214fea-kserve-5965bff56b-mkmt9" podStartSLOduration=2.016091943 podStartE2EDuration="2.016091943s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 03:32:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:32:43.014181252 +0000 UTC m=+798.480588779" watchObservedRunningTime="2026-09-17 03:32:43.016091943 +0000 UTC m=+798.482499469"
Sep 17 03:32:44.151541 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:44.151519    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c0b7e751-openshift-default-5c659ddbd8-sbrd4"
Sep 17 03:32:44.237246 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:44.237177    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5adcfc6-e1c3-4e82-b719-51e7fa179a5c-workload-socket\") pod \"e5adcfc6-e1c3-4e82-b719-51e7fa179a5c\" (UID: \"e5adcfc6-e1c3-4e82-b719-51e7fa179a5c\") "
Sep 17 03:32:44.237246 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:44.237220    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5adcfc6-e1c3-4e82-b719-51e7fa179a5c-istio-token\") pod \"e5adcfc6-e1c3-4e82-b719-51e7fa179a5c\" (UID: \"e5adcfc6-e1c3-4e82-b719-51e7fa179a5c\") "
Sep 17 03:32:44.237442 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:44.237270    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5adcfc6-e1c3-4e82-b719-51e7fa179a5c-workload-certs\") pod \"e5adcfc6-e1c3-4e82-b719-51e7fa179a5c\" (UID: \"e5adcfc6-e1c3-4e82-b719-51e7fa179a5c\") "
Sep 17 03:32:44.237442 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:44.237293    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5adcfc6-e1c3-4e82-b719-51e7fa179a5c-istio-podinfo\") pod \"e5adcfc6-e1c3-4e82-b719-51e7fa179a5c\" (UID: \"e5adcfc6-e1c3-4e82-b719-51e7fa179a5c\") "
Sep 17 03:32:44.237442 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:44.237319    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5adcfc6-e1c3-4e82-b719-51e7fa179a5c-istiod-ca-cert\") pod \"e5adcfc6-e1c3-4e82-b719-51e7fa179a5c\" (UID: \"e5adcfc6-e1c3-4e82-b719-51e7fa179a5c\") "
Sep 17 03:32:44.237442 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:44.237354    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5adcfc6-e1c3-4e82-b719-51e7fa179a5c-credential-socket\") pod \"e5adcfc6-e1c3-4e82-b719-51e7fa179a5c\" (UID: \"e5adcfc6-e1c3-4e82-b719-51e7fa179a5c\") "
Sep 17 03:32:44.237442 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:44.237415    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5adcfc6-e1c3-4e82-b719-51e7fa179a5c-istio-data\") pod \"e5adcfc6-e1c3-4e82-b719-51e7fa179a5c\" (UID: \"e5adcfc6-e1c3-4e82-b719-51e7fa179a5c\") "
Sep 17 03:32:44.237442 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:44.237421    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5adcfc6-e1c3-4e82-b719-51e7fa179a5c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e5adcfc6-e1c3-4e82-b719-51e7fa179a5c" (UID: "e5adcfc6-e1c3-4e82-b719-51e7fa179a5c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:32:44.237799 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:44.237509    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jpdtd\" (UniqueName: \"kubernetes.io/projected/e5adcfc6-e1c3-4e82-b719-51e7fa179a5c-kube-api-access-jpdtd\") pod \"e5adcfc6-e1c3-4e82-b719-51e7fa179a5c\" (UID: \"e5adcfc6-e1c3-4e82-b719-51e7fa179a5c\") "
Sep 17 03:32:44.237799 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:44.237539    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5adcfc6-e1c3-4e82-b719-51e7fa179a5c-istio-envoy\") pod \"e5adcfc6-e1c3-4e82-b719-51e7fa179a5c\" (UID: \"e5adcfc6-e1c3-4e82-b719-51e7fa179a5c\") "
Sep 17 03:32:44.237799 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:44.237616    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5adcfc6-e1c3-4e82-b719-51e7fa179a5c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e5adcfc6-e1c3-4e82-b719-51e7fa179a5c" (UID: "e5adcfc6-e1c3-4e82-b719-51e7fa179a5c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:32:44.237799 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:44.237671    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5adcfc6-e1c3-4e82-b719-51e7fa179a5c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e5adcfc6-e1c3-4e82-b719-51e7fa179a5c" (UID: "e5adcfc6-e1c3-4e82-b719-51e7fa179a5c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:32:44.237799 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:44.237761    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5adcfc6-e1c3-4e82-b719-51e7fa179a5c-istio-data" (OuterVolumeSpecName: "istio-data") pod "e5adcfc6-e1c3-4e82-b719-51e7fa179a5c" (UID: "e5adcfc6-e1c3-4e82-b719-51e7fa179a5c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:32:44.238054 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:44.237926    2552 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5adcfc6-e1c3-4e82-b719-51e7fa179a5c-workload-certs\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:32:44.238054 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:44.237944    2552 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5adcfc6-e1c3-4e82-b719-51e7fa179a5c-credential-socket\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:32:44.238054 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:44.237957    2552 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5adcfc6-e1c3-4e82-b719-51e7fa179a5c-istio-data\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:32:44.238054 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:44.237973    2552 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5adcfc6-e1c3-4e82-b719-51e7fa179a5c-workload-socket\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:32:44.238250 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:44.238215    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5adcfc6-e1c3-4e82-b719-51e7fa179a5c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e5adcfc6-e1c3-4e82-b719-51e7fa179a5c" (UID: "e5adcfc6-e1c3-4e82-b719-51e7fa179a5c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 03:32:44.240116 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:44.240080    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5adcfc6-e1c3-4e82-b719-51e7fa179a5c-istio-token" (OuterVolumeSpecName: "istio-token") pod "e5adcfc6-e1c3-4e82-b719-51e7fa179a5c" (UID: "e5adcfc6-e1c3-4e82-b719-51e7fa179a5c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 03:32:44.240573 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:44.240320    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5adcfc6-e1c3-4e82-b719-51e7fa179a5c-kube-api-access-jpdtd" (OuterVolumeSpecName: "kube-api-access-jpdtd") pod "e5adcfc6-e1c3-4e82-b719-51e7fa179a5c" (UID: "e5adcfc6-e1c3-4e82-b719-51e7fa179a5c"). InnerVolumeSpecName "kube-api-access-jpdtd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 03:32:44.240573 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:44.240321    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e5adcfc6-e1c3-4e82-b719-51e7fa179a5c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e5adcfc6-e1c3-4e82-b719-51e7fa179a5c" (UID: "e5adcfc6-e1c3-4e82-b719-51e7fa179a5c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 17 03:32:44.240870 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:44.240848    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5adcfc6-e1c3-4e82-b719-51e7fa179a5c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e5adcfc6-e1c3-4e82-b719-51e7fa179a5c" (UID: "e5adcfc6-e1c3-4e82-b719-51e7fa179a5c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:32:44.338729 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:44.338687    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jpdtd\" (UniqueName: \"kubernetes.io/projected/e5adcfc6-e1c3-4e82-b719-51e7fa179a5c-kube-api-access-jpdtd\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:32:44.338729 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:44.338723    2552 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5adcfc6-e1c3-4e82-b719-51e7fa179a5c-istio-envoy\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:32:44.338729 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:44.338735    2552 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5adcfc6-e1c3-4e82-b719-51e7fa179a5c-istio-token\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:32:44.338729 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:44.338743    2552 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5adcfc6-e1c3-4e82-b719-51e7fa179a5c-istio-podinfo\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:32:44.338995 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:44.338750    2552 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5adcfc6-e1c3-4e82-b719-51e7fa179a5c-istiod-ca-cert\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:32:45.010004 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:45.009972    2552 generic.go:356] "Generic (PLEG): container finished" podID="e5adcfc6-e1c3-4e82-b719-51e7fa179a5c" containerID="4d08ff2537519c790e3a31f93c8d9029ae671c361c9ce44bcfd7dbbf36d2caa8" exitCode=0
Sep 17 03:32:45.010160 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:45.010051    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c0b7e751-openshift-default-5c659ddbd8-sbrd4"
Sep 17 03:32:45.010160 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:45.010075    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c0b7e751-openshift-default-5c659ddbd8-sbrd4" event={"ID":"e5adcfc6-e1c3-4e82-b719-51e7fa179a5c","Type":"ContainerDied","Data":"4d08ff2537519c790e3a31f93c8d9029ae671c361c9ce44bcfd7dbbf36d2caa8"}
Sep 17 03:32:45.010269 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:45.010198    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c0b7e751-openshift-default-5c659ddbd8-sbrd4" event={"ID":"e5adcfc6-e1c3-4e82-b719-51e7fa179a5c","Type":"ContainerDied","Data":"161c0cea8b705f59dbfbacf0e0940849edec8a803fe1696aab09ed1aa13d54a7"}
Sep 17 03:32:45.010269 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:45.010260    2552 scope.go:117] "RemoveContainer" containerID="4d08ff2537519c790e3a31f93c8d9029ae671c361c9ce44bcfd7dbbf36d2caa8"
Sep 17 03:32:45.019500 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:45.019482    2552 scope.go:117] "RemoveContainer" containerID="582a4472c6ed4fc6062d650e536e32e48d9f0efbee667ee66457448fadcb2f26"
Sep 17 03:32:45.028033 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:45.028012    2552 scope.go:117] "RemoveContainer" containerID="4d08ff2537519c790e3a31f93c8d9029ae671c361c9ce44bcfd7dbbf36d2caa8"
Sep 17 03:32:45.028310 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:32:45.028287    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d08ff2537519c790e3a31f93c8d9029ae671c361c9ce44bcfd7dbbf36d2caa8\": container with ID starting with 4d08ff2537519c790e3a31f93c8d9029ae671c361c9ce44bcfd7dbbf36d2caa8 not found: ID does not exist" containerID="4d08ff2537519c790e3a31f93c8d9029ae671c361c9ce44bcfd7dbbf36d2caa8"
Sep 17 03:32:45.028381 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:45.028320    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d08ff2537519c790e3a31f93c8d9029ae671c361c9ce44bcfd7dbbf36d2caa8"} err="failed to get container status \"4d08ff2537519c790e3a31f93c8d9029ae671c361c9ce44bcfd7dbbf36d2caa8\": rpc error: code = NotFound desc = could not find container \"4d08ff2537519c790e3a31f93c8d9029ae671c361c9ce44bcfd7dbbf36d2caa8\": container with ID starting with 4d08ff2537519c790e3a31f93c8d9029ae671c361c9ce44bcfd7dbbf36d2caa8 not found: ID does not exist"
Sep 17 03:32:45.028381 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:45.028346    2552 scope.go:117] "RemoveContainer" containerID="582a4472c6ed4fc6062d650e536e32e48d9f0efbee667ee66457448fadcb2f26"
Sep 17 03:32:45.028618 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:32:45.028573    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"582a4472c6ed4fc6062d650e536e32e48d9f0efbee667ee66457448fadcb2f26\": container with ID starting with 582a4472c6ed4fc6062d650e536e32e48d9f0efbee667ee66457448fadcb2f26 not found: ID does not exist" containerID="582a4472c6ed4fc6062d650e536e32e48d9f0efbee667ee66457448fadcb2f26"
Sep 17 03:32:45.028688 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:45.028625    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"582a4472c6ed4fc6062d650e536e32e48d9f0efbee667ee66457448fadcb2f26"} err="failed to get container status \"582a4472c6ed4fc6062d650e536e32e48d9f0efbee667ee66457448fadcb2f26\": rpc error: code = NotFound desc = could not find container \"582a4472c6ed4fc6062d650e536e32e48d9f0efbee667ee66457448fadcb2f26\": container with ID starting with 582a4472c6ed4fc6062d650e536e32e48d9f0efbee667ee66457448fadcb2f26 not found: ID does not exist"
Sep 17 03:32:45.029208 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:45.029190    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-c0b7e751-openshift-default-5c659ddbd8-sbrd4"]
Sep 17 03:32:45.040083 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:45.040063    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-c0b7e751-openshift-default-5c659ddbd8-sbrd4"]
Sep 17 03:32:46.986257 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:46.986220    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e5adcfc6-e1c3-4e82-b719-51e7fa179a5c" path="/var/lib/kubelet/pods/e5adcfc6-e1c3-4e82-b719-51e7fa179a5c/volumes"
Sep 17 03:32:52.085832 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:52.085802    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-214fea/simulator-w4-214fea-kserve-5965bff56b-mkmt9"
Sep 17 03:32:52.086262 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:52.085849    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-214fea/simulator-w4-214fea-kserve-5965bff56b-mkmt9"
Sep 17 03:32:52.098415 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:52.098390    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-214fea/simulator-w4-214fea-kserve-5965bff56b-mkmt9"
Sep 17 03:32:53.059723 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:32:53.059695    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-214fea/simulator-w4-214fea-kserve-5965bff56b-mkmt9"
Sep 17 03:33:06.123114 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:06.123079    2552 generic.go:356] "Generic (PLEG): container finished" podID="0c1e74de-3dc7-463d-acde-e9ab14a22970" containerID="5679717a5818448203f9acd08d33c0af4b322b48ff873661702c4caab81bd67b" exitCode=0
Sep 17 03:33:06.123536 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:06.123152    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-0ae566a4-gw-openshift-default-7ff59f8f76-9zf22" event={"ID":"0c1e74de-3dc7-463d-acde-e9ab14a22970","Type":"ContainerDied","Data":"5679717a5818448203f9acd08d33c0af4b322b48ff873661702c4caab81bd67b"}
Sep 17 03:33:06.123584 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:06.123567    2552 scope.go:117] "RemoveContainer" containerID="5679717a5818448203f9acd08d33c0af4b322b48ff873661702c4caab81bd67b"
Sep 17 03:33:06.124716 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:06.124687    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 03:33:06.124826 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:06.124770    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 03:33:06.124826 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:06.124818    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 03:33:07.131899 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:07.131855    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-0ae566a4-gw-openshift-default-7ff59f8f76-9zf22" event={"ID":"0c1e74de-3dc7-463d-acde-e9ab14a22970","Type":"ContainerStarted","Data":"145fa649ee25077b241bed216e6df5a57ccaf89c5e87a288d63f2353292e2fc8"}
Sep 17 03:33:07.438669 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:07.435359    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0ae566a4-2hczz"]
Sep 17 03:33:07.438669 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:07.436099    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5adcfc6-e1c3-4e82-b719-51e7fa179a5c" containerName="istio-proxy"
Sep 17 03:33:07.438669 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:07.436119    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5adcfc6-e1c3-4e82-b719-51e7fa179a5c" containerName="istio-proxy"
Sep 17 03:33:07.438669 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:07.436135    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5adcfc6-e1c3-4e82-b719-51e7fa179a5c" containerName="istio-proxy"
Sep 17 03:33:07.438669 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:07.436144    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5adcfc6-e1c3-4e82-b719-51e7fa179a5c" containerName="istio-proxy"
Sep 17 03:33:07.438669 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:07.436241    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5adcfc6-e1c3-4e82-b719-51e7fa179a5c" containerName="istio-proxy"
Sep 17 03:33:07.438669 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:07.436255    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5adcfc6-e1c3-4e82-b719-51e7fa179a5c" containerName="istio-proxy"
Sep 17 03:33:07.449877 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:07.449849    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0ae566a4-2hczz"]
Sep 17 03:33:07.450107 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:07.449965    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0ae566a4-2hczz"
Sep 17 03:33:07.460025 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:07.454900    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8vwnw\""
Sep 17 03:33:07.559641 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:07.559580    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6af7334-f3d0-4e02-9f82-f4492c59cf72-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0ae566a4-2hczz\" (UID: \"b6af7334-f3d0-4e02-9f82-f4492c59cf72\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0ae566a4-2hczz"
Sep 17 03:33:07.660380 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:07.660331    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6af7334-f3d0-4e02-9f82-f4492c59cf72-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0ae566a4-2hczz\" (UID: \"b6af7334-f3d0-4e02-9f82-f4492c59cf72\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0ae566a4-2hczz"
Sep 17 03:33:07.660999 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:07.660975    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6af7334-f3d0-4e02-9f82-f4492c59cf72-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0ae566a4-2hczz\" (UID: \"b6af7334-f3d0-4e02-9f82-f4492c59cf72\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0ae566a4-2hczz"
Sep 17 03:33:07.772283 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:07.772249    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0ae566a4-2hczz"
Sep 17 03:33:08.053272 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:08.053197    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0ae566a4-2hczz"]
Sep 17 03:33:08.054359 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:33:08.054325    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6af7334_f3d0_4e02_9f82_f4492c59cf72.slice/crio-cd6f7ed13a42efadf6435249bafb502926f18ba43f986a7ee704bea91e377058 WatchSource:0}: Error finding container cd6f7ed13a42efadf6435249bafb502926f18ba43f986a7ee704bea91e377058: Status 404 returned error can't find the container with id cd6f7ed13a42efadf6435249bafb502926f18ba43f986a7ee704bea91e377058
Sep 17 03:33:08.077663 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:08.077638    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-0ae566a4-gw-openshift-default-7ff59f8f76-9zf22"
Sep 17 03:33:08.078773 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:08.078742    2552 patch_prober.go:28] interesting pod/e2e-migrate-0ae566a4-gw-openshift-default-7ff59f8f76-9zf22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Sep 17 03:33:08.078878 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:08.078815    2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-0ae566a4-gw-openshift-default-7ff59f8f76-9zf22" podUID="0c1e74de-3dc7-463d-acde-e9ab14a22970" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Sep 17 03:33:08.142232 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:08.142179    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0ae566a4-2hczz" event={"ID":"b6af7334-f3d0-4e02-9f82-f4492c59cf72","Type":"ContainerStarted","Data":"cd6f7ed13a42efadf6435249bafb502926f18ba43f986a7ee704bea91e377058"}
Sep 17 03:33:09.078392 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:09.078354    2552 patch_prober.go:28] interesting pod/e2e-migrate-0ae566a4-gw-openshift-default-7ff59f8f76-9zf22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Sep 17 03:33:09.078560 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:09.078420    2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-0ae566a4-gw-openshift-default-7ff59f8f76-9zf22" podUID="0c1e74de-3dc7-463d-acde-e9ab14a22970" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Sep 17 03:33:09.151321 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:09.151280    2552 generic.go:356] "Generic (PLEG): container finished" podID="b6af7334-f3d0-4e02-9f82-f4492c59cf72" containerID="53d8a9fce678ee7d2a52850f6d29a89c52d4a882c1e7f5078c13dd7c7fd86a37" exitCode=0
Sep 17 03:33:09.151811 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:09.151378    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0ae566a4-2hczz" event={"ID":"b6af7334-f3d0-4e02-9f82-f4492c59cf72","Type":"ContainerDied","Data":"53d8a9fce678ee7d2a52850f6d29a89c52d4a882c1e7f5078c13dd7c7fd86a37"}
Sep 17 03:33:10.078217 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:10.078180    2552 patch_prober.go:28] interesting pod/e2e-migrate-0ae566a4-gw-openshift-default-7ff59f8f76-9zf22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Sep 17 03:33:10.078413 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:10.078234    2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-0ae566a4-gw-openshift-default-7ff59f8f76-9zf22" podUID="0c1e74de-3dc7-463d-acde-e9ab14a22970" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Sep 17 03:33:10.312397 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:10.312375    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0ae566a4-2hczz"
Sep 17 03:33:10.387283 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:10.387200    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6af7334-f3d0-4e02-9f82-f4492c59cf72-maas-api-service-ca\") pod \"b6af7334-f3d0-4e02-9f82-f4492c59cf72\" (UID: \"b6af7334-f3d0-4e02-9f82-f4492c59cf72\") "
Sep 17 03:33:10.387532 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:10.387510    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6af7334-f3d0-4e02-9f82-f4492c59cf72-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b6af7334-f3d0-4e02-9f82-f4492c59cf72" (UID: "b6af7334-f3d0-4e02-9f82-f4492c59cf72"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 03:33:10.487859 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:10.487829    2552 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6af7334-f3d0-4e02-9f82-f4492c59cf72-maas-api-service-ca\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:33:11.077793 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:11.077762    2552 patch_prober.go:28] interesting pod/e2e-migrate-0ae566a4-gw-openshift-default-7ff59f8f76-9zf22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Sep 17 03:33:11.077957 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:11.077816    2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-0ae566a4-gw-openshift-default-7ff59f8f76-9zf22" podUID="0c1e74de-3dc7-463d-acde-e9ab14a22970" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Sep 17 03:33:11.163607 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:11.163564    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0ae566a4-2hczz" event={"ID":"b6af7334-f3d0-4e02-9f82-f4492c59cf72","Type":"ContainerDied","Data":"cd6f7ed13a42efadf6435249bafb502926f18ba43f986a7ee704bea91e377058"}
Sep 17 03:33:11.163774 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:11.163591    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0ae566a4-2hczz"
Sep 17 03:33:11.163774 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:11.163637    2552 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cd6f7ed13a42efadf6435249bafb502926f18ba43f986a7ee704bea91e377058"
Sep 17 03:33:11.992337 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:11.992292    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-0ae566a4-gw-openshift-default-7ff59f8f76-9zf22"
Sep 17 03:33:12.078431 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:12.078396    2552 patch_prober.go:28] interesting pod/e2e-migrate-0ae566a4-gw-openshift-default-7ff59f8f76-9zf22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Sep 17 03:33:12.078643 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:12.078466    2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-0ae566a4-gw-openshift-default-7ff59f8f76-9zf22" podUID="0c1e74de-3dc7-463d-acde-e9ab14a22970" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Sep 17 03:33:12.277940 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:12.277899    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-1afd34/simulator-w0-1afd34-kserve-6c5ffdd47b-v6h59"]
Sep 17 03:33:12.278590 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:12.278568    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6af7334-f3d0-4e02-9f82-f4492c59cf72" containerName="revoke-keys"
Sep 17 03:33:12.278649 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:12.278612    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6af7334-f3d0-4e02-9f82-f4492c59cf72" containerName="revoke-keys"
Sep 17 03:33:12.278768 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:12.278755    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="b6af7334-f3d0-4e02-9f82-f4492c59cf72" containerName="revoke-keys"
Sep 17 03:33:12.282178 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:12.282154    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1afd34/simulator-w0-1afd34-kserve-6c5ffdd47b-v6h59"
Sep 17 03:33:12.284956 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:12.284935    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-1afd34\"/\"kube-root-ca.crt\""
Sep 17 03:33:12.286515 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:12.286492    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-1afd34\"/\"default-dockercfg-29l8s\""
Sep 17 03:33:12.286866 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:12.286842    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-1afd34\"/\"simulator-w0-1afd34-kserve-self-signed-certs\""
Sep 17 03:33:12.286986 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:12.286907    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-1afd34\"/\"openshift-service-ca.crt\""
Sep 17 03:33:12.290274 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:12.289985    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-1afd34/simulator-w0-1afd34-kserve-6c5ffdd47b-v6h59"]
Sep 17 03:33:12.302315 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:12.302283    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5886143a-fc36-40eb-8818-b2c94767b529-dshm\") pod \"simulator-w0-1afd34-kserve-6c5ffdd47b-v6h59\" (UID: \"5886143a-fc36-40eb-8818-b2c94767b529\") " pod="e2e-models-e2e-worker-w0-1afd34/simulator-w0-1afd34-kserve-6c5ffdd47b-v6h59"
Sep 17 03:33:12.302498 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:12.302320    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hxq9\" (UniqueName: \"kubernetes.io/projected/5886143a-fc36-40eb-8818-b2c94767b529-kube-api-access-8hxq9\") pod \"simulator-w0-1afd34-kserve-6c5ffdd47b-v6h59\" (UID: \"5886143a-fc36-40eb-8818-b2c94767b529\") " pod="e2e-models-e2e-worker-w0-1afd34/simulator-w0-1afd34-kserve-6c5ffdd47b-v6h59"
Sep 17 03:33:12.302498 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:12.302358    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5886143a-fc36-40eb-8818-b2c94767b529-tmp-dir\") pod \"simulator-w0-1afd34-kserve-6c5ffdd47b-v6h59\" (UID: \"5886143a-fc36-40eb-8818-b2c94767b529\") " pod="e2e-models-e2e-worker-w0-1afd34/simulator-w0-1afd34-kserve-6c5ffdd47b-v6h59"
Sep 17 03:33:12.302498 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:12.302436    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5886143a-fc36-40eb-8818-b2c94767b529-home\") pod \"simulator-w0-1afd34-kserve-6c5ffdd47b-v6h59\" (UID: \"5886143a-fc36-40eb-8818-b2c94767b529\") " pod="e2e-models-e2e-worker-w0-1afd34/simulator-w0-1afd34-kserve-6c5ffdd47b-v6h59"
Sep 17 03:33:12.302720 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:12.302557    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5886143a-fc36-40eb-8818-b2c94767b529-model-cache\") pod \"simulator-w0-1afd34-kserve-6c5ffdd47b-v6h59\" (UID: \"5886143a-fc36-40eb-8818-b2c94767b529\") " pod="e2e-models-e2e-worker-w0-1afd34/simulator-w0-1afd34-kserve-6c5ffdd47b-v6h59"
Sep 17 03:33:12.302720 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:12.302615    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5886143a-fc36-40eb-8818-b2c94767b529-tls-certs\") pod \"simulator-w0-1afd34-kserve-6c5ffdd47b-v6h59\" (UID: \"5886143a-fc36-40eb-8818-b2c94767b529\") " pod="e2e-models-e2e-worker-w0-1afd34/simulator-w0-1afd34-kserve-6c5ffdd47b-v6h59"
Sep 17 03:33:12.403265 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:12.403237    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5886143a-fc36-40eb-8818-b2c94767b529-tmp-dir\") pod \"simulator-w0-1afd34-kserve-6c5ffdd47b-v6h59\" (UID: \"5886143a-fc36-40eb-8818-b2c94767b529\") " pod="e2e-models-e2e-worker-w0-1afd34/simulator-w0-1afd34-kserve-6c5ffdd47b-v6h59"
Sep 17 03:33:12.403442 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:12.403268    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5886143a-fc36-40eb-8818-b2c94767b529-home\") pod \"simulator-w0-1afd34-kserve-6c5ffdd47b-v6h59\" (UID: \"5886143a-fc36-40eb-8818-b2c94767b529\") " pod="e2e-models-e2e-worker-w0-1afd34/simulator-w0-1afd34-kserve-6c5ffdd47b-v6h59"
Sep 17 03:33:12.403442 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:12.403328    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5886143a-fc36-40eb-8818-b2c94767b529-model-cache\") pod \"simulator-w0-1afd34-kserve-6c5ffdd47b-v6h59\" (UID: \"5886143a-fc36-40eb-8818-b2c94767b529\") " pod="e2e-models-e2e-worker-w0-1afd34/simulator-w0-1afd34-kserve-6c5ffdd47b-v6h59"
Sep 17 03:33:12.403442 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:12.403351    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5886143a-fc36-40eb-8818-b2c94767b529-tls-certs\") pod \"simulator-w0-1afd34-kserve-6c5ffdd47b-v6h59\" (UID: \"5886143a-fc36-40eb-8818-b2c94767b529\") " pod="e2e-models-e2e-worker-w0-1afd34/simulator-w0-1afd34-kserve-6c5ffdd47b-v6h59"
Sep 17 03:33:12.403442 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:12.403387    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5886143a-fc36-40eb-8818-b2c94767b529-dshm\") pod \"simulator-w0-1afd34-kserve-6c5ffdd47b-v6h59\" (UID: \"5886143a-fc36-40eb-8818-b2c94767b529\") " pod="e2e-models-e2e-worker-w0-1afd34/simulator-w0-1afd34-kserve-6c5ffdd47b-v6h59"
Sep 17 03:33:12.403442 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:12.403403    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hxq9\" (UniqueName: \"kubernetes.io/projected/5886143a-fc36-40eb-8818-b2c94767b529-kube-api-access-8hxq9\") pod \"simulator-w0-1afd34-kserve-6c5ffdd47b-v6h59\" (UID: \"5886143a-fc36-40eb-8818-b2c94767b529\") " pod="e2e-models-e2e-worker-w0-1afd34/simulator-w0-1afd34-kserve-6c5ffdd47b-v6h59"
Sep 17 03:33:12.403742 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:12.403716    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5886143a-fc36-40eb-8818-b2c94767b529-tmp-dir\") pod \"simulator-w0-1afd34-kserve-6c5ffdd47b-v6h59\" (UID: \"5886143a-fc36-40eb-8818-b2c94767b529\") " pod="e2e-models-e2e-worker-w0-1afd34/simulator-w0-1afd34-kserve-6c5ffdd47b-v6h59"
Sep 17 03:33:12.403807 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:12.403744    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5886143a-fc36-40eb-8818-b2c94767b529-home\") pod \"simulator-w0-1afd34-kserve-6c5ffdd47b-v6h59\" (UID: \"5886143a-fc36-40eb-8818-b2c94767b529\") " pod="e2e-models-e2e-worker-w0-1afd34/simulator-w0-1afd34-kserve-6c5ffdd47b-v6h59"
Sep 17 03:33:12.403807 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:12.403777    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5886143a-fc36-40eb-8818-b2c94767b529-model-cache\") pod \"simulator-w0-1afd34-kserve-6c5ffdd47b-v6h59\" (UID: \"5886143a-fc36-40eb-8818-b2c94767b529\") " pod="e2e-models-e2e-worker-w0-1afd34/simulator-w0-1afd34-kserve-6c5ffdd47b-v6h59"
Sep 17 03:33:12.405636 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:12.405615    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5886143a-fc36-40eb-8818-b2c94767b529-dshm\") pod \"simulator-w0-1afd34-kserve-6c5ffdd47b-v6h59\" (UID: \"5886143a-fc36-40eb-8818-b2c94767b529\") " pod="e2e-models-e2e-worker-w0-1afd34/simulator-w0-1afd34-kserve-6c5ffdd47b-v6h59"
Sep 17 03:33:12.405981 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:12.405962    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5886143a-fc36-40eb-8818-b2c94767b529-tls-certs\") pod \"simulator-w0-1afd34-kserve-6c5ffdd47b-v6h59\" (UID: \"5886143a-fc36-40eb-8818-b2c94767b529\") " pod="e2e-models-e2e-worker-w0-1afd34/simulator-w0-1afd34-kserve-6c5ffdd47b-v6h59"
Sep 17 03:33:12.411702 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:12.411684    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hxq9\" (UniqueName: \"kubernetes.io/projected/5886143a-fc36-40eb-8818-b2c94767b529-kube-api-access-8hxq9\") pod \"simulator-w0-1afd34-kserve-6c5ffdd47b-v6h59\" (UID: \"5886143a-fc36-40eb-8818-b2c94767b529\") " pod="e2e-models-e2e-worker-w0-1afd34/simulator-w0-1afd34-kserve-6c5ffdd47b-v6h59"
Sep 17 03:33:12.593753 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:12.593663    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1afd34/simulator-w0-1afd34-kserve-6c5ffdd47b-v6h59"
Sep 17 03:33:12.867664 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:12.867637    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-1afd34/simulator-w0-1afd34-kserve-6c5ffdd47b-v6h59"]
Sep 17 03:33:12.869464 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:33:12.869438    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5886143a_fc36_40eb_8818_b2c94767b529.slice/crio-26544e43367797df736b5f932512c3ba0f5aa954bf34d8a9f7fd3670b3e08dc6 WatchSource:0}: Error finding container 26544e43367797df736b5f932512c3ba0f5aa954bf34d8a9f7fd3670b3e08dc6: Status 404 returned error can't find the container with id 26544e43367797df736b5f932512c3ba0f5aa954bf34d8a9f7fd3670b3e08dc6
Sep 17 03:33:13.082039 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:13.082010    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-0ae566a4-gw-openshift-default-7ff59f8f76-9zf22"
Sep 17 03:33:13.082902 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:13.082883    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-0ae566a4-gw-openshift-default-7ff59f8f76-9zf22"
Sep 17 03:33:13.175964 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:13.175885    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1afd34/simulator-w0-1afd34-kserve-6c5ffdd47b-v6h59" event={"ID":"5886143a-fc36-40eb-8818-b2c94767b529","Type":"ContainerStarted","Data":"b39473b5ea95e866130295afdb931f04ce58775c9297473637182719e2a79fcb"}
Sep 17 03:33:13.175964 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:13.175927    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1afd34/simulator-w0-1afd34-kserve-6c5ffdd47b-v6h59" event={"ID":"5886143a-fc36-40eb-8818-b2c94767b529","Type":"ContainerStarted","Data":"26544e43367797df736b5f932512c3ba0f5aa954bf34d8a9f7fd3670b3e08dc6"}
Sep 17 03:33:13.193976 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:13.193933    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-1afd34/simulator-w0-1afd34-kserve-6c5ffdd47b-v6h59" podStartSLOduration=1.193917501 podStartE2EDuration="1.193917501s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 03:33:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:33:13.193560431 +0000 UTC m=+828.659967958" watchObservedRunningTime="2026-09-17 03:33:13.193917501 +0000 UTC m=+828.660325085"
Sep 17 03:33:14.532083 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:14.532052    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0ae566a4-2hczz"]
Sep 17 03:33:14.536437 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:14.536411    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0ae566a4-2hczz"]
Sep 17 03:33:14.663474 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:14.663437    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-0ae566a4-gw-openshift-default-7ff59f8f76-9zf22"]
Sep 17 03:33:14.663694 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:14.663667    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-0ae566a4-gw-openshift-default-7ff59f8f76-9zf22" podUID="0c1e74de-3dc7-463d-acde-e9ab14a22970" containerName="istio-proxy" containerID="cri-o://145fa649ee25077b241bed216e6df5a57ccaf89c5e87a288d63f2353292e2fc8" gracePeriod=30
Sep 17 03:33:14.985721 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:14.985641    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b6af7334-f3d0-4e02-9f82-f4492c59cf72" path="/var/lib/kubelet/pods/b6af7334-f3d0-4e02-9f82-f4492c59cf72/volumes"
Sep 17 03:33:20.058717 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:20.058655    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-0ae566a4-gw-openshift-default-7ff59f8f76-9zf22"
Sep 17 03:33:20.181689 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:20.181648    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c1e74de-3dc7-463d-acde-e9ab14a22970-workload-socket\") pod \"0c1e74de-3dc7-463d-acde-e9ab14a22970\" (UID: \"0c1e74de-3dc7-463d-acde-e9ab14a22970\") "
Sep 17 03:33:20.181689 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:20.181687    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c1e74de-3dc7-463d-acde-e9ab14a22970-credential-socket\") pod \"0c1e74de-3dc7-463d-acde-e9ab14a22970\" (UID: \"0c1e74de-3dc7-463d-acde-e9ab14a22970\") "
Sep 17 03:33:20.181944 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:20.181704    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2nk4s\" (UniqueName: \"kubernetes.io/projected/0c1e74de-3dc7-463d-acde-e9ab14a22970-kube-api-access-2nk4s\") pod \"0c1e74de-3dc7-463d-acde-e9ab14a22970\" (UID: \"0c1e74de-3dc7-463d-acde-e9ab14a22970\") "
Sep 17 03:33:20.181944 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:20.181726    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c1e74de-3dc7-463d-acde-e9ab14a22970-istiod-ca-cert\") pod \"0c1e74de-3dc7-463d-acde-e9ab14a22970\" (UID: \"0c1e74de-3dc7-463d-acde-e9ab14a22970\") "
Sep 17 03:33:20.181944 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:20.181747    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c1e74de-3dc7-463d-acde-e9ab14a22970-workload-certs\") pod \"0c1e74de-3dc7-463d-acde-e9ab14a22970\" (UID: \"0c1e74de-3dc7-463d-acde-e9ab14a22970\") "
Sep 17 03:33:20.181944 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:20.181889    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c1e74de-3dc7-463d-acde-e9ab14a22970-istio-data\") pod \"0c1e74de-3dc7-463d-acde-e9ab14a22970\" (UID: \"0c1e74de-3dc7-463d-acde-e9ab14a22970\") "
Sep 17 03:33:20.181944 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:20.181927    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c1e74de-3dc7-463d-acde-e9ab14a22970-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0c1e74de-3dc7-463d-acde-e9ab14a22970" (UID: "0c1e74de-3dc7-463d-acde-e9ab14a22970"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:33:20.182221 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:20.181958    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c1e74de-3dc7-463d-acde-e9ab14a22970-istio-envoy\") pod \"0c1e74de-3dc7-463d-acde-e9ab14a22970\" (UID: \"0c1e74de-3dc7-463d-acde-e9ab14a22970\") "
Sep 17 03:33:20.182221 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:20.181966    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c1e74de-3dc7-463d-acde-e9ab14a22970-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0c1e74de-3dc7-463d-acde-e9ab14a22970" (UID: "0c1e74de-3dc7-463d-acde-e9ab14a22970"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:33:20.182221 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:20.182006    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c1e74de-3dc7-463d-acde-e9ab14a22970-istio-token\") pod \"0c1e74de-3dc7-463d-acde-e9ab14a22970\" (UID: \"0c1e74de-3dc7-463d-acde-e9ab14a22970\") "
Sep 17 03:33:20.182221 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:20.182078    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c1e74de-3dc7-463d-acde-e9ab14a22970-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0c1e74de-3dc7-463d-acde-e9ab14a22970" (UID: "0c1e74de-3dc7-463d-acde-e9ab14a22970"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 03:33:20.182221 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:20.182086    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c1e74de-3dc7-463d-acde-e9ab14a22970-istio-data" (OuterVolumeSpecName: "istio-data") pod "0c1e74de-3dc7-463d-acde-e9ab14a22970" (UID: "0c1e74de-3dc7-463d-acde-e9ab14a22970"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:33:20.182221 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:20.182086    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c1e74de-3dc7-463d-acde-e9ab14a22970-istio-podinfo\") pod \"0c1e74de-3dc7-463d-acde-e9ab14a22970\" (UID: \"0c1e74de-3dc7-463d-acde-e9ab14a22970\") "
Sep 17 03:33:20.182543 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:20.182334    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c1e74de-3dc7-463d-acde-e9ab14a22970-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0c1e74de-3dc7-463d-acde-e9ab14a22970" (UID: "0c1e74de-3dc7-463d-acde-e9ab14a22970"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:33:20.182543 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:20.182504    2552 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c1e74de-3dc7-463d-acde-e9ab14a22970-workload-socket\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:33:20.182543 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:20.182524    2552 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c1e74de-3dc7-463d-acde-e9ab14a22970-credential-socket\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:33:20.182543 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:20.182538    2552 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c1e74de-3dc7-463d-acde-e9ab14a22970-istiod-ca-cert\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:33:20.182776 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:20.182549    2552 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c1e74de-3dc7-463d-acde-e9ab14a22970-workload-certs\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:33:20.182776 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:20.182562    2552 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c1e74de-3dc7-463d-acde-e9ab14a22970-istio-data\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:33:20.184251 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:20.184218    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c1e74de-3dc7-463d-acde-e9ab14a22970-istio-token" (OuterVolumeSpecName: "istio-token") pod "0c1e74de-3dc7-463d-acde-e9ab14a22970" (UID: "0c1e74de-3dc7-463d-acde-e9ab14a22970"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 03:33:20.184370 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:20.184269    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0c1e74de-3dc7-463d-acde-e9ab14a22970-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0c1e74de-3dc7-463d-acde-e9ab14a22970" (UID: "0c1e74de-3dc7-463d-acde-e9ab14a22970"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 17 03:33:20.184435 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:20.184366    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c1e74de-3dc7-463d-acde-e9ab14a22970-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0c1e74de-3dc7-463d-acde-e9ab14a22970" (UID: "0c1e74de-3dc7-463d-acde-e9ab14a22970"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:33:20.184471 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:20.184437    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c1e74de-3dc7-463d-acde-e9ab14a22970-kube-api-access-2nk4s" (OuterVolumeSpecName: "kube-api-access-2nk4s") pod "0c1e74de-3dc7-463d-acde-e9ab14a22970" (UID: "0c1e74de-3dc7-463d-acde-e9ab14a22970"). InnerVolumeSpecName "kube-api-access-2nk4s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 03:33:20.214497 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:20.214471    2552 generic.go:356] "Generic (PLEG): container finished" podID="0c1e74de-3dc7-463d-acde-e9ab14a22970" containerID="145fa649ee25077b241bed216e6df5a57ccaf89c5e87a288d63f2353292e2fc8" exitCode=0
Sep 17 03:33:20.214636 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:20.214564    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-0ae566a4-gw-openshift-default-7ff59f8f76-9zf22"
Sep 17 03:33:20.214636 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:20.214567    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-0ae566a4-gw-openshift-default-7ff59f8f76-9zf22" event={"ID":"0c1e74de-3dc7-463d-acde-e9ab14a22970","Type":"ContainerDied","Data":"145fa649ee25077b241bed216e6df5a57ccaf89c5e87a288d63f2353292e2fc8"}
Sep 17 03:33:20.214744 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:20.214666    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-0ae566a4-gw-openshift-default-7ff59f8f76-9zf22" event={"ID":"0c1e74de-3dc7-463d-acde-e9ab14a22970","Type":"ContainerDied","Data":"e81fc9dd886a34c9bcfa86dcea03474ecf5b4ba12d13ea8106eb73555b28ca1d"}
Sep 17 03:33:20.214744 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:20.214692    2552 scope.go:117] "RemoveContainer" containerID="145fa649ee25077b241bed216e6df5a57ccaf89c5e87a288d63f2353292e2fc8"
Sep 17 03:33:20.227314 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:20.227298    2552 scope.go:117] "RemoveContainer" containerID="5679717a5818448203f9acd08d33c0af4b322b48ff873661702c4caab81bd67b"
Sep 17 03:33:20.238021 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:20.238002    2552 scope.go:117] "RemoveContainer" containerID="145fa649ee25077b241bed216e6df5a57ccaf89c5e87a288d63f2353292e2fc8"
Sep 17 03:33:20.238297 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:33:20.238276    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"145fa649ee25077b241bed216e6df5a57ccaf89c5e87a288d63f2353292e2fc8\": container with ID starting with 145fa649ee25077b241bed216e6df5a57ccaf89c5e87a288d63f2353292e2fc8 not found: ID does not exist" containerID="145fa649ee25077b241bed216e6df5a57ccaf89c5e87a288d63f2353292e2fc8"
Sep 17 03:33:20.238350 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:20.238307    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"145fa649ee25077b241bed216e6df5a57ccaf89c5e87a288d63f2353292e2fc8"} err="failed to get container status \"145fa649ee25077b241bed216e6df5a57ccaf89c5e87a288d63f2353292e2fc8\": rpc error: code = NotFound desc = could not find container \"145fa649ee25077b241bed216e6df5a57ccaf89c5e87a288d63f2353292e2fc8\": container with ID starting with 145fa649ee25077b241bed216e6df5a57ccaf89c5e87a288d63f2353292e2fc8 not found: ID does not exist"
Sep 17 03:33:20.238350 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:20.238331    2552 scope.go:117] "RemoveContainer" containerID="5679717a5818448203f9acd08d33c0af4b322b48ff873661702c4caab81bd67b"
Sep 17 03:33:20.238665 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:33:20.238639    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5679717a5818448203f9acd08d33c0af4b322b48ff873661702c4caab81bd67b\": container with ID starting with 5679717a5818448203f9acd08d33c0af4b322b48ff873661702c4caab81bd67b not found: ID does not exist" containerID="5679717a5818448203f9acd08d33c0af4b322b48ff873661702c4caab81bd67b"
Sep 17 03:33:20.238733 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:20.238674    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5679717a5818448203f9acd08d33c0af4b322b48ff873661702c4caab81bd67b"} err="failed to get container status \"5679717a5818448203f9acd08d33c0af4b322b48ff873661702c4caab81bd67b\": rpc error: code = NotFound desc = could not find container \"5679717a5818448203f9acd08d33c0af4b322b48ff873661702c4caab81bd67b\": container with ID starting with 5679717a5818448203f9acd08d33c0af4b322b48ff873661702c4caab81bd67b not found: ID does not exist"
Sep 17 03:33:20.239186 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:20.239169    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-0ae566a4-gw-openshift-default-7ff59f8f76-9zf22"]
Sep 17 03:33:20.250092 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:20.250071    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-0ae566a4-gw-openshift-default-7ff59f8f76-9zf22"]
Sep 17 03:33:20.283390 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:20.283358    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2nk4s\" (UniqueName: \"kubernetes.io/projected/0c1e74de-3dc7-463d-acde-e9ab14a22970-kube-api-access-2nk4s\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:33:20.283390 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:20.283384    2552 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c1e74de-3dc7-463d-acde-e9ab14a22970-istio-envoy\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:33:20.283390 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:20.283396    2552 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c1e74de-3dc7-463d-acde-e9ab14a22970-istio-token\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:33:20.283625 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:20.283403    2552 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c1e74de-3dc7-463d-acde-e9ab14a22970-istio-podinfo\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:33:20.389589 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:20.389509    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3b91ce2e-65c67fdf47-hpfw5"]
Sep 17 03:33:20.389989 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:20.389973    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c1e74de-3dc7-463d-acde-e9ab14a22970" containerName="istio-proxy"
Sep 17 03:33:20.390045 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:20.389991    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c1e74de-3dc7-463d-acde-e9ab14a22970" containerName="istio-proxy"
Sep 17 03:33:20.390081 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:20.390072    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="0c1e74de-3dc7-463d-acde-e9ab14a22970" containerName="istio-proxy"
Sep 17 03:33:20.390112 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:20.390083    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="0c1e74de-3dc7-463d-acde-e9ab14a22970" containerName="istio-proxy"
Sep 17 03:33:20.394590 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:20.394571    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3b91ce2e-65c67fdf47-hpfw5"
Sep 17 03:33:20.397446 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:20.397423    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-3b91ce2e-serving-cert\""
Sep 17 03:33:20.404378 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:20.403675    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3b91ce2e-65c67fdf47-hpfw5"]
Sep 17 03:33:20.485377 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:20.485349    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3543cd69-823f-4474-8926-dc5fe80773e4-openshift-service-ca\") pod \"maas-api-e2e-ait-3b91ce2e-65c67fdf47-hpfw5\" (UID: \"3543cd69-823f-4474-8926-dc5fe80773e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3b91ce2e-65c67fdf47-hpfw5"
Sep 17 03:33:20.485524 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:20.485393    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gk4j4\" (UniqueName: \"kubernetes.io/projected/3543cd69-823f-4474-8926-dc5fe80773e4-kube-api-access-gk4j4\") pod \"maas-api-e2e-ait-3b91ce2e-65c67fdf47-hpfw5\" (UID: \"3543cd69-823f-4474-8926-dc5fe80773e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3b91ce2e-65c67fdf47-hpfw5"
Sep 17 03:33:20.485524 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:20.485492    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3543cd69-823f-4474-8926-dc5fe80773e4-maas-api-tls\") pod \"maas-api-e2e-ait-3b91ce2e-65c67fdf47-hpfw5\" (UID: \"3543cd69-823f-4474-8926-dc5fe80773e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3b91ce2e-65c67fdf47-hpfw5"
Sep 17 03:33:20.485524 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:20.485522    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3543cd69-823f-4474-8926-dc5fe80773e4-metrics-certs\") pod \"maas-api-e2e-ait-3b91ce2e-65c67fdf47-hpfw5\" (UID: \"3543cd69-823f-4474-8926-dc5fe80773e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3b91ce2e-65c67fdf47-hpfw5"
Sep 17 03:33:20.586766 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:20.586736    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gk4j4\" (UniqueName: \"kubernetes.io/projected/3543cd69-823f-4474-8926-dc5fe80773e4-kube-api-access-gk4j4\") pod \"maas-api-e2e-ait-3b91ce2e-65c67fdf47-hpfw5\" (UID: \"3543cd69-823f-4474-8926-dc5fe80773e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3b91ce2e-65c67fdf47-hpfw5"
Sep 17 03:33:20.586932 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:20.586826    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3543cd69-823f-4474-8926-dc5fe80773e4-maas-api-tls\") pod \"maas-api-e2e-ait-3b91ce2e-65c67fdf47-hpfw5\" (UID: \"3543cd69-823f-4474-8926-dc5fe80773e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3b91ce2e-65c67fdf47-hpfw5"
Sep 17 03:33:20.586932 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:20.586859    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3543cd69-823f-4474-8926-dc5fe80773e4-metrics-certs\") pod \"maas-api-e2e-ait-3b91ce2e-65c67fdf47-hpfw5\" (UID: \"3543cd69-823f-4474-8926-dc5fe80773e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3b91ce2e-65c67fdf47-hpfw5"
Sep 17 03:33:20.586932 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:20.586896    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3543cd69-823f-4474-8926-dc5fe80773e4-openshift-service-ca\") pod \"maas-api-e2e-ait-3b91ce2e-65c67fdf47-hpfw5\" (UID: \"3543cd69-823f-4474-8926-dc5fe80773e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3b91ce2e-65c67fdf47-hpfw5"
Sep 17 03:33:20.587523 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:20.587498    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3543cd69-823f-4474-8926-dc5fe80773e4-openshift-service-ca\") pod \"maas-api-e2e-ait-3b91ce2e-65c67fdf47-hpfw5\" (UID: \"3543cd69-823f-4474-8926-dc5fe80773e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3b91ce2e-65c67fdf47-hpfw5"
Sep 17 03:33:20.589290 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:20.589270    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3543cd69-823f-4474-8926-dc5fe80773e4-maas-api-tls\") pod \"maas-api-e2e-ait-3b91ce2e-65c67fdf47-hpfw5\" (UID: \"3543cd69-823f-4474-8926-dc5fe80773e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3b91ce2e-65c67fdf47-hpfw5"
Sep 17 03:33:20.589381 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:20.589364    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3543cd69-823f-4474-8926-dc5fe80773e4-metrics-certs\") pod \"maas-api-e2e-ait-3b91ce2e-65c67fdf47-hpfw5\" (UID: \"3543cd69-823f-4474-8926-dc5fe80773e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3b91ce2e-65c67fdf47-hpfw5"
Sep 17 03:33:20.595201 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:20.595181    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gk4j4\" (UniqueName: \"kubernetes.io/projected/3543cd69-823f-4474-8926-dc5fe80773e4-kube-api-access-gk4j4\") pod \"maas-api-e2e-ait-3b91ce2e-65c67fdf47-hpfw5\" (UID: \"3543cd69-823f-4474-8926-dc5fe80773e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3b91ce2e-65c67fdf47-hpfw5"
Sep 17 03:33:20.708846 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:20.708755    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3b91ce2e-65c67fdf47-hpfw5"
Sep 17 03:33:20.986006 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:20.985929    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0c1e74de-3dc7-463d-acde-e9ab14a22970" path="/var/lib/kubelet/pods/0c1e74de-3dc7-463d-acde-e9ab14a22970/volumes"
Sep 17 03:33:21.044811 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:21.044785    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3b91ce2e-65c67fdf47-hpfw5"]
Sep 17 03:33:21.047195 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:33:21.047161    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3543cd69_823f_4474_8926_dc5fe80773e4.slice/crio-d09e049cbf9dc5f24b003458062ad863c995c8061f7a0376109597de7bd2deed WatchSource:0}: Error finding container d09e049cbf9dc5f24b003458062ad863c995c8061f7a0376109597de7bd2deed: Status 404 returned error can't find the container with id d09e049cbf9dc5f24b003458062ad863c995c8061f7a0376109597de7bd2deed
Sep 17 03:33:21.221903 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:21.221870    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3b91ce2e-65c67fdf47-hpfw5" event={"ID":"3543cd69-823f-4474-8926-dc5fe80773e4","Type":"ContainerStarted","Data":"d09e049cbf9dc5f24b003458062ad863c995c8061f7a0376109597de7bd2deed"}
Sep 17 03:33:22.228302 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:22.228265    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3b91ce2e-65c67fdf47-hpfw5" event={"ID":"3543cd69-823f-4474-8926-dc5fe80773e4","Type":"ContainerStarted","Data":"5a2073f6ca90ad3ad2343183ced24e9614d8dfb6b809447c89be6c735b818532"}
Sep 17 03:33:22.228827 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:22.228402    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3b91ce2e-65c67fdf47-hpfw5"
Sep 17 03:33:22.246903 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:22.246857    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3b91ce2e-65c67fdf47-hpfw5" podStartSLOduration=1.8283290779999999 podStartE2EDuration="2.246841564s" totalImagesPullingTime="418.512486ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 03:33:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:33:22.246316293 +0000 UTC m=+837.712723832" watchObservedRunningTime="2026-09-17 03:33:22.246841564 +0000 UTC m=+837.713249089"
Sep 17 03:33:22.593977 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:22.593945    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-1afd34/simulator-w0-1afd34-kserve-6c5ffdd47b-v6h59"
Sep 17 03:33:22.594145 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:22.593996    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1afd34/simulator-w0-1afd34-kserve-6c5ffdd47b-v6h59"
Sep 17 03:33:22.606235 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:22.606215    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-1afd34/simulator-w0-1afd34-kserve-6c5ffdd47b-v6h59"
Sep 17 03:33:23.244960 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:23.244921    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1afd34/simulator-w0-1afd34-kserve-6c5ffdd47b-v6h59"
Sep 17 03:33:27.565265 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:27.565236    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-f6a774/autores-model-f6a774-kserve-74885bd495-xwhdt"]
Sep 17 03:33:27.565635 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:27.565590    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c1e74de-3dc7-463d-acde-e9ab14a22970" containerName="istio-proxy"
Sep 17 03:33:27.565635 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:27.565620    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c1e74de-3dc7-463d-acde-e9ab14a22970" containerName="istio-proxy"
Sep 17 03:33:27.570099 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:27.570085    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-f6a774/autores-model-f6a774-kserve-74885bd495-xwhdt"
Sep 17 03:33:27.572960 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:27.572939    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-f6a774\"/\"kube-root-ca.crt\""
Sep 17 03:33:27.574309 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:27.574290    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-f6a774\"/\"autores-model-f6a774-kserve-self-signed-certs\""
Sep 17 03:33:27.574419 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:27.574329    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-f6a774\"/\"default-dockercfg-fdzpz\""
Sep 17 03:33:27.574419 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:27.574347    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-f6a774\"/\"openshift-service-ca.crt\""
Sep 17 03:33:27.579212 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:27.579192    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-f6a774/autores-model-f6a774-kserve-74885bd495-xwhdt"]
Sep 17 03:33:27.650379 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:27.650350    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zgj7\" (UniqueName: \"kubernetes.io/projected/b9576704-3a8a-4936-8c89-74b14756dc74-kube-api-access-6zgj7\") pod \"autores-model-f6a774-kserve-74885bd495-xwhdt\" (UID: \"b9576704-3a8a-4936-8c89-74b14756dc74\") " pod="ai-tenant-e2e-autores-f6a774/autores-model-f6a774-kserve-74885bd495-xwhdt"
Sep 17 03:33:27.650499 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:27.650399    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b9576704-3a8a-4936-8c89-74b14756dc74-home\") pod \"autores-model-f6a774-kserve-74885bd495-xwhdt\" (UID: \"b9576704-3a8a-4936-8c89-74b14756dc74\") " pod="ai-tenant-e2e-autores-f6a774/autores-model-f6a774-kserve-74885bd495-xwhdt"
Sep 17 03:33:27.650499 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:27.650423    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9576704-3a8a-4936-8c89-74b14756dc74-tls-certs\") pod \"autores-model-f6a774-kserve-74885bd495-xwhdt\" (UID: \"b9576704-3a8a-4936-8c89-74b14756dc74\") " pod="ai-tenant-e2e-autores-f6a774/autores-model-f6a774-kserve-74885bd495-xwhdt"
Sep 17 03:33:27.650622 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:27.650504    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b9576704-3a8a-4936-8c89-74b14756dc74-model-cache\") pod \"autores-model-f6a774-kserve-74885bd495-xwhdt\" (UID: \"b9576704-3a8a-4936-8c89-74b14756dc74\") " pod="ai-tenant-e2e-autores-f6a774/autores-model-f6a774-kserve-74885bd495-xwhdt"
Sep 17 03:33:27.650622 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:27.650538    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b9576704-3a8a-4936-8c89-74b14756dc74-dshm\") pod \"autores-model-f6a774-kserve-74885bd495-xwhdt\" (UID: \"b9576704-3a8a-4936-8c89-74b14756dc74\") " pod="ai-tenant-e2e-autores-f6a774/autores-model-f6a774-kserve-74885bd495-xwhdt"
Sep 17 03:33:27.650622 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:27.650556    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9576704-3a8a-4936-8c89-74b14756dc74-tmp-dir\") pod \"autores-model-f6a774-kserve-74885bd495-xwhdt\" (UID: \"b9576704-3a8a-4936-8c89-74b14756dc74\") " pod="ai-tenant-e2e-autores-f6a774/autores-model-f6a774-kserve-74885bd495-xwhdt"
Sep 17 03:33:27.751081 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:27.751054    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b9576704-3a8a-4936-8c89-74b14756dc74-model-cache\") pod \"autores-model-f6a774-kserve-74885bd495-xwhdt\" (UID: \"b9576704-3a8a-4936-8c89-74b14756dc74\") " pod="ai-tenant-e2e-autores-f6a774/autores-model-f6a774-kserve-74885bd495-xwhdt"
Sep 17 03:33:27.751204 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:27.751093    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b9576704-3a8a-4936-8c89-74b14756dc74-dshm\") pod \"autores-model-f6a774-kserve-74885bd495-xwhdt\" (UID: \"b9576704-3a8a-4936-8c89-74b14756dc74\") " pod="ai-tenant-e2e-autores-f6a774/autores-model-f6a774-kserve-74885bd495-xwhdt"
Sep 17 03:33:27.751204 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:27.751113    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9576704-3a8a-4936-8c89-74b14756dc74-tmp-dir\") pod \"autores-model-f6a774-kserve-74885bd495-xwhdt\" (UID: \"b9576704-3a8a-4936-8c89-74b14756dc74\") " pod="ai-tenant-e2e-autores-f6a774/autores-model-f6a774-kserve-74885bd495-xwhdt"
Sep 17 03:33:27.751204 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:27.751155    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zgj7\" (UniqueName: \"kubernetes.io/projected/b9576704-3a8a-4936-8c89-74b14756dc74-kube-api-access-6zgj7\") pod \"autores-model-f6a774-kserve-74885bd495-xwhdt\" (UID: \"b9576704-3a8a-4936-8c89-74b14756dc74\") " pod="ai-tenant-e2e-autores-f6a774/autores-model-f6a774-kserve-74885bd495-xwhdt"
Sep 17 03:33:27.751368 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:27.751210    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b9576704-3a8a-4936-8c89-74b14756dc74-home\") pod \"autores-model-f6a774-kserve-74885bd495-xwhdt\" (UID: \"b9576704-3a8a-4936-8c89-74b14756dc74\") " pod="ai-tenant-e2e-autores-f6a774/autores-model-f6a774-kserve-74885bd495-xwhdt"
Sep 17 03:33:27.751368 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:27.751232    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9576704-3a8a-4936-8c89-74b14756dc74-tls-certs\") pod \"autores-model-f6a774-kserve-74885bd495-xwhdt\" (UID: \"b9576704-3a8a-4936-8c89-74b14756dc74\") " pod="ai-tenant-e2e-autores-f6a774/autores-model-f6a774-kserve-74885bd495-xwhdt"
Sep 17 03:33:27.751543 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:27.751514    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b9576704-3a8a-4936-8c89-74b14756dc74-model-cache\") pod \"autores-model-f6a774-kserve-74885bd495-xwhdt\" (UID: \"b9576704-3a8a-4936-8c89-74b14756dc74\") " pod="ai-tenant-e2e-autores-f6a774/autores-model-f6a774-kserve-74885bd495-xwhdt"
Sep 17 03:33:27.751675 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:27.751566    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9576704-3a8a-4936-8c89-74b14756dc74-tmp-dir\") pod \"autores-model-f6a774-kserve-74885bd495-xwhdt\" (UID: \"b9576704-3a8a-4936-8c89-74b14756dc74\") " pod="ai-tenant-e2e-autores-f6a774/autores-model-f6a774-kserve-74885bd495-xwhdt"
Sep 17 03:33:27.751880 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:27.751860    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b9576704-3a8a-4936-8c89-74b14756dc74-home\") pod \"autores-model-f6a774-kserve-74885bd495-xwhdt\" (UID: \"b9576704-3a8a-4936-8c89-74b14756dc74\") " pod="ai-tenant-e2e-autores-f6a774/autores-model-f6a774-kserve-74885bd495-xwhdt"
Sep 17 03:33:27.753945 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:27.753918    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b9576704-3a8a-4936-8c89-74b14756dc74-dshm\") pod \"autores-model-f6a774-kserve-74885bd495-xwhdt\" (UID: \"b9576704-3a8a-4936-8c89-74b14756dc74\") " pod="ai-tenant-e2e-autores-f6a774/autores-model-f6a774-kserve-74885bd495-xwhdt"
Sep 17 03:33:27.754302 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:27.754282    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9576704-3a8a-4936-8c89-74b14756dc74-tls-certs\") pod \"autores-model-f6a774-kserve-74885bd495-xwhdt\" (UID: \"b9576704-3a8a-4936-8c89-74b14756dc74\") " pod="ai-tenant-e2e-autores-f6a774/autores-model-f6a774-kserve-74885bd495-xwhdt"
Sep 17 03:33:27.759243 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:27.759223    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zgj7\" (UniqueName: \"kubernetes.io/projected/b9576704-3a8a-4936-8c89-74b14756dc74-kube-api-access-6zgj7\") pod \"autores-model-f6a774-kserve-74885bd495-xwhdt\" (UID: \"b9576704-3a8a-4936-8c89-74b14756dc74\") " pod="ai-tenant-e2e-autores-f6a774/autores-model-f6a774-kserve-74885bd495-xwhdt"
Sep 17 03:33:27.879785 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:27.879707    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-f6a774/autores-model-f6a774-kserve-74885bd495-xwhdt"
Sep 17 03:33:28.149954 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:28.149931    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-f6a774/autores-model-f6a774-kserve-74885bd495-xwhdt"]
Sep 17 03:33:28.152467 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:33:28.152438    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb9576704_3a8a_4936_8c89_74b14756dc74.slice/crio-cad13a4eef5755ce45d58591ac593c9bbe3d5435a71c00d8bb3299db720697ab WatchSource:0}: Error finding container cad13a4eef5755ce45d58591ac593c9bbe3d5435a71c00d8bb3299db720697ab: Status 404 returned error can't find the container with id cad13a4eef5755ce45d58591ac593c9bbe3d5435a71c00d8bb3299db720697ab
Sep 17 03:33:28.239369 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:28.239347    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3b91ce2e-65c67fdf47-hpfw5"
Sep 17 03:33:28.260977 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:28.260937    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-f6a774/autores-model-f6a774-kserve-74885bd495-xwhdt" event={"ID":"b9576704-3a8a-4936-8c89-74b14756dc74","Type":"ContainerStarted","Data":"ba73fb1d0b3be7d4f8c5da5fefce981b33d2459fea8f0df3ecbabab7c71438ae"}
Sep 17 03:33:28.261096 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:28.260988    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-f6a774/autores-model-f6a774-kserve-74885bd495-xwhdt" event={"ID":"b9576704-3a8a-4936-8c89-74b14756dc74","Type":"ContainerStarted","Data":"cad13a4eef5755ce45d58591ac593c9bbe3d5435a71c00d8bb3299db720697ab"}
Sep 17 03:33:28.281220 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:28.281176    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-autores-f6a774/autores-model-f6a774-kserve-74885bd495-xwhdt" podStartSLOduration=1.2811612829999999 podStartE2EDuration="1.281161283s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 03:33:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:33:28.280669676 +0000 UTC m=+843.747077231" watchObservedRunningTime="2026-09-17 03:33:28.281161283 +0000 UTC m=+843.747568810"
Sep 17 03:33:37.880619 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:37.880565    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-f6a774/autores-model-f6a774-kserve-74885bd495-xwhdt"
Sep 17 03:33:37.881020 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:37.880638    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-f6a774/autores-model-f6a774-kserve-74885bd495-xwhdt"
Sep 17 03:33:37.893438 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:37.893416    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-f6a774/autores-model-f6a774-kserve-74885bd495-xwhdt"
Sep 17 03:33:38.328326 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:38.328298    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-f6a774/autores-model-f6a774-kserve-74885bd495-xwhdt"
Sep 17 03:33:50.253567 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:50.253532    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3b91ce2e-65c67fdf47-hpfw5"]
Sep 17 03:33:50.254013 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:50.253808    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3b91ce2e-65c67fdf47-hpfw5" podUID="3543cd69-823f-4474-8926-dc5fe80773e4" containerName="maas-api" containerID="cri-o://5a2073f6ca90ad3ad2343183ced24e9614d8dfb6b809447c89be6c735b818532" gracePeriod=30
Sep 17 03:33:50.381511 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:50.381470    2552 generic.go:356] "Generic (PLEG): container finished" podID="3543cd69-823f-4474-8926-dc5fe80773e4" containerID="5a2073f6ca90ad3ad2343183ced24e9614d8dfb6b809447c89be6c735b818532" exitCode=0
Sep 17 03:33:50.381676 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:50.381513    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3b91ce2e-65c67fdf47-hpfw5" event={"ID":"3543cd69-823f-4474-8926-dc5fe80773e4","Type":"ContainerDied","Data":"5a2073f6ca90ad3ad2343183ced24e9614d8dfb6b809447c89be6c735b818532"}
Sep 17 03:33:50.504128 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:50.504070    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3b91ce2e-65c67fdf47-hpfw5"
Sep 17 03:33:50.661218 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:50.661185    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3543cd69-823f-4474-8926-dc5fe80773e4-metrics-certs\") pod \"3543cd69-823f-4474-8926-dc5fe80773e4\" (UID: \"3543cd69-823f-4474-8926-dc5fe80773e4\") "
Sep 17 03:33:50.661399 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:50.661250    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3543cd69-823f-4474-8926-dc5fe80773e4-openshift-service-ca\") pod \"3543cd69-823f-4474-8926-dc5fe80773e4\" (UID: \"3543cd69-823f-4474-8926-dc5fe80773e4\") "
Sep 17 03:33:50.661399 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:50.661361    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3543cd69-823f-4474-8926-dc5fe80773e4-maas-api-tls\") pod \"3543cd69-823f-4474-8926-dc5fe80773e4\" (UID: \"3543cd69-823f-4474-8926-dc5fe80773e4\") "
Sep 17 03:33:50.661517 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:50.661427    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gk4j4\" (UniqueName: \"kubernetes.io/projected/3543cd69-823f-4474-8926-dc5fe80773e4-kube-api-access-gk4j4\") pod \"3543cd69-823f-4474-8926-dc5fe80773e4\" (UID: \"3543cd69-823f-4474-8926-dc5fe80773e4\") "
Sep 17 03:33:50.661686 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:50.661653    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3543cd69-823f-4474-8926-dc5fe80773e4-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "3543cd69-823f-4474-8926-dc5fe80773e4" (UID: "3543cd69-823f-4474-8926-dc5fe80773e4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 03:33:50.661861 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:50.661840    2552 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3543cd69-823f-4474-8926-dc5fe80773e4-openshift-service-ca\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:33:50.663444 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:50.663420    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3543cd69-823f-4474-8926-dc5fe80773e4-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "3543cd69-823f-4474-8926-dc5fe80773e4" (UID: "3543cd69-823f-4474-8926-dc5fe80773e4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 03:33:50.663627 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:50.663586    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3543cd69-823f-4474-8926-dc5fe80773e4-kube-api-access-gk4j4" (OuterVolumeSpecName: "kube-api-access-gk4j4") pod "3543cd69-823f-4474-8926-dc5fe80773e4" (UID: "3543cd69-823f-4474-8926-dc5fe80773e4"). InnerVolumeSpecName "kube-api-access-gk4j4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 03:33:50.663732 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:50.663714    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3543cd69-823f-4474-8926-dc5fe80773e4-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "3543cd69-823f-4474-8926-dc5fe80773e4" (UID: "3543cd69-823f-4474-8926-dc5fe80773e4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 03:33:50.763099 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:50.763024    2552 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3543cd69-823f-4474-8926-dc5fe80773e4-maas-api-tls\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:33:50.763099 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:50.763052    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gk4j4\" (UniqueName: \"kubernetes.io/projected/3543cd69-823f-4474-8926-dc5fe80773e4-kube-api-access-gk4j4\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:33:50.763099 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:50.763066    2552 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3543cd69-823f-4474-8926-dc5fe80773e4-metrics-certs\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:33:51.389311 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:51.389282    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3b91ce2e-65c67fdf47-hpfw5" event={"ID":"3543cd69-823f-4474-8926-dc5fe80773e4","Type":"ContainerDied","Data":"d09e049cbf9dc5f24b003458062ad863c995c8061f7a0376109597de7bd2deed"}
Sep 17 03:33:51.389739 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:51.389315    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3b91ce2e-65c67fdf47-hpfw5"
Sep 17 03:33:51.389739 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:51.389338    2552 scope.go:117] "RemoveContainer" containerID="5a2073f6ca90ad3ad2343183ced24e9614d8dfb6b809447c89be6c735b818532"
Sep 17 03:33:51.406716 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:51.406692    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3b91ce2e-65c67fdf47-hpfw5"]
Sep 17 03:33:51.410486 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:51.410467    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3b91ce2e-65c67fdf47-hpfw5"]
Sep 17 03:33:52.401212 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:52.401172    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-acf311cd-openshift-default-5bf744fb4d-vl6f2"]
Sep 17 03:33:52.401835 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:52.401617    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3543cd69-823f-4474-8926-dc5fe80773e4" containerName="maas-api"
Sep 17 03:33:52.401835 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:52.401630    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="3543cd69-823f-4474-8926-dc5fe80773e4" containerName="maas-api"
Sep 17 03:33:52.401835 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:52.401702    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="3543cd69-823f-4474-8926-dc5fe80773e4" containerName="maas-api"
Sep 17 03:33:52.405726 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:52.405707    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-acf311cd-openshift-default-5bf744fb4d-vl6f2"
Sep 17 03:33:52.409479 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:52.409463    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-acf311cd-openshift-default-dockercfg-vq6vq\""
Sep 17 03:33:52.425057 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:52.425034    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-acf311cd-openshift-default-5bf744fb4d-vl6f2"]
Sep 17 03:33:52.581998 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:52.581970    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08d183e7-7176-4eaa-8843-747fb05618d7-istiod-ca-cert\") pod \"e2e-derive-acf311cd-openshift-default-5bf744fb4d-vl6f2\" (UID: \"08d183e7-7176-4eaa-8843-747fb05618d7\") " pod="openshift-ingress/e2e-derive-acf311cd-openshift-default-5bf744fb4d-vl6f2"
Sep 17 03:33:52.581998 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:52.582001    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08d183e7-7176-4eaa-8843-747fb05618d7-istio-podinfo\") pod \"e2e-derive-acf311cd-openshift-default-5bf744fb4d-vl6f2\" (UID: \"08d183e7-7176-4eaa-8843-747fb05618d7\") " pod="openshift-ingress/e2e-derive-acf311cd-openshift-default-5bf744fb4d-vl6f2"
Sep 17 03:33:52.582220 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:52.582041    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08d183e7-7176-4eaa-8843-747fb05618d7-credential-socket\") pod \"e2e-derive-acf311cd-openshift-default-5bf744fb4d-vl6f2\" (UID: \"08d183e7-7176-4eaa-8843-747fb05618d7\") " pod="openshift-ingress/e2e-derive-acf311cd-openshift-default-5bf744fb4d-vl6f2"
Sep 17 03:33:52.582220 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:52.582060    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08d183e7-7176-4eaa-8843-747fb05618d7-istio-data\") pod \"e2e-derive-acf311cd-openshift-default-5bf744fb4d-vl6f2\" (UID: \"08d183e7-7176-4eaa-8843-747fb05618d7\") " pod="openshift-ingress/e2e-derive-acf311cd-openshift-default-5bf744fb4d-vl6f2"
Sep 17 03:33:52.582220 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:52.582126    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08d183e7-7176-4eaa-8843-747fb05618d7-workload-socket\") pod \"e2e-derive-acf311cd-openshift-default-5bf744fb4d-vl6f2\" (UID: \"08d183e7-7176-4eaa-8843-747fb05618d7\") " pod="openshift-ingress/e2e-derive-acf311cd-openshift-default-5bf744fb4d-vl6f2"
Sep 17 03:33:52.582220 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:52.582160    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08d183e7-7176-4eaa-8843-747fb05618d7-workload-certs\") pod \"e2e-derive-acf311cd-openshift-default-5bf744fb4d-vl6f2\" (UID: \"08d183e7-7176-4eaa-8843-747fb05618d7\") " pod="openshift-ingress/e2e-derive-acf311cd-openshift-default-5bf744fb4d-vl6f2"
Sep 17 03:33:52.582220 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:52.582179    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hzdz\" (UniqueName: \"kubernetes.io/projected/08d183e7-7176-4eaa-8843-747fb05618d7-kube-api-access-9hzdz\") pod \"e2e-derive-acf311cd-openshift-default-5bf744fb4d-vl6f2\" (UID: \"08d183e7-7176-4eaa-8843-747fb05618d7\") " pod="openshift-ingress/e2e-derive-acf311cd-openshift-default-5bf744fb4d-vl6f2"
Sep 17 03:33:52.582220 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:52.582203    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08d183e7-7176-4eaa-8843-747fb05618d7-istio-envoy\") pod \"e2e-derive-acf311cd-openshift-default-5bf744fb4d-vl6f2\" (UID: \"08d183e7-7176-4eaa-8843-747fb05618d7\") " pod="openshift-ingress/e2e-derive-acf311cd-openshift-default-5bf744fb4d-vl6f2"
Sep 17 03:33:52.582447 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:52.582233    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08d183e7-7176-4eaa-8843-747fb05618d7-istio-token\") pod \"e2e-derive-acf311cd-openshift-default-5bf744fb4d-vl6f2\" (UID: \"08d183e7-7176-4eaa-8843-747fb05618d7\") " pod="openshift-ingress/e2e-derive-acf311cd-openshift-default-5bf744fb4d-vl6f2"
Sep 17 03:33:52.682746 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:52.682673    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08d183e7-7176-4eaa-8843-747fb05618d7-workload-socket\") pod \"e2e-derive-acf311cd-openshift-default-5bf744fb4d-vl6f2\" (UID: \"08d183e7-7176-4eaa-8843-747fb05618d7\") " pod="openshift-ingress/e2e-derive-acf311cd-openshift-default-5bf744fb4d-vl6f2"
Sep 17 03:33:52.682746 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:52.682702    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08d183e7-7176-4eaa-8843-747fb05618d7-workload-certs\") pod \"e2e-derive-acf311cd-openshift-default-5bf744fb4d-vl6f2\" (UID: \"08d183e7-7176-4eaa-8843-747fb05618d7\") " pod="openshift-ingress/e2e-derive-acf311cd-openshift-default-5bf744fb4d-vl6f2"
Sep 17 03:33:52.682746 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:52.682724    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hzdz\" (UniqueName: \"kubernetes.io/projected/08d183e7-7176-4eaa-8843-747fb05618d7-kube-api-access-9hzdz\") pod \"e2e-derive-acf311cd-openshift-default-5bf744fb4d-vl6f2\" (UID: \"08d183e7-7176-4eaa-8843-747fb05618d7\") " pod="openshift-ingress/e2e-derive-acf311cd-openshift-default-5bf744fb4d-vl6f2"
Sep 17 03:33:52.682746 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:52.682745    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08d183e7-7176-4eaa-8843-747fb05618d7-istio-envoy\") pod \"e2e-derive-acf311cd-openshift-default-5bf744fb4d-vl6f2\" (UID: \"08d183e7-7176-4eaa-8843-747fb05618d7\") " pod="openshift-ingress/e2e-derive-acf311cd-openshift-default-5bf744fb4d-vl6f2"
Sep 17 03:33:52.683017 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:52.682785    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08d183e7-7176-4eaa-8843-747fb05618d7-istio-token\") pod \"e2e-derive-acf311cd-openshift-default-5bf744fb4d-vl6f2\" (UID: \"08d183e7-7176-4eaa-8843-747fb05618d7\") " pod="openshift-ingress/e2e-derive-acf311cd-openshift-default-5bf744fb4d-vl6f2"
Sep 17 03:33:52.683017 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:52.682856    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08d183e7-7176-4eaa-8843-747fb05618d7-istiod-ca-cert\") pod \"e2e-derive-acf311cd-openshift-default-5bf744fb4d-vl6f2\" (UID: \"08d183e7-7176-4eaa-8843-747fb05618d7\") " pod="openshift-ingress/e2e-derive-acf311cd-openshift-default-5bf744fb4d-vl6f2"
Sep 17 03:33:52.683017 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:52.682879    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08d183e7-7176-4eaa-8843-747fb05618d7-istio-podinfo\") pod \"e2e-derive-acf311cd-openshift-default-5bf744fb4d-vl6f2\" (UID: \"08d183e7-7176-4eaa-8843-747fb05618d7\") " pod="openshift-ingress/e2e-derive-acf311cd-openshift-default-5bf744fb4d-vl6f2"
Sep 17 03:33:52.683017 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:52.682932    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08d183e7-7176-4eaa-8843-747fb05618d7-credential-socket\") pod \"e2e-derive-acf311cd-openshift-default-5bf744fb4d-vl6f2\" (UID: \"08d183e7-7176-4eaa-8843-747fb05618d7\") " pod="openshift-ingress/e2e-derive-acf311cd-openshift-default-5bf744fb4d-vl6f2"
Sep 17 03:33:52.683230 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:52.683054    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08d183e7-7176-4eaa-8843-747fb05618d7-istio-data\") pod \"e2e-derive-acf311cd-openshift-default-5bf744fb4d-vl6f2\" (UID: \"08d183e7-7176-4eaa-8843-747fb05618d7\") " pod="openshift-ingress/e2e-derive-acf311cd-openshift-default-5bf744fb4d-vl6f2"
Sep 17 03:33:52.683230 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:52.683123    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08d183e7-7176-4eaa-8843-747fb05618d7-workload-socket\") pod \"e2e-derive-acf311cd-openshift-default-5bf744fb4d-vl6f2\" (UID: \"08d183e7-7176-4eaa-8843-747fb05618d7\") " pod="openshift-ingress/e2e-derive-acf311cd-openshift-default-5bf744fb4d-vl6f2"
Sep 17 03:33:52.683230 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:52.683173    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08d183e7-7176-4eaa-8843-747fb05618d7-workload-certs\") pod \"e2e-derive-acf311cd-openshift-default-5bf744fb4d-vl6f2\" (UID: \"08d183e7-7176-4eaa-8843-747fb05618d7\") " pod="openshift-ingress/e2e-derive-acf311cd-openshift-default-5bf744fb4d-vl6f2"
Sep 17 03:33:52.683230 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:52.683218    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08d183e7-7176-4eaa-8843-747fb05618d7-credential-socket\") pod \"e2e-derive-acf311cd-openshift-default-5bf744fb4d-vl6f2\" (UID: \"08d183e7-7176-4eaa-8843-747fb05618d7\") " pod="openshift-ingress/e2e-derive-acf311cd-openshift-default-5bf744fb4d-vl6f2"
Sep 17 03:33:52.683444 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:52.683368    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08d183e7-7176-4eaa-8843-747fb05618d7-istio-data\") pod \"e2e-derive-acf311cd-openshift-default-5bf744fb4d-vl6f2\" (UID: \"08d183e7-7176-4eaa-8843-747fb05618d7\") " pod="openshift-ingress/e2e-derive-acf311cd-openshift-default-5bf744fb4d-vl6f2"
Sep 17 03:33:52.683695 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:52.683663    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08d183e7-7176-4eaa-8843-747fb05618d7-istiod-ca-cert\") pod \"e2e-derive-acf311cd-openshift-default-5bf744fb4d-vl6f2\" (UID: \"08d183e7-7176-4eaa-8843-747fb05618d7\") " pod="openshift-ingress/e2e-derive-acf311cd-openshift-default-5bf744fb4d-vl6f2"
Sep 17 03:33:52.685182 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:52.685154    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08d183e7-7176-4eaa-8843-747fb05618d7-istio-envoy\") pod \"e2e-derive-acf311cd-openshift-default-5bf744fb4d-vl6f2\" (UID: \"08d183e7-7176-4eaa-8843-747fb05618d7\") " pod="openshift-ingress/e2e-derive-acf311cd-openshift-default-5bf744fb4d-vl6f2"
Sep 17 03:33:52.685407 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:52.685386    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08d183e7-7176-4eaa-8843-747fb05618d7-istio-podinfo\") pod \"e2e-derive-acf311cd-openshift-default-5bf744fb4d-vl6f2\" (UID: \"08d183e7-7176-4eaa-8843-747fb05618d7\") " pod="openshift-ingress/e2e-derive-acf311cd-openshift-default-5bf744fb4d-vl6f2"
Sep 17 03:33:52.691222 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:52.691202    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08d183e7-7176-4eaa-8843-747fb05618d7-istio-token\") pod \"e2e-derive-acf311cd-openshift-default-5bf744fb4d-vl6f2\" (UID: \"08d183e7-7176-4eaa-8843-747fb05618d7\") " pod="openshift-ingress/e2e-derive-acf311cd-openshift-default-5bf744fb4d-vl6f2"
Sep 17 03:33:52.691346 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:52.691329    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hzdz\" (UniqueName: \"kubernetes.io/projected/08d183e7-7176-4eaa-8843-747fb05618d7-kube-api-access-9hzdz\") pod \"e2e-derive-acf311cd-openshift-default-5bf744fb4d-vl6f2\" (UID: \"08d183e7-7176-4eaa-8843-747fb05618d7\") " pod="openshift-ingress/e2e-derive-acf311cd-openshift-default-5bf744fb4d-vl6f2"
Sep 17 03:33:52.715739 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:52.715721    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-acf311cd-openshift-default-5bf744fb4d-vl6f2"
Sep 17 03:33:52.985661 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:52.985553    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3543cd69-823f-4474-8926-dc5fe80773e4" path="/var/lib/kubelet/pods/3543cd69-823f-4474-8926-dc5fe80773e4/volumes"
Sep 17 03:33:52.997950 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:52.997927    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-acf311cd-openshift-default-5bf744fb4d-vl6f2"]
Sep 17 03:33:52.999686 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:33:52.999658    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08d183e7_7176_4eaa_8843_747fb05618d7.slice/crio-269a51b72db52e6b717f9d5ff54ff844eed34ab515ae676d7b60568db4b2d80b WatchSource:0}: Error finding container 269a51b72db52e6b717f9d5ff54ff844eed34ab515ae676d7b60568db4b2d80b: Status 404 returned error can't find the container with id 269a51b72db52e6b717f9d5ff54ff844eed34ab515ae676d7b60568db4b2d80b
Sep 17 03:33:53.001856 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:53.001826    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 03:33:53.001944 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:53.001888    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 03:33:53.001944 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:53.001918    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 03:33:53.404274 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:53.404236    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-acf311cd-openshift-default-5bf744fb4d-vl6f2" event={"ID":"08d183e7-7176-4eaa-8843-747fb05618d7","Type":"ContainerStarted","Data":"76573d3a3aef61ec8f9d66177c620fc0d7b443aa40b034ca4fa698766be3e10c"}
Sep 17 03:33:53.404712 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:53.404321    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-acf311cd-openshift-default-5bf744fb4d-vl6f2" event={"ID":"08d183e7-7176-4eaa-8843-747fb05618d7","Type":"ContainerStarted","Data":"269a51b72db52e6b717f9d5ff54ff844eed34ab515ae676d7b60568db4b2d80b"}
Sep 17 03:33:53.426886 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:53.426845    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-acf311cd-openshift-default-5bf744fb4d-vl6f2" podStartSLOduration=1.426832858 podStartE2EDuration="1.426832858s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 03:33:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:33:53.425455698 +0000 UTC m=+868.891863237" watchObservedRunningTime="2026-09-17 03:33:53.426832858 +0000 UTC m=+868.893240385"
Sep 17 03:33:53.716211 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:53.716131    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-acf311cd-openshift-default-5bf744fb4d-vl6f2"
Sep 17 03:33:54.721119 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:54.721091    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-acf311cd-openshift-default-5bf744fb4d-vl6f2"
Sep 17 03:33:55.415125 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:55.415098    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-acf311cd-openshift-default-5bf744fb4d-vl6f2"
Sep 17 03:33:55.416176 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:55.416153    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-acf311cd-openshift-default-5bf744fb4d-vl6f2"
Sep 17 03:33:56.187335 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:56.187302    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-acf311cd-b97c76cd-w8znq"]
Sep 17 03:33:56.191342 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:56.191323    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-acf311cd-b97c76cd-w8znq"
Sep 17 03:33:56.194581 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:56.194556    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-acf311cd-serving-cert\""
Sep 17 03:33:56.202693 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:56.202667    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-acf311cd-b97c76cd-w8znq"]
Sep 17 03:33:56.218699 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:56.218677    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbe608d1-8ab8-48f7-b12a-08f76cb35aa6-openshift-service-ca\") pod \"maas-api-e2e-derive-acf311cd-b97c76cd-w8znq\" (UID: \"bbe608d1-8ab8-48f7-b12a-08f76cb35aa6\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-acf311cd-b97c76cd-w8znq"
Sep 17 03:33:56.218794 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:56.218738    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bbe608d1-8ab8-48f7-b12a-08f76cb35aa6-maas-api-tls\") pod \"maas-api-e2e-derive-acf311cd-b97c76cd-w8znq\" (UID: \"bbe608d1-8ab8-48f7-b12a-08f76cb35aa6\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-acf311cd-b97c76cd-w8znq"
Sep 17 03:33:56.218837 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:56.218805    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bbe608d1-8ab8-48f7-b12a-08f76cb35aa6-metrics-certs\") pod \"maas-api-e2e-derive-acf311cd-b97c76cd-w8znq\" (UID: \"bbe608d1-8ab8-48f7-b12a-08f76cb35aa6\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-acf311cd-b97c76cd-w8znq"
Sep 17 03:33:56.218911 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:56.218894    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62x4f\" (UniqueName: \"kubernetes.io/projected/bbe608d1-8ab8-48f7-b12a-08f76cb35aa6-kube-api-access-62x4f\") pod \"maas-api-e2e-derive-acf311cd-b97c76cd-w8znq\" (UID: \"bbe608d1-8ab8-48f7-b12a-08f76cb35aa6\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-acf311cd-b97c76cd-w8znq"
Sep 17 03:33:56.320261 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:56.320230    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bbe608d1-8ab8-48f7-b12a-08f76cb35aa6-maas-api-tls\") pod \"maas-api-e2e-derive-acf311cd-b97c76cd-w8znq\" (UID: \"bbe608d1-8ab8-48f7-b12a-08f76cb35aa6\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-acf311cd-b97c76cd-w8znq"
Sep 17 03:33:56.320430 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:56.320269    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bbe608d1-8ab8-48f7-b12a-08f76cb35aa6-metrics-certs\") pod \"maas-api-e2e-derive-acf311cd-b97c76cd-w8znq\" (UID: \"bbe608d1-8ab8-48f7-b12a-08f76cb35aa6\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-acf311cd-b97c76cd-w8znq"
Sep 17 03:33:56.320430 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:56.320323    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62x4f\" (UniqueName: \"kubernetes.io/projected/bbe608d1-8ab8-48f7-b12a-08f76cb35aa6-kube-api-access-62x4f\") pod \"maas-api-e2e-derive-acf311cd-b97c76cd-w8znq\" (UID: \"bbe608d1-8ab8-48f7-b12a-08f76cb35aa6\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-acf311cd-b97c76cd-w8znq"
Sep 17 03:33:56.320430 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:56.320356    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbe608d1-8ab8-48f7-b12a-08f76cb35aa6-openshift-service-ca\") pod \"maas-api-e2e-derive-acf311cd-b97c76cd-w8znq\" (UID: \"bbe608d1-8ab8-48f7-b12a-08f76cb35aa6\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-acf311cd-b97c76cd-w8znq"
Sep 17 03:33:56.320430 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:33:56.320376    2552 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-acf311cd-serving-cert: secret "maas-api-e2e-derive-acf311cd-serving-cert" not found
Sep 17 03:33:56.320569 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:33:56.320452    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bbe608d1-8ab8-48f7-b12a-08f76cb35aa6-maas-api-tls podName:bbe608d1-8ab8-48f7-b12a-08f76cb35aa6 nodeName:}" failed. No retries permitted until 2026-09-17 03:33:56.820435811 +0000 UTC m=+872.286843330 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/bbe608d1-8ab8-48f7-b12a-08f76cb35aa6-maas-api-tls") pod "maas-api-e2e-derive-acf311cd-b97c76cd-w8znq" (UID: "bbe608d1-8ab8-48f7-b12a-08f76cb35aa6") : secret "maas-api-e2e-derive-acf311cd-serving-cert" not found
Sep 17 03:33:56.321040 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:56.321020    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbe608d1-8ab8-48f7-b12a-08f76cb35aa6-openshift-service-ca\") pod \"maas-api-e2e-derive-acf311cd-b97c76cd-w8znq\" (UID: \"bbe608d1-8ab8-48f7-b12a-08f76cb35aa6\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-acf311cd-b97c76cd-w8znq"
Sep 17 03:33:56.322653 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:56.322636    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bbe608d1-8ab8-48f7-b12a-08f76cb35aa6-metrics-certs\") pod \"maas-api-e2e-derive-acf311cd-b97c76cd-w8znq\" (UID: \"bbe608d1-8ab8-48f7-b12a-08f76cb35aa6\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-acf311cd-b97c76cd-w8znq"
Sep 17 03:33:56.329460 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:56.329444    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-62x4f\" (UniqueName: \"kubernetes.io/projected/bbe608d1-8ab8-48f7-b12a-08f76cb35aa6-kube-api-access-62x4f\") pod \"maas-api-e2e-derive-acf311cd-b97c76cd-w8znq\" (UID: \"bbe608d1-8ab8-48f7-b12a-08f76cb35aa6\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-acf311cd-b97c76cd-w8znq"
Sep 17 03:33:56.823977 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:56.823941    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bbe608d1-8ab8-48f7-b12a-08f76cb35aa6-maas-api-tls\") pod \"maas-api-e2e-derive-acf311cd-b97c76cd-w8znq\" (UID: \"bbe608d1-8ab8-48f7-b12a-08f76cb35aa6\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-acf311cd-b97c76cd-w8znq"
Sep 17 03:33:56.826332 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:56.826298    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bbe608d1-8ab8-48f7-b12a-08f76cb35aa6-maas-api-tls\") pod \"maas-api-e2e-derive-acf311cd-b97c76cd-w8znq\" (UID: \"bbe608d1-8ab8-48f7-b12a-08f76cb35aa6\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-acf311cd-b97c76cd-w8znq"
Sep 17 03:33:57.104199 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:57.104112    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-acf311cd-b97c76cd-w8znq"
Sep 17 03:33:57.385284 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:57.385253    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-acf311cd-b97c76cd-w8znq"]
Sep 17 03:33:57.386705 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:33:57.386668    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbe608d1_8ab8_48f7_b12a_08f76cb35aa6.slice/crio-35f017871fd3146f3b8e2a97d41766b1a4d650fe09a509a993a0550dfbf3429a WatchSource:0}: Error finding container 35f017871fd3146f3b8e2a97d41766b1a4d650fe09a509a993a0550dfbf3429a: Status 404 returned error can't find the container with id 35f017871fd3146f3b8e2a97d41766b1a4d650fe09a509a993a0550dfbf3429a
Sep 17 03:33:57.426451 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:57.426414    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-acf311cd-b97c76cd-w8znq" event={"ID":"bbe608d1-8ab8-48f7-b12a-08f76cb35aa6","Type":"ContainerStarted","Data":"35f017871fd3146f3b8e2a97d41766b1a4d650fe09a509a993a0550dfbf3429a"}
Sep 17 03:33:58.432742 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:58.432703    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-acf311cd-b97c76cd-w8znq" event={"ID":"bbe608d1-8ab8-48f7-b12a-08f76cb35aa6","Type":"ContainerStarted","Data":"6c80f9d0b12183c377e04d403bf6441905746fc6a6fdc8d86976579b90dba02c"}
Sep 17 03:33:58.433217 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:58.432808    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-acf311cd-b97c76cd-w8znq"
Sep 17 03:33:58.451918 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:33:58.451870    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-acf311cd-b97c76cd-w8znq" podStartSLOduration=1.94822039 podStartE2EDuration="2.451855398s" totalImagesPullingTime="503.635008ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 03:33:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:33:58.450192409 +0000 UTC m=+873.916599939" watchObservedRunningTime="2026-09-17 03:33:58.451855398 +0000 UTC m=+873.918262924"
Sep 17 03:34:04.444730 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:34:04.444648    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-acf311cd-b97c76cd-w8znq"
Sep 17 03:34:06.858374 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:34:06.858341    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-f6a774/autores-model-f6a774-kserve-74885bd495-xwhdt"]
Sep 17 03:34:14.934746 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:34:14.934709    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-f6a774-64f58fc685-4x5ng"]
Sep 17 03:34:14.935565 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:34:14.935522    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-f6a774-64f58fc685-4x5ng" podUID="145c1a8c-a6e6-4910-98e5-58ee2b6753d3" containerName="maas-api" containerID="cri-o://8e05b4f624230081f71f947face151188a8c3e37cab72334cd359c4331a2f926" gracePeriod=30
Sep 17 03:34:15.253643 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:34:15.253612    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-f6a774-64f58fc685-4x5ng"
Sep 17 03:34:15.298230 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:34:15.298196    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/145c1a8c-a6e6-4910-98e5-58ee2b6753d3-metrics-certs\") pod \"145c1a8c-a6e6-4910-98e5-58ee2b6753d3\" (UID: \"145c1a8c-a6e6-4910-98e5-58ee2b6753d3\") "
Sep 17 03:34:15.298417 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:34:15.298251    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/145c1a8c-a6e6-4910-98e5-58ee2b6753d3-maas-api-tls\") pod \"145c1a8c-a6e6-4910-98e5-58ee2b6753d3\" (UID: \"145c1a8c-a6e6-4910-98e5-58ee2b6753d3\") "
Sep 17 03:34:15.298417 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:34:15.298333    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/145c1a8c-a6e6-4910-98e5-58ee2b6753d3-openshift-service-ca\") pod \"145c1a8c-a6e6-4910-98e5-58ee2b6753d3\" (UID: \"145c1a8c-a6e6-4910-98e5-58ee2b6753d3\") "
Sep 17 03:34:15.298417 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:34:15.298367    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2p9jg\" (UniqueName: \"kubernetes.io/projected/145c1a8c-a6e6-4910-98e5-58ee2b6753d3-kube-api-access-2p9jg\") pod \"145c1a8c-a6e6-4910-98e5-58ee2b6753d3\" (UID: \"145c1a8c-a6e6-4910-98e5-58ee2b6753d3\") "
Sep 17 03:34:15.298790 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:34:15.298762    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/145c1a8c-a6e6-4910-98e5-58ee2b6753d3-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "145c1a8c-a6e6-4910-98e5-58ee2b6753d3" (UID: "145c1a8c-a6e6-4910-98e5-58ee2b6753d3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 03:34:15.300308 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:34:15.300280    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/145c1a8c-a6e6-4910-98e5-58ee2b6753d3-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "145c1a8c-a6e6-4910-98e5-58ee2b6753d3" (UID: "145c1a8c-a6e6-4910-98e5-58ee2b6753d3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 03:34:15.300624 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:34:15.300587    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/145c1a8c-a6e6-4910-98e5-58ee2b6753d3-kube-api-access-2p9jg" (OuterVolumeSpecName: "kube-api-access-2p9jg") pod "145c1a8c-a6e6-4910-98e5-58ee2b6753d3" (UID: "145c1a8c-a6e6-4910-98e5-58ee2b6753d3"). InnerVolumeSpecName "kube-api-access-2p9jg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 03:34:15.300744 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:34:15.300587    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/145c1a8c-a6e6-4910-98e5-58ee2b6753d3-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "145c1a8c-a6e6-4910-98e5-58ee2b6753d3" (UID: "145c1a8c-a6e6-4910-98e5-58ee2b6753d3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 03:34:15.399574 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:34:15.399540    2552 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/145c1a8c-a6e6-4910-98e5-58ee2b6753d3-openshift-service-ca\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:34:15.399574 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:34:15.399565    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2p9jg\" (UniqueName: \"kubernetes.io/projected/145c1a8c-a6e6-4910-98e5-58ee2b6753d3-kube-api-access-2p9jg\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:34:15.399574 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:34:15.399576    2552 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/145c1a8c-a6e6-4910-98e5-58ee2b6753d3-metrics-certs\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:34:15.399574 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:34:15.399585    2552 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/145c1a8c-a6e6-4910-98e5-58ee2b6753d3-maas-api-tls\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:34:15.523740 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:34:15.523707    2552 generic.go:356] "Generic (PLEG): container finished" podID="145c1a8c-a6e6-4910-98e5-58ee2b6753d3" containerID="8e05b4f624230081f71f947face151188a8c3e37cab72334cd359c4331a2f926" exitCode=0
Sep 17 03:34:15.523910 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:34:15.523835    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-f6a774-64f58fc685-4x5ng" event={"ID":"145c1a8c-a6e6-4910-98e5-58ee2b6753d3","Type":"ContainerDied","Data":"8e05b4f624230081f71f947face151188a8c3e37cab72334cd359c4331a2f926"}
Sep 17 03:34:15.523910 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:34:15.523898    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-f6a774-64f58fc685-4x5ng" event={"ID":"145c1a8c-a6e6-4910-98e5-58ee2b6753d3","Type":"ContainerDied","Data":"83eee38a215003031949ed8013339dd7e03dbe71301bad6b8f4f5d32effd227b"}
Sep 17 03:34:15.523990 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:34:15.523919    2552 scope.go:117] "RemoveContainer" containerID="8e05b4f624230081f71f947face151188a8c3e37cab72334cd359c4331a2f926"
Sep 17 03:34:15.524054 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:34:15.524031    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-f6a774-64f58fc685-4x5ng"
Sep 17 03:34:15.551529 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:34:15.551232    2552 scope.go:117] "RemoveContainer" containerID="8e05b4f624230081f71f947face151188a8c3e37cab72334cd359c4331a2f926"
Sep 17 03:34:15.551829 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:34:15.551801    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e05b4f624230081f71f947face151188a8c3e37cab72334cd359c4331a2f926\": container with ID starting with 8e05b4f624230081f71f947face151188a8c3e37cab72334cd359c4331a2f926 not found: ID does not exist" containerID="8e05b4f624230081f71f947face151188a8c3e37cab72334cd359c4331a2f926"
Sep 17 03:34:15.552026 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:34:15.551987    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e05b4f624230081f71f947face151188a8c3e37cab72334cd359c4331a2f926"} err="failed to get container status \"8e05b4f624230081f71f947face151188a8c3e37cab72334cd359c4331a2f926\": rpc error: code = NotFound desc = could not find container \"8e05b4f624230081f71f947face151188a8c3e37cab72334cd359c4331a2f926\": container with ID starting with 8e05b4f624230081f71f947face151188a8c3e37cab72334cd359c4331a2f926 not found: ID does not exist"
Sep 17 03:34:15.563057 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:34:15.563030    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-f6a774-64f58fc685-4x5ng"]
Sep 17 03:34:15.566458 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:34:15.566404    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-f6a774-64f58fc685-4x5ng"]
Sep 17 03:34:16.985789 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:34:16.985751    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="145c1a8c-a6e6-4910-98e5-58ee2b6753d3" path="/var/lib/kubelet/pods/145c1a8c-a6e6-4910-98e5-58ee2b6753d3/volumes"
Sep 17 03:34:21.878139 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:34:21.878087    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-autores-f6a774/autores-model-f6a774-kserve-74885bd495-xwhdt" podUID="b9576704-3a8a-4936-8c89-74b14756dc74" containerName="main" containerID="cri-o://ba73fb1d0b3be7d4f8c5da5fefce981b33d2459fea8f0df3ecbabab7c71438ae" gracePeriod=45
Sep 17 03:34:22.133517 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:34:22.133494    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-f6a774/autores-model-f6a774-kserve-74885bd495-xwhdt"
Sep 17 03:34:22.261448 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:34:22.261415    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b9576704-3a8a-4936-8c89-74b14756dc74-model-cache\") pod \"b9576704-3a8a-4936-8c89-74b14756dc74\" (UID: \"b9576704-3a8a-4936-8c89-74b14756dc74\") "
Sep 17 03:34:22.261641 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:34:22.261493    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9576704-3a8a-4936-8c89-74b14756dc74-tmp-dir\") pod \"b9576704-3a8a-4936-8c89-74b14756dc74\" (UID: \"b9576704-3a8a-4936-8c89-74b14756dc74\") "
Sep 17 03:34:22.261641 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:34:22.261524    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9576704-3a8a-4936-8c89-74b14756dc74-tls-certs\") pod \"b9576704-3a8a-4936-8c89-74b14756dc74\" (UID: \"b9576704-3a8a-4936-8c89-74b14756dc74\") "
Sep 17 03:34:22.261641 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:34:22.261569    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b9576704-3a8a-4936-8c89-74b14756dc74-home\") pod \"b9576704-3a8a-4936-8c89-74b14756dc74\" (UID: \"b9576704-3a8a-4936-8c89-74b14756dc74\") "
Sep 17 03:34:22.261641 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:34:22.261589    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b9576704-3a8a-4936-8c89-74b14756dc74-dshm\") pod \"b9576704-3a8a-4936-8c89-74b14756dc74\" (UID: \"b9576704-3a8a-4936-8c89-74b14756dc74\") "
Sep 17 03:34:22.261641 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:34:22.261632    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6zgj7\" (UniqueName: \"kubernetes.io/projected/b9576704-3a8a-4936-8c89-74b14756dc74-kube-api-access-6zgj7\") pod \"b9576704-3a8a-4936-8c89-74b14756dc74\" (UID: \"b9576704-3a8a-4936-8c89-74b14756dc74\") "
Sep 17 03:34:22.261898 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:34:22.261724    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9576704-3a8a-4936-8c89-74b14756dc74-model-cache" (OuterVolumeSpecName: "model-cache") pod "b9576704-3a8a-4936-8c89-74b14756dc74" (UID: "b9576704-3a8a-4936-8c89-74b14756dc74"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:34:22.261898 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:34:22.261819    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9576704-3a8a-4936-8c89-74b14756dc74-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b9576704-3a8a-4936-8c89-74b14756dc74" (UID: "b9576704-3a8a-4936-8c89-74b14756dc74"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:34:22.261898 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:34:22.261854    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9576704-3a8a-4936-8c89-74b14756dc74-home" (OuterVolumeSpecName: "home") pod "b9576704-3a8a-4936-8c89-74b14756dc74" (UID: "b9576704-3a8a-4936-8c89-74b14756dc74"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:34:22.262046 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:34:22.261964    2552 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9576704-3a8a-4936-8c89-74b14756dc74-tmp-dir\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:34:22.262046 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:34:22.261976    2552 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b9576704-3a8a-4936-8c89-74b14756dc74-home\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:34:22.262046 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:34:22.261985    2552 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b9576704-3a8a-4936-8c89-74b14756dc74-model-cache\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:34:22.263903 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:34:22.263878    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9576704-3a8a-4936-8c89-74b14756dc74-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b9576704-3a8a-4936-8c89-74b14756dc74" (UID: "b9576704-3a8a-4936-8c89-74b14756dc74"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 03:34:22.264012 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:34:22.263931    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9576704-3a8a-4936-8c89-74b14756dc74-dshm" (OuterVolumeSpecName: "dshm") pod "b9576704-3a8a-4936-8c89-74b14756dc74" (UID: "b9576704-3a8a-4936-8c89-74b14756dc74"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:34:22.264012 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:34:22.263946    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9576704-3a8a-4936-8c89-74b14756dc74-kube-api-access-6zgj7" (OuterVolumeSpecName: "kube-api-access-6zgj7") pod "b9576704-3a8a-4936-8c89-74b14756dc74" (UID: "b9576704-3a8a-4936-8c89-74b14756dc74"). InnerVolumeSpecName "kube-api-access-6zgj7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 03:34:22.362543 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:34:22.362485    2552 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9576704-3a8a-4936-8c89-74b14756dc74-tls-certs\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:34:22.362543 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:34:22.362506    2552 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b9576704-3a8a-4936-8c89-74b14756dc74-dshm\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:34:22.362543 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:34:22.362514    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6zgj7\" (UniqueName: \"kubernetes.io/projected/b9576704-3a8a-4936-8c89-74b14756dc74-kube-api-access-6zgj7\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:34:22.570947 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:34:22.570914    2552 generic.go:356] "Generic (PLEG): container finished" podID="b9576704-3a8a-4936-8c89-74b14756dc74" containerID="ba73fb1d0b3be7d4f8c5da5fefce981b33d2459fea8f0df3ecbabab7c71438ae" exitCode=0
Sep 17 03:34:22.571081 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:34:22.570961    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-f6a774/autores-model-f6a774-kserve-74885bd495-xwhdt"
Sep 17 03:34:22.571081 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:34:22.570998    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-f6a774/autores-model-f6a774-kserve-74885bd495-xwhdt" event={"ID":"b9576704-3a8a-4936-8c89-74b14756dc74","Type":"ContainerDied","Data":"ba73fb1d0b3be7d4f8c5da5fefce981b33d2459fea8f0df3ecbabab7c71438ae"}
Sep 17 03:34:22.571081 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:34:22.571046    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-f6a774/autores-model-f6a774-kserve-74885bd495-xwhdt" event={"ID":"b9576704-3a8a-4936-8c89-74b14756dc74","Type":"ContainerDied","Data":"cad13a4eef5755ce45d58591ac593c9bbe3d5435a71c00d8bb3299db720697ab"}
Sep 17 03:34:22.571081 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:34:22.571072    2552 scope.go:117] "RemoveContainer" containerID="ba73fb1d0b3be7d4f8c5da5fefce981b33d2459fea8f0df3ecbabab7c71438ae"
Sep 17 03:34:22.572577 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:34:22.572553    2552 generic.go:356] "Generic (PLEG): container finished" podID="08d183e7-7176-4eaa-8843-747fb05618d7" containerID="76573d3a3aef61ec8f9d66177c620fc0d7b443aa40b034ca4fa698766be3e10c" exitCode=0
Sep 17 03:34:22.572710 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:34:22.572592    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-acf311cd-openshift-default-5bf744fb4d-vl6f2" event={"ID":"08d183e7-7176-4eaa-8843-747fb05618d7","Type":"ContainerDied","Data":"76573d3a3aef61ec8f9d66177c620fc0d7b443aa40b034ca4fa698766be3e10c"}
Sep 17 03:34:22.573021 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:34:22.573006    2552 scope.go:117] "RemoveContainer" containerID="76573d3a3aef61ec8f9d66177c620fc0d7b443aa40b034ca4fa698766be3e10c"
Sep 17 03:34:22.574051 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:34:22.574025    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 03:34:22.574142 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:34:22.574087    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 03:34:22.574142 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:34:22.574119    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 03:34:22.579865 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:34:22.579851    2552 scope.go:117] "RemoveContainer" containerID="ba73fb1d0b3be7d4f8c5da5fefce981b33d2459fea8f0df3ecbabab7c71438ae"
Sep 17 03:34:22.580107 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:34:22.580089    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba73fb1d0b3be7d4f8c5da5fefce981b33d2459fea8f0df3ecbabab7c71438ae\": container with ID starting with ba73fb1d0b3be7d4f8c5da5fefce981b33d2459fea8f0df3ecbabab7c71438ae not found: ID does not exist" containerID="ba73fb1d0b3be7d4f8c5da5fefce981b33d2459fea8f0df3ecbabab7c71438ae"
Sep 17 03:34:22.580162 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:34:22.580114    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba73fb1d0b3be7d4f8c5da5fefce981b33d2459fea8f0df3ecbabab7c71438ae"} err="failed to get container status \"ba73fb1d0b3be7d4f8c5da5fefce981b33d2459fea8f0df3ecbabab7c71438ae\": rpc error: code = NotFound desc = could not find container \"ba73fb1d0b3be7d4f8c5da5fefce981b33d2459fea8f0df3ecbabab7c71438ae\": container with ID starting with ba73fb1d0b3be7d4f8c5da5fefce981b33d2459fea8f0df3ecbabab7c71438ae not found: ID does not exist"
Sep 17 03:34:22.612668 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:34:22.612590    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-f6a774/autores-model-f6a774-kserve-74885bd495-xwhdt"]
Sep 17 03:34:22.617406 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:34:22.617381    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-f6a774/autores-model-f6a774-kserve-74885bd495-xwhdt"]
Sep 17 03:34:22.986583 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:34:22.986507    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b9576704-3a8a-4936-8c89-74b14756dc74" path="/var/lib/kubelet/pods/b9576704-3a8a-4936-8c89-74b14756dc74/volumes"
Sep 17 03:34:23.580557 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:34:23.580521    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-acf311cd-openshift-default-5bf744fb4d-vl6f2" event={"ID":"08d183e7-7176-4eaa-8843-747fb05618d7","Type":"ContainerStarted","Data":"dc50ebc4954e78bcff1ccd53853e4715b4440c95f7ab1003a8c0e6b4ab64a8bd"}
Sep 17 03:34:23.716741 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:34:23.716704    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-acf311cd-openshift-default-5bf744fb4d-vl6f2"
Sep 17 03:34:23.718096 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:34:23.718072    2552 patch_prober.go:28] interesting pod/e2e-derive-acf311cd-openshift-default-5bf744fb4d-vl6f2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Sep 17 03:34:23.718255 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:34:23.718118    2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-acf311cd-openshift-default-5bf744fb4d-vl6f2" podUID="08d183e7-7176-4eaa-8843-747fb05618d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Sep 17 03:34:24.716520 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:34:24.716486    2552 patch_prober.go:28] interesting pod/e2e-derive-acf311cd-openshift-default-5bf744fb4d-vl6f2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Sep 17 03:34:24.716925 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:34:24.716539    2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-acf311cd-openshift-default-5bf744fb4d-vl6f2" podUID="08d183e7-7176-4eaa-8843-747fb05618d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Sep 17 03:34:25.061498 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:34:25.061356    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-acf311cd-b97c76cd-w8znq"]
Sep 17 03:34:25.061651 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:34:25.061618    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-acf311cd-b97c76cd-w8znq" podUID="bbe608d1-8ab8-48f7-b12a-08f76cb35aa6" containerName="maas-api" containerID="cri-o://6c80f9d0b12183c377e04d403bf6441905746fc6a6fdc8d86976579b90dba02c" gracePeriod=30
Sep 17 03:34:25.359465 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:34:25.359437    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-acf311cd-b97c76cd-w8znq"
Sep 17 03:34:25.415174 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:34:25.415141    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-acf311cd-openshift-default-5bf744fb4d-vl6f2"
Sep 17 03:34:25.442632 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:34:25.437848    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-acf311cd-openshift-default-5bf744fb4d-vl6f2"]
Sep 17 03:34:25.493032 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:34:25.493003    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bbe608d1-8ab8-48f7-b12a-08f76cb35aa6-metrics-certs\") pod \"bbe608d1-8ab8-48f7-b12a-08f76cb35aa6\" (UID: \"bbe608d1-8ab8-48f7-b12a-08f76cb35aa6\") "
Sep 17 03:34:25.493188 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:34:25.493071    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bbe608d1-8ab8-48f7-b12a-08f76cb35aa6-maas-api-tls\") pod \"bbe608d1-8ab8-48f7-b12a-08f76cb35aa6\" (UID: \"bbe608d1-8ab8-48f7-b12a-08f76cb35aa6\") "
Sep 17 03:34:25.493188 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:34:25.493100    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-62x4f\" (UniqueName: \"kubernetes.io/projected/bbe608d1-8ab8-48f7-b12a-08f76cb35aa6-kube-api-access-62x4f\") pod \"bbe608d1-8ab8-48f7-b12a-08f76cb35aa6\" (UID: \"bbe608d1-8ab8-48f7-b12a-08f76cb35aa6\") "
Sep 17 03:34:25.493188 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:34:25.493167    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbe608d1-8ab8-48f7-b12a-08f76cb35aa6-openshift-service-ca\") pod \"bbe608d1-8ab8-48f7-b12a-08f76cb35aa6\" (UID: \"bbe608d1-8ab8-48f7-b12a-08f76cb35aa6\") "
Sep 17 03:34:25.493571 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:34:25.493539    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbe608d1-8ab8-48f7-b12a-08f76cb35aa6-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "bbe608d1-8ab8-48f7-b12a-08f76cb35aa6" (UID: "bbe608d1-8ab8-48f7-b12a-08f76cb35aa6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 03:34:25.495289 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:34:25.495261    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bbe608d1-8ab8-48f7-b12a-08f76cb35aa6-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "bbe608d1-8ab8-48f7-b12a-08f76cb35aa6" (UID: "bbe608d1-8ab8-48f7-b12a-08f76cb35aa6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 03:34:25.495409 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:34:25.495295    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbe608d1-8ab8-48f7-b12a-08f76cb35aa6-kube-api-access-62x4f" (OuterVolumeSpecName: "kube-api-access-62x4f") pod "bbe608d1-8ab8-48f7-b12a-08f76cb35aa6" (UID: "bbe608d1-8ab8-48f7-b12a-08f76cb35aa6"). InnerVolumeSpecName "kube-api-access-62x4f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 03:34:25.495572 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:34:25.495552    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bbe608d1-8ab8-48f7-b12a-08f76cb35aa6-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "bbe608d1-8ab8-48f7-b12a-08f76cb35aa6" (UID: "bbe608d1-8ab8-48f7-b12a-08f76cb35aa6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 03:34:25.591991 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:34:25.591907    2552 generic.go:356] "Generic (PLEG): container finished" podID="bbe608d1-8ab8-48f7-b12a-08f76cb35aa6" containerID="6c80f9d0b12183c377e04d403bf6441905746fc6a6fdc8d86976579b90dba02c" exitCode=0
Sep 17 03:34:25.591991 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:34:25.591956    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-acf311cd-b97c76cd-w8znq"
Sep 17 03:34:25.592183 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:34:25.591994    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-acf311cd-b97c76cd-w8znq" event={"ID":"bbe608d1-8ab8-48f7-b12a-08f76cb35aa6","Type":"ContainerDied","Data":"6c80f9d0b12183c377e04d403bf6441905746fc6a6fdc8d86976579b90dba02c"}
Sep 17 03:34:25.592183 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:34:25.592048    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-acf311cd-b97c76cd-w8znq" event={"ID":"bbe608d1-8ab8-48f7-b12a-08f76cb35aa6","Type":"ContainerDied","Data":"35f017871fd3146f3b8e2a97d41766b1a4d650fe09a509a993a0550dfbf3429a"}
Sep 17 03:34:25.592183 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:34:25.592077    2552 scope.go:117] "RemoveContainer" containerID="6c80f9d0b12183c377e04d403bf6441905746fc6a6fdc8d86976579b90dba02c"
Sep 17 03:34:25.592510 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:34:25.592485    2552 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-derive-acf311cd-openshift-default-5bf744fb4d-vl6f2" secret="" err="secret \"e2e-derive-acf311cd-openshift-default-dockercfg-vq6vq\" not found"
Sep 17 03:34:25.594051 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:34:25.594026    2552 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbe608d1-8ab8-48f7-b12a-08f76cb35aa6-openshift-service-ca\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:34:25.594128 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:34:25.594060    2552 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bbe608d1-8ab8-48f7-b12a-08f76cb35aa6-metrics-certs\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:34:25.594128 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:34:25.594075    2552 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bbe608d1-8ab8-48f7-b12a-08f76cb35aa6-maas-api-tls\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:34:25.594128 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:34:25.594089    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-62x4f\" (UniqueName: \"kubernetes.io/projected/bbe608d1-8ab8-48f7-b12a-08f76cb35aa6-kube-api-access-62x4f\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:34:25.601584 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:34:25.601568    2552 scope.go:117] "RemoveContainer" containerID="6c80f9d0b12183c377e04d403bf6441905746fc6a6fdc8d86976579b90dba02c"
Sep 17 03:34:25.601884 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:34:25.601867    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c80f9d0b12183c377e04d403bf6441905746fc6a6fdc8d86976579b90dba02c\": container with ID starting with 6c80f9d0b12183c377e04d403bf6441905746fc6a6fdc8d86976579b90dba02c not found: ID does not exist" containerID="6c80f9d0b12183c377e04d403bf6441905746fc6a6fdc8d86976579b90dba02c"
Sep 17 03:34:25.601938 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:34:25.601892    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c80f9d0b12183c377e04d403bf6441905746fc6a6fdc8d86976579b90dba02c"} err="failed to get container status \"6c80f9d0b12183c377e04d403bf6441905746fc6a6fdc8d86976579b90dba02c\": rpc error: code = NotFound desc = could not find container \"6c80f9d0b12183c377e04d403bf6441905746fc6a6fdc8d86976579b90dba02c\": container with ID starting with 6c80f9d0b12183c377e04d403bf6441905746fc6a6fdc8d86976579b90dba02c not found: ID does not exist"
Sep 17 03:34:25.614268 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:34:25.614248    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-acf311cd-b97c76cd-w8znq"]
Sep 17 03:34:25.618331 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:34:25.618310    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-acf311cd-b97c76cd-w8znq"]
Sep 17 03:34:25.716343 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:34:25.716315    2552 patch_prober.go:28] interesting pod/e2e-derive-acf311cd-openshift-default-5bf744fb4d-vl6f2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Sep 17 03:34:25.716512 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:34:25.716369    2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-acf311cd-openshift-default-5bf744fb4d-vl6f2" podUID="08d183e7-7176-4eaa-8843-747fb05618d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Sep 17 03:34:26.598298 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:34:26.598263    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-acf311cd-openshift-default-5bf744fb4d-vl6f2" podUID="08d183e7-7176-4eaa-8843-747fb05618d7" containerName="istio-proxy" containerID="cri-o://dc50ebc4954e78bcff1ccd53853e4715b4440c95f7ab1003a8c0e6b4ab64a8bd" gracePeriod=30
Sep 17 03:34:26.985404 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:34:26.985322    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bbe608d1-8ab8-48f7-b12a-08f76cb35aa6" path="/var/lib/kubelet/pods/bbe608d1-8ab8-48f7-b12a-08f76cb35aa6/volumes"
Sep 17 03:34:28.370650 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:34:28.370615    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-214fea/premium-w4-214fea-kserve-589dcd865f-hkkwm"]
Sep 17 03:34:28.371028 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:34:28.371019    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="145c1a8c-a6e6-4910-98e5-58ee2b6753d3" containerName="maas-api"
Sep 17 03:34:28.371071 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:34:28.371031    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="145c1a8c-a6e6-4910-98e5-58ee2b6753d3" containerName="maas-api"
Sep 17 03:34:28.371071 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:34:28.371040    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9576704-3a8a-4936-8c89-74b14756dc74" containerName="main"
Sep 17 03:34:28.371071 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:34:28.371045    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9576704-3a8a-4936-8c89-74b14756dc74" containerName="main"
Sep 17 03:34:28.371071 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:34:28.371051    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bbe608d1-8ab8-48f7-b12a-08f76cb35aa6" containerName="maas-api"
Sep 17 03:34:28.371071 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:34:28.371057    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="bbe608d1-8ab8-48f7-b12a-08f76cb35aa6" containerName="maas-api"
Sep 17 03:34:28.371220 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:34:28.371126    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="145c1a8c-a6e6-4910-98e5-58ee2b6753d3" containerName="maas-api"
Sep 17 03:34:28.371220 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:34:28.371133    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9576704-3a8a-4936-8c89-74b14756dc74" containerName="main"
Sep 17 03:34:28.371220 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:34:28.371141    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="bbe608d1-8ab8-48f7-b12a-08f76cb35aa6" containerName="maas-api"
Sep 17 03:34:28.381298 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:34:28.381265    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-214fea/premium-w4-214fea-kserve-589dcd865f-hkkwm"
Sep 17 03:34:28.383988 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:34:28.383962    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-214fea\"/\"premium-w4-214fea-kserve-self-signed-certs\""
Sep 17 03:34:28.387975 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:34:28.387950    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-214fea/premium-w4-214fea-kserve-589dcd865f-hkkwm"]
Sep 17 03:34:28.521619 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:34:28.521574    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/469467a8-4b70-4019-a870-e8e600f9f840-tmp-dir\") pod \"premium-w4-214fea-kserve-589dcd865f-hkkwm\" (UID: \"469467a8-4b70-4019-a870-e8e600f9f840\") " pod="e2e-models-e2e-worker-w4-214fea/premium-w4-214fea-kserve-589dcd865f-hkkwm"
Sep 17 03:34:28.521793 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:34:28.521639    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tklb9\" (UniqueName: \"kubernetes.io/projected/469467a8-4b70-4019-a870-e8e600f9f840-kube-api-access-tklb9\") pod \"premium-w4-214fea-kserve-589dcd865f-hkkwm\" (UID: \"469467a8-4b70-4019-a870-e8e600f9f840\") " pod="e2e-models-e2e-worker-w4-214fea/premium-w4-214fea-kserve-589dcd865f-hkkwm"
Sep 17 03:34:28.521793 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:34:28.521723    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/469467a8-4b70-4019-a870-e8e600f9f840-home\") pod \"premium-w4-214fea-kserve-589dcd865f-hkkwm\" (UID: \"469467a8-4b70-4019-a870-e8e600f9f840\") " pod="e2e-models-e2e-worker-w4-214fea/premium-w4-214fea-kserve-589dcd865f-hkkwm"
Sep 17 03:34:28.521908 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:34:28.521797    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/469467a8-4b70-4019-a870-e8e600f9f840-dshm\") pod \"premium-w4-214fea-kserve-589dcd865f-hkkwm\" (UID: \"469467a8-4b70-4019-a870-e8e600f9f840\") " pod="e2e-models-e2e-worker-w4-214fea/premium-w4-214fea-kserve-589dcd865f-hkkwm"
Sep 17 03:34:28.521908 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:34:28.521825    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/469467a8-4b70-4019-a870-e8e600f9f840-tls-certs\") pod \"premium-w4-214fea-kserve-589dcd865f-hkkwm\" (UID: \"469467a8-4b70-4019-a870-e8e600f9f840\") " pod="e2e-models-e2e-worker-w4-214fea/premium-w4-214fea-kserve-589dcd865f-hkkwm"
Sep 17 03:34:28.521908 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:34:28.521876    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/469467a8-4b70-4019-a870-e8e600f9f840-model-cache\") pod \"premium-w4-214fea-kserve-589dcd865f-hkkwm\" (UID: \"469467a8-4b70-4019-a870-e8e600f9f840\") " pod="e2e-models-e2e-worker-w4-214fea/premium-w4-214fea-kserve-589dcd865f-hkkwm"
Sep 17 03:34:28.622630 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:34:28.622521    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/469467a8-4b70-4019-a870-e8e600f9f840-dshm\") pod \"premium-w4-214fea-kserve-589dcd865f-hkkwm\" (UID: \"469467a8-4b70-4019-a870-e8e600f9f840\") " pod="e2e-models-e2e-worker-w4-214fea/premium-w4-214fea-kserve-589dcd865f-hkkwm"
Sep 17 03:34:28.622630 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:34:28.622562    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/469467a8-4b70-4019-a870-e8e600f9f840-tls-certs\") pod \"premium-w4-214fea-kserve-589dcd865f-hkkwm\" (UID: \"469467a8-4b70-4019-a870-e8e600f9f840\") " pod="e2e-models-e2e-worker-w4-214fea/premium-w4-214fea-kserve-589dcd865f-hkkwm"
Sep 17 03:34:28.622630 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:34:28.622624    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/469467a8-4b70-4019-a870-e8e600f9f840-model-cache\") pod \"premium-w4-214fea-kserve-589dcd865f-hkkwm\" (UID: \"469467a8-4b70-4019-a870-e8e600f9f840\") " pod="e2e-models-e2e-worker-w4-214fea/premium-w4-214fea-kserve-589dcd865f-hkkwm"
Sep 17 03:34:28.622927 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:34:28.622821    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/469467a8-4b70-4019-a870-e8e600f9f840-tmp-dir\") pod \"premium-w4-214fea-kserve-589dcd865f-hkkwm\" (UID: \"469467a8-4b70-4019-a870-e8e600f9f840\") " pod="e2e-models-e2e-worker-w4-214fea/premium-w4-214fea-kserve-589dcd865f-hkkwm"
Sep 17 03:34:28.622927 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:34:28.622889    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tklb9\" (UniqueName: \"kubernetes.io/projected/469467a8-4b70-4019-a870-e8e600f9f840-kube-api-access-tklb9\") pod \"premium-w4-214fea-kserve-589dcd865f-hkkwm\" (UID: \"469467a8-4b70-4019-a870-e8e600f9f840\") " pod="e2e-models-e2e-worker-w4-214fea/premium-w4-214fea-kserve-589dcd865f-hkkwm"
Sep 17 03:34:28.622927 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:34:28.622925    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/469467a8-4b70-4019-a870-e8e600f9f840-home\") pod \"premium-w4-214fea-kserve-589dcd865f-hkkwm\" (UID: \"469467a8-4b70-4019-a870-e8e600f9f840\") " pod="e2e-models-e2e-worker-w4-214fea/premium-w4-214fea-kserve-589dcd865f-hkkwm"
Sep 17 03:34:28.623084 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:34:28.623056    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/469467a8-4b70-4019-a870-e8e600f9f840-model-cache\") pod \"premium-w4-214fea-kserve-589dcd865f-hkkwm\" (UID: \"469467a8-4b70-4019-a870-e8e600f9f840\") " pod="e2e-models-e2e-worker-w4-214fea/premium-w4-214fea-kserve-589dcd865f-hkkwm"
Sep 17 03:34:28.623177 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:34:28.623157    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/469467a8-4b70-4019-a870-e8e600f9f840-tmp-dir\") pod \"premium-w4-214fea-kserve-589dcd865f-hkkwm\" (UID: \"469467a8-4b70-4019-a870-e8e600f9f840\") " pod="e2e-models-e2e-worker-w4-214fea/premium-w4-214fea-kserve-589dcd865f-hkkwm"
Sep 17 03:34:28.623351 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:34:28.623326    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/469467a8-4b70-4019-a870-e8e600f9f840-home\") pod \"premium-w4-214fea-kserve-589dcd865f-hkkwm\" (UID: \"469467a8-4b70-4019-a870-e8e600f9f840\") " pod="e2e-models-e2e-worker-w4-214fea/premium-w4-214fea-kserve-589dcd865f-hkkwm"
Sep 17 03:34:28.625414 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:34:28.625385    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/469467a8-4b70-4019-a870-e8e600f9f840-dshm\") pod \"premium-w4-214fea-kserve-589dcd865f-hkkwm\" (UID: \"469467a8-4b70-4019-a870-e8e600f9f840\") " pod="e2e-models-e2e-worker-w4-214fea/premium-w4-214fea-kserve-589dcd865f-hkkwm"
Sep 17 03:34:28.625588 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:34:28.625572    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/469467a8-4b70-4019-a870-e8e600f9f840-tls-certs\") pod \"premium-w4-214fea-kserve-589dcd865f-hkkwm\" (UID: \"469467a8-4b70-4019-a870-e8e600f9f840\") " pod="e2e-models-e2e-worker-w4-214fea/premium-w4-214fea-kserve-589dcd865f-hkkwm"
Sep 17 03:34:28.634319 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:34:28.634293    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tklb9\" (UniqueName: \"kubernetes.io/projected/469467a8-4b70-4019-a870-e8e600f9f840-kube-api-access-tklb9\") pod \"premium-w4-214fea-kserve-589dcd865f-hkkwm\" (UID: \"469467a8-4b70-4019-a870-e8e600f9f840\") " pod="e2e-models-e2e-worker-w4-214fea/premium-w4-214fea-kserve-589dcd865f-hkkwm"
Sep 17 03:34:28.692682 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:34:28.692649    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-214fea/premium-w4-214fea-kserve-589dcd865f-hkkwm"
Sep 17 03:34:28.972378 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:34:28.972305    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-214fea/premium-w4-214fea-kserve-589dcd865f-hkkwm"]
Sep 17 03:34:28.975417 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:34:28.975380    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod469467a8_4b70_4019_a870_e8e600f9f840.slice/crio-e75aeb83fe2cc95c1359dff83526eaceb3a949a0d5809f7dcc15205ee3f0f64f WatchSource:0}: Error finding container e75aeb83fe2cc95c1359dff83526eaceb3a949a0d5809f7dcc15205ee3f0f64f: Status 404 returned error can't find the container with id e75aeb83fe2cc95c1359dff83526eaceb3a949a0d5809f7dcc15205ee3f0f64f
Sep 17 03:34:29.615489 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:34:29.615453    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-214fea/premium-w4-214fea-kserve-589dcd865f-hkkwm" event={"ID":"469467a8-4b70-4019-a870-e8e600f9f840","Type":"ContainerStarted","Data":"51ba05a0267aa4f6465a1201ce577023b024740d86548a5e7b76a0fbc2863af3"}
Sep 17 03:34:29.615489 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:34:29.615493    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-214fea/premium-w4-214fea-kserve-589dcd865f-hkkwm" event={"ID":"469467a8-4b70-4019-a870-e8e600f9f840","Type":"ContainerStarted","Data":"e75aeb83fe2cc95c1359dff83526eaceb3a949a0d5809f7dcc15205ee3f0f64f"}
Sep 17 03:34:29.632636 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:34:29.632562    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-214fea/premium-w4-214fea-kserve-589dcd865f-hkkwm" podStartSLOduration=1.632544829 podStartE2EDuration="1.632544829s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 03:34:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:34:29.632087025 +0000 UTC m=+905.098494543" watchObservedRunningTime="2026-09-17 03:34:29.632544829 +0000 UTC m=+905.098952357"
Sep 17 03:34:33.941496 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:34:33.941473    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-acf311cd-openshift-default-5bf744fb4d-vl6f2"
Sep 17 03:34:34.071032 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:34:34.071001    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08d183e7-7176-4eaa-8843-747fb05618d7-istio-data\") pod \"08d183e7-7176-4eaa-8843-747fb05618d7\" (UID: \"08d183e7-7176-4eaa-8843-747fb05618d7\") "
Sep 17 03:34:34.071032 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:34:34.071033    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08d183e7-7176-4eaa-8843-747fb05618d7-istio-envoy\") pod \"08d183e7-7176-4eaa-8843-747fb05618d7\" (UID: \"08d183e7-7176-4eaa-8843-747fb05618d7\") "
Sep 17 03:34:34.071270 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:34:34.071057    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08d183e7-7176-4eaa-8843-747fb05618d7-istio-token\") pod \"08d183e7-7176-4eaa-8843-747fb05618d7\" (UID: \"08d183e7-7176-4eaa-8843-747fb05618d7\") "
Sep 17 03:34:34.071270 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:34:34.071083    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08d183e7-7176-4eaa-8843-747fb05618d7-credential-socket\") pod \"08d183e7-7176-4eaa-8843-747fb05618d7\" (UID: \"08d183e7-7176-4eaa-8843-747fb05618d7\") "
Sep 17 03:34:34.071270 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:34:34.071134    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9hzdz\" (UniqueName: \"kubernetes.io/projected/08d183e7-7176-4eaa-8843-747fb05618d7-kube-api-access-9hzdz\") pod \"08d183e7-7176-4eaa-8843-747fb05618d7\" (UID: \"08d183e7-7176-4eaa-8843-747fb05618d7\") "
Sep 17 03:34:34.071270 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:34:34.071168    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08d183e7-7176-4eaa-8843-747fb05618d7-istio-podinfo\") pod \"08d183e7-7176-4eaa-8843-747fb05618d7\" (UID: \"08d183e7-7176-4eaa-8843-747fb05618d7\") "
Sep 17 03:34:34.071270 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:34:34.071190    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08d183e7-7176-4eaa-8843-747fb05618d7-workload-certs\") pod \"08d183e7-7176-4eaa-8843-747fb05618d7\" (UID: \"08d183e7-7176-4eaa-8843-747fb05618d7\") "
Sep 17 03:34:34.071270 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:34:34.071240    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08d183e7-7176-4eaa-8843-747fb05618d7-istiod-ca-cert\") pod \"08d183e7-7176-4eaa-8843-747fb05618d7\" (UID: \"08d183e7-7176-4eaa-8843-747fb05618d7\") "
Sep 17 03:34:34.071565 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:34:34.071296    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08d183e7-7176-4eaa-8843-747fb05618d7-istio-data" (OuterVolumeSpecName: "istio-data") pod "08d183e7-7176-4eaa-8843-747fb05618d7" (UID: "08d183e7-7176-4eaa-8843-747fb05618d7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:34:34.071565 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:34:34.071356    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08d183e7-7176-4eaa-8843-747fb05618d7-workload-socket\") pod \"08d183e7-7176-4eaa-8843-747fb05618d7\" (UID: \"08d183e7-7176-4eaa-8843-747fb05618d7\") "
Sep 17 03:34:34.071565 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:34:34.071442    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08d183e7-7176-4eaa-8843-747fb05618d7-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "08d183e7-7176-4eaa-8843-747fb05618d7" (UID: "08d183e7-7176-4eaa-8843-747fb05618d7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:34:34.071565 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:34:34.071485    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08d183e7-7176-4eaa-8843-747fb05618d7-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "08d183e7-7176-4eaa-8843-747fb05618d7" (UID: "08d183e7-7176-4eaa-8843-747fb05618d7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:34:34.071803 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:34:34.071698    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08d183e7-7176-4eaa-8843-747fb05618d7-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "08d183e7-7176-4eaa-8843-747fb05618d7" (UID: "08d183e7-7176-4eaa-8843-747fb05618d7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:34:34.071882 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:34:34.071861    2552 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08d183e7-7176-4eaa-8843-747fb05618d7-istio-data\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:34:34.071948 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:34:34.071889    2552 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08d183e7-7176-4eaa-8843-747fb05618d7-credential-socket\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:34:34.071948 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:34:34.071906    2552 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08d183e7-7176-4eaa-8843-747fb05618d7-workload-certs\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:34:34.071948 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:34:34.071921    2552 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08d183e7-7176-4eaa-8843-747fb05618d7-workload-socket\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:34:34.072211 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:34:34.072188    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08d183e7-7176-4eaa-8843-747fb05618d7-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "08d183e7-7176-4eaa-8843-747fb05618d7" (UID: "08d183e7-7176-4eaa-8843-747fb05618d7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 03:34:34.073296 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:34:34.073274    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08d183e7-7176-4eaa-8843-747fb05618d7-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "08d183e7-7176-4eaa-8843-747fb05618d7" (UID: "08d183e7-7176-4eaa-8843-747fb05618d7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:34:34.073472 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:34:34.073448    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/08d183e7-7176-4eaa-8843-747fb05618d7-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "08d183e7-7176-4eaa-8843-747fb05618d7" (UID: "08d183e7-7176-4eaa-8843-747fb05618d7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 17 03:34:34.073738 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:34:34.073717    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08d183e7-7176-4eaa-8843-747fb05618d7-istio-token" (OuterVolumeSpecName: "istio-token") pod "08d183e7-7176-4eaa-8843-747fb05618d7" (UID: "08d183e7-7176-4eaa-8843-747fb05618d7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 03:34:34.073813 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:34:34.073756    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08d183e7-7176-4eaa-8843-747fb05618d7-kube-api-access-9hzdz" (OuterVolumeSpecName: "kube-api-access-9hzdz") pod "08d183e7-7176-4eaa-8843-747fb05618d7" (UID: "08d183e7-7176-4eaa-8843-747fb05618d7"). InnerVolumeSpecName "kube-api-access-9hzdz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 03:34:34.172937 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:34:34.172900    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9hzdz\" (UniqueName: \"kubernetes.io/projected/08d183e7-7176-4eaa-8843-747fb05618d7-kube-api-access-9hzdz\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:34:34.172937 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:34:34.172936    2552 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08d183e7-7176-4eaa-8843-747fb05618d7-istio-podinfo\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:34:34.172937 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:34:34.172947    2552 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08d183e7-7176-4eaa-8843-747fb05618d7-istiod-ca-cert\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:34:34.173132 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:34:34.172955    2552 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08d183e7-7176-4eaa-8843-747fb05618d7-istio-envoy\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:34:34.173132 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:34:34.172963    2552 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08d183e7-7176-4eaa-8843-747fb05618d7-istio-token\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:34:34.644071 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:34:34.644039    2552 generic.go:356] "Generic (PLEG): container finished" podID="08d183e7-7176-4eaa-8843-747fb05618d7" containerID="dc50ebc4954e78bcff1ccd53853e4715b4440c95f7ab1003a8c0e6b4ab64a8bd" exitCode=0
Sep 17 03:34:34.644249 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:34:34.644094    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-acf311cd-openshift-default-5bf744fb4d-vl6f2" event={"ID":"08d183e7-7176-4eaa-8843-747fb05618d7","Type":"ContainerDied","Data":"dc50ebc4954e78bcff1ccd53853e4715b4440c95f7ab1003a8c0e6b4ab64a8bd"}
Sep 17 03:34:34.644249 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:34:34.644106    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-acf311cd-openshift-default-5bf744fb4d-vl6f2"
Sep 17 03:34:34.644249 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:34:34.644162    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-acf311cd-openshift-default-5bf744fb4d-vl6f2" event={"ID":"08d183e7-7176-4eaa-8843-747fb05618d7","Type":"ContainerDied","Data":"269a51b72db52e6b717f9d5ff54ff844eed34ab515ae676d7b60568db4b2d80b"}
Sep 17 03:34:34.644249 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:34:34.644185    2552 scope.go:117] "RemoveContainer" containerID="dc50ebc4954e78bcff1ccd53853e4715b4440c95f7ab1003a8c0e6b4ab64a8bd"
Sep 17 03:34:34.653975 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:34:34.653955    2552 scope.go:117] "RemoveContainer" containerID="76573d3a3aef61ec8f9d66177c620fc0d7b443aa40b034ca4fa698766be3e10c"
Sep 17 03:34:34.661895 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:34:34.661878    2552 scope.go:117] "RemoveContainer" containerID="dc50ebc4954e78bcff1ccd53853e4715b4440c95f7ab1003a8c0e6b4ab64a8bd"
Sep 17 03:34:34.662154 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:34:34.662135    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc50ebc4954e78bcff1ccd53853e4715b4440c95f7ab1003a8c0e6b4ab64a8bd\": container with ID starting with dc50ebc4954e78bcff1ccd53853e4715b4440c95f7ab1003a8c0e6b4ab64a8bd not found: ID does not exist" containerID="dc50ebc4954e78bcff1ccd53853e4715b4440c95f7ab1003a8c0e6b4ab64a8bd"
Sep 17 03:34:34.662200 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:34:34.662164    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc50ebc4954e78bcff1ccd53853e4715b4440c95f7ab1003a8c0e6b4ab64a8bd"} err="failed to get container status \"dc50ebc4954e78bcff1ccd53853e4715b4440c95f7ab1003a8c0e6b4ab64a8bd\": rpc error: code = NotFound desc = could not find container \"dc50ebc4954e78bcff1ccd53853e4715b4440c95f7ab1003a8c0e6b4ab64a8bd\": container with ID starting with dc50ebc4954e78bcff1ccd53853e4715b4440c95f7ab1003a8c0e6b4ab64a8bd not found: ID does not exist"
Sep 17 03:34:34.662200 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:34:34.662180    2552 scope.go:117] "RemoveContainer" containerID="76573d3a3aef61ec8f9d66177c620fc0d7b443aa40b034ca4fa698766be3e10c"
Sep 17 03:34:34.662418 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:34:34.662401    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76573d3a3aef61ec8f9d66177c620fc0d7b443aa40b034ca4fa698766be3e10c\": container with ID starting with 76573d3a3aef61ec8f9d66177c620fc0d7b443aa40b034ca4fa698766be3e10c not found: ID does not exist" containerID="76573d3a3aef61ec8f9d66177c620fc0d7b443aa40b034ca4fa698766be3e10c"
Sep 17 03:34:34.662483 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:34:34.662422    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76573d3a3aef61ec8f9d66177c620fc0d7b443aa40b034ca4fa698766be3e10c"} err="failed to get container status \"76573d3a3aef61ec8f9d66177c620fc0d7b443aa40b034ca4fa698766be3e10c\": rpc error: code = NotFound desc = could not find container \"76573d3a3aef61ec8f9d66177c620fc0d7b443aa40b034ca4fa698766be3e10c\": container with ID starting with 76573d3a3aef61ec8f9d66177c620fc0d7b443aa40b034ca4fa698766be3e10c not found: ID does not exist"
Sep 17 03:34:34.683822 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:34:34.683801    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-acf311cd-openshift-default-5bf744fb4d-vl6f2"]
Sep 17 03:34:34.693074 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:34:34.693046    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-acf311cd-openshift-default-5bf744fb4d-vl6f2"]
Sep 17 03:34:34.986699 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:34:34.986621    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="08d183e7-7176-4eaa-8843-747fb05618d7" path="/var/lib/kubelet/pods/08d183e7-7176-4eaa-8843-747fb05618d7/volumes"
Sep 17 03:34:38.693759 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:34:38.693727    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-214fea/premium-w4-214fea-kserve-589dcd865f-hkkwm"
Sep 17 03:34:38.694278 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:34:38.693780    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-214fea/premium-w4-214fea-kserve-589dcd865f-hkkwm"
Sep 17 03:34:38.706365 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:34:38.706339    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-214fea/premium-w4-214fea-kserve-589dcd865f-hkkwm"
Sep 17 03:34:39.682525 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:34:39.682496    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-214fea/premium-w4-214fea-kserve-589dcd865f-hkkwm"
Sep 17 03:35:07.384219 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:35:07.384189    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-1afd34/premium-w0-1afd34-kserve-85c998d845-r5zq9"]
Sep 17 03:35:07.384644 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:35:07.384629    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08d183e7-7176-4eaa-8843-747fb05618d7" containerName="istio-proxy"
Sep 17 03:35:07.384696 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:35:07.384646    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="08d183e7-7176-4eaa-8843-747fb05618d7" containerName="istio-proxy"
Sep 17 03:35:07.384696 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:35:07.384666    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08d183e7-7176-4eaa-8843-747fb05618d7" containerName="istio-proxy"
Sep 17 03:35:07.384696 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:35:07.384671    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="08d183e7-7176-4eaa-8843-747fb05618d7" containerName="istio-proxy"
Sep 17 03:35:07.384880 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:35:07.384746    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="08d183e7-7176-4eaa-8843-747fb05618d7" containerName="istio-proxy"
Sep 17 03:35:07.384880 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:35:07.384761    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="08d183e7-7176-4eaa-8843-747fb05618d7" containerName="istio-proxy"
Sep 17 03:35:07.388089 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:35:07.388069    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1afd34/premium-w0-1afd34-kserve-85c998d845-r5zq9"
Sep 17 03:35:07.390842 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:35:07.390823    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-1afd34\"/\"premium-w0-1afd34-kserve-self-signed-certs\""
Sep 17 03:35:07.399002 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:35:07.398979    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-1afd34/premium-w0-1afd34-kserve-85c998d845-r5zq9"]
Sep 17 03:35:07.486858 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:35:07.486830    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d3098da-c426-4ed1-8932-737fb644755c-tls-certs\") pod \"premium-w0-1afd34-kserve-85c998d845-r5zq9\" (UID: \"6d3098da-c426-4ed1-8932-737fb644755c\") " pod="e2e-models-e2e-worker-w0-1afd34/premium-w0-1afd34-kserve-85c998d845-r5zq9"
Sep 17 03:35:07.486976 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:35:07.486887    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d3098da-c426-4ed1-8932-737fb644755c-home\") pod \"premium-w0-1afd34-kserve-85c998d845-r5zq9\" (UID: \"6d3098da-c426-4ed1-8932-737fb644755c\") " pod="e2e-models-e2e-worker-w0-1afd34/premium-w0-1afd34-kserve-85c998d845-r5zq9"
Sep 17 03:35:07.487025 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:35:07.486984    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcncx\" (UniqueName: \"kubernetes.io/projected/6d3098da-c426-4ed1-8932-737fb644755c-kube-api-access-vcncx\") pod \"premium-w0-1afd34-kserve-85c998d845-r5zq9\" (UID: \"6d3098da-c426-4ed1-8932-737fb644755c\") " pod="e2e-models-e2e-worker-w0-1afd34/premium-w0-1afd34-kserve-85c998d845-r5zq9"
Sep 17 03:35:07.487078 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:35:07.487023    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d3098da-c426-4ed1-8932-737fb644755c-model-cache\") pod \"premium-w0-1afd34-kserve-85c998d845-r5zq9\" (UID: \"6d3098da-c426-4ed1-8932-737fb644755c\") " pod="e2e-models-e2e-worker-w0-1afd34/premium-w0-1afd34-kserve-85c998d845-r5zq9"
Sep 17 03:35:07.487078 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:35:07.487043    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d3098da-c426-4ed1-8932-737fb644755c-dshm\") pod \"premium-w0-1afd34-kserve-85c998d845-r5zq9\" (UID: \"6d3098da-c426-4ed1-8932-737fb644755c\") " pod="e2e-models-e2e-worker-w0-1afd34/premium-w0-1afd34-kserve-85c998d845-r5zq9"
Sep 17 03:35:07.487078 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:35:07.487072    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d3098da-c426-4ed1-8932-737fb644755c-tmp-dir\") pod \"premium-w0-1afd34-kserve-85c998d845-r5zq9\" (UID: \"6d3098da-c426-4ed1-8932-737fb644755c\") " pod="e2e-models-e2e-worker-w0-1afd34/premium-w0-1afd34-kserve-85c998d845-r5zq9"
Sep 17 03:35:07.587863 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:35:07.587831    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcncx\" (UniqueName: \"kubernetes.io/projected/6d3098da-c426-4ed1-8932-737fb644755c-kube-api-access-vcncx\") pod \"premium-w0-1afd34-kserve-85c998d845-r5zq9\" (UID: \"6d3098da-c426-4ed1-8932-737fb644755c\") " pod="e2e-models-e2e-worker-w0-1afd34/premium-w0-1afd34-kserve-85c998d845-r5zq9"
Sep 17 03:35:07.588026 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:35:07.587873    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d3098da-c426-4ed1-8932-737fb644755c-model-cache\") pod \"premium-w0-1afd34-kserve-85c998d845-r5zq9\" (UID: \"6d3098da-c426-4ed1-8932-737fb644755c\") " pod="e2e-models-e2e-worker-w0-1afd34/premium-w0-1afd34-kserve-85c998d845-r5zq9"
Sep 17 03:35:07.588026 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:35:07.587891    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d3098da-c426-4ed1-8932-737fb644755c-dshm\") pod \"premium-w0-1afd34-kserve-85c998d845-r5zq9\" (UID: \"6d3098da-c426-4ed1-8932-737fb644755c\") " pod="e2e-models-e2e-worker-w0-1afd34/premium-w0-1afd34-kserve-85c998d845-r5zq9"
Sep 17 03:35:07.588026 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:35:07.587913    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d3098da-c426-4ed1-8932-737fb644755c-tmp-dir\") pod \"premium-w0-1afd34-kserve-85c998d845-r5zq9\" (UID: \"6d3098da-c426-4ed1-8932-737fb644755c\") " pod="e2e-models-e2e-worker-w0-1afd34/premium-w0-1afd34-kserve-85c998d845-r5zq9"
Sep 17 03:35:07.588026 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:35:07.587963    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d3098da-c426-4ed1-8932-737fb644755c-tls-certs\") pod \"premium-w0-1afd34-kserve-85c998d845-r5zq9\" (UID: \"6d3098da-c426-4ed1-8932-737fb644755c\") " pod="e2e-models-e2e-worker-w0-1afd34/premium-w0-1afd34-kserve-85c998d845-r5zq9"
Sep 17 03:35:07.588026 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:35:07.588019    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d3098da-c426-4ed1-8932-737fb644755c-home\") pod \"premium-w0-1afd34-kserve-85c998d845-r5zq9\" (UID: \"6d3098da-c426-4ed1-8932-737fb644755c\") " pod="e2e-models-e2e-worker-w0-1afd34/premium-w0-1afd34-kserve-85c998d845-r5zq9"
Sep 17 03:35:07.588386 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:35:07.588363    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d3098da-c426-4ed1-8932-737fb644755c-model-cache\") pod \"premium-w0-1afd34-kserve-85c998d845-r5zq9\" (UID: \"6d3098da-c426-4ed1-8932-737fb644755c\") " pod="e2e-models-e2e-worker-w0-1afd34/premium-w0-1afd34-kserve-85c998d845-r5zq9"
Sep 17 03:35:07.588466 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:35:07.588378    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d3098da-c426-4ed1-8932-737fb644755c-tmp-dir\") pod \"premium-w0-1afd34-kserve-85c998d845-r5zq9\" (UID: \"6d3098da-c426-4ed1-8932-737fb644755c\") " pod="e2e-models-e2e-worker-w0-1afd34/premium-w0-1afd34-kserve-85c998d845-r5zq9"
Sep 17 03:35:07.588466 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:35:07.588434    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d3098da-c426-4ed1-8932-737fb644755c-home\") pod \"premium-w0-1afd34-kserve-85c998d845-r5zq9\" (UID: \"6d3098da-c426-4ed1-8932-737fb644755c\") " pod="e2e-models-e2e-worker-w0-1afd34/premium-w0-1afd34-kserve-85c998d845-r5zq9"
Sep 17 03:35:07.590350 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:35:07.590324    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d3098da-c426-4ed1-8932-737fb644755c-dshm\") pod \"premium-w0-1afd34-kserve-85c998d845-r5zq9\" (UID: \"6d3098da-c426-4ed1-8932-737fb644755c\") " pod="e2e-models-e2e-worker-w0-1afd34/premium-w0-1afd34-kserve-85c998d845-r5zq9"
Sep 17 03:35:07.590555 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:35:07.590537    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d3098da-c426-4ed1-8932-737fb644755c-tls-certs\") pod \"premium-w0-1afd34-kserve-85c998d845-r5zq9\" (UID: \"6d3098da-c426-4ed1-8932-737fb644755c\") " pod="e2e-models-e2e-worker-w0-1afd34/premium-w0-1afd34-kserve-85c998d845-r5zq9"
Sep 17 03:35:07.596638 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:35:07.596590    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcncx\" (UniqueName: \"kubernetes.io/projected/6d3098da-c426-4ed1-8932-737fb644755c-kube-api-access-vcncx\") pod \"premium-w0-1afd34-kserve-85c998d845-r5zq9\" (UID: \"6d3098da-c426-4ed1-8932-737fb644755c\") " pod="e2e-models-e2e-worker-w0-1afd34/premium-w0-1afd34-kserve-85c998d845-r5zq9"
Sep 17 03:35:07.698508 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:35:07.698430    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1afd34/premium-w0-1afd34-kserve-85c998d845-r5zq9"
Sep 17 03:35:07.971121 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:35:07.971099    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-1afd34/premium-w0-1afd34-kserve-85c998d845-r5zq9"]
Sep 17 03:35:07.974308 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:35:07.974269    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6d3098da_c426_4ed1_8932_737fb644755c.slice/crio-76ee1f0fd34b3962c74eab7838f74f7619c5878e23eda1e6d200addd6b47f80f WatchSource:0}: Error finding container 76ee1f0fd34b3962c74eab7838f74f7619c5878e23eda1e6d200addd6b47f80f: Status 404 returned error can't find the container with id 76ee1f0fd34b3962c74eab7838f74f7619c5878e23eda1e6d200addd6b47f80f
Sep 17 03:35:08.820137 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:35:08.820100    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1afd34/premium-w0-1afd34-kserve-85c998d845-r5zq9" event={"ID":"6d3098da-c426-4ed1-8932-737fb644755c","Type":"ContainerStarted","Data":"2ca1069b5dcba8a8513036fa09f56698a813a07d12d8932fe80da3941b2489f2"}
Sep 17 03:35:08.820507 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:35:08.820146    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1afd34/premium-w0-1afd34-kserve-85c998d845-r5zq9" event={"ID":"6d3098da-c426-4ed1-8932-737fb644755c","Type":"ContainerStarted","Data":"76ee1f0fd34b3962c74eab7838f74f7619c5878e23eda1e6d200addd6b47f80f"}
Sep 17 03:35:08.837492 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:35:08.837447    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-1afd34/premium-w0-1afd34-kserve-85c998d845-r5zq9" podStartSLOduration=1.837434013 podStartE2EDuration="1.837434013s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 03:35:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:35:08.835934799 +0000 UTC m=+944.302342321" watchObservedRunningTime="2026-09-17 03:35:08.837434013 +0000 UTC m=+944.303841538"
Sep 17 03:35:17.698557 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:35:17.698517    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-1afd34/premium-w0-1afd34-kserve-85c998d845-r5zq9"
Sep 17 03:35:17.699031 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:35:17.698574    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1afd34/premium-w0-1afd34-kserve-85c998d845-r5zq9"
Sep 17 03:35:17.710973 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:35:17.710949    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-1afd34/premium-w0-1afd34-kserve-85c998d845-r5zq9"
Sep 17 03:35:17.878698 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:35:17.878675    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1afd34/premium-w0-1afd34-kserve-85c998d845-r5zq9"
Sep 17 03:35:22.481996 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:35:22.481964    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-3d0c3c/test-model-3d0c3c-kserve-56f86bc989-bhcdt"]
Sep 17 03:35:22.485490 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:35:22.485474    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-3d0c3c/test-model-3d0c3c-kserve-56f86bc989-bhcdt"
Sep 17 03:35:22.488373 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:35:22.488355    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-3d0c3c\"/\"default-dockercfg-xqwlz\""
Sep 17 03:35:22.488815 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:35:22.488796    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-3d0c3c\"/\"openshift-service-ca.crt\""
Sep 17 03:35:22.489524 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:35:22.489505    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-3d0c3c\"/\"kube-root-ca.crt\""
Sep 17 03:35:22.489667 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:35:22.489648    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-3d0c3c\"/\"test-model-3d0c3c-kserve-self-signed-certs\""
Sep 17 03:35:22.496183 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:35:22.496160    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-3d0c3c/test-model-3d0c3c-kserve-56f86bc989-bhcdt"]
Sep 17 03:35:22.628145 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:35:22.628115    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b5f7f2e-b36d-46c4-9f37-8aae4d8722b3-model-cache\") pod \"test-model-3d0c3c-kserve-56f86bc989-bhcdt\" (UID: \"3b5f7f2e-b36d-46c4-9f37-8aae4d8722b3\") " pod="ai-tenant-e2e-inf-b-3d0c3c/test-model-3d0c3c-kserve-56f86bc989-bhcdt"
Sep 17 03:35:22.628302 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:35:22.628147    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxf7t\" (UniqueName: \"kubernetes.io/projected/3b5f7f2e-b36d-46c4-9f37-8aae4d8722b3-kube-api-access-mxf7t\") pod \"test-model-3d0c3c-kserve-56f86bc989-bhcdt\" (UID: \"3b5f7f2e-b36d-46c4-9f37-8aae4d8722b3\") " pod="ai-tenant-e2e-inf-b-3d0c3c/test-model-3d0c3c-kserve-56f86bc989-bhcdt"
Sep 17 03:35:22.628302 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:35:22.628188    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b5f7f2e-b36d-46c4-9f37-8aae4d8722b3-home\") pod \"test-model-3d0c3c-kserve-56f86bc989-bhcdt\" (UID: \"3b5f7f2e-b36d-46c4-9f37-8aae4d8722b3\") " pod="ai-tenant-e2e-inf-b-3d0c3c/test-model-3d0c3c-kserve-56f86bc989-bhcdt"
Sep 17 03:35:22.628302 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:35:22.628230    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b5f7f2e-b36d-46c4-9f37-8aae4d8722b3-tmp-dir\") pod \"test-model-3d0c3c-kserve-56f86bc989-bhcdt\" (UID: \"3b5f7f2e-b36d-46c4-9f37-8aae4d8722b3\") " pod="ai-tenant-e2e-inf-b-3d0c3c/test-model-3d0c3c-kserve-56f86bc989-bhcdt"
Sep 17 03:35:22.628464 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:35:22.628323    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b5f7f2e-b36d-46c4-9f37-8aae4d8722b3-dshm\") pod \"test-model-3d0c3c-kserve-56f86bc989-bhcdt\" (UID: \"3b5f7f2e-b36d-46c4-9f37-8aae4d8722b3\") " pod="ai-tenant-e2e-inf-b-3d0c3c/test-model-3d0c3c-kserve-56f86bc989-bhcdt"
Sep 17 03:35:22.628464 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:35:22.628364    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b5f7f2e-b36d-46c4-9f37-8aae4d8722b3-tls-certs\") pod \"test-model-3d0c3c-kserve-56f86bc989-bhcdt\" (UID: \"3b5f7f2e-b36d-46c4-9f37-8aae4d8722b3\") " pod="ai-tenant-e2e-inf-b-3d0c3c/test-model-3d0c3c-kserve-56f86bc989-bhcdt"
Sep 17 03:35:22.729780 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:35:22.729746    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b5f7f2e-b36d-46c4-9f37-8aae4d8722b3-dshm\") pod \"test-model-3d0c3c-kserve-56f86bc989-bhcdt\" (UID: \"3b5f7f2e-b36d-46c4-9f37-8aae4d8722b3\") " pod="ai-tenant-e2e-inf-b-3d0c3c/test-model-3d0c3c-kserve-56f86bc989-bhcdt"
Sep 17 03:35:22.729780 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:35:22.729784    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b5f7f2e-b36d-46c4-9f37-8aae4d8722b3-tls-certs\") pod \"test-model-3d0c3c-kserve-56f86bc989-bhcdt\" (UID: \"3b5f7f2e-b36d-46c4-9f37-8aae4d8722b3\") " pod="ai-tenant-e2e-inf-b-3d0c3c/test-model-3d0c3c-kserve-56f86bc989-bhcdt"
Sep 17 03:35:22.730017 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:35:22.729920    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b5f7f2e-b36d-46c4-9f37-8aae4d8722b3-model-cache\") pod \"test-model-3d0c3c-kserve-56f86bc989-bhcdt\" (UID: \"3b5f7f2e-b36d-46c4-9f37-8aae4d8722b3\") " pod="ai-tenant-e2e-inf-b-3d0c3c/test-model-3d0c3c-kserve-56f86bc989-bhcdt"
Sep 17 03:35:22.730017 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:35:22.729949    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxf7t\" (UniqueName: \"kubernetes.io/projected/3b5f7f2e-b36d-46c4-9f37-8aae4d8722b3-kube-api-access-mxf7t\") pod \"test-model-3d0c3c-kserve-56f86bc989-bhcdt\" (UID: \"3b5f7f2e-b36d-46c4-9f37-8aae4d8722b3\") " pod="ai-tenant-e2e-inf-b-3d0c3c/test-model-3d0c3c-kserve-56f86bc989-bhcdt"
Sep 17 03:35:22.730017 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:35:22.729985    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b5f7f2e-b36d-46c4-9f37-8aae4d8722b3-home\") pod \"test-model-3d0c3c-kserve-56f86bc989-bhcdt\" (UID: \"3b5f7f2e-b36d-46c4-9f37-8aae4d8722b3\") " pod="ai-tenant-e2e-inf-b-3d0c3c/test-model-3d0c3c-kserve-56f86bc989-bhcdt"
Sep 17 03:35:22.730187 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:35:22.730015    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b5f7f2e-b36d-46c4-9f37-8aae4d8722b3-tmp-dir\") pod \"test-model-3d0c3c-kserve-56f86bc989-bhcdt\" (UID: \"3b5f7f2e-b36d-46c4-9f37-8aae4d8722b3\") " pod="ai-tenant-e2e-inf-b-3d0c3c/test-model-3d0c3c-kserve-56f86bc989-bhcdt"
Sep 17 03:35:22.730297 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:35:22.730270    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b5f7f2e-b36d-46c4-9f37-8aae4d8722b3-model-cache\") pod \"test-model-3d0c3c-kserve-56f86bc989-bhcdt\" (UID: \"3b5f7f2e-b36d-46c4-9f37-8aae4d8722b3\") " pod="ai-tenant-e2e-inf-b-3d0c3c/test-model-3d0c3c-kserve-56f86bc989-bhcdt"
Sep 17 03:35:22.730350 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:35:22.730294    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b5f7f2e-b36d-46c4-9f37-8aae4d8722b3-home\") pod \"test-model-3d0c3c-kserve-56f86bc989-bhcdt\" (UID: \"3b5f7f2e-b36d-46c4-9f37-8aae4d8722b3\") " pod="ai-tenant-e2e-inf-b-3d0c3c/test-model-3d0c3c-kserve-56f86bc989-bhcdt"
Sep 17 03:35:22.730350 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:35:22.730324    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b5f7f2e-b36d-46c4-9f37-8aae4d8722b3-tmp-dir\") pod \"test-model-3d0c3c-kserve-56f86bc989-bhcdt\" (UID: \"3b5f7f2e-b36d-46c4-9f37-8aae4d8722b3\") " pod="ai-tenant-e2e-inf-b-3d0c3c/test-model-3d0c3c-kserve-56f86bc989-bhcdt"
Sep 17 03:35:22.732075 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:35:22.732020    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b5f7f2e-b36d-46c4-9f37-8aae4d8722b3-dshm\") pod \"test-model-3d0c3c-kserve-56f86bc989-bhcdt\" (UID: \"3b5f7f2e-b36d-46c4-9f37-8aae4d8722b3\") " pod="ai-tenant-e2e-inf-b-3d0c3c/test-model-3d0c3c-kserve-56f86bc989-bhcdt"
Sep 17 03:35:22.732358 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:35:22.732342    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b5f7f2e-b36d-46c4-9f37-8aae4d8722b3-tls-certs\") pod \"test-model-3d0c3c-kserve-56f86bc989-bhcdt\" (UID: \"3b5f7f2e-b36d-46c4-9f37-8aae4d8722b3\") " pod="ai-tenant-e2e-inf-b-3d0c3c/test-model-3d0c3c-kserve-56f86bc989-bhcdt"
Sep 17 03:35:22.739670 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:35:22.739647    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxf7t\" (UniqueName: \"kubernetes.io/projected/3b5f7f2e-b36d-46c4-9f37-8aae4d8722b3-kube-api-access-mxf7t\") pod \"test-model-3d0c3c-kserve-56f86bc989-bhcdt\" (UID: \"3b5f7f2e-b36d-46c4-9f37-8aae4d8722b3\") " pod="ai-tenant-e2e-inf-b-3d0c3c/test-model-3d0c3c-kserve-56f86bc989-bhcdt"
Sep 17 03:35:22.796301 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:35:22.796278    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-3d0c3c/test-model-3d0c3c-kserve-56f86bc989-bhcdt"
Sep 17 03:35:23.070418 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:35:23.070394    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-3d0c3c/test-model-3d0c3c-kserve-56f86bc989-bhcdt"]
Sep 17 03:35:23.072122 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:35:23.072101    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3b5f7f2e_b36d_46c4_9f37_8aae4d8722b3.slice/crio-0777a6eab0240501d7309a804f2f69ece9455a35e47bda723dd0c2dee06c2f2d WatchSource:0}: Error finding container 0777a6eab0240501d7309a804f2f69ece9455a35e47bda723dd0c2dee06c2f2d: Status 404 returned error can't find the container with id 0777a6eab0240501d7309a804f2f69ece9455a35e47bda723dd0c2dee06c2f2d
Sep 17 03:35:23.905804 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:35:23.905766    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-3d0c3c/test-model-3d0c3c-kserve-56f86bc989-bhcdt" event={"ID":"3b5f7f2e-b36d-46c4-9f37-8aae4d8722b3","Type":"ContainerStarted","Data":"3f6340e972d307af61e65e4ce5599b7b506856c9b4db8a2b505e6a77c0a497b4"}
Sep 17 03:35:23.905804 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:35:23.905808    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-3d0c3c/test-model-3d0c3c-kserve-56f86bc989-bhcdt" event={"ID":"3b5f7f2e-b36d-46c4-9f37-8aae4d8722b3","Type":"ContainerStarted","Data":"0777a6eab0240501d7309a804f2f69ece9455a35e47bda723dd0c2dee06c2f2d"}
Sep 17 03:35:23.927859 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:35:23.927813    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-3d0c3c/test-model-3d0c3c-kserve-56f86bc989-bhcdt" podStartSLOduration=1.927798005 podStartE2EDuration="1.927798005s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 03:35:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:35:23.925653301 +0000 UTC m=+959.392060826" watchObservedRunningTime="2026-09-17 03:35:23.927798005 +0000 UTC m=+959.394205528"
Sep 17 03:35:32.796495 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:35:32.796458    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-3d0c3c/test-model-3d0c3c-kserve-56f86bc989-bhcdt"
Sep 17 03:35:32.796906 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:35:32.796507    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-3d0c3c/test-model-3d0c3c-kserve-56f86bc989-bhcdt"
Sep 17 03:35:32.809128 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:35:32.809104    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-3d0c3c/test-model-3d0c3c-kserve-56f86bc989-bhcdt"
Sep 17 03:35:32.959611 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:35:32.959573    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-3d0c3c/test-model-3d0c3c-kserve-56f86bc989-bhcdt"
Sep 17 03:35:36.463750 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:35:36.463718    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-5062bf/explicit-model-5062bf-kserve-58678f7547-qhwp7"]
Sep 17 03:35:36.468258 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:35:36.468242    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-5062bf/explicit-model-5062bf-kserve-58678f7547-qhwp7"
Sep 17 03:35:36.472238 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:35:36.472205    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-5062bf\"/\"openshift-service-ca.crt\""
Sep 17 03:35:36.472238 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:35:36.472235    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-5062bf\"/\"kube-root-ca.crt\""
Sep 17 03:35:36.472419 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:35:36.472214    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-5062bf\"/\"default-dockercfg-smlww\""
Sep 17 03:35:36.472419 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:35:36.472247    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-5062bf\"/\"explicit-model-5062bf-kserve-self-signed-certs\""
Sep 17 03:35:36.477557 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:35:36.477533    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-5062bf/explicit-model-5062bf-kserve-58678f7547-qhwp7"]
Sep 17 03:35:36.556995 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:35:36.556957    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdwgb\" (UniqueName: \"kubernetes.io/projected/d9628dd8-0bff-47e7-84aa-1b072b47c4a8-kube-api-access-cdwgb\") pod \"explicit-model-5062bf-kserve-58678f7547-qhwp7\" (UID: \"d9628dd8-0bff-47e7-84aa-1b072b47c4a8\") " pod="ai-tenant-e2e-explicit-5062bf/explicit-model-5062bf-kserve-58678f7547-qhwp7"
Sep 17 03:35:36.557178 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:35:36.557014    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9628dd8-0bff-47e7-84aa-1b072b47c4a8-tls-certs\") pod \"explicit-model-5062bf-kserve-58678f7547-qhwp7\" (UID: \"d9628dd8-0bff-47e7-84aa-1b072b47c4a8\") " pod="ai-tenant-e2e-explicit-5062bf/explicit-model-5062bf-kserve-58678f7547-qhwp7"
Sep 17 03:35:36.557178 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:35:36.557096    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9628dd8-0bff-47e7-84aa-1b072b47c4a8-tmp-dir\") pod \"explicit-model-5062bf-kserve-58678f7547-qhwp7\" (UID: \"d9628dd8-0bff-47e7-84aa-1b072b47c4a8\") " pod="ai-tenant-e2e-explicit-5062bf/explicit-model-5062bf-kserve-58678f7547-qhwp7"
Sep 17 03:35:36.557178 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:35:36.557132    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9628dd8-0bff-47e7-84aa-1b072b47c4a8-model-cache\") pod \"explicit-model-5062bf-kserve-58678f7547-qhwp7\" (UID: \"d9628dd8-0bff-47e7-84aa-1b072b47c4a8\") " pod="ai-tenant-e2e-explicit-5062bf/explicit-model-5062bf-kserve-58678f7547-qhwp7"
Sep 17 03:35:36.557178 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:35:36.557152    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9628dd8-0bff-47e7-84aa-1b072b47c4a8-home\") pod \"explicit-model-5062bf-kserve-58678f7547-qhwp7\" (UID: \"d9628dd8-0bff-47e7-84aa-1b072b47c4a8\") " pod="ai-tenant-e2e-explicit-5062bf/explicit-model-5062bf-kserve-58678f7547-qhwp7"
Sep 17 03:35:36.557367 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:35:36.557248    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9628dd8-0bff-47e7-84aa-1b072b47c4a8-dshm\") pod \"explicit-model-5062bf-kserve-58678f7547-qhwp7\" (UID: \"d9628dd8-0bff-47e7-84aa-1b072b47c4a8\") " pod="ai-tenant-e2e-explicit-5062bf/explicit-model-5062bf-kserve-58678f7547-qhwp7"
Sep 17 03:35:36.658535 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:35:36.658497    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9628dd8-0bff-47e7-84aa-1b072b47c4a8-tmp-dir\") pod \"explicit-model-5062bf-kserve-58678f7547-qhwp7\" (UID: \"d9628dd8-0bff-47e7-84aa-1b072b47c4a8\") " pod="ai-tenant-e2e-explicit-5062bf/explicit-model-5062bf-kserve-58678f7547-qhwp7"
Sep 17 03:35:36.658700 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:35:36.658646    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9628dd8-0bff-47e7-84aa-1b072b47c4a8-model-cache\") pod \"explicit-model-5062bf-kserve-58678f7547-qhwp7\" (UID: \"d9628dd8-0bff-47e7-84aa-1b072b47c4a8\") " pod="ai-tenant-e2e-explicit-5062bf/explicit-model-5062bf-kserve-58678f7547-qhwp7"
Sep 17 03:35:36.658757 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:35:36.658701    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9628dd8-0bff-47e7-84aa-1b072b47c4a8-home\") pod \"explicit-model-5062bf-kserve-58678f7547-qhwp7\" (UID: \"d9628dd8-0bff-47e7-84aa-1b072b47c4a8\") " pod="ai-tenant-e2e-explicit-5062bf/explicit-model-5062bf-kserve-58678f7547-qhwp7"
Sep 17 03:35:36.658814 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:35:36.658796    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9628dd8-0bff-47e7-84aa-1b072b47c4a8-dshm\") pod \"explicit-model-5062bf-kserve-58678f7547-qhwp7\" (UID: \"d9628dd8-0bff-47e7-84aa-1b072b47c4a8\") " pod="ai-tenant-e2e-explicit-5062bf/explicit-model-5062bf-kserve-58678f7547-qhwp7"
Sep 17 03:35:36.658908 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:35:36.658890    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdwgb\" (UniqueName: \"kubernetes.io/projected/d9628dd8-0bff-47e7-84aa-1b072b47c4a8-kube-api-access-cdwgb\") pod \"explicit-model-5062bf-kserve-58678f7547-qhwp7\" (UID: \"d9628dd8-0bff-47e7-84aa-1b072b47c4a8\") " pod="ai-tenant-e2e-explicit-5062bf/explicit-model-5062bf-kserve-58678f7547-qhwp7"
Sep 17 03:35:36.658969 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:35:36.658927    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9628dd8-0bff-47e7-84aa-1b072b47c4a8-tmp-dir\") pod \"explicit-model-5062bf-kserve-58678f7547-qhwp7\" (UID: \"d9628dd8-0bff-47e7-84aa-1b072b47c4a8\") " pod="ai-tenant-e2e-explicit-5062bf/explicit-model-5062bf-kserve-58678f7547-qhwp7"
Sep 17 03:35:36.658969 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:35:36.658956    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9628dd8-0bff-47e7-84aa-1b072b47c4a8-tls-certs\") pod \"explicit-model-5062bf-kserve-58678f7547-qhwp7\" (UID: \"d9628dd8-0bff-47e7-84aa-1b072b47c4a8\") " pod="ai-tenant-e2e-explicit-5062bf/explicit-model-5062bf-kserve-58678f7547-qhwp7"
Sep 17 03:35:36.659063 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:35:36.659033    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9628dd8-0bff-47e7-84aa-1b072b47c4a8-model-cache\") pod \"explicit-model-5062bf-kserve-58678f7547-qhwp7\" (UID: \"d9628dd8-0bff-47e7-84aa-1b072b47c4a8\") " pod="ai-tenant-e2e-explicit-5062bf/explicit-model-5062bf-kserve-58678f7547-qhwp7"
Sep 17 03:35:36.659115 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:35:36.659101    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9628dd8-0bff-47e7-84aa-1b072b47c4a8-home\") pod \"explicit-model-5062bf-kserve-58678f7547-qhwp7\" (UID: \"d9628dd8-0bff-47e7-84aa-1b072b47c4a8\") " pod="ai-tenant-e2e-explicit-5062bf/explicit-model-5062bf-kserve-58678f7547-qhwp7"
Sep 17 03:35:36.661090 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:35:36.661062    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9628dd8-0bff-47e7-84aa-1b072b47c4a8-dshm\") pod \"explicit-model-5062bf-kserve-58678f7547-qhwp7\" (UID: \"d9628dd8-0bff-47e7-84aa-1b072b47c4a8\") " pod="ai-tenant-e2e-explicit-5062bf/explicit-model-5062bf-kserve-58678f7547-qhwp7"
Sep 17 03:35:36.661342 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:35:36.661325    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9628dd8-0bff-47e7-84aa-1b072b47c4a8-tls-certs\") pod \"explicit-model-5062bf-kserve-58678f7547-qhwp7\" (UID: \"d9628dd8-0bff-47e7-84aa-1b072b47c4a8\") " pod="ai-tenant-e2e-explicit-5062bf/explicit-model-5062bf-kserve-58678f7547-qhwp7"
Sep 17 03:35:36.668258 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:35:36.668238    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdwgb\" (UniqueName: \"kubernetes.io/projected/d9628dd8-0bff-47e7-84aa-1b072b47c4a8-kube-api-access-cdwgb\") pod \"explicit-model-5062bf-kserve-58678f7547-qhwp7\" (UID: \"d9628dd8-0bff-47e7-84aa-1b072b47c4a8\") " pod="ai-tenant-e2e-explicit-5062bf/explicit-model-5062bf-kserve-58678f7547-qhwp7"
Sep 17 03:35:36.778620 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:35:36.778566    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-5062bf/explicit-model-5062bf-kserve-58678f7547-qhwp7"
Sep 17 03:35:37.054210 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:35:37.054184    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-5062bf/explicit-model-5062bf-kserve-58678f7547-qhwp7"]
Sep 17 03:35:37.056642 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:35:37.056588    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd9628dd8_0bff_47e7_84aa_1b072b47c4a8.slice/crio-6747469a0cb64ab6023a035a036c74b5a994965c03f771234b150ae671cb97a5 WatchSource:0}: Error finding container 6747469a0cb64ab6023a035a036c74b5a994965c03f771234b150ae671cb97a5: Status 404 returned error can't find the container with id 6747469a0cb64ab6023a035a036c74b5a994965c03f771234b150ae671cb97a5
Sep 17 03:35:37.975893 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:35:37.975857    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-5062bf/explicit-model-5062bf-kserve-58678f7547-qhwp7" event={"ID":"d9628dd8-0bff-47e7-84aa-1b072b47c4a8","Type":"ContainerStarted","Data":"a9840841f63c5a6de51200466d40523f858d2f5b06239b946dceb75636701b0f"}
Sep 17 03:35:37.975893 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:35:37.975899    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-5062bf/explicit-model-5062bf-kserve-58678f7547-qhwp7" event={"ID":"d9628dd8-0bff-47e7-84aa-1b072b47c4a8","Type":"ContainerStarted","Data":"6747469a0cb64ab6023a035a036c74b5a994965c03f771234b150ae671cb97a5"}
Sep 17 03:35:37.994313 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:35:37.994267    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-5062bf/explicit-model-5062bf-kserve-58678f7547-qhwp7" podStartSLOduration=1.994253185 podStartE2EDuration="1.994253185s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 03:35:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:35:37.992320247 +0000 UTC m=+973.458727785" watchObservedRunningTime="2026-09-17 03:35:37.994253185 +0000 UTC m=+973.460660712"
Sep 17 03:35:46.779587 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:35:46.779545    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-explicit-5062bf/explicit-model-5062bf-kserve-58678f7547-qhwp7"
Sep 17 03:35:46.780008 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:35:46.779591    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-5062bf/explicit-model-5062bf-kserve-58678f7547-qhwp7"
Sep 17 03:35:46.791940 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:35:46.791910    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-explicit-5062bf/explicit-model-5062bf-kserve-58678f7547-qhwp7"
Sep 17 03:35:47.034138 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:35:47.034057    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-explicit-5062bf/explicit-model-5062bf-kserve-58678f7547-qhwp7"
Sep 17 03:36:08.686572 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:36:08.686536    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-5062bf-xd7rn"]
Sep 17 03:36:08.690220 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:36:08.690199    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-5062bf-xd7rn"
Sep 17 03:36:08.693425 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:36:08.693403    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8vwnw\""
Sep 17 03:36:08.701348 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:36:08.701325    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-5062bf-xd7rn"]
Sep 17 03:36:08.777866 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:36:08.777833    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/59867e92-c252-4d1d-b927-ef43a70f3646-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-5062bf-xd7rn\" (UID: \"59867e92-c252-4d1d-b927-ef43a70f3646\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-5062bf-xd7rn"
Sep 17 03:36:08.878406 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:36:08.878374    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/59867e92-c252-4d1d-b927-ef43a70f3646-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-5062bf-xd7rn\" (UID: \"59867e92-c252-4d1d-b927-ef43a70f3646\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-5062bf-xd7rn"
Sep 17 03:36:08.879022 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:36:08.879004    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/59867e92-c252-4d1d-b927-ef43a70f3646-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-5062bf-xd7rn\" (UID: \"59867e92-c252-4d1d-b927-ef43a70f3646\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-5062bf-xd7rn"
Sep 17 03:36:09.003568 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:36:09.003545    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-5062bf-xd7rn"
Sep 17 03:36:09.281150 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:36:09.281122    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-5062bf-xd7rn"]
Sep 17 03:36:09.282946 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:36:09.282923    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59867e92_c252_4d1d_b927_ef43a70f3646.slice/crio-a3a4f7b4c3008379873558a0ad013a38b75c8f639b9c9e1906a79ba1d7aa3dbd WatchSource:0}: Error finding container a3a4f7b4c3008379873558a0ad013a38b75c8f639b9c9e1906a79ba1d7aa3dbd: Status 404 returned error can't find the container with id a3a4f7b4c3008379873558a0ad013a38b75c8f639b9c9e1906a79ba1d7aa3dbd
Sep 17 03:36:10.147039 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:36:10.146939    2552 generic.go:356] "Generic (PLEG): container finished" podID="59867e92-c252-4d1d-b927-ef43a70f3646" containerID="ea33a6266aa0d4ee0a29121eca11dd48aee436025c713d17d70290f6db2dd1e2" exitCode=0
Sep 17 03:36:10.147039 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:36:10.146986    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-5062bf-xd7rn" event={"ID":"59867e92-c252-4d1d-b927-ef43a70f3646","Type":"ContainerDied","Data":"ea33a6266aa0d4ee0a29121eca11dd48aee436025c713d17d70290f6db2dd1e2"}
Sep 17 03:36:10.147492 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:36:10.147040    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-5062bf-xd7rn" event={"ID":"59867e92-c252-4d1d-b927-ef43a70f3646","Type":"ContainerStarted","Data":"a3a4f7b4c3008379873558a0ad013a38b75c8f639b9c9e1906a79ba1d7aa3dbd"}
Sep 17 03:36:11.285719 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:36:11.285694    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-5062bf-xd7rn"
Sep 17 03:36:11.301467 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:36:11.301449    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/59867e92-c252-4d1d-b927-ef43a70f3646-maas-api-service-ca\") pod \"59867e92-c252-4d1d-b927-ef43a70f3646\" (UID: \"59867e92-c252-4d1d-b927-ef43a70f3646\") "
Sep 17 03:36:11.301775 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:36:11.301756    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59867e92-c252-4d1d-b927-ef43a70f3646-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "59867e92-c252-4d1d-b927-ef43a70f3646" (UID: "59867e92-c252-4d1d-b927-ef43a70f3646"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 03:36:11.402492 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:36:11.402403    2552 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/59867e92-c252-4d1d-b927-ef43a70f3646-maas-api-service-ca\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:36:12.158427 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:36:12.158391    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-5062bf-xd7rn"
Sep 17 03:36:12.158636 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:36:12.158391    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-5062bf-xd7rn" event={"ID":"59867e92-c252-4d1d-b927-ef43a70f3646","Type":"ContainerDied","Data":"a3a4f7b4c3008379873558a0ad013a38b75c8f639b9c9e1906a79ba1d7aa3dbd"}
Sep 17 03:36:12.158636 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:36:12.158496    2552 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a3a4f7b4c3008379873558a0ad013a38b75c8f639b9c9e1906a79ba1d7aa3dbd"
Sep 17 03:36:13.284998 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:36:13.284967    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-cdaa7ee2-86d4b79b6d-w7t2l"]
Sep 17 03:36:13.285429 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:36:13.285415    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59867e92-c252-4d1d-b927-ef43a70f3646" containerName="revoke-keys"
Sep 17 03:36:13.285490 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:36:13.285431    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="59867e92-c252-4d1d-b927-ef43a70f3646" containerName="revoke-keys"
Sep 17 03:36:13.285533 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:36:13.285511    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="59867e92-c252-4d1d-b927-ef43a70f3646" containerName="revoke-keys"
Sep 17 03:36:13.288702 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:36:13.288683    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cdaa7ee2-86d4b79b6d-w7t2l"
Sep 17 03:36:13.291954 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:36:13.291924    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-cdaa7ee2-serving-cert\""
Sep 17 03:36:13.311002 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:36:13.310981    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-cdaa7ee2-86d4b79b6d-w7t2l"]
Sep 17 03:36:13.323918 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:36:13.323898    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/58e03c55-ec75-4d27-a39a-c9f8697c96dc-metrics-certs\") pod \"maas-api-e2e-mt-cdaa7ee2-86d4b79b6d-w7t2l\" (UID: \"58e03c55-ec75-4d27-a39a-c9f8697c96dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cdaa7ee2-86d4b79b6d-w7t2l"
Sep 17 03:36:13.324054 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:36:13.323933    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kn57z\" (UniqueName: \"kubernetes.io/projected/58e03c55-ec75-4d27-a39a-c9f8697c96dc-kube-api-access-kn57z\") pod \"maas-api-e2e-mt-cdaa7ee2-86d4b79b6d-w7t2l\" (UID: \"58e03c55-ec75-4d27-a39a-c9f8697c96dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cdaa7ee2-86d4b79b6d-w7t2l"
Sep 17 03:36:13.324054 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:36:13.323973    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58e03c55-ec75-4d27-a39a-c9f8697c96dc-maas-api-tls\") pod \"maas-api-e2e-mt-cdaa7ee2-86d4b79b6d-w7t2l\" (UID: \"58e03c55-ec75-4d27-a39a-c9f8697c96dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cdaa7ee2-86d4b79b6d-w7t2l"
Sep 17 03:36:13.324054 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:36:13.324017    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/58e03c55-ec75-4d27-a39a-c9f8697c96dc-openshift-service-ca\") pod \"maas-api-e2e-mt-cdaa7ee2-86d4b79b6d-w7t2l\" (UID: \"58e03c55-ec75-4d27-a39a-c9f8697c96dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cdaa7ee2-86d4b79b6d-w7t2l"
Sep 17 03:36:13.425094 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:36:13.425058    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/58e03c55-ec75-4d27-a39a-c9f8697c96dc-openshift-service-ca\") pod \"maas-api-e2e-mt-cdaa7ee2-86d4b79b6d-w7t2l\" (UID: \"58e03c55-ec75-4d27-a39a-c9f8697c96dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cdaa7ee2-86d4b79b6d-w7t2l"
Sep 17 03:36:13.425266 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:36:13.425163    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/58e03c55-ec75-4d27-a39a-c9f8697c96dc-metrics-certs\") pod \"maas-api-e2e-mt-cdaa7ee2-86d4b79b6d-w7t2l\" (UID: \"58e03c55-ec75-4d27-a39a-c9f8697c96dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cdaa7ee2-86d4b79b6d-w7t2l"
Sep 17 03:36:13.425266 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:36:13.425191    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kn57z\" (UniqueName: \"kubernetes.io/projected/58e03c55-ec75-4d27-a39a-c9f8697c96dc-kube-api-access-kn57z\") pod \"maas-api-e2e-mt-cdaa7ee2-86d4b79b6d-w7t2l\" (UID: \"58e03c55-ec75-4d27-a39a-c9f8697c96dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cdaa7ee2-86d4b79b6d-w7t2l"
Sep 17 03:36:13.425266 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:36:13.425230    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58e03c55-ec75-4d27-a39a-c9f8697c96dc-maas-api-tls\") pod \"maas-api-e2e-mt-cdaa7ee2-86d4b79b6d-w7t2l\" (UID: \"58e03c55-ec75-4d27-a39a-c9f8697c96dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cdaa7ee2-86d4b79b6d-w7t2l"
Sep 17 03:36:13.425384 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:36:13.425346    2552 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-cdaa7ee2-serving-cert: secret "maas-api-e2e-mt-cdaa7ee2-serving-cert" not found
Sep 17 03:36:13.425424 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:36:13.425414    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/58e03c55-ec75-4d27-a39a-c9f8697c96dc-maas-api-tls podName:58e03c55-ec75-4d27-a39a-c9f8697c96dc nodeName:}" failed. No retries permitted until 2026-09-17 03:36:13.925392096 +0000 UTC m=+1009.391799601 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/58e03c55-ec75-4d27-a39a-c9f8697c96dc-maas-api-tls") pod "maas-api-e2e-mt-cdaa7ee2-86d4b79b6d-w7t2l" (UID: "58e03c55-ec75-4d27-a39a-c9f8697c96dc") : secret "maas-api-e2e-mt-cdaa7ee2-serving-cert" not found
Sep 17 03:36:13.425768 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:36:13.425746    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/58e03c55-ec75-4d27-a39a-c9f8697c96dc-openshift-service-ca\") pod \"maas-api-e2e-mt-cdaa7ee2-86d4b79b6d-w7t2l\" (UID: \"58e03c55-ec75-4d27-a39a-c9f8697c96dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cdaa7ee2-86d4b79b6d-w7t2l"
Sep 17 03:36:13.427592 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:36:13.427575    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/58e03c55-ec75-4d27-a39a-c9f8697c96dc-metrics-certs\") pod \"maas-api-e2e-mt-cdaa7ee2-86d4b79b6d-w7t2l\" (UID: \"58e03c55-ec75-4d27-a39a-c9f8697c96dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cdaa7ee2-86d4b79b6d-w7t2l"
Sep 17 03:36:13.436973 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:36:13.436951    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kn57z\" (UniqueName: \"kubernetes.io/projected/58e03c55-ec75-4d27-a39a-c9f8697c96dc-kube-api-access-kn57z\") pod \"maas-api-e2e-mt-cdaa7ee2-86d4b79b6d-w7t2l\" (UID: \"58e03c55-ec75-4d27-a39a-c9f8697c96dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cdaa7ee2-86d4b79b6d-w7t2l"
Sep 17 03:36:13.930545 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:36:13.930509    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58e03c55-ec75-4d27-a39a-c9f8697c96dc-maas-api-tls\") pod \"maas-api-e2e-mt-cdaa7ee2-86d4b79b6d-w7t2l\" (UID: \"58e03c55-ec75-4d27-a39a-c9f8697c96dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cdaa7ee2-86d4b79b6d-w7t2l"
Sep 17 03:36:13.932979 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:36:13.932956    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58e03c55-ec75-4d27-a39a-c9f8697c96dc-maas-api-tls\") pod \"maas-api-e2e-mt-cdaa7ee2-86d4b79b6d-w7t2l\" (UID: \"58e03c55-ec75-4d27-a39a-c9f8697c96dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cdaa7ee2-86d4b79b6d-w7t2l"
Sep 17 03:36:14.199369 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:36:14.199275    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cdaa7ee2-86d4b79b6d-w7t2l"
Sep 17 03:36:14.482308 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:36:14.482283    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-cdaa7ee2-86d4b79b6d-w7t2l"]
Sep 17 03:36:14.484319 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:36:14.484286    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58e03c55_ec75_4d27_a39a_c9f8697c96dc.slice/crio-7ecbc27fae6207435389f2dc4af32efddcffb66897782e29196c5f72851637d5 WatchSource:0}: Error finding container 7ecbc27fae6207435389f2dc4af32efddcffb66897782e29196c5f72851637d5: Status 404 returned error can't find the container with id 7ecbc27fae6207435389f2dc4af32efddcffb66897782e29196c5f72851637d5
Sep 17 03:36:14.485958 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:36:14.485935    2552 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 03:36:15.175239 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:36:15.175152    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cdaa7ee2-86d4b79b6d-w7t2l" event={"ID":"58e03c55-ec75-4d27-a39a-c9f8697c96dc","Type":"ContainerStarted","Data":"6d0755e5f563ff17c20e3ef666beb6d61a939488a851d98776d51866bfc55dd8"}
Sep 17 03:36:15.175239 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:36:15.175202    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cdaa7ee2-86d4b79b6d-w7t2l" event={"ID":"58e03c55-ec75-4d27-a39a-c9f8697c96dc","Type":"ContainerStarted","Data":"7ecbc27fae6207435389f2dc4af32efddcffb66897782e29196c5f72851637d5"}
Sep 17 03:36:15.175405 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:36:15.175238    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cdaa7ee2-86d4b79b6d-w7t2l"
Sep 17 03:36:15.199784 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:36:15.199732    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cdaa7ee2-86d4b79b6d-w7t2l" podStartSLOduration=1.775625902 podStartE2EDuration="2.199717798s" totalImagesPullingTime="424.091896ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 03:36:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:36:15.197710169 +0000 UTC m=+1010.664117695" watchObservedRunningTime="2026-09-17 03:36:15.199717798 +0000 UTC m=+1010.666125323"
Sep 17 03:36:19.597805 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:36:19.597775    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-5062bf-xd7rn"]
Sep 17 03:36:19.601123 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:36:19.601096    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-5062bf-xd7rn"]
Sep 17 03:36:20.986737 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:36:20.986708    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="59867e92-c252-4d1d-b927-ef43a70f3646" path="/var/lib/kubelet/pods/59867e92-c252-4d1d-b927-ef43a70f3646/volumes"
Sep 17 03:36:21.187088 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:36:21.187061    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cdaa7ee2-86d4b79b6d-w7t2l"
Sep 17 03:36:24.976480 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:36:24.976444    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-5062bf/explicit-model-5062bf-kserve-58678f7547-qhwp7"]
Sep 17 03:36:39.996388 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:36:39.996340    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-5062bf/explicit-model-5062bf-kserve-58678f7547-qhwp7" podUID="d9628dd8-0bff-47e7-84aa-1b072b47c4a8" containerName="main" containerID="cri-o://a9840841f63c5a6de51200466d40523f858d2f5b06239b946dceb75636701b0f" gracePeriod=45
Sep 17 03:36:40.246340 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:36:40.246316    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-5062bf/explicit-model-5062bf-kserve-58678f7547-qhwp7"
Sep 17 03:36:40.310102 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:36:40.310063    2552 generic.go:356] "Generic (PLEG): container finished" podID="d9628dd8-0bff-47e7-84aa-1b072b47c4a8" containerID="a9840841f63c5a6de51200466d40523f858d2f5b06239b946dceb75636701b0f" exitCode=0
Sep 17 03:36:40.310102 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:36:40.310106    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-5062bf/explicit-model-5062bf-kserve-58678f7547-qhwp7"
Sep 17 03:36:40.310354 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:36:40.310148    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-5062bf/explicit-model-5062bf-kserve-58678f7547-qhwp7" event={"ID":"d9628dd8-0bff-47e7-84aa-1b072b47c4a8","Type":"ContainerDied","Data":"a9840841f63c5a6de51200466d40523f858d2f5b06239b946dceb75636701b0f"}
Sep 17 03:36:40.310354 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:36:40.310200    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-5062bf/explicit-model-5062bf-kserve-58678f7547-qhwp7" event={"ID":"d9628dd8-0bff-47e7-84aa-1b072b47c4a8","Type":"ContainerDied","Data":"6747469a0cb64ab6023a035a036c74b5a994965c03f771234b150ae671cb97a5"}
Sep 17 03:36:40.310354 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:36:40.310222    2552 scope.go:117] "RemoveContainer" containerID="a9840841f63c5a6de51200466d40523f858d2f5b06239b946dceb75636701b0f"
Sep 17 03:36:40.319013 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:36:40.318998    2552 scope.go:117] "RemoveContainer" containerID="a9840841f63c5a6de51200466d40523f858d2f5b06239b946dceb75636701b0f"
Sep 17 03:36:40.319257 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:36:40.319241    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9840841f63c5a6de51200466d40523f858d2f5b06239b946dceb75636701b0f\": container with ID starting with a9840841f63c5a6de51200466d40523f858d2f5b06239b946dceb75636701b0f not found: ID does not exist" containerID="a9840841f63c5a6de51200466d40523f858d2f5b06239b946dceb75636701b0f"
Sep 17 03:36:40.319302 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:36:40.319263    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9840841f63c5a6de51200466d40523f858d2f5b06239b946dceb75636701b0f"} err="failed to get container status \"a9840841f63c5a6de51200466d40523f858d2f5b06239b946dceb75636701b0f\": rpc error: code = NotFound desc = could not find container \"a9840841f63c5a6de51200466d40523f858d2f5b06239b946dceb75636701b0f\": container with ID starting with a9840841f63c5a6de51200466d40523f858d2f5b06239b946dceb75636701b0f not found: ID does not exist"
Sep 17 03:36:40.399883 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:36:40.399852    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9628dd8-0bff-47e7-84aa-1b072b47c4a8-tmp-dir\") pod \"d9628dd8-0bff-47e7-84aa-1b072b47c4a8\" (UID: \"d9628dd8-0bff-47e7-84aa-1b072b47c4a8\") "
Sep 17 03:36:40.400033 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:36:40.399935    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9628dd8-0bff-47e7-84aa-1b072b47c4a8-dshm\") pod \"d9628dd8-0bff-47e7-84aa-1b072b47c4a8\" (UID: \"d9628dd8-0bff-47e7-84aa-1b072b47c4a8\") "
Sep 17 03:36:40.400033 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:36:40.399959    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9628dd8-0bff-47e7-84aa-1b072b47c4a8-model-cache\") pod \"d9628dd8-0bff-47e7-84aa-1b072b47c4a8\" (UID: \"d9628dd8-0bff-47e7-84aa-1b072b47c4a8\") "
Sep 17 03:36:40.400033 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:36:40.400024    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9628dd8-0bff-47e7-84aa-1b072b47c4a8-home\") pod \"d9628dd8-0bff-47e7-84aa-1b072b47c4a8\" (UID: \"d9628dd8-0bff-47e7-84aa-1b072b47c4a8\") "
Sep 17 03:36:40.400158 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:36:40.400045    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9628dd8-0bff-47e7-84aa-1b072b47c4a8-tls-certs\") pod \"d9628dd8-0bff-47e7-84aa-1b072b47c4a8\" (UID: \"d9628dd8-0bff-47e7-84aa-1b072b47c4a8\") "
Sep 17 03:36:40.400158 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:36:40.400078    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cdwgb\" (UniqueName: \"kubernetes.io/projected/d9628dd8-0bff-47e7-84aa-1b072b47c4a8-kube-api-access-cdwgb\") pod \"d9628dd8-0bff-47e7-84aa-1b072b47c4a8\" (UID: \"d9628dd8-0bff-47e7-84aa-1b072b47c4a8\") "
Sep 17 03:36:40.400158 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:36:40.400097    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9628dd8-0bff-47e7-84aa-1b072b47c4a8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d9628dd8-0bff-47e7-84aa-1b072b47c4a8" (UID: "d9628dd8-0bff-47e7-84aa-1b072b47c4a8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:36:40.400319 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:36:40.400273    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9628dd8-0bff-47e7-84aa-1b072b47c4a8-home" (OuterVolumeSpecName: "home") pod "d9628dd8-0bff-47e7-84aa-1b072b47c4a8" (UID: "d9628dd8-0bff-47e7-84aa-1b072b47c4a8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:36:40.400379 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:36:40.400329    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9628dd8-0bff-47e7-84aa-1b072b47c4a8-model-cache" (OuterVolumeSpecName: "model-cache") pod "d9628dd8-0bff-47e7-84aa-1b072b47c4a8" (UID: "d9628dd8-0bff-47e7-84aa-1b072b47c4a8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:36:40.400529 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:36:40.400504    2552 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9628dd8-0bff-47e7-84aa-1b072b47c4a8-model-cache\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:36:40.400529 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:36:40.400523    2552 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9628dd8-0bff-47e7-84aa-1b072b47c4a8-home\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:36:40.400529 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:36:40.400533    2552 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9628dd8-0bff-47e7-84aa-1b072b47c4a8-tmp-dir\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:36:40.402276 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:36:40.402254    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9628dd8-0bff-47e7-84aa-1b072b47c4a8-dshm" (OuterVolumeSpecName: "dshm") pod "d9628dd8-0bff-47e7-84aa-1b072b47c4a8" (UID: "d9628dd8-0bff-47e7-84aa-1b072b47c4a8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:36:40.402378 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:36:40.402316    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9628dd8-0bff-47e7-84aa-1b072b47c4a8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d9628dd8-0bff-47e7-84aa-1b072b47c4a8" (UID: "d9628dd8-0bff-47e7-84aa-1b072b47c4a8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 03:36:40.402378 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:36:40.402355    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9628dd8-0bff-47e7-84aa-1b072b47c4a8-kube-api-access-cdwgb" (OuterVolumeSpecName: "kube-api-access-cdwgb") pod "d9628dd8-0bff-47e7-84aa-1b072b47c4a8" (UID: "d9628dd8-0bff-47e7-84aa-1b072b47c4a8"). InnerVolumeSpecName "kube-api-access-cdwgb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 03:36:40.501794 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:36:40.501765    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cdwgb\" (UniqueName: \"kubernetes.io/projected/d9628dd8-0bff-47e7-84aa-1b072b47c4a8-kube-api-access-cdwgb\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:36:40.501794 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:36:40.501790    2552 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9628dd8-0bff-47e7-84aa-1b072b47c4a8-dshm\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:36:40.501985 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:36:40.501804    2552 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9628dd8-0bff-47e7-84aa-1b072b47c4a8-tls-certs\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:36:40.631241 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:36:40.631213    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-5062bf/explicit-model-5062bf-kserve-58678f7547-qhwp7"]
Sep 17 03:36:40.636655 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:36:40.636632    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-5062bf/explicit-model-5062bf-kserve-58678f7547-qhwp7"]
Sep 17 03:36:40.960786 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:36:40.960702    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-cdaa7ee2-86d4b79b6d-w7t2l"]
Sep 17 03:36:40.961006 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:36:40.960974    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cdaa7ee2-86d4b79b6d-w7t2l" podUID="58e03c55-ec75-4d27-a39a-c9f8697c96dc" containerName="maas-api" containerID="cri-o://6d0755e5f563ff17c20e3ef666beb6d61a939488a851d98776d51866bfc55dd8" gracePeriod=30
Sep 17 03:36:40.987077 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:36:40.987044    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d9628dd8-0bff-47e7-84aa-1b072b47c4a8" path="/var/lib/kubelet/pods/d9628dd8-0bff-47e7-84aa-1b072b47c4a8/volumes"
Sep 17 03:36:41.222700 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:36:41.222641    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cdaa7ee2-86d4b79b6d-w7t2l"
Sep 17 03:36:41.308903 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:36:41.308871    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/58e03c55-ec75-4d27-a39a-c9f8697c96dc-metrics-certs\") pod \"58e03c55-ec75-4d27-a39a-c9f8697c96dc\" (UID: \"58e03c55-ec75-4d27-a39a-c9f8697c96dc\") "
Sep 17 03:36:41.309075 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:36:41.308968    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/58e03c55-ec75-4d27-a39a-c9f8697c96dc-openshift-service-ca\") pod \"58e03c55-ec75-4d27-a39a-c9f8697c96dc\" (UID: \"58e03c55-ec75-4d27-a39a-c9f8697c96dc\") "
Sep 17 03:36:41.309075 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:36:41.309000    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58e03c55-ec75-4d27-a39a-c9f8697c96dc-maas-api-tls\") pod \"58e03c55-ec75-4d27-a39a-c9f8697c96dc\" (UID: \"58e03c55-ec75-4d27-a39a-c9f8697c96dc\") "
Sep 17 03:36:41.309353 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:36:41.309328    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58e03c55-ec75-4d27-a39a-c9f8697c96dc-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "58e03c55-ec75-4d27-a39a-c9f8697c96dc" (UID: "58e03c55-ec75-4d27-a39a-c9f8697c96dc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 03:36:41.311168 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:36:41.311137    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58e03c55-ec75-4d27-a39a-c9f8697c96dc-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "58e03c55-ec75-4d27-a39a-c9f8697c96dc" (UID: "58e03c55-ec75-4d27-a39a-c9f8697c96dc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 03:36:41.311282 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:36:41.311224    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58e03c55-ec75-4d27-a39a-c9f8697c96dc-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "58e03c55-ec75-4d27-a39a-c9f8697c96dc" (UID: "58e03c55-ec75-4d27-a39a-c9f8697c96dc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 03:36:41.317070 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:36:41.317044    2552 generic.go:356] "Generic (PLEG): container finished" podID="58e03c55-ec75-4d27-a39a-c9f8697c96dc" containerID="6d0755e5f563ff17c20e3ef666beb6d61a939488a851d98776d51866bfc55dd8" exitCode=0
Sep 17 03:36:41.317200 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:36:41.317083    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cdaa7ee2-86d4b79b6d-w7t2l"
Sep 17 03:36:41.317200 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:36:41.317129    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cdaa7ee2-86d4b79b6d-w7t2l" event={"ID":"58e03c55-ec75-4d27-a39a-c9f8697c96dc","Type":"ContainerDied","Data":"6d0755e5f563ff17c20e3ef666beb6d61a939488a851d98776d51866bfc55dd8"}
Sep 17 03:36:41.317200 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:36:41.317191    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cdaa7ee2-86d4b79b6d-w7t2l" event={"ID":"58e03c55-ec75-4d27-a39a-c9f8697c96dc","Type":"ContainerDied","Data":"7ecbc27fae6207435389f2dc4af32efddcffb66897782e29196c5f72851637d5"}
Sep 17 03:36:41.317345 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:36:41.317221    2552 scope.go:117] "RemoveContainer" containerID="6d0755e5f563ff17c20e3ef666beb6d61a939488a851d98776d51866bfc55dd8"
Sep 17 03:36:41.331218 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:36:41.331201    2552 scope.go:117] "RemoveContainer" containerID="6d0755e5f563ff17c20e3ef666beb6d61a939488a851d98776d51866bfc55dd8"
Sep 17 03:36:41.331471 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:36:41.331446    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d0755e5f563ff17c20e3ef666beb6d61a939488a851d98776d51866bfc55dd8\": container with ID starting with 6d0755e5f563ff17c20e3ef666beb6d61a939488a851d98776d51866bfc55dd8 not found: ID does not exist" containerID="6d0755e5f563ff17c20e3ef666beb6d61a939488a851d98776d51866bfc55dd8"
Sep 17 03:36:41.331535 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:36:41.331484    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d0755e5f563ff17c20e3ef666beb6d61a939488a851d98776d51866bfc55dd8"} err="failed to get container status \"6d0755e5f563ff17c20e3ef666beb6d61a939488a851d98776d51866bfc55dd8\": rpc error: code = NotFound desc = could not find container \"6d0755e5f563ff17c20e3ef666beb6d61a939488a851d98776d51866bfc55dd8\": container with ID starting with 6d0755e5f563ff17c20e3ef666beb6d61a939488a851d98776d51866bfc55dd8 not found: ID does not exist"
Sep 17 03:36:41.409986 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:36:41.409950    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kn57z\" (UniqueName: \"kubernetes.io/projected/58e03c55-ec75-4d27-a39a-c9f8697c96dc-kube-api-access-kn57z\") pod \"58e03c55-ec75-4d27-a39a-c9f8697c96dc\" (UID: \"58e03c55-ec75-4d27-a39a-c9f8697c96dc\") "
Sep 17 03:36:41.410288 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:36:41.410270    2552 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/58e03c55-ec75-4d27-a39a-c9f8697c96dc-metrics-certs\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:36:41.410367 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:36:41.410292    2552 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/58e03c55-ec75-4d27-a39a-c9f8697c96dc-openshift-service-ca\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:36:41.410367 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:36:41.410312    2552 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58e03c55-ec75-4d27-a39a-c9f8697c96dc-maas-api-tls\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:36:41.412106 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:36:41.412078    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58e03c55-ec75-4d27-a39a-c9f8697c96dc-kube-api-access-kn57z" (OuterVolumeSpecName: "kube-api-access-kn57z") pod "58e03c55-ec75-4d27-a39a-c9f8697c96dc" (UID: "58e03c55-ec75-4d27-a39a-c9f8697c96dc"). InnerVolumeSpecName "kube-api-access-kn57z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 03:36:41.511686 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:36:41.511649    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kn57z\" (UniqueName: \"kubernetes.io/projected/58e03c55-ec75-4d27-a39a-c9f8697c96dc-kube-api-access-kn57z\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:36:41.643324 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:36:41.643298    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-cdaa7ee2-86d4b79b6d-w7t2l"]
Sep 17 03:36:41.645229 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:36:41.645208    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-cdaa7ee2-86d4b79b6d-w7t2l"]
Sep 17 03:36:42.985876 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:36:42.985836    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="58e03c55-ec75-4d27-a39a-c9f8697c96dc" path="/var/lib/kubelet/pods/58e03c55-ec75-4d27-a39a-c9f8697c96dc/volumes"
Sep 17 03:36:44.181826 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:36:44.181772    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cdaa7ee2-86d4b79b6d-w7t2l" podUID="58e03c55-ec75-4d27-a39a-c9f8697c96dc" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.63:8443/health\": context deadline exceeded"
Sep 17 03:37:03.352529 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:37:03.352497    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-35cf45-openshift-default-bfd7bddcb-b8ddw"]
Sep 17 03:37:03.352937 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:37:03.352923    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58e03c55-ec75-4d27-a39a-c9f8697c96dc" containerName="maas-api"
Sep 17 03:37:03.352983 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:37:03.352939    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="58e03c55-ec75-4d27-a39a-c9f8697c96dc" containerName="maas-api"
Sep 17 03:37:03.352983 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:37:03.352968    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9628dd8-0bff-47e7-84aa-1b072b47c4a8" containerName="main"
Sep 17 03:37:03.352983 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:37:03.352973    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9628dd8-0bff-47e7-84aa-1b072b47c4a8" containerName="main"
Sep 17 03:37:03.353074 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:37:03.353034    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="58e03c55-ec75-4d27-a39a-c9f8697c96dc" containerName="maas-api"
Sep 17 03:37:03.353074 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:37:03.353046    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9628dd8-0bff-47e7-84aa-1b072b47c4a8" containerName="main"
Sep 17 03:37:03.356137 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:37:03.356121    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-35cf45-openshift-default-bfd7bddcb-b8ddw"
Sep 17 03:37:03.358795 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:37:03.358772    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-35cf45-openshift-default-dockercfg-xpvz6\""
Sep 17 03:37:03.367237 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:37:03.367216    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-35cf45-openshift-default-bfd7bddcb-b8ddw"]
Sep 17 03:37:03.419896 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:37:03.419860    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a816967-43ac-4537-ae0d-f8dd289ad8ed-istio-token\") pod \"e2e-noten-35cf45-openshift-default-bfd7bddcb-b8ddw\" (UID: \"1a816967-43ac-4537-ae0d-f8dd289ad8ed\") " pod="openshift-ingress/e2e-noten-35cf45-openshift-default-bfd7bddcb-b8ddw"
Sep 17 03:37:03.419896 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:37:03.419893    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a816967-43ac-4537-ae0d-f8dd289ad8ed-istiod-ca-cert\") pod \"e2e-noten-35cf45-openshift-default-bfd7bddcb-b8ddw\" (UID: \"1a816967-43ac-4537-ae0d-f8dd289ad8ed\") " pod="openshift-ingress/e2e-noten-35cf45-openshift-default-bfd7bddcb-b8ddw"
Sep 17 03:37:03.420104 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:37:03.419959    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a816967-43ac-4537-ae0d-f8dd289ad8ed-credential-socket\") pod \"e2e-noten-35cf45-openshift-default-bfd7bddcb-b8ddw\" (UID: \"1a816967-43ac-4537-ae0d-f8dd289ad8ed\") " pod="openshift-ingress/e2e-noten-35cf45-openshift-default-bfd7bddcb-b8ddw"
Sep 17 03:37:03.420104 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:37:03.420041    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a816967-43ac-4537-ae0d-f8dd289ad8ed-workload-socket\") pod \"e2e-noten-35cf45-openshift-default-bfd7bddcb-b8ddw\" (UID: \"1a816967-43ac-4537-ae0d-f8dd289ad8ed\") " pod="openshift-ingress/e2e-noten-35cf45-openshift-default-bfd7bddcb-b8ddw"
Sep 17 03:37:03.420104 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:37:03.420073    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a816967-43ac-4537-ae0d-f8dd289ad8ed-istio-envoy\") pod \"e2e-noten-35cf45-openshift-default-bfd7bddcb-b8ddw\" (UID: \"1a816967-43ac-4537-ae0d-f8dd289ad8ed\") " pod="openshift-ingress/e2e-noten-35cf45-openshift-default-bfd7bddcb-b8ddw"
Sep 17 03:37:03.420207 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:37:03.420104    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdxzf\" (UniqueName: \"kubernetes.io/projected/1a816967-43ac-4537-ae0d-f8dd289ad8ed-kube-api-access-zdxzf\") pod \"e2e-noten-35cf45-openshift-default-bfd7bddcb-b8ddw\" (UID: \"1a816967-43ac-4537-ae0d-f8dd289ad8ed\") " pod="openshift-ingress/e2e-noten-35cf45-openshift-default-bfd7bddcb-b8ddw"
Sep 17 03:37:03.420207 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:37:03.420146    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a816967-43ac-4537-ae0d-f8dd289ad8ed-istio-podinfo\") pod \"e2e-noten-35cf45-openshift-default-bfd7bddcb-b8ddw\" (UID: \"1a816967-43ac-4537-ae0d-f8dd289ad8ed\") " pod="openshift-ingress/e2e-noten-35cf45-openshift-default-bfd7bddcb-b8ddw"
Sep 17 03:37:03.420207 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:37:03.420172    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a816967-43ac-4537-ae0d-f8dd289ad8ed-workload-certs\") pod \"e2e-noten-35cf45-openshift-default-bfd7bddcb-b8ddw\" (UID: \"1a816967-43ac-4537-ae0d-f8dd289ad8ed\") " pod="openshift-ingress/e2e-noten-35cf45-openshift-default-bfd7bddcb-b8ddw"
Sep 17 03:37:03.420207 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:37:03.420187    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a816967-43ac-4537-ae0d-f8dd289ad8ed-istio-data\") pod \"e2e-noten-35cf45-openshift-default-bfd7bddcb-b8ddw\" (UID: \"1a816967-43ac-4537-ae0d-f8dd289ad8ed\") " pod="openshift-ingress/e2e-noten-35cf45-openshift-default-bfd7bddcb-b8ddw"
Sep 17 03:37:03.521824 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:37:03.521786    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a816967-43ac-4537-ae0d-f8dd289ad8ed-istio-token\") pod \"e2e-noten-35cf45-openshift-default-bfd7bddcb-b8ddw\" (UID: \"1a816967-43ac-4537-ae0d-f8dd289ad8ed\") " pod="openshift-ingress/e2e-noten-35cf45-openshift-default-bfd7bddcb-b8ddw"
Sep 17 03:37:03.522033 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:37:03.521836    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a816967-43ac-4537-ae0d-f8dd289ad8ed-istiod-ca-cert\") pod \"e2e-noten-35cf45-openshift-default-bfd7bddcb-b8ddw\" (UID: \"1a816967-43ac-4537-ae0d-f8dd289ad8ed\") " pod="openshift-ingress/e2e-noten-35cf45-openshift-default-bfd7bddcb-b8ddw"
Sep 17 03:37:03.522033 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:37:03.521888    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a816967-43ac-4537-ae0d-f8dd289ad8ed-credential-socket\") pod \"e2e-noten-35cf45-openshift-default-bfd7bddcb-b8ddw\" (UID: \"1a816967-43ac-4537-ae0d-f8dd289ad8ed\") " pod="openshift-ingress/e2e-noten-35cf45-openshift-default-bfd7bddcb-b8ddw"
Sep 17 03:37:03.522033 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:37:03.521948    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a816967-43ac-4537-ae0d-f8dd289ad8ed-workload-socket\") pod \"e2e-noten-35cf45-openshift-default-bfd7bddcb-b8ddw\" (UID: \"1a816967-43ac-4537-ae0d-f8dd289ad8ed\") " pod="openshift-ingress/e2e-noten-35cf45-openshift-default-bfd7bddcb-b8ddw"
Sep 17 03:37:03.522033 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:37:03.521979    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a816967-43ac-4537-ae0d-f8dd289ad8ed-istio-envoy\") pod \"e2e-noten-35cf45-openshift-default-bfd7bddcb-b8ddw\" (UID: \"1a816967-43ac-4537-ae0d-f8dd289ad8ed\") " pod="openshift-ingress/e2e-noten-35cf45-openshift-default-bfd7bddcb-b8ddw"
Sep 17 03:37:03.522033 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:37:03.522018    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdxzf\" (UniqueName: \"kubernetes.io/projected/1a816967-43ac-4537-ae0d-f8dd289ad8ed-kube-api-access-zdxzf\") pod \"e2e-noten-35cf45-openshift-default-bfd7bddcb-b8ddw\" (UID: \"1a816967-43ac-4537-ae0d-f8dd289ad8ed\") " pod="openshift-ingress/e2e-noten-35cf45-openshift-default-bfd7bddcb-b8ddw"
Sep 17 03:37:03.522340 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:37:03.522063    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a816967-43ac-4537-ae0d-f8dd289ad8ed-istio-podinfo\") pod \"e2e-noten-35cf45-openshift-default-bfd7bddcb-b8ddw\" (UID: \"1a816967-43ac-4537-ae0d-f8dd289ad8ed\") " pod="openshift-ingress/e2e-noten-35cf45-openshift-default-bfd7bddcb-b8ddw"
Sep 17 03:37:03.522340 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:37:03.522100    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a816967-43ac-4537-ae0d-f8dd289ad8ed-workload-certs\") pod \"e2e-noten-35cf45-openshift-default-bfd7bddcb-b8ddw\" (UID: \"1a816967-43ac-4537-ae0d-f8dd289ad8ed\") " pod="openshift-ingress/e2e-noten-35cf45-openshift-default-bfd7bddcb-b8ddw"
Sep 17 03:37:03.522340 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:37:03.522127    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a816967-43ac-4537-ae0d-f8dd289ad8ed-istio-data\") pod \"e2e-noten-35cf45-openshift-default-bfd7bddcb-b8ddw\" (UID: \"1a816967-43ac-4537-ae0d-f8dd289ad8ed\") " pod="openshift-ingress/e2e-noten-35cf45-openshift-default-bfd7bddcb-b8ddw"
Sep 17 03:37:03.522652 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:37:03.522627    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a816967-43ac-4537-ae0d-f8dd289ad8ed-credential-socket\") pod \"e2e-noten-35cf45-openshift-default-bfd7bddcb-b8ddw\" (UID: \"1a816967-43ac-4537-ae0d-f8dd289ad8ed\") " pod="openshift-ingress/e2e-noten-35cf45-openshift-default-bfd7bddcb-b8ddw"
Sep 17 03:37:03.522779 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:37:03.522699    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a816967-43ac-4537-ae0d-f8dd289ad8ed-workload-socket\") pod \"e2e-noten-35cf45-openshift-default-bfd7bddcb-b8ddw\" (UID: \"1a816967-43ac-4537-ae0d-f8dd289ad8ed\") " pod="openshift-ingress/e2e-noten-35cf45-openshift-default-bfd7bddcb-b8ddw"
Sep 17 03:37:03.522841 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:37:03.522716    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a816967-43ac-4537-ae0d-f8dd289ad8ed-workload-certs\") pod \"e2e-noten-35cf45-openshift-default-bfd7bddcb-b8ddw\" (UID: \"1a816967-43ac-4537-ae0d-f8dd289ad8ed\") " pod="openshift-ingress/e2e-noten-35cf45-openshift-default-bfd7bddcb-b8ddw"
Sep 17 03:37:03.522841 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:37:03.522790    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a816967-43ac-4537-ae0d-f8dd289ad8ed-istio-data\") pod \"e2e-noten-35cf45-openshift-default-bfd7bddcb-b8ddw\" (UID: \"1a816967-43ac-4537-ae0d-f8dd289ad8ed\") " pod="openshift-ingress/e2e-noten-35cf45-openshift-default-bfd7bddcb-b8ddw"
Sep 17 03:37:03.522841 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:37:03.522790    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a816967-43ac-4537-ae0d-f8dd289ad8ed-istiod-ca-cert\") pod \"e2e-noten-35cf45-openshift-default-bfd7bddcb-b8ddw\" (UID: \"1a816967-43ac-4537-ae0d-f8dd289ad8ed\") " pod="openshift-ingress/e2e-noten-35cf45-openshift-default-bfd7bddcb-b8ddw"
Sep 17 03:37:03.524504 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:37:03.524482    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a816967-43ac-4537-ae0d-f8dd289ad8ed-istio-envoy\") pod \"e2e-noten-35cf45-openshift-default-bfd7bddcb-b8ddw\" (UID: \"1a816967-43ac-4537-ae0d-f8dd289ad8ed\") " pod="openshift-ingress/e2e-noten-35cf45-openshift-default-bfd7bddcb-b8ddw"
Sep 17 03:37:03.524808 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:37:03.524791    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a816967-43ac-4537-ae0d-f8dd289ad8ed-istio-podinfo\") pod \"e2e-noten-35cf45-openshift-default-bfd7bddcb-b8ddw\" (UID: \"1a816967-43ac-4537-ae0d-f8dd289ad8ed\") " pod="openshift-ingress/e2e-noten-35cf45-openshift-default-bfd7bddcb-b8ddw"
Sep 17 03:37:03.529793 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:37:03.529773    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a816967-43ac-4537-ae0d-f8dd289ad8ed-istio-token\") pod \"e2e-noten-35cf45-openshift-default-bfd7bddcb-b8ddw\" (UID: \"1a816967-43ac-4537-ae0d-f8dd289ad8ed\") " pod="openshift-ingress/e2e-noten-35cf45-openshift-default-bfd7bddcb-b8ddw"
Sep 17 03:37:03.529956 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:37:03.529937    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdxzf\" (UniqueName: \"kubernetes.io/projected/1a816967-43ac-4537-ae0d-f8dd289ad8ed-kube-api-access-zdxzf\") pod \"e2e-noten-35cf45-openshift-default-bfd7bddcb-b8ddw\" (UID: \"1a816967-43ac-4537-ae0d-f8dd289ad8ed\") " pod="openshift-ingress/e2e-noten-35cf45-openshift-default-bfd7bddcb-b8ddw"
Sep 17 03:37:03.668165 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:37:03.668079    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-35cf45-openshift-default-bfd7bddcb-b8ddw"
Sep 17 03:37:03.948143 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:37:03.948114    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-35cf45-openshift-default-bfd7bddcb-b8ddw"]
Sep 17 03:37:03.949558 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:37:03.949531    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a816967_43ac_4537_ae0d_f8dd289ad8ed.slice/crio-6085d2cbc4bb01ca21c892ceb67569f51eb7e4cc7eebaf4062e9fb76fd21f7e9 WatchSource:0}: Error finding container 6085d2cbc4bb01ca21c892ceb67569f51eb7e4cc7eebaf4062e9fb76fd21f7e9: Status 404 returned error can't find the container with id 6085d2cbc4bb01ca21c892ceb67569f51eb7e4cc7eebaf4062e9fb76fd21f7e9
Sep 17 03:37:03.951897 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:37:03.951861    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 03:37:03.952018 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:37:03.951929    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 03:37:03.952018 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:37:03.951957    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 03:37:04.445582 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:37:04.445543    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-35cf45-openshift-default-bfd7bddcb-b8ddw" event={"ID":"1a816967-43ac-4537-ae0d-f8dd289ad8ed","Type":"ContainerStarted","Data":"2cd88d9ee79a4fe97beec70cd5aca953f0acb582af8ba967e3e422ed7680ecd6"}
Sep 17 03:37:04.445972 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:37:04.445607    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-35cf45-openshift-default-bfd7bddcb-b8ddw" event={"ID":"1a816967-43ac-4537-ae0d-f8dd289ad8ed","Type":"ContainerStarted","Data":"6085d2cbc4bb01ca21c892ceb67569f51eb7e4cc7eebaf4062e9fb76fd21f7e9"}
Sep 17 03:37:04.470301 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:37:04.470248    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-35cf45-openshift-default-bfd7bddcb-b8ddw" podStartSLOduration=1.470230451 podStartE2EDuration="1.470230451s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 03:37:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:37:04.46747574 +0000 UTC m=+1059.933883358" watchObservedRunningTime="2026-09-17 03:37:04.470230451 +0000 UTC m=+1059.936637976"
Sep 17 03:37:04.668750 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:37:04.668702    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-35cf45-openshift-default-bfd7bddcb-b8ddw"
Sep 17 03:37:05.674393 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:37:05.674364    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-35cf45-openshift-default-bfd7bddcb-b8ddw"
Sep 17 03:37:06.455881 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:37:06.455845    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-35cf45-openshift-default-bfd7bddcb-b8ddw"
Sep 17 03:37:06.456770 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:37:06.456750    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-35cf45-openshift-default-bfd7bddcb-b8ddw"
Sep 17 03:37:54.573546 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:37:54.573511    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-297324-6sbvq"]
Sep 17 03:37:54.576939 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:37:54.576925    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-297324-6sbvq"
Sep 17 03:37:54.579517 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:37:54.579491    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8vwnw\""
Sep 17 03:37:54.583609 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:37:54.583572    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-297324-6sbvq"]
Sep 17 03:37:54.637151 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:37:54.637116    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd2bfc16-3307-4d39-b461-8494befcd34a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-297324-6sbvq\" (UID: \"dd2bfc16-3307-4d39-b461-8494befcd34a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-297324-6sbvq"
Sep 17 03:37:54.737999 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:37:54.737962    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd2bfc16-3307-4d39-b461-8494befcd34a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-297324-6sbvq\" (UID: \"dd2bfc16-3307-4d39-b461-8494befcd34a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-297324-6sbvq"
Sep 17 03:37:54.738690 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:37:54.738669    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd2bfc16-3307-4d39-b461-8494befcd34a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-297324-6sbvq\" (UID: \"dd2bfc16-3307-4d39-b461-8494befcd34a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-297324-6sbvq"
Sep 17 03:37:54.890107 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:37:54.890019    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-297324-6sbvq"
Sep 17 03:37:55.164394 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:37:55.164369    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-297324-6sbvq"]
Sep 17 03:37:55.166230 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:37:55.166203    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd2bfc16_3307_4d39_b461_8494befcd34a.slice/crio-3b8fd824cc134d00fb3d3c346af00fd9a1b1e9545ce265605bd6c847ccc990b8 WatchSource:0}: Error finding container 3b8fd824cc134d00fb3d3c346af00fd9a1b1e9545ce265605bd6c847ccc990b8: Status 404 returned error can't find the container with id 3b8fd824cc134d00fb3d3c346af00fd9a1b1e9545ce265605bd6c847ccc990b8
Sep 17 03:37:55.729240 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:37:55.729207    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-297324-6sbvq" event={"ID":"dd2bfc16-3307-4d39-b461-8494befcd34a","Type":"ContainerStarted","Data":"383741806db3a8b09734d42bc46507ba82fe319e08889d73fd63370f2712974b"}
Sep 17 03:37:55.729672 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:37:55.729257    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-297324-6sbvq" event={"ID":"dd2bfc16-3307-4d39-b461-8494befcd34a","Type":"ContainerStarted","Data":"3b8fd824cc134d00fb3d3c346af00fd9a1b1e9545ce265605bd6c847ccc990b8"}
Sep 17 03:37:55.746262 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:37:55.746216    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-297324-6sbvq" podStartSLOduration=1.74619923 podStartE2EDuration="1.74619923s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 03:37:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:37:55.744239724 +0000 UTC m=+1111.210647245" watchObservedRunningTime="2026-09-17 03:37:55.74619923 +0000 UTC m=+1111.212606756"
Sep 17 03:37:56.735763 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:37:56.735726    2552 generic.go:356] "Generic (PLEG): container finished" podID="dd2bfc16-3307-4d39-b461-8494befcd34a" containerID="383741806db3a8b09734d42bc46507ba82fe319e08889d73fd63370f2712974b" exitCode=0
Sep 17 03:37:56.736249 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:37:56.735780    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-297324-6sbvq" event={"ID":"dd2bfc16-3307-4d39-b461-8494befcd34a","Type":"ContainerDied","Data":"383741806db3a8b09734d42bc46507ba82fe319e08889d73fd63370f2712974b"}
Sep 17 03:37:57.877287 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:37:57.877257    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-297324-6sbvq"
Sep 17 03:37:57.971029 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:37:57.970991    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd2bfc16-3307-4d39-b461-8494befcd34a-maas-api-service-ca\") pod \"dd2bfc16-3307-4d39-b461-8494befcd34a\" (UID: \"dd2bfc16-3307-4d39-b461-8494befcd34a\") "
Sep 17 03:37:57.971375 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:37:57.971351    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd2bfc16-3307-4d39-b461-8494befcd34a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "dd2bfc16-3307-4d39-b461-8494befcd34a" (UID: "dd2bfc16-3307-4d39-b461-8494befcd34a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 03:37:58.072014 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:37:58.071981    2552 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd2bfc16-3307-4d39-b461-8494befcd34a-maas-api-service-ca\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:37:58.748693 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:37:58.748660    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-297324-6sbvq"
Sep 17 03:37:58.748693 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:37:58.748688    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-297324-6sbvq" event={"ID":"dd2bfc16-3307-4d39-b461-8494befcd34a","Type":"ContainerDied","Data":"3b8fd824cc134d00fb3d3c346af00fd9a1b1e9545ce265605bd6c847ccc990b8"}
Sep 17 03:37:58.748994 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:37:58.748738    2552 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3b8fd824cc134d00fb3d3c346af00fd9a1b1e9545ce265605bd6c847ccc990b8"
Sep 17 03:38:04.473547 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:04.473472    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7743-7d8e51"]
Sep 17 03:38:04.473948 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:04.473934    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd2bfc16-3307-4d39-b461-8494befcd34a" containerName="revoke-keys"
Sep 17 03:38:04.473948 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:04.473949    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd2bfc16-3307-4d39-b461-8494befcd34a" containerName="revoke-keys"
Sep 17 03:38:04.474040 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:04.474031    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd2bfc16-3307-4d39-b461-8494befcd34a" containerName="revoke-keys"
Sep 17 03:38:04.477318 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:04.477299    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7743-7d8e51"
Sep 17 03:38:04.480257 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:04.480234    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-s27lz\""
Sep 17 03:38:04.482548 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:04.482520    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7743-7d8e51"]
Sep 17 03:38:04.532393 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:04.532364    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkvd8\" (UniqueName: \"kubernetes.io/projected/24e0647b-3b33-4a26-a8f4-af1425313252-kube-api-access-pkvd8\") pod \"test-curl-7743-7d8e51\" (UID: \"24e0647b-3b33-4a26-a8f4-af1425313252\") " pod="opendatahub/test-curl-7743-7d8e51"
Sep 17 03:38:04.633130 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:04.633099    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkvd8\" (UniqueName: \"kubernetes.io/projected/24e0647b-3b33-4a26-a8f4-af1425313252-kube-api-access-pkvd8\") pod \"test-curl-7743-7d8e51\" (UID: \"24e0647b-3b33-4a26-a8f4-af1425313252\") " pod="opendatahub/test-curl-7743-7d8e51"
Sep 17 03:38:04.640522 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:04.640489    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkvd8\" (UniqueName: \"kubernetes.io/projected/24e0647b-3b33-4a26-a8f4-af1425313252-kube-api-access-pkvd8\") pod \"test-curl-7743-7d8e51\" (UID: \"24e0647b-3b33-4a26-a8f4-af1425313252\") " pod="opendatahub/test-curl-7743-7d8e51"
Sep 17 03:38:04.779482 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:04.779451    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-297324-67cdb564cf-cxrsw"]
Sep 17 03:38:04.779706 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:04.779677    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-297324-67cdb564cf-cxrsw" podUID="eb8c83fc-9471-4e90-88b4-b0e0de8d4803" containerName="payload-processing" containerID="cri-o://aa1f6b20abd15406d4e168b02d70bfec851f3c470106afbebca888fb5255119b" gracePeriod=30
Sep 17 03:38:04.787804 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:04.787782    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7743-7d8e51"
Sep 17 03:38:04.933673 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:04.933615    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-297324-6sbvq"]
Sep 17 03:38:04.935936 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:04.935903    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-297324-6sbvq"]
Sep 17 03:38:04.988239 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:04.988207    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dd2bfc16-3307-4d39-b461-8494befcd34a" path="/var/lib/kubelet/pods/dd2bfc16-3307-4d39-b461-8494befcd34a/volumes"
Sep 17 03:38:05.035865 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:05.035797    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-297324-67cdb564cf-cxrsw"
Sep 17 03:38:05.069247 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:05.069225    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7743-7d8e51"]
Sep 17 03:38:05.071856 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:38:05.071834    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod24e0647b_3b33_4a26_a8f4_af1425313252.slice/crio-9f4c460dbe270054e134d9e9f9bc9dbd328a5a1d302f8e1932464cfc32064655 WatchSource:0}: Error finding container 9f4c460dbe270054e134d9e9f9bc9dbd328a5a1d302f8e1932464cfc32064655: Status 404 returned error can't find the container with id 9f4c460dbe270054e134d9e9f9bc9dbd328a5a1d302f8e1932464cfc32064655
Sep 17 03:38:05.138612 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:05.138575    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb8c83fc-9471-4e90-88b4-b0e0de8d4803-plugins-config-volume\") pod \"eb8c83fc-9471-4e90-88b4-b0e0de8d4803\" (UID: \"eb8c83fc-9471-4e90-88b4-b0e0de8d4803\") "
Sep 17 03:38:05.138829 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:05.138816    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9h92d\" (UniqueName: \"kubernetes.io/projected/eb8c83fc-9471-4e90-88b4-b0e0de8d4803-kube-api-access-9h92d\") pod \"eb8c83fc-9471-4e90-88b4-b0e0de8d4803\" (UID: \"eb8c83fc-9471-4e90-88b4-b0e0de8d4803\") "
Sep 17 03:38:05.138874 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:05.138853    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb8c83fc-9471-4e90-88b4-b0e0de8d4803-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "eb8c83fc-9471-4e90-88b4-b0e0de8d4803" (UID: "eb8c83fc-9471-4e90-88b4-b0e0de8d4803"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 03:38:05.139037 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:05.139024    2552 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb8c83fc-9471-4e90-88b4-b0e0de8d4803-plugins-config-volume\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:38:05.140674 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:05.140649    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb8c83fc-9471-4e90-88b4-b0e0de8d4803-kube-api-access-9h92d" (OuterVolumeSpecName: "kube-api-access-9h92d") pod "eb8c83fc-9471-4e90-88b4-b0e0de8d4803" (UID: "eb8c83fc-9471-4e90-88b4-b0e0de8d4803"). InnerVolumeSpecName "kube-api-access-9h92d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 03:38:05.240286 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:05.240253    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9h92d\" (UniqueName: \"kubernetes.io/projected/eb8c83fc-9471-4e90-88b4-b0e0de8d4803-kube-api-access-9h92d\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:38:05.791292 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:05.791253    2552 generic.go:356] "Generic (PLEG): container finished" podID="eb8c83fc-9471-4e90-88b4-b0e0de8d4803" containerID="aa1f6b20abd15406d4e168b02d70bfec851f3c470106afbebca888fb5255119b" exitCode=0
Sep 17 03:38:05.791846 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:05.791307    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-297324-67cdb564cf-cxrsw"
Sep 17 03:38:05.791846 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:05.791346    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-297324-67cdb564cf-cxrsw" event={"ID":"eb8c83fc-9471-4e90-88b4-b0e0de8d4803","Type":"ContainerDied","Data":"aa1f6b20abd15406d4e168b02d70bfec851f3c470106afbebca888fb5255119b"}
Sep 17 03:38:05.791846 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:05.791413    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-297324-67cdb564cf-cxrsw" event={"ID":"eb8c83fc-9471-4e90-88b4-b0e0de8d4803","Type":"ContainerDied","Data":"98b127155ecc226d44f4b17d0a53fddec9b632adaf3592037ae7f5e1c350a9e3"}
Sep 17 03:38:05.791846 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:05.791449    2552 scope.go:117] "RemoveContainer" containerID="aa1f6b20abd15406d4e168b02d70bfec851f3c470106afbebca888fb5255119b"
Sep 17 03:38:05.793232 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:05.793208    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7743-7d8e51" event={"ID":"24e0647b-3b33-4a26-a8f4-af1425313252","Type":"ContainerStarted","Data":"f289d3fb05adf1ffc93c94c097db93ddd616425a3cf1005c2f1563d7773a9f50"}
Sep 17 03:38:05.793327 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:05.793246    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7743-7d8e51" event={"ID":"24e0647b-3b33-4a26-a8f4-af1425313252","Type":"ContainerStarted","Data":"9f4c460dbe270054e134d9e9f9bc9dbd328a5a1d302f8e1932464cfc32064655"}
Sep 17 03:38:05.800541 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:05.800523    2552 scope.go:117] "RemoveContainer" containerID="aa1f6b20abd15406d4e168b02d70bfec851f3c470106afbebca888fb5255119b"
Sep 17 03:38:05.800828 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:38:05.800808    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa1f6b20abd15406d4e168b02d70bfec851f3c470106afbebca888fb5255119b\": container with ID starting with aa1f6b20abd15406d4e168b02d70bfec851f3c470106afbebca888fb5255119b not found: ID does not exist" containerID="aa1f6b20abd15406d4e168b02d70bfec851f3c470106afbebca888fb5255119b"
Sep 17 03:38:05.800896 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:05.800837    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa1f6b20abd15406d4e168b02d70bfec851f3c470106afbebca888fb5255119b"} err="failed to get container status \"aa1f6b20abd15406d4e168b02d70bfec851f3c470106afbebca888fb5255119b\": rpc error: code = NotFound desc = could not find container \"aa1f6b20abd15406d4e168b02d70bfec851f3c470106afbebca888fb5255119b\": container with ID starting with aa1f6b20abd15406d4e168b02d70bfec851f3c470106afbebca888fb5255119b not found: ID does not exist"
Sep 17 03:38:05.808435 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:05.808393    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7743-7d8e51" podStartSLOduration=1.8083809419999999 podStartE2EDuration="1.808380942s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 03:38:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:38:05.807254843 +0000 UTC m=+1121.273662369" watchObservedRunningTime="2026-09-17 03:38:05.808380942 +0000 UTC m=+1121.274788487"
Sep 17 03:38:05.819943 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:05.819916    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-297324-67cdb564cf-cxrsw"]
Sep 17 03:38:05.821998 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:05.821982    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-297324-67cdb564cf-cxrsw"]
Sep 17 03:38:06.375059 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:06.375021    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7743-7d8e51"]
Sep 17 03:38:06.380336 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:06.380309    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7743-7d8e51"]
Sep 17 03:38:06.495704 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:06.495668    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7743-49e2fc"]
Sep 17 03:38:06.496162 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:06.496143    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb8c83fc-9471-4e90-88b4-b0e0de8d4803" containerName="payload-processing"
Sep 17 03:38:06.496261 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:06.496164    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb8c83fc-9471-4e90-88b4-b0e0de8d4803" containerName="payload-processing"
Sep 17 03:38:06.496261 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:06.496187    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24e0647b-3b33-4a26-a8f4-af1425313252" containerName="test-curl-7743-7d8e51"
Sep 17 03:38:06.496261 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:06.496196    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="24e0647b-3b33-4a26-a8f4-af1425313252" containerName="test-curl-7743-7d8e51"
Sep 17 03:38:06.496407 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:06.496310    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb8c83fc-9471-4e90-88b4-b0e0de8d4803" containerName="payload-processing"
Sep 17 03:38:06.496407 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:06.496328    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="24e0647b-3b33-4a26-a8f4-af1425313252" containerName="test-curl-7743-7d8e51"
Sep 17 03:38:06.500704 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:06.500683    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7743-49e2fc"
Sep 17 03:38:06.507446 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:06.507422    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7743-49e2fc"]
Sep 17 03:38:06.553418 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:06.553387    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrtpz\" (UniqueName: \"kubernetes.io/projected/9fb3f37e-675b-4dc3-add5-280c8a8b02d5-kube-api-access-lrtpz\") pod \"test-curl-7743-49e2fc\" (UID: \"9fb3f37e-675b-4dc3-add5-280c8a8b02d5\") " pod="opendatahub/test-curl-7743-49e2fc"
Sep 17 03:38:06.654879 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:06.654783    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrtpz\" (UniqueName: \"kubernetes.io/projected/9fb3f37e-675b-4dc3-add5-280c8a8b02d5-kube-api-access-lrtpz\") pod \"test-curl-7743-49e2fc\" (UID: \"9fb3f37e-675b-4dc3-add5-280c8a8b02d5\") " pod="opendatahub/test-curl-7743-49e2fc"
Sep 17 03:38:06.662349 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:06.662326    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrtpz\" (UniqueName: \"kubernetes.io/projected/9fb3f37e-675b-4dc3-add5-280c8a8b02d5-kube-api-access-lrtpz\") pod \"test-curl-7743-49e2fc\" (UID: \"9fb3f37e-675b-4dc3-add5-280c8a8b02d5\") " pod="opendatahub/test-curl-7743-49e2fc"
Sep 17 03:38:06.799851 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:38:06.799816    2552 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7743-7d8e51" container="test-curl-7743-7d8e51"
Sep 17 03:38:06.810934 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:06.810911    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7743-49e2fc"
Sep 17 03:38:06.986203 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:06.986124    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eb8c83fc-9471-4e90-88b4-b0e0de8d4803" path="/var/lib/kubelet/pods/eb8c83fc-9471-4e90-88b4-b0e0de8d4803/volumes"
Sep 17 03:38:07.077703 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:07.077680    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7743-49e2fc"]
Sep 17 03:38:07.079927 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:38:07.079901    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9fb3f37e_675b_4dc3_add5_280c8a8b02d5.slice/crio-c4b49498f658d73885f01675d561b70b46d07e4734bc389cf3d987ed97e285ce WatchSource:0}: Error finding container c4b49498f658d73885f01675d561b70b46d07e4734bc389cf3d987ed97e285ce: Status 404 returned error can't find the container with id c4b49498f658d73885f01675d561b70b46d07e4734bc389cf3d987ed97e285ce
Sep 17 03:38:07.806793 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:07.806753    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7743-49e2fc" event={"ID":"9fb3f37e-675b-4dc3-add5-280c8a8b02d5","Type":"ContainerStarted","Data":"efdc60290c678df2bcf639ea1ff40de33a0e4c61c8278ad19082b70d14df2cc4"}
Sep 17 03:38:07.806793 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:07.806788    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7743-7d8e51" podUID="24e0647b-3b33-4a26-a8f4-af1425313252" containerName="test-curl-7743-7d8e51" containerID="cri-o://f289d3fb05adf1ffc93c94c097db93ddd616425a3cf1005c2f1563d7773a9f50" gracePeriod=2
Sep 17 03:38:07.806793 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:07.806805    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7743-49e2fc" event={"ID":"9fb3f37e-675b-4dc3-add5-280c8a8b02d5","Type":"ContainerStarted","Data":"c4b49498f658d73885f01675d561b70b46d07e4734bc389cf3d987ed97e285ce"}
Sep 17 03:38:07.823870 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:07.823690    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7743-49e2fc" podStartSLOduration=1.823676117 podStartE2EDuration="1.823676117s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 03:38:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:38:07.823325776 +0000 UTC m=+1123.289733302" watchObservedRunningTime="2026-09-17 03:38:07.823676117 +0000 UTC m=+1123.290083639"
Sep 17 03:38:08.380164 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:08.380133    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7743-49e2fc"]
Sep 17 03:38:08.386488 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:08.386463    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7743-49e2fc"]
Sep 17 03:38:08.501117 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:08.501085    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7743-f7fb0f"]
Sep 17 03:38:08.501572 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:08.501555    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9fb3f37e-675b-4dc3-add5-280c8a8b02d5" containerName="test-curl-7743-49e2fc"
Sep 17 03:38:08.501672 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:08.501574    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="9fb3f37e-675b-4dc3-add5-280c8a8b02d5" containerName="test-curl-7743-49e2fc"
Sep 17 03:38:08.501735 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:08.501687    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="9fb3f37e-675b-4dc3-add5-280c8a8b02d5" containerName="test-curl-7743-49e2fc"
Sep 17 03:38:08.504870 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:08.504846    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7743-f7fb0f"
Sep 17 03:38:08.510327 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:08.510304    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7743-f7fb0f"]
Sep 17 03:38:08.576355 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:08.576325    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfcb8\" (UniqueName: \"kubernetes.io/projected/265e9c39-6e10-45f1-bcac-4f69c998ebea-kube-api-access-pfcb8\") pod \"test-curl-7743-f7fb0f\" (UID: \"265e9c39-6e10-45f1-bcac-4f69c998ebea\") " pod="opendatahub/test-curl-7743-f7fb0f"
Sep 17 03:38:08.677128 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:08.677086    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfcb8\" (UniqueName: \"kubernetes.io/projected/265e9c39-6e10-45f1-bcac-4f69c998ebea-kube-api-access-pfcb8\") pod \"test-curl-7743-f7fb0f\" (UID: \"265e9c39-6e10-45f1-bcac-4f69c998ebea\") " pod="opendatahub/test-curl-7743-f7fb0f"
Sep 17 03:38:08.685610 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:08.685576    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfcb8\" (UniqueName: \"kubernetes.io/projected/265e9c39-6e10-45f1-bcac-4f69c998ebea-kube-api-access-pfcb8\") pod \"test-curl-7743-f7fb0f\" (UID: \"265e9c39-6e10-45f1-bcac-4f69c998ebea\") " pod="opendatahub/test-curl-7743-f7fb0f"
Sep 17 03:38:08.812823 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:38:08.812737    2552 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7743-49e2fc" container="test-curl-7743-49e2fc"
Sep 17 03:38:08.816580 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:08.816562    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7743-f7fb0f"
Sep 17 03:38:09.087082 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:09.087055    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7743-f7fb0f"]
Sep 17 03:38:09.089320 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:38:09.089293    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod265e9c39_6e10_45f1_bcac_4f69c998ebea.slice/crio-cbc466fe91ec3e8d6d62c1dd4ab4701537ce5b285dbb08adedf54d9336783582 WatchSource:0}: Error finding container cbc466fe91ec3e8d6d62c1dd4ab4701537ce5b285dbb08adedf54d9336783582: Status 404 returned error can't find the container with id cbc466fe91ec3e8d6d62c1dd4ab4701537ce5b285dbb08adedf54d9336783582
Sep 17 03:38:09.820346 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:09.820312    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7743-f7fb0f" event={"ID":"265e9c39-6e10-45f1-bcac-4f69c998ebea","Type":"ContainerStarted","Data":"5b7395bbbcae5febdff8b0b30c06a5f99a6121189bd863577e140e0073f42fb0"}
Sep 17 03:38:09.820725 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:09.820362    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7743-f7fb0f" event={"ID":"265e9c39-6e10-45f1-bcac-4f69c998ebea","Type":"ContainerStarted","Data":"cbc466fe91ec3e8d6d62c1dd4ab4701537ce5b285dbb08adedf54d9336783582"}
Sep 17 03:38:09.820725 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:09.820417    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7743-49e2fc" podUID="9fb3f37e-675b-4dc3-add5-280c8a8b02d5" containerName="test-curl-7743-49e2fc" containerID="cri-o://efdc60290c678df2bcf639ea1ff40de33a0e4c61c8278ad19082b70d14df2cc4" gracePeriod=2
Sep 17 03:38:09.835529 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:09.835482    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7743-f7fb0f" podStartSLOduration=1.835468057 podStartE2EDuration="1.835468057s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 03:38:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:38:09.832893341 +0000 UTC m=+1125.299300868" watchObservedRunningTime="2026-09-17 03:38:09.835468057 +0000 UTC m=+1125.301875582"
Sep 17 03:38:09.982615 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:09.982570    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7743-7d8e51"
Sep 17 03:38:10.092546 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:10.092472    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pkvd8\" (UniqueName: \"kubernetes.io/projected/24e0647b-3b33-4a26-a8f4-af1425313252-kube-api-access-pkvd8\") pod \"24e0647b-3b33-4a26-a8f4-af1425313252\" (UID: \"24e0647b-3b33-4a26-a8f4-af1425313252\") "
Sep 17 03:38:10.094405 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:10.094382    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24e0647b-3b33-4a26-a8f4-af1425313252-kube-api-access-pkvd8" (OuterVolumeSpecName: "kube-api-access-pkvd8") pod "24e0647b-3b33-4a26-a8f4-af1425313252" (UID: "24e0647b-3b33-4a26-a8f4-af1425313252"). InnerVolumeSpecName "kube-api-access-pkvd8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 03:38:10.193417 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:10.193385    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pkvd8\" (UniqueName: \"kubernetes.io/projected/24e0647b-3b33-4a26-a8f4-af1425313252-kube-api-access-pkvd8\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:38:10.410856 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:10.410771    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7743-f7fb0f"]
Sep 17 03:38:10.415310 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:10.415281    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7743-f7fb0f"]
Sep 17 03:38:10.525408 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:10.525373    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7743-8520ea"]
Sep 17 03:38:10.525810 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:10.525793    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="265e9c39-6e10-45f1-bcac-4f69c998ebea" containerName="test-curl-7743-f7fb0f"
Sep 17 03:38:10.525810 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:10.525808    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="265e9c39-6e10-45f1-bcac-4f69c998ebea" containerName="test-curl-7743-f7fb0f"
Sep 17 03:38:10.525963 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:10.525873    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="265e9c39-6e10-45f1-bcac-4f69c998ebea" containerName="test-curl-7743-f7fb0f"
Sep 17 03:38:10.528981 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:10.528966    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7743-8520ea"
Sep 17 03:38:10.532910 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:10.532886    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7743-8520ea"]
Sep 17 03:38:10.598275 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:10.598247    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rqkj\" (UniqueName: \"kubernetes.io/projected/8c8c8d16-290a-4815-99ba-d4d771f8eec0-kube-api-access-8rqkj\") pod \"test-curl-7743-8520ea\" (UID: \"8c8c8d16-290a-4815-99ba-d4d771f8eec0\") " pod="opendatahub/test-curl-7743-8520ea"
Sep 17 03:38:10.699826 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:10.699746    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rqkj\" (UniqueName: \"kubernetes.io/projected/8c8c8d16-290a-4815-99ba-d4d771f8eec0-kube-api-access-8rqkj\") pod \"test-curl-7743-8520ea\" (UID: \"8c8c8d16-290a-4815-99ba-d4d771f8eec0\") " pod="opendatahub/test-curl-7743-8520ea"
Sep 17 03:38:10.707637 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:10.707611    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rqkj\" (UniqueName: \"kubernetes.io/projected/8c8c8d16-290a-4815-99ba-d4d771f8eec0-kube-api-access-8rqkj\") pod \"test-curl-7743-8520ea\" (UID: \"8c8c8d16-290a-4815-99ba-d4d771f8eec0\") " pod="opendatahub/test-curl-7743-8520ea"
Sep 17 03:38:10.827484 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:10.827452    2552 generic.go:356] "Generic (PLEG): container finished" podID="24e0647b-3b33-4a26-a8f4-af1425313252" containerID="f289d3fb05adf1ffc93c94c097db93ddd616425a3cf1005c2f1563d7773a9f50" exitCode=137
Sep 17 03:38:10.827923 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:10.827494    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7743-7d8e51"
Sep 17 03:38:10.827923 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:10.827551    2552 scope.go:117] "RemoveContainer" containerID="f289d3fb05adf1ffc93c94c097db93ddd616425a3cf1005c2f1563d7773a9f50"
Sep 17 03:38:10.827923 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:38:10.827893    2552 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7743-f7fb0f" container="test-curl-7743-f7fb0f"
Sep 17 03:38:10.837714 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:10.837696    2552 scope.go:117] "RemoveContainer" containerID="f289d3fb05adf1ffc93c94c097db93ddd616425a3cf1005c2f1563d7773a9f50"
Sep 17 03:38:10.837970 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:38:10.837954    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f289d3fb05adf1ffc93c94c097db93ddd616425a3cf1005c2f1563d7773a9f50\": container with ID starting with f289d3fb05adf1ffc93c94c097db93ddd616425a3cf1005c2f1563d7773a9f50 not found: ID does not exist" containerID="f289d3fb05adf1ffc93c94c097db93ddd616425a3cf1005c2f1563d7773a9f50"
Sep 17 03:38:10.838007 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:10.837981    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f289d3fb05adf1ffc93c94c097db93ddd616425a3cf1005c2f1563d7773a9f50"} err="failed to get container status \"f289d3fb05adf1ffc93c94c097db93ddd616425a3cf1005c2f1563d7773a9f50\": rpc error: code = NotFound desc = could not find container \"f289d3fb05adf1ffc93c94c097db93ddd616425a3cf1005c2f1563d7773a9f50\": container with ID starting with f289d3fb05adf1ffc93c94c097db93ddd616425a3cf1005c2f1563d7773a9f50 not found: ID does not exist"
Sep 17 03:38:10.839134 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:10.839121    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7743-8520ea"
Sep 17 03:38:10.839173 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:38:10.839127    2552 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7743-7d8e51\" is forbidden: User \"system:node:ip-10-0-139-53.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-53.ec2.internal' and this object" podUID="24e0647b-3b33-4a26-a8f4-af1425313252" pod="opendatahub/test-curl-7743-7d8e51"
Sep 17 03:38:10.985966 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:10.985873    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="24e0647b-3b33-4a26-a8f4-af1425313252" path="/var/lib/kubelet/pods/24e0647b-3b33-4a26-a8f4-af1425313252/volumes"
Sep 17 03:38:11.110703 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:11.110649    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7743-8520ea"]
Sep 17 03:38:11.112982 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:38:11.112955    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8c8c8d16_290a_4815_99ba_d4d771f8eec0.slice/crio-a6727cc0300ebc5d9b073ea3c02ef6d1f8c4041a7fabab3ff747c844986cdfe2 WatchSource:0}: Error finding container a6727cc0300ebc5d9b073ea3c02ef6d1f8c4041a7fabab3ff747c844986cdfe2: Status 404 returned error can't find the container with id a6727cc0300ebc5d9b073ea3c02ef6d1f8c4041a7fabab3ff747c844986cdfe2
Sep 17 03:38:11.511204 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:11.511171    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3d0c3c-m8n7h"]
Sep 17 03:38:11.514816 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:11.514801    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3d0c3c-m8n7h"
Sep 17 03:38:11.517180 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:11.517158    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8vwnw\""
Sep 17 03:38:11.521674 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:11.521654    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3d0c3c-m8n7h"]
Sep 17 03:38:11.608061 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:11.608034    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbc3e9cb-f483-47cc-940a-79d564a91719-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-3d0c3c-m8n7h\" (UID: \"cbc3e9cb-f483-47cc-940a-79d564a91719\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3d0c3c-m8n7h"
Sep 17 03:38:11.709022 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:11.708993    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbc3e9cb-f483-47cc-940a-79d564a91719-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-3d0c3c-m8n7h\" (UID: \"cbc3e9cb-f483-47cc-940a-79d564a91719\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3d0c3c-m8n7h"
Sep 17 03:38:11.709615 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:11.709581    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbc3e9cb-f483-47cc-940a-79d564a91719-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-3d0c3c-m8n7h\" (UID: \"cbc3e9cb-f483-47cc-940a-79d564a91719\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3d0c3c-m8n7h"
Sep 17 03:38:11.826474 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:11.826324    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3d0c3c-m8n7h"
Sep 17 03:38:11.836422 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:11.836391    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7743-8520ea" event={"ID":"8c8c8d16-290a-4815-99ba-d4d771f8eec0","Type":"ContainerStarted","Data":"23d302ea69d3643d511308990cf204cae0d54061df9439e13422d351e8a4ab49"}
Sep 17 03:38:11.836840 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:11.836434    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7743-8520ea" event={"ID":"8c8c8d16-290a-4815-99ba-d4d771f8eec0","Type":"ContainerStarted","Data":"a6727cc0300ebc5d9b073ea3c02ef6d1f8c4041a7fabab3ff747c844986cdfe2"}
Sep 17 03:38:11.836840 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:11.836500    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7743-f7fb0f" podUID="265e9c39-6e10-45f1-bcac-4f69c998ebea" containerName="test-curl-7743-f7fb0f" containerID="cri-o://5b7395bbbcae5febdff8b0b30c06a5f99a6121189bd863577e140e0073f42fb0" gracePeriod=2
Sep 17 03:38:11.852003 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:11.851950    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7743-8520ea" podStartSLOduration=1.8519304650000001 podStartE2EDuration="1.851930465s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 03:38:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:38:11.849761583 +0000 UTC m=+1127.316169111" watchObservedRunningTime="2026-09-17 03:38:11.851930465 +0000 UTC m=+1127.318337993"
Sep 17 03:38:12.020719 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:12.020694    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7743-49e2fc"
Sep 17 03:38:12.098257 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:12.098231    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3d0c3c-m8n7h"]
Sep 17 03:38:12.099853 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:38:12.099818    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbc3e9cb_f483_47cc_940a_79d564a91719.slice/crio-025a84d34e44e7846430f9ee982238187c0bb85f0bb31543fc0b74fc8de33863 WatchSource:0}: Error finding container 025a84d34e44e7846430f9ee982238187c0bb85f0bb31543fc0b74fc8de33863: Status 404 returned error can't find the container with id 025a84d34e44e7846430f9ee982238187c0bb85f0bb31543fc0b74fc8de33863
Sep 17 03:38:12.113091 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:12.113069    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lrtpz\" (UniqueName: \"kubernetes.io/projected/9fb3f37e-675b-4dc3-add5-280c8a8b02d5-kube-api-access-lrtpz\") pod \"9fb3f37e-675b-4dc3-add5-280c8a8b02d5\" (UID: \"9fb3f37e-675b-4dc3-add5-280c8a8b02d5\") "
Sep 17 03:38:12.115059 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:12.115032    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fb3f37e-675b-4dc3-add5-280c8a8b02d5-kube-api-access-lrtpz" (OuterVolumeSpecName: "kube-api-access-lrtpz") pod "9fb3f37e-675b-4dc3-add5-280c8a8b02d5" (UID: "9fb3f37e-675b-4dc3-add5-280c8a8b02d5"). InnerVolumeSpecName "kube-api-access-lrtpz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 03:38:12.214124 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:12.214098    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrtpz\" (UniqueName: \"kubernetes.io/projected/9fb3f37e-675b-4dc3-add5-280c8a8b02d5-kube-api-access-lrtpz\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:38:12.412248 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:12.412167    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7743-8520ea"]
Sep 17 03:38:12.416064 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:12.416038    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7743-8520ea"]
Sep 17 03:38:12.843139 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:12.843112    2552 generic.go:356] "Generic (PLEG): container finished" podID="9fb3f37e-675b-4dc3-add5-280c8a8b02d5" containerID="efdc60290c678df2bcf639ea1ff40de33a0e4c61c8278ad19082b70d14df2cc4" exitCode=137
Sep 17 03:38:12.843515 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:12.843167    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7743-49e2fc"
Sep 17 03:38:12.843515 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:12.843223    2552 scope.go:117] "RemoveContainer" containerID="efdc60290c678df2bcf639ea1ff40de33a0e4c61c8278ad19082b70d14df2cc4"
Sep 17 03:38:12.844781 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:12.844753    2552 generic.go:356] "Generic (PLEG): container finished" podID="cbc3e9cb-f483-47cc-940a-79d564a91719" containerID="d8122c179bb57a4714b93f4e2e17f768ef9a84194e63090c10b0939e7dc57e86" exitCode=0
Sep 17 03:38:12.844895 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:12.844819    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3d0c3c-m8n7h" event={"ID":"cbc3e9cb-f483-47cc-940a-79d564a91719","Type":"ContainerDied","Data":"d8122c179bb57a4714b93f4e2e17f768ef9a84194e63090c10b0939e7dc57e86"}
Sep 17 03:38:12.844964 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:12.844904    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3d0c3c-m8n7h" event={"ID":"cbc3e9cb-f483-47cc-940a-79d564a91719","Type":"ContainerStarted","Data":"025a84d34e44e7846430f9ee982238187c0bb85f0bb31543fc0b74fc8de33863"}
Sep 17 03:38:12.845059 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:38:12.845040    2552 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7743-8520ea" container="test-curl-7743-8520ea"
Sep 17 03:38:12.886083 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:38:12.886055    2552 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7743-49e2fc\" is forbidden: User \"system:node:ip-10-0-139-53.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-53.ec2.internal' and this object" podUID="9fb3f37e-675b-4dc3-add5-280c8a8b02d5" pod="opendatahub/test-curl-7743-49e2fc"
Sep 17 03:38:12.886669 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:12.886644    2552 scope.go:117] "RemoveContainer" containerID="efdc60290c678df2bcf639ea1ff40de33a0e4c61c8278ad19082b70d14df2cc4"
Sep 17 03:38:12.886935 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:38:12.886917    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efdc60290c678df2bcf639ea1ff40de33a0e4c61c8278ad19082b70d14df2cc4\": container with ID starting with efdc60290c678df2bcf639ea1ff40de33a0e4c61c8278ad19082b70d14df2cc4 not found: ID does not exist" containerID="efdc60290c678df2bcf639ea1ff40de33a0e4c61c8278ad19082b70d14df2cc4"
Sep 17 03:38:12.887013 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:12.886945    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efdc60290c678df2bcf639ea1ff40de33a0e4c61c8278ad19082b70d14df2cc4"} err="failed to get container status \"efdc60290c678df2bcf639ea1ff40de33a0e4c61c8278ad19082b70d14df2cc4\": rpc error: code = NotFound desc = could not find container \"efdc60290c678df2bcf639ea1ff40de33a0e4c61c8278ad19082b70d14df2cc4\": container with ID starting with efdc60290c678df2bcf639ea1ff40de33a0e4c61c8278ad19082b70d14df2cc4 not found: ID does not exist"
Sep 17 03:38:12.985907 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:12.985877    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9fb3f37e-675b-4dc3-add5-280c8a8b02d5" path="/var/lib/kubelet/pods/9fb3f37e-675b-4dc3-add5-280c8a8b02d5/volumes"
Sep 17 03:38:13.223856 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:13.223773    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-c10ab8fb-openshift-default-8654fcf896-qxlrg"]
Sep 17 03:38:13.224391 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:13.224372    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c8c8d16-290a-4815-99ba-d4d771f8eec0" containerName="test-curl-7743-8520ea"
Sep 17 03:38:13.224434 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:13.224393    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c8c8d16-290a-4815-99ba-d4d771f8eec0" containerName="test-curl-7743-8520ea"
Sep 17 03:38:13.224486 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:13.224476    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c8c8d16-290a-4815-99ba-d4d771f8eec0" containerName="test-curl-7743-8520ea"
Sep 17 03:38:13.228923 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:13.228905    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c10ab8fb-openshift-default-8654fcf896-qxlrg"
Sep 17 03:38:13.231675 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:13.231655    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-c10ab8fb-openshift-default-dockercfg-6dlnb\""
Sep 17 03:38:13.238654 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:13.238632    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-c10ab8fb-openshift-default-8654fcf896-qxlrg"]
Sep 17 03:38:13.325432 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:13.325399    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78c4208f-673e-4ab0-b439-63c3a10b9cac-istio-token\") pod \"e2e-mt-c10ab8fb-openshift-default-8654fcf896-qxlrg\" (UID: \"78c4208f-673e-4ab0-b439-63c3a10b9cac\") " pod="openshift-ingress/e2e-mt-c10ab8fb-openshift-default-8654fcf896-qxlrg"
Sep 17 03:38:13.325628 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:13.325445    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78c4208f-673e-4ab0-b439-63c3a10b9cac-istio-podinfo\") pod \"e2e-mt-c10ab8fb-openshift-default-8654fcf896-qxlrg\" (UID: \"78c4208f-673e-4ab0-b439-63c3a10b9cac\") " pod="openshift-ingress/e2e-mt-c10ab8fb-openshift-default-8654fcf896-qxlrg"
Sep 17 03:38:13.325628 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:13.325465    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78c4208f-673e-4ab0-b439-63c3a10b9cac-istio-data\") pod \"e2e-mt-c10ab8fb-openshift-default-8654fcf896-qxlrg\" (UID: \"78c4208f-673e-4ab0-b439-63c3a10b9cac\") " pod="openshift-ingress/e2e-mt-c10ab8fb-openshift-default-8654fcf896-qxlrg"
Sep 17 03:38:13.325628 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:13.325552    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78c4208f-673e-4ab0-b439-63c3a10b9cac-istiod-ca-cert\") pod \"e2e-mt-c10ab8fb-openshift-default-8654fcf896-qxlrg\" (UID: \"78c4208f-673e-4ab0-b439-63c3a10b9cac\") " pod="openshift-ingress/e2e-mt-c10ab8fb-openshift-default-8654fcf896-qxlrg"
Sep 17 03:38:13.325628 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:13.325587    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78c4208f-673e-4ab0-b439-63c3a10b9cac-istio-envoy\") pod \"e2e-mt-c10ab8fb-openshift-default-8654fcf896-qxlrg\" (UID: \"78c4208f-673e-4ab0-b439-63c3a10b9cac\") " pod="openshift-ingress/e2e-mt-c10ab8fb-openshift-default-8654fcf896-qxlrg"
Sep 17 03:38:13.325802 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:13.325682    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78c4208f-673e-4ab0-b439-63c3a10b9cac-credential-socket\") pod \"e2e-mt-c10ab8fb-openshift-default-8654fcf896-qxlrg\" (UID: \"78c4208f-673e-4ab0-b439-63c3a10b9cac\") " pod="openshift-ingress/e2e-mt-c10ab8fb-openshift-default-8654fcf896-qxlrg"
Sep 17 03:38:13.325802 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:13.325733    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78c4208f-673e-4ab0-b439-63c3a10b9cac-workload-certs\") pod \"e2e-mt-c10ab8fb-openshift-default-8654fcf896-qxlrg\" (UID: \"78c4208f-673e-4ab0-b439-63c3a10b9cac\") " pod="openshift-ingress/e2e-mt-c10ab8fb-openshift-default-8654fcf896-qxlrg"
Sep 17 03:38:13.325802 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:13.325750    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58fct\" (UniqueName: \"kubernetes.io/projected/78c4208f-673e-4ab0-b439-63c3a10b9cac-kube-api-access-58fct\") pod \"e2e-mt-c10ab8fb-openshift-default-8654fcf896-qxlrg\" (UID: \"78c4208f-673e-4ab0-b439-63c3a10b9cac\") " pod="openshift-ingress/e2e-mt-c10ab8fb-openshift-default-8654fcf896-qxlrg"
Sep 17 03:38:13.325947 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:13.325838    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78c4208f-673e-4ab0-b439-63c3a10b9cac-workload-socket\") pod \"e2e-mt-c10ab8fb-openshift-default-8654fcf896-qxlrg\" (UID: \"78c4208f-673e-4ab0-b439-63c3a10b9cac\") " pod="openshift-ingress/e2e-mt-c10ab8fb-openshift-default-8654fcf896-qxlrg"
Sep 17 03:38:13.427031 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:13.426995    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78c4208f-673e-4ab0-b439-63c3a10b9cac-istio-envoy\") pod \"e2e-mt-c10ab8fb-openshift-default-8654fcf896-qxlrg\" (UID: \"78c4208f-673e-4ab0-b439-63c3a10b9cac\") " pod="openshift-ingress/e2e-mt-c10ab8fb-openshift-default-8654fcf896-qxlrg"
Sep 17 03:38:13.427228 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:13.427050    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78c4208f-673e-4ab0-b439-63c3a10b9cac-credential-socket\") pod \"e2e-mt-c10ab8fb-openshift-default-8654fcf896-qxlrg\" (UID: \"78c4208f-673e-4ab0-b439-63c3a10b9cac\") " pod="openshift-ingress/e2e-mt-c10ab8fb-openshift-default-8654fcf896-qxlrg"
Sep 17 03:38:13.427228 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:13.427093    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78c4208f-673e-4ab0-b439-63c3a10b9cac-workload-certs\") pod \"e2e-mt-c10ab8fb-openshift-default-8654fcf896-qxlrg\" (UID: \"78c4208f-673e-4ab0-b439-63c3a10b9cac\") " pod="openshift-ingress/e2e-mt-c10ab8fb-openshift-default-8654fcf896-qxlrg"
Sep 17 03:38:13.427228 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:13.427116    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58fct\" (UniqueName: \"kubernetes.io/projected/78c4208f-673e-4ab0-b439-63c3a10b9cac-kube-api-access-58fct\") pod \"e2e-mt-c10ab8fb-openshift-default-8654fcf896-qxlrg\" (UID: \"78c4208f-673e-4ab0-b439-63c3a10b9cac\") " pod="openshift-ingress/e2e-mt-c10ab8fb-openshift-default-8654fcf896-qxlrg"
Sep 17 03:38:13.427228 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:13.427199    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78c4208f-673e-4ab0-b439-63c3a10b9cac-workload-socket\") pod \"e2e-mt-c10ab8fb-openshift-default-8654fcf896-qxlrg\" (UID: \"78c4208f-673e-4ab0-b439-63c3a10b9cac\") " pod="openshift-ingress/e2e-mt-c10ab8fb-openshift-default-8654fcf896-qxlrg"
Sep 17 03:38:13.427448 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:13.427285    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78c4208f-673e-4ab0-b439-63c3a10b9cac-istio-token\") pod \"e2e-mt-c10ab8fb-openshift-default-8654fcf896-qxlrg\" (UID: \"78c4208f-673e-4ab0-b439-63c3a10b9cac\") " pod="openshift-ingress/e2e-mt-c10ab8fb-openshift-default-8654fcf896-qxlrg"
Sep 17 03:38:13.427448 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:13.427363    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78c4208f-673e-4ab0-b439-63c3a10b9cac-istio-podinfo\") pod \"e2e-mt-c10ab8fb-openshift-default-8654fcf896-qxlrg\" (UID: \"78c4208f-673e-4ab0-b439-63c3a10b9cac\") " pod="openshift-ingress/e2e-mt-c10ab8fb-openshift-default-8654fcf896-qxlrg"
Sep 17 03:38:13.427448 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:13.427393    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78c4208f-673e-4ab0-b439-63c3a10b9cac-istio-data\") pod \"e2e-mt-c10ab8fb-openshift-default-8654fcf896-qxlrg\" (UID: \"78c4208f-673e-4ab0-b439-63c3a10b9cac\") " pod="openshift-ingress/e2e-mt-c10ab8fb-openshift-default-8654fcf896-qxlrg"
Sep 17 03:38:13.427649 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:13.427454    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78c4208f-673e-4ab0-b439-63c3a10b9cac-istiod-ca-cert\") pod \"e2e-mt-c10ab8fb-openshift-default-8654fcf896-qxlrg\" (UID: \"78c4208f-673e-4ab0-b439-63c3a10b9cac\") " pod="openshift-ingress/e2e-mt-c10ab8fb-openshift-default-8654fcf896-qxlrg"
Sep 17 03:38:13.427649 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:13.427572    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78c4208f-673e-4ab0-b439-63c3a10b9cac-workload-certs\") pod \"e2e-mt-c10ab8fb-openshift-default-8654fcf896-qxlrg\" (UID: \"78c4208f-673e-4ab0-b439-63c3a10b9cac\") " pod="openshift-ingress/e2e-mt-c10ab8fb-openshift-default-8654fcf896-qxlrg"
Sep 17 03:38:13.427649 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:13.427572    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78c4208f-673e-4ab0-b439-63c3a10b9cac-credential-socket\") pod \"e2e-mt-c10ab8fb-openshift-default-8654fcf896-qxlrg\" (UID: \"78c4208f-673e-4ab0-b439-63c3a10b9cac\") " pod="openshift-ingress/e2e-mt-c10ab8fb-openshift-default-8654fcf896-qxlrg"
Sep 17 03:38:13.427830 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:13.427641    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78c4208f-673e-4ab0-b439-63c3a10b9cac-workload-socket\") pod \"e2e-mt-c10ab8fb-openshift-default-8654fcf896-qxlrg\" (UID: \"78c4208f-673e-4ab0-b439-63c3a10b9cac\") " pod="openshift-ingress/e2e-mt-c10ab8fb-openshift-default-8654fcf896-qxlrg"
Sep 17 03:38:13.427884 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:13.427849    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78c4208f-673e-4ab0-b439-63c3a10b9cac-istio-data\") pod \"e2e-mt-c10ab8fb-openshift-default-8654fcf896-qxlrg\" (UID: \"78c4208f-673e-4ab0-b439-63c3a10b9cac\") " pod="openshift-ingress/e2e-mt-c10ab8fb-openshift-default-8654fcf896-qxlrg"
Sep 17 03:38:13.428282 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:13.428259    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78c4208f-673e-4ab0-b439-63c3a10b9cac-istiod-ca-cert\") pod \"e2e-mt-c10ab8fb-openshift-default-8654fcf896-qxlrg\" (UID: \"78c4208f-673e-4ab0-b439-63c3a10b9cac\") " pod="openshift-ingress/e2e-mt-c10ab8fb-openshift-default-8654fcf896-qxlrg"
Sep 17 03:38:13.429666 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:13.429644    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78c4208f-673e-4ab0-b439-63c3a10b9cac-istio-envoy\") pod \"e2e-mt-c10ab8fb-openshift-default-8654fcf896-qxlrg\" (UID: \"78c4208f-673e-4ab0-b439-63c3a10b9cac\") " pod="openshift-ingress/e2e-mt-c10ab8fb-openshift-default-8654fcf896-qxlrg"
Sep 17 03:38:13.430046 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:13.430024    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78c4208f-673e-4ab0-b439-63c3a10b9cac-istio-podinfo\") pod \"e2e-mt-c10ab8fb-openshift-default-8654fcf896-qxlrg\" (UID: \"78c4208f-673e-4ab0-b439-63c3a10b9cac\") " pod="openshift-ingress/e2e-mt-c10ab8fb-openshift-default-8654fcf896-qxlrg"
Sep 17 03:38:13.440824 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:13.440805    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-58fct\" (UniqueName: \"kubernetes.io/projected/78c4208f-673e-4ab0-b439-63c3a10b9cac-kube-api-access-58fct\") pod \"e2e-mt-c10ab8fb-openshift-default-8654fcf896-qxlrg\" (UID: \"78c4208f-673e-4ab0-b439-63c3a10b9cac\") " pod="openshift-ingress/e2e-mt-c10ab8fb-openshift-default-8654fcf896-qxlrg"
Sep 17 03:38:13.441203 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:13.441183    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78c4208f-673e-4ab0-b439-63c3a10b9cac-istio-token\") pod \"e2e-mt-c10ab8fb-openshift-default-8654fcf896-qxlrg\" (UID: \"78c4208f-673e-4ab0-b439-63c3a10b9cac\") " pod="openshift-ingress/e2e-mt-c10ab8fb-openshift-default-8654fcf896-qxlrg"
Sep 17 03:38:13.541474 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:13.541440    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c10ab8fb-openshift-default-8654fcf896-qxlrg"
Sep 17 03:38:13.837689 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:13.837661    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-c10ab8fb-openshift-default-8654fcf896-qxlrg"]
Sep 17 03:38:13.840247 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:38:13.840223    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78c4208f_673e_4ab0_b439_63c3a10b9cac.slice/crio-c03833cef72702c88129e0d98546e933e47f77b30e5f39bd8b24a5dec8ac7bbf WatchSource:0}: Error finding container c03833cef72702c88129e0d98546e933e47f77b30e5f39bd8b24a5dec8ac7bbf: Status 404 returned error can't find the container with id c03833cef72702c88129e0d98546e933e47f77b30e5f39bd8b24a5dec8ac7bbf
Sep 17 03:38:13.842896 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:13.842860    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 03:38:13.842971 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:13.842929    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 03:38:13.842971 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:13.842957    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 03:38:13.852887 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:13.852821    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c10ab8fb-openshift-default-8654fcf896-qxlrg" event={"ID":"78c4208f-673e-4ab0-b439-63c3a10b9cac","Type":"ContainerStarted","Data":"c03833cef72702c88129e0d98546e933e47f77b30e5f39bd8b24a5dec8ac7bbf"}
Sep 17 03:38:13.854590 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:13.854559    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7743-8520ea" podUID="8c8c8d16-290a-4815-99ba-d4d771f8eec0" containerName="test-curl-7743-8520ea" containerID="cri-o://23d302ea69d3643d511308990cf204cae0d54061df9439e13422d351e8a4ab49" gracePeriod=2
Sep 17 03:38:14.011076 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:14.011044    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3d0c3c-m8n7h"
Sep 17 03:38:14.046216 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:14.046190    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7743-f7fb0f"
Sep 17 03:38:14.136137 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:14.136046    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pfcb8\" (UniqueName: \"kubernetes.io/projected/265e9c39-6e10-45f1-bcac-4f69c998ebea-kube-api-access-pfcb8\") pod \"265e9c39-6e10-45f1-bcac-4f69c998ebea\" (UID: \"265e9c39-6e10-45f1-bcac-4f69c998ebea\") "
Sep 17 03:38:14.136137 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:14.136105    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbc3e9cb-f483-47cc-940a-79d564a91719-maas-api-service-ca\") pod \"cbc3e9cb-f483-47cc-940a-79d564a91719\" (UID: \"cbc3e9cb-f483-47cc-940a-79d564a91719\") "
Sep 17 03:38:14.136474 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:14.136450    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbc3e9cb-f483-47cc-940a-79d564a91719-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "cbc3e9cb-f483-47cc-940a-79d564a91719" (UID: "cbc3e9cb-f483-47cc-940a-79d564a91719"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 03:38:14.138166 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:14.138146    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/265e9c39-6e10-45f1-bcac-4f69c998ebea-kube-api-access-pfcb8" (OuterVolumeSpecName: "kube-api-access-pfcb8") pod "265e9c39-6e10-45f1-bcac-4f69c998ebea" (UID: "265e9c39-6e10-45f1-bcac-4f69c998ebea"). InnerVolumeSpecName "kube-api-access-pfcb8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 03:38:14.237780 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:14.237745    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pfcb8\" (UniqueName: \"kubernetes.io/projected/265e9c39-6e10-45f1-bcac-4f69c998ebea-kube-api-access-pfcb8\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:38:14.237933 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:14.237785    2552 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbc3e9cb-f483-47cc-940a-79d564a91719-maas-api-service-ca\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:38:14.861331 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:14.861292    2552 generic.go:356] "Generic (PLEG): container finished" podID="265e9c39-6e10-45f1-bcac-4f69c998ebea" containerID="5b7395bbbcae5febdff8b0b30c06a5f99a6121189bd863577e140e0073f42fb0" exitCode=137
Sep 17 03:38:14.861331 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:14.861333    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7743-f7fb0f"
Sep 17 03:38:14.861901 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:14.861394    2552 scope.go:117] "RemoveContainer" containerID="5b7395bbbcae5febdff8b0b30c06a5f99a6121189bd863577e140e0073f42fb0"
Sep 17 03:38:14.862945 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:14.862923    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3d0c3c-m8n7h"
Sep 17 03:38:14.863059 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:14.862988    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3d0c3c-m8n7h" event={"ID":"cbc3e9cb-f483-47cc-940a-79d564a91719","Type":"ContainerDied","Data":"025a84d34e44e7846430f9ee982238187c0bb85f0bb31543fc0b74fc8de33863"}
Sep 17 03:38:14.863059 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:14.863034    2552 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="025a84d34e44e7846430f9ee982238187c0bb85f0bb31543fc0b74fc8de33863"
Sep 17 03:38:14.864992 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:14.864970    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c10ab8fb-openshift-default-8654fcf896-qxlrg" event={"ID":"78c4208f-673e-4ab0-b439-63c3a10b9cac","Type":"ContainerStarted","Data":"45708f7aaf5a723f7db716c3845aa51e67e1bbea8bf560f2cc1b13c6aa43d061"}
Sep 17 03:38:14.871166 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:14.871150    2552 scope.go:117] "RemoveContainer" containerID="5b7395bbbcae5febdff8b0b30c06a5f99a6121189bd863577e140e0073f42fb0"
Sep 17 03:38:14.871445 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:38:14.871422    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b7395bbbcae5febdff8b0b30c06a5f99a6121189bd863577e140e0073f42fb0\": container with ID starting with 5b7395bbbcae5febdff8b0b30c06a5f99a6121189bd863577e140e0073f42fb0 not found: ID does not exist" containerID="5b7395bbbcae5febdff8b0b30c06a5f99a6121189bd863577e140e0073f42fb0"
Sep 17 03:38:14.871516 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:14.871453    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b7395bbbcae5febdff8b0b30c06a5f99a6121189bd863577e140e0073f42fb0"} err="failed to get container status \"5b7395bbbcae5febdff8b0b30c06a5f99a6121189bd863577e140e0073f42fb0\": rpc error: code = NotFound desc = could not find container \"5b7395bbbcae5febdff8b0b30c06a5f99a6121189bd863577e140e0073f42fb0\": container with ID starting with 5b7395bbbcae5febdff8b0b30c06a5f99a6121189bd863577e140e0073f42fb0 not found: ID does not exist"
Sep 17 03:38:14.885517 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:14.885459    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-c10ab8fb-openshift-default-8654fcf896-qxlrg" podStartSLOduration=1.885447193 podStartE2EDuration="1.885447193s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 03:38:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:38:14.883775385 +0000 UTC m=+1130.350182910" watchObservedRunningTime="2026-09-17 03:38:14.885447193 +0000 UTC m=+1130.351854719"
Sep 17 03:38:14.885748 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:38:14.885726    2552 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7743-f7fb0f\" is forbidden: User \"system:node:ip-10-0-139-53.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-53.ec2.internal' and this object" podUID="265e9c39-6e10-45f1-bcac-4f69c998ebea" pod="opendatahub/test-curl-7743-f7fb0f"
Sep 17 03:38:14.984838 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:38:14.984803    2552 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7743-f7fb0f\" is forbidden: User \"system:node:ip-10-0-139-53.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-53.ec2.internal' and this object" podUID="265e9c39-6e10-45f1-bcac-4f69c998ebea" pod="opendatahub/test-curl-7743-f7fb0f"
Sep 17 03:38:14.986833 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:14.986805    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="265e9c39-6e10-45f1-bcac-4f69c998ebea" path="/var/lib/kubelet/pods/265e9c39-6e10-45f1-bcac-4f69c998ebea/volumes"
Sep 17 03:38:15.541878 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:15.541847    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-c10ab8fb-openshift-default-8654fcf896-qxlrg"
Sep 17 03:38:15.548055 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:15.548029    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-c10ab8fb-openshift-default-8654fcf896-qxlrg"
Sep 17 03:38:15.871992 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:15.871971    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-c10ab8fb-openshift-default-8654fcf896-qxlrg"
Sep 17 03:38:15.873001 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:15.872981    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-c10ab8fb-openshift-default-8654fcf896-qxlrg"
Sep 17 03:38:16.041882 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:16.041857    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7743-8520ea"
Sep 17 03:38:16.159438 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:16.159350    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8rqkj\" (UniqueName: \"kubernetes.io/projected/8c8c8d16-290a-4815-99ba-d4d771f8eec0-kube-api-access-8rqkj\") pod \"8c8c8d16-290a-4815-99ba-d4d771f8eec0\" (UID: \"8c8c8d16-290a-4815-99ba-d4d771f8eec0\") "
Sep 17 03:38:16.161452 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:16.161421    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c8c8d16-290a-4815-99ba-d4d771f8eec0-kube-api-access-8rqkj" (OuterVolumeSpecName: "kube-api-access-8rqkj") pod "8c8c8d16-290a-4815-99ba-d4d771f8eec0" (UID: "8c8c8d16-290a-4815-99ba-d4d771f8eec0"). InnerVolumeSpecName "kube-api-access-8rqkj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 03:38:16.261134 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:16.261095    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8rqkj\" (UniqueName: \"kubernetes.io/projected/8c8c8d16-290a-4815-99ba-d4d771f8eec0-kube-api-access-8rqkj\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:38:16.878314 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:16.878276    2552 generic.go:356] "Generic (PLEG): container finished" podID="8c8c8d16-290a-4815-99ba-d4d771f8eec0" containerID="23d302ea69d3643d511308990cf204cae0d54061df9439e13422d351e8a4ab49" exitCode=137
Sep 17 03:38:16.878314 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:16.878315    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7743-8520ea"
Sep 17 03:38:16.878865 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:16.878379    2552 scope.go:117] "RemoveContainer" containerID="23d302ea69d3643d511308990cf204cae0d54061df9439e13422d351e8a4ab49"
Sep 17 03:38:16.888067 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:16.888050    2552 scope.go:117] "RemoveContainer" containerID="23d302ea69d3643d511308990cf204cae0d54061df9439e13422d351e8a4ab49"
Sep 17 03:38:16.888319 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:38:16.888301    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23d302ea69d3643d511308990cf204cae0d54061df9439e13422d351e8a4ab49\": container with ID starting with 23d302ea69d3643d511308990cf204cae0d54061df9439e13422d351e8a4ab49 not found: ID does not exist" containerID="23d302ea69d3643d511308990cf204cae0d54061df9439e13422d351e8a4ab49"
Sep 17 03:38:16.888385 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:16.888332    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23d302ea69d3643d511308990cf204cae0d54061df9439e13422d351e8a4ab49"} err="failed to get container status \"23d302ea69d3643d511308990cf204cae0d54061df9439e13422d351e8a4ab49\": rpc error: code = NotFound desc = could not find container \"23d302ea69d3643d511308990cf204cae0d54061df9439e13422d351e8a4ab49\": container with ID starting with 23d302ea69d3643d511308990cf204cae0d54061df9439e13422d351e8a4ab49 not found: ID does not exist"
Sep 17 03:38:16.889004 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:38:16.888981    2552 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7743-8520ea\" is forbidden: User \"system:node:ip-10-0-139-53.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-53.ec2.internal' and this object" podUID="8c8c8d16-290a-4815-99ba-d4d771f8eec0" pod="opendatahub/test-curl-7743-8520ea"
Sep 17 03:38:16.986379 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:16.986343    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8c8c8d16-290a-4815-99ba-d4d771f8eec0" path="/var/lib/kubelet/pods/8c8c8d16-290a-4815-99ba-d4d771f8eec0/volumes"
Sep 17 03:38:22.280664 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:22.280626    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3d0c3c-m8n7h"]
Sep 17 03:38:22.282829 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:22.282809    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3d0c3c-m8n7h"]
Sep 17 03:38:22.988744 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:22.988697    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cbc3e9cb-f483-47cc-940a-79d564a91719" path="/var/lib/kubelet/pods/cbc3e9cb-f483-47cc-940a-79d564a91719/volumes"
Sep 17 03:38:27.616721 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:27.616681    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-3d0c3c/test-model-3d0c3c-kserve-56f86bc989-bhcdt"]
Sep 17 03:38:41.008420 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:41.008384    2552 generic.go:356] "Generic (PLEG): container finished" podID="78c4208f-673e-4ab0-b439-63c3a10b9cac" containerID="45708f7aaf5a723f7db716c3845aa51e67e1bbea8bf560f2cc1b13c6aa43d061" exitCode=0
Sep 17 03:38:41.008420 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:41.008429    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c10ab8fb-openshift-default-8654fcf896-qxlrg" event={"ID":"78c4208f-673e-4ab0-b439-63c3a10b9cac","Type":"ContainerDied","Data":"45708f7aaf5a723f7db716c3845aa51e67e1bbea8bf560f2cc1b13c6aa43d061"}
Sep 17 03:38:41.008914 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:41.008806    2552 scope.go:117] "RemoveContainer" containerID="45708f7aaf5a723f7db716c3845aa51e67e1bbea8bf560f2cc1b13c6aa43d061"
Sep 17 03:38:41.010001 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:41.009972    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 03:38:41.010067 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:41.010043    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 03:38:41.010102 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:41.010073    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 03:38:42.015895 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:42.015861    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c10ab8fb-openshift-default-8654fcf896-qxlrg" event={"ID":"78c4208f-673e-4ab0-b439-63c3a10b9cac","Type":"ContainerStarted","Data":"3850974e33e1c1d1fdc1bf9e64ef484d62b08fffe7f6f65551b28a008d8787c5"}
Sep 17 03:38:42.542644 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:42.542586    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-c10ab8fb-openshift-default-8654fcf896-qxlrg"
Sep 17 03:38:42.543778 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:42.543747    2552 patch_prober.go:28] interesting pod/e2e-mt-c10ab8fb-openshift-default-8654fcf896-qxlrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Sep 17 03:38:42.543905 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:42.543812    2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c10ab8fb-openshift-default-8654fcf896-qxlrg" podUID="78c4208f-673e-4ab0-b439-63c3a10b9cac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Sep 17 03:38:42.652330 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:42.652146    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-3d0c3c/test-model-3d0c3c-kserve-56f86bc989-bhcdt" podUID="3b5f7f2e-b36d-46c4-9f37-8aae4d8722b3" containerName="main" containerID="cri-o://3f6340e972d307af61e65e4ce5599b7b506856c9b4db8a2b505e6a77c0a497b4" gracePeriod=45
Sep 17 03:38:42.946621 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:42.944932    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-3d0c3c/test-model-3d0c3c-kserve-56f86bc989-bhcdt"
Sep 17 03:38:43.023667 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:43.023625    2552 generic.go:356] "Generic (PLEG): container finished" podID="3b5f7f2e-b36d-46c4-9f37-8aae4d8722b3" containerID="3f6340e972d307af61e65e4ce5599b7b506856c9b4db8a2b505e6a77c0a497b4" exitCode=0
Sep 17 03:38:43.024055 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:43.023674    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-3d0c3c/test-model-3d0c3c-kserve-56f86bc989-bhcdt"
Sep 17 03:38:43.024055 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:43.023715    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-3d0c3c/test-model-3d0c3c-kserve-56f86bc989-bhcdt" event={"ID":"3b5f7f2e-b36d-46c4-9f37-8aae4d8722b3","Type":"ContainerDied","Data":"3f6340e972d307af61e65e4ce5599b7b506856c9b4db8a2b505e6a77c0a497b4"}
Sep 17 03:38:43.024055 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:43.023762    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-3d0c3c/test-model-3d0c3c-kserve-56f86bc989-bhcdt" event={"ID":"3b5f7f2e-b36d-46c4-9f37-8aae4d8722b3","Type":"ContainerDied","Data":"0777a6eab0240501d7309a804f2f69ece9455a35e47bda723dd0c2dee06c2f2d"}
Sep 17 03:38:43.024055 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:43.023780    2552 scope.go:117] "RemoveContainer" containerID="3f6340e972d307af61e65e4ce5599b7b506856c9b4db8a2b505e6a77c0a497b4"
Sep 17 03:38:43.055806 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:43.055772    2552 scope.go:117] "RemoveContainer" containerID="3f6340e972d307af61e65e4ce5599b7b506856c9b4db8a2b505e6a77c0a497b4"
Sep 17 03:38:43.056174 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:38:43.056154    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f6340e972d307af61e65e4ce5599b7b506856c9b4db8a2b505e6a77c0a497b4\": container with ID starting with 3f6340e972d307af61e65e4ce5599b7b506856c9b4db8a2b505e6a77c0a497b4 not found: ID does not exist" containerID="3f6340e972d307af61e65e4ce5599b7b506856c9b4db8a2b505e6a77c0a497b4"
Sep 17 03:38:43.056247 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:43.056185    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f6340e972d307af61e65e4ce5599b7b506856c9b4db8a2b505e6a77c0a497b4"} err="failed to get container status \"3f6340e972d307af61e65e4ce5599b7b506856c9b4db8a2b505e6a77c0a497b4\": rpc error: code = NotFound desc = could not find container \"3f6340e972d307af61e65e4ce5599b7b506856c9b4db8a2b505e6a77c0a497b4\": container with ID starting with 3f6340e972d307af61e65e4ce5599b7b506856c9b4db8a2b505e6a77c0a497b4 not found: ID does not exist"
Sep 17 03:38:43.059499 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:43.059478    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b5f7f2e-b36d-46c4-9f37-8aae4d8722b3-dshm\") pod \"3b5f7f2e-b36d-46c4-9f37-8aae4d8722b3\" (UID: \"3b5f7f2e-b36d-46c4-9f37-8aae4d8722b3\") "
Sep 17 03:38:43.059609 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:43.059520    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b5f7f2e-b36d-46c4-9f37-8aae4d8722b3-model-cache\") pod \"3b5f7f2e-b36d-46c4-9f37-8aae4d8722b3\" (UID: \"3b5f7f2e-b36d-46c4-9f37-8aae4d8722b3\") "
Sep 17 03:38:43.059609 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:43.059540    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mxf7t\" (UniqueName: \"kubernetes.io/projected/3b5f7f2e-b36d-46c4-9f37-8aae4d8722b3-kube-api-access-mxf7t\") pod \"3b5f7f2e-b36d-46c4-9f37-8aae4d8722b3\" (UID: \"3b5f7f2e-b36d-46c4-9f37-8aae4d8722b3\") "
Sep 17 03:38:43.059609 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:43.059582    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b5f7f2e-b36d-46c4-9f37-8aae4d8722b3-home\") pod \"3b5f7f2e-b36d-46c4-9f37-8aae4d8722b3\" (UID: \"3b5f7f2e-b36d-46c4-9f37-8aae4d8722b3\") "
Sep 17 03:38:43.059784 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:43.059678    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b5f7f2e-b36d-46c4-9f37-8aae4d8722b3-tmp-dir\") pod \"3b5f7f2e-b36d-46c4-9f37-8aae4d8722b3\" (UID: \"3b5f7f2e-b36d-46c4-9f37-8aae4d8722b3\") "
Sep 17 03:38:43.059784 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:43.059728    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b5f7f2e-b36d-46c4-9f37-8aae4d8722b3-tls-certs\") pod \"3b5f7f2e-b36d-46c4-9f37-8aae4d8722b3\" (UID: \"3b5f7f2e-b36d-46c4-9f37-8aae4d8722b3\") "
Sep 17 03:38:43.059898 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:43.059781    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b5f7f2e-b36d-46c4-9f37-8aae4d8722b3-model-cache" (OuterVolumeSpecName: "model-cache") pod "3b5f7f2e-b36d-46c4-9f37-8aae4d8722b3" (UID: "3b5f7f2e-b36d-46c4-9f37-8aae4d8722b3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:38:43.059951 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:43.059904    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b5f7f2e-b36d-46c4-9f37-8aae4d8722b3-home" (OuterVolumeSpecName: "home") pod "3b5f7f2e-b36d-46c4-9f37-8aae4d8722b3" (UID: "3b5f7f2e-b36d-46c4-9f37-8aae4d8722b3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:38:43.060008 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:43.059946    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b5f7f2e-b36d-46c4-9f37-8aae4d8722b3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3b5f7f2e-b36d-46c4-9f37-8aae4d8722b3" (UID: "3b5f7f2e-b36d-46c4-9f37-8aae4d8722b3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:38:43.060160 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:43.060138    2552 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b5f7f2e-b36d-46c4-9f37-8aae4d8722b3-tmp-dir\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:38:43.060225 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:43.060165    2552 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b5f7f2e-b36d-46c4-9f37-8aae4d8722b3-model-cache\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:38:43.060225 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:43.060179    2552 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b5f7f2e-b36d-46c4-9f37-8aae4d8722b3-home\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:38:43.061816 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:43.061790    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b5f7f2e-b36d-46c4-9f37-8aae4d8722b3-dshm" (OuterVolumeSpecName: "dshm") pod "3b5f7f2e-b36d-46c4-9f37-8aae4d8722b3" (UID: "3b5f7f2e-b36d-46c4-9f37-8aae4d8722b3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:38:43.061920 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:43.061838    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b5f7f2e-b36d-46c4-9f37-8aae4d8722b3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3b5f7f2e-b36d-46c4-9f37-8aae4d8722b3" (UID: "3b5f7f2e-b36d-46c4-9f37-8aae4d8722b3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 03:38:43.061920 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:43.061844    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b5f7f2e-b36d-46c4-9f37-8aae4d8722b3-kube-api-access-mxf7t" (OuterVolumeSpecName: "kube-api-access-mxf7t") pod "3b5f7f2e-b36d-46c4-9f37-8aae4d8722b3" (UID: "3b5f7f2e-b36d-46c4-9f37-8aae4d8722b3"). InnerVolumeSpecName "kube-api-access-mxf7t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 03:38:43.161406 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:43.161364    2552 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b5f7f2e-b36d-46c4-9f37-8aae4d8722b3-tls-certs\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:38:43.161406 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:43.161403    2552 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b5f7f2e-b36d-46c4-9f37-8aae4d8722b3-dshm\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:38:43.161406 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:43.161417    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxf7t\" (UniqueName: \"kubernetes.io/projected/3b5f7f2e-b36d-46c4-9f37-8aae4d8722b3-kube-api-access-mxf7t\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:38:43.349970 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:43.349926    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-3d0c3c/test-model-3d0c3c-kserve-56f86bc989-bhcdt"]
Sep 17 03:38:43.353987 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:43.353950    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-3d0c3c/test-model-3d0c3c-kserve-56f86bc989-bhcdt"]
Sep 17 03:38:43.543054 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:43.542539    2552 patch_prober.go:28] interesting pod/e2e-mt-c10ab8fb-openshift-default-8654fcf896-qxlrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Sep 17 03:38:43.543054 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:43.542647    2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c10ab8fb-openshift-default-8654fcf896-qxlrg" podUID="78c4208f-673e-4ab0-b439-63c3a10b9cac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Sep 17 03:38:44.542234 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:44.542187    2552 patch_prober.go:28] interesting pod/e2e-mt-c10ab8fb-openshift-default-8654fcf896-qxlrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Sep 17 03:38:44.542784 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:44.542274    2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c10ab8fb-openshift-default-8654fcf896-qxlrg" podUID="78c4208f-673e-4ab0-b439-63c3a10b9cac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Sep 17 03:38:44.993273 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:44.993186    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3b5f7f2e-b36d-46c4-9f37-8aae4d8722b3" path="/var/lib/kubelet/pods/3b5f7f2e-b36d-46c4-9f37-8aae4d8722b3/volumes"
Sep 17 03:38:45.542725 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:45.542689    2552 patch_prober.go:28] interesting pod/e2e-mt-c10ab8fb-openshift-default-8654fcf896-qxlrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Sep 17 03:38:45.543343 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:45.543290    2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c10ab8fb-openshift-default-8654fcf896-qxlrg" podUID="78c4208f-673e-4ab0-b439-63c3a10b9cac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Sep 17 03:38:45.872758 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:45.872661    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-c10ab8fb-openshift-default-8654fcf896-qxlrg"
Sep 17 03:38:46.542395 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:46.542353    2552 patch_prober.go:28] interesting pod/e2e-mt-c10ab8fb-openshift-default-8654fcf896-qxlrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Sep 17 03:38:46.542747 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:46.542714    2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c10ab8fb-openshift-default-8654fcf896-qxlrg" podUID="78c4208f-673e-4ab0-b439-63c3a10b9cac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Sep 17 03:38:47.545849 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:47.545774    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-c10ab8fb-openshift-default-8654fcf896-qxlrg"
Sep 17 03:38:47.546798 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:47.546774    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-c10ab8fb-openshift-default-8654fcf896-qxlrg"
Sep 17 03:38:54.376746 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:54.376717    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-35cf45/noten-model-35cf45-kserve-64cf784dc5-vlm7n"]
Sep 17 03:38:54.377246 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:54.377229    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b5f7f2e-b36d-46c4-9f37-8aae4d8722b3" containerName="main"
Sep 17 03:38:54.377296 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:54.377248    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b5f7f2e-b36d-46c4-9f37-8aae4d8722b3" containerName="main"
Sep 17 03:38:54.377296 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:54.377263    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cbc3e9cb-f483-47cc-940a-79d564a91719" containerName="revoke-keys"
Sep 17 03:38:54.377296 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:54.377269    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="cbc3e9cb-f483-47cc-940a-79d564a91719" containerName="revoke-keys"
Sep 17 03:38:54.377391 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:54.377346    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b5f7f2e-b36d-46c4-9f37-8aae4d8722b3" containerName="main"
Sep 17 03:38:54.377391 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:54.377356    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="cbc3e9cb-f483-47cc-940a-79d564a91719" containerName="revoke-keys"
Sep 17 03:38:54.380640 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:54.380618    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-35cf45/noten-model-35cf45-kserve-64cf784dc5-vlm7n"
Sep 17 03:38:54.384895 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:54.384868    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-35cf45\"/\"kube-root-ca.crt\""
Sep 17 03:38:54.385019 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:54.384890    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-35cf45\"/\"noten-model-35cf45-kserve-self-signed-certs\""
Sep 17 03:38:54.385019 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:54.384875    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-35cf45\"/\"default-dockercfg-wjcjq\""
Sep 17 03:38:54.385019 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:54.384898    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-35cf45\"/\"openshift-service-ca.crt\""
Sep 17 03:38:54.390520 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:54.390499    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-35cf45/noten-model-35cf45-kserve-64cf784dc5-vlm7n"]
Sep 17 03:38:54.473225 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:54.473187    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f08d64ea-b3e4-439d-bab2-03011a9683b8-dshm\") pod \"noten-model-35cf45-kserve-64cf784dc5-vlm7n\" (UID: \"f08d64ea-b3e4-439d-bab2-03011a9683b8\") " pod="ai-tenant-e2e-noten-35cf45/noten-model-35cf45-kserve-64cf784dc5-vlm7n"
Sep 17 03:38:54.473401 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:54.473268    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f08d64ea-b3e4-439d-bab2-03011a9683b8-tmp-dir\") pod \"noten-model-35cf45-kserve-64cf784dc5-vlm7n\" (UID: \"f08d64ea-b3e4-439d-bab2-03011a9683b8\") " pod="ai-tenant-e2e-noten-35cf45/noten-model-35cf45-kserve-64cf784dc5-vlm7n"
Sep 17 03:38:54.473401 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:54.473294    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f08d64ea-b3e4-439d-bab2-03011a9683b8-model-cache\") pod \"noten-model-35cf45-kserve-64cf784dc5-vlm7n\" (UID: \"f08d64ea-b3e4-439d-bab2-03011a9683b8\") " pod="ai-tenant-e2e-noten-35cf45/noten-model-35cf45-kserve-64cf784dc5-vlm7n"
Sep 17 03:38:54.473401 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:54.473326    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvk64\" (UniqueName: \"kubernetes.io/projected/f08d64ea-b3e4-439d-bab2-03011a9683b8-kube-api-access-gvk64\") pod \"noten-model-35cf45-kserve-64cf784dc5-vlm7n\" (UID: \"f08d64ea-b3e4-439d-bab2-03011a9683b8\") " pod="ai-tenant-e2e-noten-35cf45/noten-model-35cf45-kserve-64cf784dc5-vlm7n"
Sep 17 03:38:54.473401 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:54.473386    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f08d64ea-b3e4-439d-bab2-03011a9683b8-home\") pod \"noten-model-35cf45-kserve-64cf784dc5-vlm7n\" (UID: \"f08d64ea-b3e4-439d-bab2-03011a9683b8\") " pod="ai-tenant-e2e-noten-35cf45/noten-model-35cf45-kserve-64cf784dc5-vlm7n"
Sep 17 03:38:54.473538 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:54.473473    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f08d64ea-b3e4-439d-bab2-03011a9683b8-tls-certs\") pod \"noten-model-35cf45-kserve-64cf784dc5-vlm7n\" (UID: \"f08d64ea-b3e4-439d-bab2-03011a9683b8\") " pod="ai-tenant-e2e-noten-35cf45/noten-model-35cf45-kserve-64cf784dc5-vlm7n"
Sep 17 03:38:54.574150 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:54.574115    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f08d64ea-b3e4-439d-bab2-03011a9683b8-tmp-dir\") pod \"noten-model-35cf45-kserve-64cf784dc5-vlm7n\" (UID: \"f08d64ea-b3e4-439d-bab2-03011a9683b8\") " pod="ai-tenant-e2e-noten-35cf45/noten-model-35cf45-kserve-64cf784dc5-vlm7n"
Sep 17 03:38:54.574150 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:54.574150    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f08d64ea-b3e4-439d-bab2-03011a9683b8-model-cache\") pod \"noten-model-35cf45-kserve-64cf784dc5-vlm7n\" (UID: \"f08d64ea-b3e4-439d-bab2-03011a9683b8\") " pod="ai-tenant-e2e-noten-35cf45/noten-model-35cf45-kserve-64cf784dc5-vlm7n"
Sep 17 03:38:54.574386 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:54.574180    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvk64\" (UniqueName: \"kubernetes.io/projected/f08d64ea-b3e4-439d-bab2-03011a9683b8-kube-api-access-gvk64\") pod \"noten-model-35cf45-kserve-64cf784dc5-vlm7n\" (UID: \"f08d64ea-b3e4-439d-bab2-03011a9683b8\") " pod="ai-tenant-e2e-noten-35cf45/noten-model-35cf45-kserve-64cf784dc5-vlm7n"
Sep 17 03:38:54.574386 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:54.574200    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f08d64ea-b3e4-439d-bab2-03011a9683b8-home\") pod \"noten-model-35cf45-kserve-64cf784dc5-vlm7n\" (UID: \"f08d64ea-b3e4-439d-bab2-03011a9683b8\") " pod="ai-tenant-e2e-noten-35cf45/noten-model-35cf45-kserve-64cf784dc5-vlm7n"
Sep 17 03:38:54.574386 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:54.574259    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f08d64ea-b3e4-439d-bab2-03011a9683b8-tls-certs\") pod \"noten-model-35cf45-kserve-64cf784dc5-vlm7n\" (UID: \"f08d64ea-b3e4-439d-bab2-03011a9683b8\") " pod="ai-tenant-e2e-noten-35cf45/noten-model-35cf45-kserve-64cf784dc5-vlm7n"
Sep 17 03:38:54.574386 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:54.574289    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f08d64ea-b3e4-439d-bab2-03011a9683b8-dshm\") pod \"noten-model-35cf45-kserve-64cf784dc5-vlm7n\" (UID: \"f08d64ea-b3e4-439d-bab2-03011a9683b8\") " pod="ai-tenant-e2e-noten-35cf45/noten-model-35cf45-kserve-64cf784dc5-vlm7n"
Sep 17 03:38:54.574637 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:54.574545    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f08d64ea-b3e4-439d-bab2-03011a9683b8-model-cache\") pod \"noten-model-35cf45-kserve-64cf784dc5-vlm7n\" (UID: \"f08d64ea-b3e4-439d-bab2-03011a9683b8\") " pod="ai-tenant-e2e-noten-35cf45/noten-model-35cf45-kserve-64cf784dc5-vlm7n"
Sep 17 03:38:54.574637 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:54.574625    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f08d64ea-b3e4-439d-bab2-03011a9683b8-tmp-dir\") pod \"noten-model-35cf45-kserve-64cf784dc5-vlm7n\" (UID: \"f08d64ea-b3e4-439d-bab2-03011a9683b8\") " pod="ai-tenant-e2e-noten-35cf45/noten-model-35cf45-kserve-64cf784dc5-vlm7n"
Sep 17 03:38:54.574762 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:54.574688    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f08d64ea-b3e4-439d-bab2-03011a9683b8-home\") pod \"noten-model-35cf45-kserve-64cf784dc5-vlm7n\" (UID: \"f08d64ea-b3e4-439d-bab2-03011a9683b8\") " pod="ai-tenant-e2e-noten-35cf45/noten-model-35cf45-kserve-64cf784dc5-vlm7n"
Sep 17 03:38:54.576549 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:54.576530    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f08d64ea-b3e4-439d-bab2-03011a9683b8-dshm\") pod \"noten-model-35cf45-kserve-64cf784dc5-vlm7n\" (UID: \"f08d64ea-b3e4-439d-bab2-03011a9683b8\") " pod="ai-tenant-e2e-noten-35cf45/noten-model-35cf45-kserve-64cf784dc5-vlm7n"
Sep 17 03:38:54.576929 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:54.576909    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f08d64ea-b3e4-439d-bab2-03011a9683b8-tls-certs\") pod \"noten-model-35cf45-kserve-64cf784dc5-vlm7n\" (UID: \"f08d64ea-b3e4-439d-bab2-03011a9683b8\") " pod="ai-tenant-e2e-noten-35cf45/noten-model-35cf45-kserve-64cf784dc5-vlm7n"
Sep 17 03:38:54.582259 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:54.582239    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvk64\" (UniqueName: \"kubernetes.io/projected/f08d64ea-b3e4-439d-bab2-03011a9683b8-kube-api-access-gvk64\") pod \"noten-model-35cf45-kserve-64cf784dc5-vlm7n\" (UID: \"f08d64ea-b3e4-439d-bab2-03011a9683b8\") " pod="ai-tenant-e2e-noten-35cf45/noten-model-35cf45-kserve-64cf784dc5-vlm7n"
Sep 17 03:38:54.690843 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:54.690747    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-35cf45/noten-model-35cf45-kserve-64cf784dc5-vlm7n"
Sep 17 03:38:54.964760 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:54.964731    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-35cf45/noten-model-35cf45-kserve-64cf784dc5-vlm7n"]
Sep 17 03:38:54.966442 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:38:54.966418    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf08d64ea_b3e4_439d_bab2_03011a9683b8.slice/crio-ff148db29cefb1d74d3094cdf53bee2161c1267f2cfbbe041e95f743e91a6b0d WatchSource:0}: Error finding container ff148db29cefb1d74d3094cdf53bee2161c1267f2cfbbe041e95f743e91a6b0d: Status 404 returned error can't find the container with id ff148db29cefb1d74d3094cdf53bee2161c1267f2cfbbe041e95f743e91a6b0d
Sep 17 03:38:55.108224 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:55.108187    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-35cf45/noten-model-35cf45-kserve-64cf784dc5-vlm7n" event={"ID":"f08d64ea-b3e4-439d-bab2-03011a9683b8","Type":"ContainerStarted","Data":"de5c2801b174991cfd7d73828fa7eed390bef3f2d090b19c876ce0ca274e9832"}
Sep 17 03:38:55.108411 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:55.108234    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-35cf45/noten-model-35cf45-kserve-64cf784dc5-vlm7n" event={"ID":"f08d64ea-b3e4-439d-bab2-03011a9683b8","Type":"ContainerStarted","Data":"ff148db29cefb1d74d3094cdf53bee2161c1267f2cfbbe041e95f743e91a6b0d"}
Sep 17 03:38:55.125402 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:38:55.125351    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-noten-35cf45/noten-model-35cf45-kserve-64cf784dc5-vlm7n" podStartSLOduration=1.125335958 podStartE2EDuration="1.125335958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 03:38:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:38:55.123388705 +0000 UTC m=+1170.589796233" watchObservedRunningTime="2026-09-17 03:38:55.125335958 +0000 UTC m=+1170.591743485"
Sep 17 03:39:04.691772 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:04.691725    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-noten-35cf45/noten-model-35cf45-kserve-64cf784dc5-vlm7n"
Sep 17 03:39:04.692159 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:04.691784    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-35cf45/noten-model-35cf45-kserve-64cf784dc5-vlm7n"
Sep 17 03:39:04.704429 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:04.704396    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-noten-35cf45/noten-model-35cf45-kserve-64cf784dc5-vlm7n"
Sep 17 03:39:05.173012 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:05.172984    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-noten-35cf45/noten-model-35cf45-kserve-64cf784dc5-vlm7n"
Sep 17 03:39:10.779668 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:10.779631    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-3d0c3c-openshift-default-7bb86f7b56-zbt2v"]
Sep 17 03:39:10.780260 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:10.779912    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-3d0c3c-openshift-default-7bb86f7b56-zbt2v" podUID="8b5b7160-05b5-4218-ab7b-0e7cdf8383a2" containerName="istio-proxy" containerID="cri-o://09cb90acbcd3bcf39f778c365bde47570dcefbd34599ff8643894e330642d123" gracePeriod=30
Sep 17 03:39:11.672942 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:11.672908    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-22e045-openshift-default-5f598cd8fb-zz7g6"]
Sep 17 03:39:11.676725 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:11.676709    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-22e045-openshift-default-5f598cd8fb-zz7g6"
Sep 17 03:39:11.679215 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:11.679193    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-22e045-openshift-default-dockercfg-csqb2\""
Sep 17 03:39:11.686009 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:11.685986    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-22e045-openshift-default-5f598cd8fb-zz7g6"]
Sep 17 03:39:11.757946 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:11.757910    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67x7d\" (UniqueName: \"kubernetes.io/projected/c4d4225c-5283-4143-a89e-9c2fbc0eb172-kube-api-access-67x7d\") pod \"e2e-shared-a-w3-22e045-openshift-default-5f598cd8fb-zz7g6\" (UID: \"c4d4225c-5283-4143-a89e-9c2fbc0eb172\") " pod="openshift-ingress/e2e-shared-a-w3-22e045-openshift-default-5f598cd8fb-zz7g6"
Sep 17 03:39:11.758147 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:11.757969    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4d4225c-5283-4143-a89e-9c2fbc0eb172-credential-socket\") pod \"e2e-shared-a-w3-22e045-openshift-default-5f598cd8fb-zz7g6\" (UID: \"c4d4225c-5283-4143-a89e-9c2fbc0eb172\") " pod="openshift-ingress/e2e-shared-a-w3-22e045-openshift-default-5f598cd8fb-zz7g6"
Sep 17 03:39:11.758147 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:11.757994    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4d4225c-5283-4143-a89e-9c2fbc0eb172-workload-certs\") pod \"e2e-shared-a-w3-22e045-openshift-default-5f598cd8fb-zz7g6\" (UID: \"c4d4225c-5283-4143-a89e-9c2fbc0eb172\") " pod="openshift-ingress/e2e-shared-a-w3-22e045-openshift-default-5f598cd8fb-zz7g6"
Sep 17 03:39:11.758147 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:11.758015    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4d4225c-5283-4143-a89e-9c2fbc0eb172-istio-envoy\") pod \"e2e-shared-a-w3-22e045-openshift-default-5f598cd8fb-zz7g6\" (UID: \"c4d4225c-5283-4143-a89e-9c2fbc0eb172\") " pod="openshift-ingress/e2e-shared-a-w3-22e045-openshift-default-5f598cd8fb-zz7g6"
Sep 17 03:39:11.758147 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:11.758089    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4d4225c-5283-4143-a89e-9c2fbc0eb172-istiod-ca-cert\") pod \"e2e-shared-a-w3-22e045-openshift-default-5f598cd8fb-zz7g6\" (UID: \"c4d4225c-5283-4143-a89e-9c2fbc0eb172\") " pod="openshift-ingress/e2e-shared-a-w3-22e045-openshift-default-5f598cd8fb-zz7g6"
Sep 17 03:39:11.758331 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:11.758187    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4d4225c-5283-4143-a89e-9c2fbc0eb172-istio-podinfo\") pod \"e2e-shared-a-w3-22e045-openshift-default-5f598cd8fb-zz7g6\" (UID: \"c4d4225c-5283-4143-a89e-9c2fbc0eb172\") " pod="openshift-ingress/e2e-shared-a-w3-22e045-openshift-default-5f598cd8fb-zz7g6"
Sep 17 03:39:11.758331 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:11.758224    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4d4225c-5283-4143-a89e-9c2fbc0eb172-istio-data\") pod \"e2e-shared-a-w3-22e045-openshift-default-5f598cd8fb-zz7g6\" (UID: \"c4d4225c-5283-4143-a89e-9c2fbc0eb172\") " pod="openshift-ingress/e2e-shared-a-w3-22e045-openshift-default-5f598cd8fb-zz7g6"
Sep 17 03:39:11.758331 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:11.758245    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4d4225c-5283-4143-a89e-9c2fbc0eb172-istio-token\") pod \"e2e-shared-a-w3-22e045-openshift-default-5f598cd8fb-zz7g6\" (UID: \"c4d4225c-5283-4143-a89e-9c2fbc0eb172\") " pod="openshift-ingress/e2e-shared-a-w3-22e045-openshift-default-5f598cd8fb-zz7g6"
Sep 17 03:39:11.758331 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:11.758323    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4d4225c-5283-4143-a89e-9c2fbc0eb172-workload-socket\") pod \"e2e-shared-a-w3-22e045-openshift-default-5f598cd8fb-zz7g6\" (UID: \"c4d4225c-5283-4143-a89e-9c2fbc0eb172\") " pod="openshift-ingress/e2e-shared-a-w3-22e045-openshift-default-5f598cd8fb-zz7g6"
Sep 17 03:39:11.859094 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:11.859054    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4d4225c-5283-4143-a89e-9c2fbc0eb172-istiod-ca-cert\") pod \"e2e-shared-a-w3-22e045-openshift-default-5f598cd8fb-zz7g6\" (UID: \"c4d4225c-5283-4143-a89e-9c2fbc0eb172\") " pod="openshift-ingress/e2e-shared-a-w3-22e045-openshift-default-5f598cd8fb-zz7g6"
Sep 17 03:39:11.859543 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:11.859112    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4d4225c-5283-4143-a89e-9c2fbc0eb172-istio-podinfo\") pod \"e2e-shared-a-w3-22e045-openshift-default-5f598cd8fb-zz7g6\" (UID: \"c4d4225c-5283-4143-a89e-9c2fbc0eb172\") " pod="openshift-ingress/e2e-shared-a-w3-22e045-openshift-default-5f598cd8fb-zz7g6"
Sep 17 03:39:11.859543 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:11.859137    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4d4225c-5283-4143-a89e-9c2fbc0eb172-istio-data\") pod \"e2e-shared-a-w3-22e045-openshift-default-5f598cd8fb-zz7g6\" (UID: \"c4d4225c-5283-4143-a89e-9c2fbc0eb172\") " pod="openshift-ingress/e2e-shared-a-w3-22e045-openshift-default-5f598cd8fb-zz7g6"
Sep 17 03:39:11.859543 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:11.859154    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4d4225c-5283-4143-a89e-9c2fbc0eb172-istio-token\") pod \"e2e-shared-a-w3-22e045-openshift-default-5f598cd8fb-zz7g6\" (UID: \"c4d4225c-5283-4143-a89e-9c2fbc0eb172\") " pod="openshift-ingress/e2e-shared-a-w3-22e045-openshift-default-5f598cd8fb-zz7g6"
Sep 17 03:39:11.859543 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:11.859192    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4d4225c-5283-4143-a89e-9c2fbc0eb172-workload-socket\") pod \"e2e-shared-a-w3-22e045-openshift-default-5f598cd8fb-zz7g6\" (UID: \"c4d4225c-5283-4143-a89e-9c2fbc0eb172\") " pod="openshift-ingress/e2e-shared-a-w3-22e045-openshift-default-5f598cd8fb-zz7g6"
Sep 17 03:39:11.859543 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:11.859230    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67x7d\" (UniqueName: \"kubernetes.io/projected/c4d4225c-5283-4143-a89e-9c2fbc0eb172-kube-api-access-67x7d\") pod \"e2e-shared-a-w3-22e045-openshift-default-5f598cd8fb-zz7g6\" (UID: \"c4d4225c-5283-4143-a89e-9c2fbc0eb172\") " pod="openshift-ingress/e2e-shared-a-w3-22e045-openshift-default-5f598cd8fb-zz7g6"
Sep 17 03:39:11.859543 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:11.859269    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4d4225c-5283-4143-a89e-9c2fbc0eb172-credential-socket\") pod \"e2e-shared-a-w3-22e045-openshift-default-5f598cd8fb-zz7g6\" (UID: \"c4d4225c-5283-4143-a89e-9c2fbc0eb172\") " pod="openshift-ingress/e2e-shared-a-w3-22e045-openshift-default-5f598cd8fb-zz7g6"
Sep 17 03:39:11.859543 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:11.859290    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4d4225c-5283-4143-a89e-9c2fbc0eb172-workload-certs\") pod \"e2e-shared-a-w3-22e045-openshift-default-5f598cd8fb-zz7g6\" (UID: \"c4d4225c-5283-4143-a89e-9c2fbc0eb172\") " pod="openshift-ingress/e2e-shared-a-w3-22e045-openshift-default-5f598cd8fb-zz7g6"
Sep 17 03:39:11.859543 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:11.859311    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4d4225c-5283-4143-a89e-9c2fbc0eb172-istio-envoy\") pod \"e2e-shared-a-w3-22e045-openshift-default-5f598cd8fb-zz7g6\" (UID: \"c4d4225c-5283-4143-a89e-9c2fbc0eb172\") " pod="openshift-ingress/e2e-shared-a-w3-22e045-openshift-default-5f598cd8fb-zz7g6"
Sep 17 03:39:11.859963 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:11.859653    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4d4225c-5283-4143-a89e-9c2fbc0eb172-workload-socket\") pod \"e2e-shared-a-w3-22e045-openshift-default-5f598cd8fb-zz7g6\" (UID: \"c4d4225c-5283-4143-a89e-9c2fbc0eb172\") " pod="openshift-ingress/e2e-shared-a-w3-22e045-openshift-default-5f598cd8fb-zz7g6"
Sep 17 03:39:11.859963 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:11.859770    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4d4225c-5283-4143-a89e-9c2fbc0eb172-credential-socket\") pod \"e2e-shared-a-w3-22e045-openshift-default-5f598cd8fb-zz7g6\" (UID: \"c4d4225c-5283-4143-a89e-9c2fbc0eb172\") " pod="openshift-ingress/e2e-shared-a-w3-22e045-openshift-default-5f598cd8fb-zz7g6"
Sep 17 03:39:11.859963 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:11.859788    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4d4225c-5283-4143-a89e-9c2fbc0eb172-workload-certs\") pod \"e2e-shared-a-w3-22e045-openshift-default-5f598cd8fb-zz7g6\" (UID: \"c4d4225c-5283-4143-a89e-9c2fbc0eb172\") " pod="openshift-ingress/e2e-shared-a-w3-22e045-openshift-default-5f598cd8fb-zz7g6"
Sep 17 03:39:11.859963 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:11.859901    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4d4225c-5283-4143-a89e-9c2fbc0eb172-istio-data\") pod \"e2e-shared-a-w3-22e045-openshift-default-5f598cd8fb-zz7g6\" (UID: \"c4d4225c-5283-4143-a89e-9c2fbc0eb172\") " pod="openshift-ingress/e2e-shared-a-w3-22e045-openshift-default-5f598cd8fb-zz7g6"
Sep 17 03:39:11.859963 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:11.859918    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4d4225c-5283-4143-a89e-9c2fbc0eb172-istiod-ca-cert\") pod \"e2e-shared-a-w3-22e045-openshift-default-5f598cd8fb-zz7g6\" (UID: \"c4d4225c-5283-4143-a89e-9c2fbc0eb172\") " pod="openshift-ingress/e2e-shared-a-w3-22e045-openshift-default-5f598cd8fb-zz7g6"
Sep 17 03:39:11.861697 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:11.861676    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4d4225c-5283-4143-a89e-9c2fbc0eb172-istio-envoy\") pod \"e2e-shared-a-w3-22e045-openshift-default-5f598cd8fb-zz7g6\" (UID: \"c4d4225c-5283-4143-a89e-9c2fbc0eb172\") " pod="openshift-ingress/e2e-shared-a-w3-22e045-openshift-default-5f598cd8fb-zz7g6"
Sep 17 03:39:11.861805 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:11.861777    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4d4225c-5283-4143-a89e-9c2fbc0eb172-istio-podinfo\") pod \"e2e-shared-a-w3-22e045-openshift-default-5f598cd8fb-zz7g6\" (UID: \"c4d4225c-5283-4143-a89e-9c2fbc0eb172\") " pod="openshift-ingress/e2e-shared-a-w3-22e045-openshift-default-5f598cd8fb-zz7g6"
Sep 17 03:39:11.866720 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:11.866699    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4d4225c-5283-4143-a89e-9c2fbc0eb172-istio-token\") pod \"e2e-shared-a-w3-22e045-openshift-default-5f598cd8fb-zz7g6\" (UID: \"c4d4225c-5283-4143-a89e-9c2fbc0eb172\") " pod="openshift-ingress/e2e-shared-a-w3-22e045-openshift-default-5f598cd8fb-zz7g6"
Sep 17 03:39:11.866836 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:11.866785    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-67x7d\" (UniqueName: \"kubernetes.io/projected/c4d4225c-5283-4143-a89e-9c2fbc0eb172-kube-api-access-67x7d\") pod \"e2e-shared-a-w3-22e045-openshift-default-5f598cd8fb-zz7g6\" (UID: \"c4d4225c-5283-4143-a89e-9c2fbc0eb172\") " pod="openshift-ingress/e2e-shared-a-w3-22e045-openshift-default-5f598cd8fb-zz7g6"
Sep 17 03:39:11.988693 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:11.988569    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-22e045-openshift-default-5f598cd8fb-zz7g6"
Sep 17 03:39:12.266963 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:12.266936    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-22e045-openshift-default-5f598cd8fb-zz7g6"]
Sep 17 03:39:12.268969 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:39:12.268942    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4d4225c_5283_4143_a89e_9c2fbc0eb172.slice/crio-1736feeec2b6eeb24a6d1fde1a81775f37bed8ed84566f9fe2c856327cf1fba1 WatchSource:0}: Error finding container 1736feeec2b6eeb24a6d1fde1a81775f37bed8ed84566f9fe2c856327cf1fba1: Status 404 returned error can't find the container with id 1736feeec2b6eeb24a6d1fde1a81775f37bed8ed84566f9fe2c856327cf1fba1
Sep 17 03:39:12.271195 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:12.271159    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 03:39:12.271298 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:12.271224    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 03:39:12.271298 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:12.271259    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 03:39:13.213385 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:13.213350    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-22e045-openshift-default-5f598cd8fb-zz7g6" event={"ID":"c4d4225c-5283-4143-a89e-9c2fbc0eb172","Type":"ContainerStarted","Data":"e1b6f851237769b107bbdca0c10b962d54e00b828de8f82dbfbc38d4138b2d4f"}
Sep 17 03:39:13.213823 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:13.213402    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-22e045-openshift-default-5f598cd8fb-zz7g6" event={"ID":"c4d4225c-5283-4143-a89e-9c2fbc0eb172","Type":"ContainerStarted","Data":"1736feeec2b6eeb24a6d1fde1a81775f37bed8ed84566f9fe2c856327cf1fba1"}
Sep 17 03:39:13.233134 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:13.233080    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-22e045-openshift-default-5f598cd8fb-zz7g6" podStartSLOduration=2.233064596 podStartE2EDuration="2.233064596s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 03:39:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:39:13.231943547 +0000 UTC m=+1188.698351073" watchObservedRunningTime="2026-09-17 03:39:13.233064596 +0000 UTC m=+1188.699472133"
Sep 17 03:39:13.989754 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:13.989710    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-22e045-openshift-default-5f598cd8fb-zz7g6"
Sep 17 03:39:13.994970 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:13.994944    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-22e045-openshift-default-5f598cd8fb-zz7g6"
Sep 17 03:39:14.219083 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:14.219051    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-22e045-openshift-default-5f598cd8fb-zz7g6"
Sep 17 03:39:14.220080 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:14.220060    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-22e045-openshift-default-5f598cd8fb-zz7g6"
Sep 17 03:39:16.032739 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:16.032715    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-3d0c3c-openshift-default-7bb86f7b56-zbt2v"
Sep 17 03:39:16.100302 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:16.100219    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b5b7160-05b5-4218-ab7b-0e7cdf8383a2-istio-data\") pod \"8b5b7160-05b5-4218-ab7b-0e7cdf8383a2\" (UID: \"8b5b7160-05b5-4218-ab7b-0e7cdf8383a2\") "
Sep 17 03:39:16.100302 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:16.100262    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-scppn\" (UniqueName: \"kubernetes.io/projected/8b5b7160-05b5-4218-ab7b-0e7cdf8383a2-kube-api-access-scppn\") pod \"8b5b7160-05b5-4218-ab7b-0e7cdf8383a2\" (UID: \"8b5b7160-05b5-4218-ab7b-0e7cdf8383a2\") "
Sep 17 03:39:16.100302 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:16.100290    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b5b7160-05b5-4218-ab7b-0e7cdf8383a2-istiod-ca-cert\") pod \"8b5b7160-05b5-4218-ab7b-0e7cdf8383a2\" (UID: \"8b5b7160-05b5-4218-ab7b-0e7cdf8383a2\") "
Sep 17 03:39:16.100569 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:16.100330    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b5b7160-05b5-4218-ab7b-0e7cdf8383a2-istio-token\") pod \"8b5b7160-05b5-4218-ab7b-0e7cdf8383a2\" (UID: \"8b5b7160-05b5-4218-ab7b-0e7cdf8383a2\") "
Sep 17 03:39:16.100569 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:16.100350    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b5b7160-05b5-4218-ab7b-0e7cdf8383a2-workload-socket\") pod \"8b5b7160-05b5-4218-ab7b-0e7cdf8383a2\" (UID: \"8b5b7160-05b5-4218-ab7b-0e7cdf8383a2\") "
Sep 17 03:39:16.100569 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:16.100405    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b5b7160-05b5-4218-ab7b-0e7cdf8383a2-istio-podinfo\") pod \"8b5b7160-05b5-4218-ab7b-0e7cdf8383a2\" (UID: \"8b5b7160-05b5-4218-ab7b-0e7cdf8383a2\") "
Sep 17 03:39:16.100569 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:16.100429    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b5b7160-05b5-4218-ab7b-0e7cdf8383a2-istio-envoy\") pod \"8b5b7160-05b5-4218-ab7b-0e7cdf8383a2\" (UID: \"8b5b7160-05b5-4218-ab7b-0e7cdf8383a2\") "
Sep 17 03:39:16.100569 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:16.100464    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b5b7160-05b5-4218-ab7b-0e7cdf8383a2-workload-certs\") pod \"8b5b7160-05b5-4218-ab7b-0e7cdf8383a2\" (UID: \"8b5b7160-05b5-4218-ab7b-0e7cdf8383a2\") "
Sep 17 03:39:16.100569 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:16.100505    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b5b7160-05b5-4218-ab7b-0e7cdf8383a2-credential-socket\") pod \"8b5b7160-05b5-4218-ab7b-0e7cdf8383a2\" (UID: \"8b5b7160-05b5-4218-ab7b-0e7cdf8383a2\") "
Sep 17 03:39:16.100895 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:16.100591    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b5b7160-05b5-4218-ab7b-0e7cdf8383a2-istio-data" (OuterVolumeSpecName: "istio-data") pod "8b5b7160-05b5-4218-ab7b-0e7cdf8383a2" (UID: "8b5b7160-05b5-4218-ab7b-0e7cdf8383a2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:39:16.100895 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:16.100704    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b5b7160-05b5-4218-ab7b-0e7cdf8383a2-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8b5b7160-05b5-4218-ab7b-0e7cdf8383a2" (UID: "8b5b7160-05b5-4218-ab7b-0e7cdf8383a2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:39:16.100895 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:16.100784    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b5b7160-05b5-4218-ab7b-0e7cdf8383a2-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8b5b7160-05b5-4218-ab7b-0e7cdf8383a2" (UID: "8b5b7160-05b5-4218-ab7b-0e7cdf8383a2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 03:39:16.100895 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:16.100857    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b5b7160-05b5-4218-ab7b-0e7cdf8383a2-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8b5b7160-05b5-4218-ab7b-0e7cdf8383a2" (UID: "8b5b7160-05b5-4218-ab7b-0e7cdf8383a2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:39:16.101076 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:16.101040    2552 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b5b7160-05b5-4218-ab7b-0e7cdf8383a2-workload-certs\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:39:16.101076 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:16.101059    2552 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b5b7160-05b5-4218-ab7b-0e7cdf8383a2-istio-data\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:39:16.101076 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:16.101071    2552 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b5b7160-05b5-4218-ab7b-0e7cdf8383a2-istiod-ca-cert\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:39:16.101221 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:16.101083    2552 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b5b7160-05b5-4218-ab7b-0e7cdf8383a2-workload-socket\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:39:16.101221 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:16.101116    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b5b7160-05b5-4218-ab7b-0e7cdf8383a2-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8b5b7160-05b5-4218-ab7b-0e7cdf8383a2" (UID: "8b5b7160-05b5-4218-ab7b-0e7cdf8383a2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:39:16.102686 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:16.102654    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8b5b7160-05b5-4218-ab7b-0e7cdf8383a2-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8b5b7160-05b5-4218-ab7b-0e7cdf8383a2" (UID: "8b5b7160-05b5-4218-ab7b-0e7cdf8383a2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 17 03:39:16.102819 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:16.102693    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b5b7160-05b5-4218-ab7b-0e7cdf8383a2-kube-api-access-scppn" (OuterVolumeSpecName: "kube-api-access-scppn") pod "8b5b7160-05b5-4218-ab7b-0e7cdf8383a2" (UID: "8b5b7160-05b5-4218-ab7b-0e7cdf8383a2"). InnerVolumeSpecName "kube-api-access-scppn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 03:39:16.102955 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:16.102941    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b5b7160-05b5-4218-ab7b-0e7cdf8383a2-istio-token" (OuterVolumeSpecName: "istio-token") pod "8b5b7160-05b5-4218-ab7b-0e7cdf8383a2" (UID: "8b5b7160-05b5-4218-ab7b-0e7cdf8383a2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 03:39:16.102999 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:16.102989    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b5b7160-05b5-4218-ab7b-0e7cdf8383a2-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8b5b7160-05b5-4218-ab7b-0e7cdf8383a2" (UID: "8b5b7160-05b5-4218-ab7b-0e7cdf8383a2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:39:16.201959 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:16.201925    2552 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b5b7160-05b5-4218-ab7b-0e7cdf8383a2-istio-podinfo\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:39:16.201959 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:16.201954    2552 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b5b7160-05b5-4218-ab7b-0e7cdf8383a2-istio-envoy\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:39:16.201959 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:16.201964    2552 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b5b7160-05b5-4218-ab7b-0e7cdf8383a2-credential-socket\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:39:16.202244 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:16.201974    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-scppn\" (UniqueName: \"kubernetes.io/projected/8b5b7160-05b5-4218-ab7b-0e7cdf8383a2-kube-api-access-scppn\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:39:16.202244 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:16.201982    2552 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b5b7160-05b5-4218-ab7b-0e7cdf8383a2-istio-token\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:39:16.231488 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:16.231454    2552 generic.go:356] "Generic (PLEG): container finished" podID="8b5b7160-05b5-4218-ab7b-0e7cdf8383a2" containerID="09cb90acbcd3bcf39f778c365bde47570dcefbd34599ff8643894e330642d123" exitCode=0
Sep 17 03:39:16.231658 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:16.231500    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-3d0c3c-openshift-default-7bb86f7b56-zbt2v"
Sep 17 03:39:16.231658 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:16.231537    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-3d0c3c-openshift-default-7bb86f7b56-zbt2v" event={"ID":"8b5b7160-05b5-4218-ab7b-0e7cdf8383a2","Type":"ContainerDied","Data":"09cb90acbcd3bcf39f778c365bde47570dcefbd34599ff8643894e330642d123"}
Sep 17 03:39:16.231658 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:16.231616    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-3d0c3c-openshift-default-7bb86f7b56-zbt2v" event={"ID":"8b5b7160-05b5-4218-ab7b-0e7cdf8383a2","Type":"ContainerDied","Data":"16aa2262f9f573dda9994e330948dfccbd7c815aac35dce92cea346119604ff6"}
Sep 17 03:39:16.231658 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:16.231648    2552 scope.go:117] "RemoveContainer" containerID="09cb90acbcd3bcf39f778c365bde47570dcefbd34599ff8643894e330642d123"
Sep 17 03:39:16.241807 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:16.241790    2552 scope.go:117] "RemoveContainer" containerID="1d6e97980462193c02b6a8371658c294ac8ab6b69706403ad3c2f021f8908caf"
Sep 17 03:39:16.250329 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:16.250313    2552 scope.go:117] "RemoveContainer" containerID="09cb90acbcd3bcf39f778c365bde47570dcefbd34599ff8643894e330642d123"
Sep 17 03:39:16.250643 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:39:16.250616    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09cb90acbcd3bcf39f778c365bde47570dcefbd34599ff8643894e330642d123\": container with ID starting with 09cb90acbcd3bcf39f778c365bde47570dcefbd34599ff8643894e330642d123 not found: ID does not exist" containerID="09cb90acbcd3bcf39f778c365bde47570dcefbd34599ff8643894e330642d123"
Sep 17 03:39:16.250770 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:16.250649    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09cb90acbcd3bcf39f778c365bde47570dcefbd34599ff8643894e330642d123"} err="failed to get container status \"09cb90acbcd3bcf39f778c365bde47570dcefbd34599ff8643894e330642d123\": rpc error: code = NotFound desc = could not find container \"09cb90acbcd3bcf39f778c365bde47570dcefbd34599ff8643894e330642d123\": container with ID starting with 09cb90acbcd3bcf39f778c365bde47570dcefbd34599ff8643894e330642d123 not found: ID does not exist"
Sep 17 03:39:16.250770 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:16.250673    2552 scope.go:117] "RemoveContainer" containerID="1d6e97980462193c02b6a8371658c294ac8ab6b69706403ad3c2f021f8908caf"
Sep 17 03:39:16.250990 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:39:16.250969    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d6e97980462193c02b6a8371658c294ac8ab6b69706403ad3c2f021f8908caf\": container with ID starting with 1d6e97980462193c02b6a8371658c294ac8ab6b69706403ad3c2f021f8908caf not found: ID does not exist" containerID="1d6e97980462193c02b6a8371658c294ac8ab6b69706403ad3c2f021f8908caf"
Sep 17 03:39:16.251085 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:16.250996    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d6e97980462193c02b6a8371658c294ac8ab6b69706403ad3c2f021f8908caf"} err="failed to get container status \"1d6e97980462193c02b6a8371658c294ac8ab6b69706403ad3c2f021f8908caf\": rpc error: code = NotFound desc = could not find container \"1d6e97980462193c02b6a8371658c294ac8ab6b69706403ad3c2f021f8908caf\": container with ID starting with 1d6e97980462193c02b6a8371658c294ac8ab6b69706403ad3c2f021f8908caf not found: ID does not exist"
Sep 17 03:39:16.253859 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:16.253838    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-3d0c3c-openshift-default-7bb86f7b56-zbt2v"]
Sep 17 03:39:16.257490 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:16.257471    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-3d0c3c-openshift-default-7bb86f7b56-zbt2v"]
Sep 17 03:39:16.986524 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:16.986489    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8b5b7160-05b5-4218-ab7b-0e7cdf8383a2" path="/var/lib/kubelet/pods/8b5b7160-05b5-4218-ab7b-0e7cdf8383a2/volumes"
Sep 17 03:39:21.764848 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:21.764812    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-1afd34/distinct-2-w0-1afd34-kserve-5457449c9-bqjmv"]
Sep 17 03:39:21.765248 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:21.765234    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b5b7160-05b5-4218-ab7b-0e7cdf8383a2" containerName="istio-proxy"
Sep 17 03:39:21.765248 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:21.765249    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b5b7160-05b5-4218-ab7b-0e7cdf8383a2" containerName="istio-proxy"
Sep 17 03:39:21.765334 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:21.765325    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b5b7160-05b5-4218-ab7b-0e7cdf8383a2" containerName="istio-proxy"
Sep 17 03:39:21.769765 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:21.769736    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1afd34/distinct-2-w0-1afd34-kserve-5457449c9-bqjmv"
Sep 17 03:39:21.773711 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:21.773688    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-1afd34\"/\"distinct-2-w0-1afd34-kserve-self-signed-certs\""
Sep 17 03:39:21.778807 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:21.778785    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-1afd34/distinct-2-w0-1afd34-kserve-5457449c9-bqjmv"]
Sep 17 03:39:21.861949 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:21.861915    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a2ec719-0a31-439c-96af-ee004af31201-dshm\") pod \"distinct-2-w0-1afd34-kserve-5457449c9-bqjmv\" (UID: \"2a2ec719-0a31-439c-96af-ee004af31201\") " pod="e2e-models-e2e-worker-w0-1afd34/distinct-2-w0-1afd34-kserve-5457449c9-bqjmv"
Sep 17 03:39:21.862139 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:21.861982    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a2ec719-0a31-439c-96af-ee004af31201-model-cache\") pod \"distinct-2-w0-1afd34-kserve-5457449c9-bqjmv\" (UID: \"2a2ec719-0a31-439c-96af-ee004af31201\") " pod="e2e-models-e2e-worker-w0-1afd34/distinct-2-w0-1afd34-kserve-5457449c9-bqjmv"
Sep 17 03:39:21.862139 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:21.862014    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nwlc\" (UniqueName: \"kubernetes.io/projected/2a2ec719-0a31-439c-96af-ee004af31201-kube-api-access-4nwlc\") pod \"distinct-2-w0-1afd34-kserve-5457449c9-bqjmv\" (UID: \"2a2ec719-0a31-439c-96af-ee004af31201\") " pod="e2e-models-e2e-worker-w0-1afd34/distinct-2-w0-1afd34-kserve-5457449c9-bqjmv"
Sep 17 03:39:21.862139 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:21.862043    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a2ec719-0a31-439c-96af-ee004af31201-home\") pod \"distinct-2-w0-1afd34-kserve-5457449c9-bqjmv\" (UID: \"2a2ec719-0a31-439c-96af-ee004af31201\") " pod="e2e-models-e2e-worker-w0-1afd34/distinct-2-w0-1afd34-kserve-5457449c9-bqjmv"
Sep 17 03:39:21.862265 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:21.862184    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a2ec719-0a31-439c-96af-ee004af31201-tmp-dir\") pod \"distinct-2-w0-1afd34-kserve-5457449c9-bqjmv\" (UID: \"2a2ec719-0a31-439c-96af-ee004af31201\") " pod="e2e-models-e2e-worker-w0-1afd34/distinct-2-w0-1afd34-kserve-5457449c9-bqjmv"
Sep 17 03:39:21.862265 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:21.862218    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a2ec719-0a31-439c-96af-ee004af31201-tls-certs\") pod \"distinct-2-w0-1afd34-kserve-5457449c9-bqjmv\" (UID: \"2a2ec719-0a31-439c-96af-ee004af31201\") " pod="e2e-models-e2e-worker-w0-1afd34/distinct-2-w0-1afd34-kserve-5457449c9-bqjmv"
Sep 17 03:39:21.963534 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:21.963492    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a2ec719-0a31-439c-96af-ee004af31201-model-cache\") pod \"distinct-2-w0-1afd34-kserve-5457449c9-bqjmv\" (UID: \"2a2ec719-0a31-439c-96af-ee004af31201\") " pod="e2e-models-e2e-worker-w0-1afd34/distinct-2-w0-1afd34-kserve-5457449c9-bqjmv"
Sep 17 03:39:21.963534 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:21.963536    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nwlc\" (UniqueName: \"kubernetes.io/projected/2a2ec719-0a31-439c-96af-ee004af31201-kube-api-access-4nwlc\") pod \"distinct-2-w0-1afd34-kserve-5457449c9-bqjmv\" (UID: \"2a2ec719-0a31-439c-96af-ee004af31201\") " pod="e2e-models-e2e-worker-w0-1afd34/distinct-2-w0-1afd34-kserve-5457449c9-bqjmv"
Sep 17 03:39:21.963812 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:21.963572    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a2ec719-0a31-439c-96af-ee004af31201-home\") pod \"distinct-2-w0-1afd34-kserve-5457449c9-bqjmv\" (UID: \"2a2ec719-0a31-439c-96af-ee004af31201\") " pod="e2e-models-e2e-worker-w0-1afd34/distinct-2-w0-1afd34-kserve-5457449c9-bqjmv"
Sep 17 03:39:21.963812 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:21.963660    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a2ec719-0a31-439c-96af-ee004af31201-tmp-dir\") pod \"distinct-2-w0-1afd34-kserve-5457449c9-bqjmv\" (UID: \"2a2ec719-0a31-439c-96af-ee004af31201\") " pod="e2e-models-e2e-worker-w0-1afd34/distinct-2-w0-1afd34-kserve-5457449c9-bqjmv"
Sep 17 03:39:21.963812 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:21.963686    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a2ec719-0a31-439c-96af-ee004af31201-tls-certs\") pod \"distinct-2-w0-1afd34-kserve-5457449c9-bqjmv\" (UID: \"2a2ec719-0a31-439c-96af-ee004af31201\") " pod="e2e-models-e2e-worker-w0-1afd34/distinct-2-w0-1afd34-kserve-5457449c9-bqjmv"
Sep 17 03:39:21.963812 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:21.963750    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a2ec719-0a31-439c-96af-ee004af31201-dshm\") pod \"distinct-2-w0-1afd34-kserve-5457449c9-bqjmv\" (UID: \"2a2ec719-0a31-439c-96af-ee004af31201\") " pod="e2e-models-e2e-worker-w0-1afd34/distinct-2-w0-1afd34-kserve-5457449c9-bqjmv"
Sep 17 03:39:21.964005 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:21.963979    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a2ec719-0a31-439c-96af-ee004af31201-model-cache\") pod \"distinct-2-w0-1afd34-kserve-5457449c9-bqjmv\" (UID: \"2a2ec719-0a31-439c-96af-ee004af31201\") " pod="e2e-models-e2e-worker-w0-1afd34/distinct-2-w0-1afd34-kserve-5457449c9-bqjmv"
Sep 17 03:39:21.964042 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:21.964011    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a2ec719-0a31-439c-96af-ee004af31201-home\") pod \"distinct-2-w0-1afd34-kserve-5457449c9-bqjmv\" (UID: \"2a2ec719-0a31-439c-96af-ee004af31201\") " pod="e2e-models-e2e-worker-w0-1afd34/distinct-2-w0-1afd34-kserve-5457449c9-bqjmv"
Sep 17 03:39:21.964077 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:21.964056    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a2ec719-0a31-439c-96af-ee004af31201-tmp-dir\") pod \"distinct-2-w0-1afd34-kserve-5457449c9-bqjmv\" (UID: \"2a2ec719-0a31-439c-96af-ee004af31201\") " pod="e2e-models-e2e-worker-w0-1afd34/distinct-2-w0-1afd34-kserve-5457449c9-bqjmv"
Sep 17 03:39:21.966052 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:21.966028    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a2ec719-0a31-439c-96af-ee004af31201-dshm\") pod \"distinct-2-w0-1afd34-kserve-5457449c9-bqjmv\" (UID: \"2a2ec719-0a31-439c-96af-ee004af31201\") " pod="e2e-models-e2e-worker-w0-1afd34/distinct-2-w0-1afd34-kserve-5457449c9-bqjmv"
Sep 17 03:39:21.966350 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:21.966328    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a2ec719-0a31-439c-96af-ee004af31201-tls-certs\") pod \"distinct-2-w0-1afd34-kserve-5457449c9-bqjmv\" (UID: \"2a2ec719-0a31-439c-96af-ee004af31201\") " pod="e2e-models-e2e-worker-w0-1afd34/distinct-2-w0-1afd34-kserve-5457449c9-bqjmv"
Sep 17 03:39:21.971579 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:21.971557    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nwlc\" (UniqueName: \"kubernetes.io/projected/2a2ec719-0a31-439c-96af-ee004af31201-kube-api-access-4nwlc\") pod \"distinct-2-w0-1afd34-kserve-5457449c9-bqjmv\" (UID: \"2a2ec719-0a31-439c-96af-ee004af31201\") " pod="e2e-models-e2e-worker-w0-1afd34/distinct-2-w0-1afd34-kserve-5457449c9-bqjmv"
Sep 17 03:39:22.079271 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:22.079182    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1afd34/distinct-2-w0-1afd34-kserve-5457449c9-bqjmv"
Sep 17 03:39:22.357732 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:22.357707    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-1afd34/distinct-2-w0-1afd34-kserve-5457449c9-bqjmv"]
Sep 17 03:39:22.359216 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:39:22.359180    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2a2ec719_0a31_439c_96af_ee004af31201.slice/crio-efb7d5c0cd0eb006278e9cd092ec10884cab6fd87573821675326c555c3eb241 WatchSource:0}: Error finding container efb7d5c0cd0eb006278e9cd092ec10884cab6fd87573821675326c555c3eb241: Status 404 returned error can't find the container with id efb7d5c0cd0eb006278e9cd092ec10884cab6fd87573821675326c555c3eb241
Sep 17 03:39:23.272981 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:23.272946    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1afd34/distinct-2-w0-1afd34-kserve-5457449c9-bqjmv" event={"ID":"2a2ec719-0a31-439c-96af-ee004af31201","Type":"ContainerStarted","Data":"13207976dd7484b88ff439e9edcc3cbc78ed36e7f21b5370d004552f1d53e287"}
Sep 17 03:39:23.273360 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:23.272997    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1afd34/distinct-2-w0-1afd34-kserve-5457449c9-bqjmv" event={"ID":"2a2ec719-0a31-439c-96af-ee004af31201","Type":"ContainerStarted","Data":"efb7d5c0cd0eb006278e9cd092ec10884cab6fd87573821675326c555c3eb241"}
Sep 17 03:39:23.290082 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:23.290034    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-1afd34/distinct-2-w0-1afd34-kserve-5457449c9-bqjmv" podStartSLOduration=2.290019614 podStartE2EDuration="2.290019614s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 03:39:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:39:23.288354508 +0000 UTC m=+1198.754762033" watchObservedRunningTime="2026-09-17 03:39:23.290019614 +0000 UTC m=+1198.756427139"
Sep 17 03:39:25.244129 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:25.244085    2552 scope.go:117] "RemoveContainer" containerID="53d8a9fce678ee7d2a52850f6d29a89c52d4a882c1e7f5078c13dd7c7fd86a37"
Sep 17 03:39:28.512536 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:28.512501    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-35cf45-openshift-default-bfd7bddcb-b8ddw"]
Sep 17 03:39:28.512989 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:28.512760    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-35cf45-openshift-default-bfd7bddcb-b8ddw" podUID="1a816967-43ac-4537-ae0d-f8dd289ad8ed" containerName="istio-proxy" containerID="cri-o://2cd88d9ee79a4fe97beec70cd5aca953f0acb582af8ba967e3e422ed7680ecd6" gracePeriod=30
Sep 17 03:39:32.080104 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:32.080070    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-1afd34/distinct-2-w0-1afd34-kserve-5457449c9-bqjmv"
Sep 17 03:39:32.080466 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:32.080270    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1afd34/distinct-2-w0-1afd34-kserve-5457449c9-bqjmv"
Sep 17 03:39:32.092892 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:32.092864    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-1afd34/distinct-2-w0-1afd34-kserve-5457449c9-bqjmv"
Sep 17 03:39:32.355475 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:32.355382    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1afd34/distinct-2-w0-1afd34-kserve-5457449c9-bqjmv"
Sep 17 03:39:33.808266 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:33.808244    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-35cf45-openshift-default-bfd7bddcb-b8ddw"
Sep 17 03:39:33.851270 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:33.850499    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-35cf45/noten-model-35cf45-kserve-64cf784dc5-vlm7n"]
Sep 17 03:39:33.889031 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:33.888999    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a816967-43ac-4537-ae0d-f8dd289ad8ed-istiod-ca-cert\") pod \"1a816967-43ac-4537-ae0d-f8dd289ad8ed\" (UID: \"1a816967-43ac-4537-ae0d-f8dd289ad8ed\") "
Sep 17 03:39:33.889223 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:33.889041    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a816967-43ac-4537-ae0d-f8dd289ad8ed-workload-certs\") pod \"1a816967-43ac-4537-ae0d-f8dd289ad8ed\" (UID: \"1a816967-43ac-4537-ae0d-f8dd289ad8ed\") "
Sep 17 03:39:33.889223 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:33.889078    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a816967-43ac-4537-ae0d-f8dd289ad8ed-istio-podinfo\") pod \"1a816967-43ac-4537-ae0d-f8dd289ad8ed\" (UID: \"1a816967-43ac-4537-ae0d-f8dd289ad8ed\") "
Sep 17 03:39:33.889223 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:33.889109    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a816967-43ac-4537-ae0d-f8dd289ad8ed-istio-token\") pod \"1a816967-43ac-4537-ae0d-f8dd289ad8ed\" (UID: \"1a816967-43ac-4537-ae0d-f8dd289ad8ed\") "
Sep 17 03:39:33.889223 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:33.889138    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a816967-43ac-4537-ae0d-f8dd289ad8ed-istio-data\") pod \"1a816967-43ac-4537-ae0d-f8dd289ad8ed\" (UID: \"1a816967-43ac-4537-ae0d-f8dd289ad8ed\") "
Sep 17 03:39:33.889223 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:33.889175    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a816967-43ac-4537-ae0d-f8dd289ad8ed-workload-socket\") pod \"1a816967-43ac-4537-ae0d-f8dd289ad8ed\" (UID: \"1a816967-43ac-4537-ae0d-f8dd289ad8ed\") "
Sep 17 03:39:33.889223 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:33.889209    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a816967-43ac-4537-ae0d-f8dd289ad8ed-istio-envoy\") pod \"1a816967-43ac-4537-ae0d-f8dd289ad8ed\" (UID: \"1a816967-43ac-4537-ae0d-f8dd289ad8ed\") "
Sep 17 03:39:33.889531 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:33.889271    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zdxzf\" (UniqueName: \"kubernetes.io/projected/1a816967-43ac-4537-ae0d-f8dd289ad8ed-kube-api-access-zdxzf\") pod \"1a816967-43ac-4537-ae0d-f8dd289ad8ed\" (UID: \"1a816967-43ac-4537-ae0d-f8dd289ad8ed\") "
Sep 17 03:39:33.889531 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:33.889342    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a816967-43ac-4537-ae0d-f8dd289ad8ed-credential-socket\") pod \"1a816967-43ac-4537-ae0d-f8dd289ad8ed\" (UID: \"1a816967-43ac-4537-ae0d-f8dd289ad8ed\") "
Sep 17 03:39:33.889531 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:33.889392    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a816967-43ac-4537-ae0d-f8dd289ad8ed-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "1a816967-43ac-4537-ae0d-f8dd289ad8ed" (UID: "1a816967-43ac-4537-ae0d-f8dd289ad8ed"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:39:33.889531 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:33.889415    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a816967-43ac-4537-ae0d-f8dd289ad8ed-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "1a816967-43ac-4537-ae0d-f8dd289ad8ed" (UID: "1a816967-43ac-4537-ae0d-f8dd289ad8ed"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 03:39:33.889531 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:33.889477    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a816967-43ac-4537-ae0d-f8dd289ad8ed-istio-data" (OuterVolumeSpecName: "istio-data") pod "1a816967-43ac-4537-ae0d-f8dd289ad8ed" (UID: "1a816967-43ac-4537-ae0d-f8dd289ad8ed"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:39:33.889824 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:33.889527    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a816967-43ac-4537-ae0d-f8dd289ad8ed-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "1a816967-43ac-4537-ae0d-f8dd289ad8ed" (UID: "1a816967-43ac-4537-ae0d-f8dd289ad8ed"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:39:33.889824 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:33.889680    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a816967-43ac-4537-ae0d-f8dd289ad8ed-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "1a816967-43ac-4537-ae0d-f8dd289ad8ed" (UID: "1a816967-43ac-4537-ae0d-f8dd289ad8ed"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:39:33.889926 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:33.889887    2552 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a816967-43ac-4537-ae0d-f8dd289ad8ed-credential-socket\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:39:33.889926 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:33.889907    2552 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a816967-43ac-4537-ae0d-f8dd289ad8ed-istiod-ca-cert\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:39:33.889926 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:33.889922    2552 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a816967-43ac-4537-ae0d-f8dd289ad8ed-workload-certs\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:39:33.890054 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:33.889935    2552 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a816967-43ac-4537-ae0d-f8dd289ad8ed-istio-data\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:39:33.890054 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:33.889947    2552 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a816967-43ac-4537-ae0d-f8dd289ad8ed-workload-socket\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:39:33.891326 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:33.891303    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1a816967-43ac-4537-ae0d-f8dd289ad8ed-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "1a816967-43ac-4537-ae0d-f8dd289ad8ed" (UID: "1a816967-43ac-4537-ae0d-f8dd289ad8ed"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 17 03:39:33.891449 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:33.891416    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a816967-43ac-4537-ae0d-f8dd289ad8ed-kube-api-access-zdxzf" (OuterVolumeSpecName: "kube-api-access-zdxzf") pod "1a816967-43ac-4537-ae0d-f8dd289ad8ed" (UID: "1a816967-43ac-4537-ae0d-f8dd289ad8ed"). InnerVolumeSpecName "kube-api-access-zdxzf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 03:39:33.891751 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:33.891724    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a816967-43ac-4537-ae0d-f8dd289ad8ed-istio-token" (OuterVolumeSpecName: "istio-token") pod "1a816967-43ac-4537-ae0d-f8dd289ad8ed" (UID: "1a816967-43ac-4537-ae0d-f8dd289ad8ed"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 03:39:33.891751 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:33.891740    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a816967-43ac-4537-ae0d-f8dd289ad8ed-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "1a816967-43ac-4537-ae0d-f8dd289ad8ed" (UID: "1a816967-43ac-4537-ae0d-f8dd289ad8ed"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:39:33.991099 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:33.991064    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zdxzf\" (UniqueName: \"kubernetes.io/projected/1a816967-43ac-4537-ae0d-f8dd289ad8ed-kube-api-access-zdxzf\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:39:33.991099 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:33.991099    2552 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a816967-43ac-4537-ae0d-f8dd289ad8ed-istio-podinfo\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:39:33.991321 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:33.991113    2552 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a816967-43ac-4537-ae0d-f8dd289ad8ed-istio-token\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:39:33.991321 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:33.991124    2552 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a816967-43ac-4537-ae0d-f8dd289ad8ed-istio-envoy\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:39:34.332817 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:34.332781    2552 generic.go:356] "Generic (PLEG): container finished" podID="1a816967-43ac-4537-ae0d-f8dd289ad8ed" containerID="2cd88d9ee79a4fe97beec70cd5aca953f0acb582af8ba967e3e422ed7680ecd6" exitCode=0
Sep 17 03:39:34.332988 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:34.332842    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-35cf45-openshift-default-bfd7bddcb-b8ddw"
Sep 17 03:39:34.332988 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:34.332846    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-35cf45-openshift-default-bfd7bddcb-b8ddw" event={"ID":"1a816967-43ac-4537-ae0d-f8dd289ad8ed","Type":"ContainerDied","Data":"2cd88d9ee79a4fe97beec70cd5aca953f0acb582af8ba967e3e422ed7680ecd6"}
Sep 17 03:39:34.332988 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:34.332896    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-35cf45-openshift-default-bfd7bddcb-b8ddw" event={"ID":"1a816967-43ac-4537-ae0d-f8dd289ad8ed","Type":"ContainerDied","Data":"6085d2cbc4bb01ca21c892ceb67569f51eb7e4cc7eebaf4062e9fb76fd21f7e9"}
Sep 17 03:39:34.332988 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:34.332917    2552 scope.go:117] "RemoveContainer" containerID="2cd88d9ee79a4fe97beec70cd5aca953f0acb582af8ba967e3e422ed7680ecd6"
Sep 17 03:39:34.342528 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:34.342511    2552 scope.go:117] "RemoveContainer" containerID="2cd88d9ee79a4fe97beec70cd5aca953f0acb582af8ba967e3e422ed7680ecd6"
Sep 17 03:39:34.342847 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:39:34.342823    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2cd88d9ee79a4fe97beec70cd5aca953f0acb582af8ba967e3e422ed7680ecd6\": container with ID starting with 2cd88d9ee79a4fe97beec70cd5aca953f0acb582af8ba967e3e422ed7680ecd6 not found: ID does not exist" containerID="2cd88d9ee79a4fe97beec70cd5aca953f0acb582af8ba967e3e422ed7680ecd6"
Sep 17 03:39:34.342920 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:34.342859    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2cd88d9ee79a4fe97beec70cd5aca953f0acb582af8ba967e3e422ed7680ecd6"} err="failed to get container status \"2cd88d9ee79a4fe97beec70cd5aca953f0acb582af8ba967e3e422ed7680ecd6\": rpc error: code = NotFound desc = could not find container \"2cd88d9ee79a4fe97beec70cd5aca953f0acb582af8ba967e3e422ed7680ecd6\": container with ID starting with 2cd88d9ee79a4fe97beec70cd5aca953f0acb582af8ba967e3e422ed7680ecd6 not found: ID does not exist"
Sep 17 03:39:34.358396 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:34.358372    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-35cf45-openshift-default-bfd7bddcb-b8ddw"]
Sep 17 03:39:34.362044 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:34.362022    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-35cf45-openshift-default-bfd7bddcb-b8ddw"]
Sep 17 03:39:34.988501 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:34.988466    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1a816967-43ac-4537-ae0d-f8dd289ad8ed" path="/var/lib/kubelet/pods/1a816967-43ac-4537-ae0d-f8dd289ad8ed/volumes"
Sep 17 03:39:37.353215 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:37.353182    2552 generic.go:356] "Generic (PLEG): container finished" podID="c4d4225c-5283-4143-a89e-9c2fbc0eb172" containerID="e1b6f851237769b107bbdca0c10b962d54e00b828de8f82dbfbc38d4138b2d4f" exitCode=0
Sep 17 03:39:37.353682 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:37.353254    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-22e045-openshift-default-5f598cd8fb-zz7g6" event={"ID":"c4d4225c-5283-4143-a89e-9c2fbc0eb172","Type":"ContainerDied","Data":"e1b6f851237769b107bbdca0c10b962d54e00b828de8f82dbfbc38d4138b2d4f"}
Sep 17 03:39:37.353834 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:37.353817    2552 scope.go:117] "RemoveContainer" containerID="e1b6f851237769b107bbdca0c10b962d54e00b828de8f82dbfbc38d4138b2d4f"
Sep 17 03:39:37.355148 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:37.355110    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 03:39:37.355236 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:37.355207    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 03:39:37.355274 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:37.355254    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 03:39:38.360642 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:38.360589    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-22e045-openshift-default-5f598cd8fb-zz7g6" event={"ID":"c4d4225c-5283-4143-a89e-9c2fbc0eb172","Type":"ContainerStarted","Data":"44455b80e068b92576da9d313a0f7409fff0b057cae9bb6c6a045561bd1e5588"}
Sep 17 03:39:38.989737 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:38.989705    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-22e045-openshift-default-5f598cd8fb-zz7g6"
Sep 17 03:39:38.989924 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:38.989900    2552 patch_prober.go:28] interesting pod/e2e-shared-a-w3-22e045-openshift-default-5f598cd8fb-zz7g6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Sep 17 03:39:38.989976 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:38.989943    2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-22e045-openshift-default-5f598cd8fb-zz7g6" podUID="c4d4225c-5283-4143-a89e-9c2fbc0eb172" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Sep 17 03:39:39.989961 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:39.989926    2552 patch_prober.go:28] interesting pod/e2e-shared-a-w3-22e045-openshift-default-5f598cd8fb-zz7g6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Sep 17 03:39:39.990393 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:39.989983    2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-22e045-openshift-default-5f598cd8fb-zz7g6" podUID="c4d4225c-5283-4143-a89e-9c2fbc0eb172" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Sep 17 03:39:40.646093 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:40.646054    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-f6d581-openshift-default-6955dfcbb6-szl9x"]
Sep 17 03:39:40.647006 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:40.646982    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b5b7160-05b5-4218-ab7b-0e7cdf8383a2" containerName="istio-proxy"
Sep 17 03:39:40.647168 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:40.647155    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b5b7160-05b5-4218-ab7b-0e7cdf8383a2" containerName="istio-proxy"
Sep 17 03:39:40.647290 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:40.647279    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a816967-43ac-4537-ae0d-f8dd289ad8ed" containerName="istio-proxy"
Sep 17 03:39:40.647379 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:40.647369    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a816967-43ac-4537-ae0d-f8dd289ad8ed" containerName="istio-proxy"
Sep 17 03:39:40.647566 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:40.647554    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b5b7160-05b5-4218-ab7b-0e7cdf8383a2" containerName="istio-proxy"
Sep 17 03:39:40.647693 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:40.647681    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a816967-43ac-4537-ae0d-f8dd289ad8ed" containerName="istio-proxy"
Sep 17 03:39:40.651973 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:40.651954    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-f6d581-openshift-default-6955dfcbb6-szl9x"
Sep 17 03:39:40.658976 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:40.658956    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-f6d581-openshift-default-dockercfg-dhjkm\""
Sep 17 03:39:40.667937 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:40.667670    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-f6d581-openshift-default-6955dfcbb6-szl9x"]
Sep 17 03:39:40.766856 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:40.766821    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c871cdcf-6205-4f39-809a-7a72f976e145-istiod-ca-cert\") pod \"e2e-shared-b-w3-f6d581-openshift-default-6955dfcbb6-szl9x\" (UID: \"c871cdcf-6205-4f39-809a-7a72f976e145\") " pod="openshift-ingress/e2e-shared-b-w3-f6d581-openshift-default-6955dfcbb6-szl9x"
Sep 17 03:39:40.767049 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:40.766875    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c871cdcf-6205-4f39-809a-7a72f976e145-workload-socket\") pod \"e2e-shared-b-w3-f6d581-openshift-default-6955dfcbb6-szl9x\" (UID: \"c871cdcf-6205-4f39-809a-7a72f976e145\") " pod="openshift-ingress/e2e-shared-b-w3-f6d581-openshift-default-6955dfcbb6-szl9x"
Sep 17 03:39:40.767049 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:40.766903    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmnxb\" (UniqueName: \"kubernetes.io/projected/c871cdcf-6205-4f39-809a-7a72f976e145-kube-api-access-lmnxb\") pod \"e2e-shared-b-w3-f6d581-openshift-default-6955dfcbb6-szl9x\" (UID: \"c871cdcf-6205-4f39-809a-7a72f976e145\") " pod="openshift-ingress/e2e-shared-b-w3-f6d581-openshift-default-6955dfcbb6-szl9x"
Sep 17 03:39:40.767049 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:40.766926    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c871cdcf-6205-4f39-809a-7a72f976e145-istio-envoy\") pod \"e2e-shared-b-w3-f6d581-openshift-default-6955dfcbb6-szl9x\" (UID: \"c871cdcf-6205-4f39-809a-7a72f976e145\") " pod="openshift-ingress/e2e-shared-b-w3-f6d581-openshift-default-6955dfcbb6-szl9x"
Sep 17 03:39:40.767049 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:40.766964    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c871cdcf-6205-4f39-809a-7a72f976e145-workload-certs\") pod \"e2e-shared-b-w3-f6d581-openshift-default-6955dfcbb6-szl9x\" (UID: \"c871cdcf-6205-4f39-809a-7a72f976e145\") " pod="openshift-ingress/e2e-shared-b-w3-f6d581-openshift-default-6955dfcbb6-szl9x"
Sep 17 03:39:40.767049 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:40.766996    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c871cdcf-6205-4f39-809a-7a72f976e145-istio-token\") pod \"e2e-shared-b-w3-f6d581-openshift-default-6955dfcbb6-szl9x\" (UID: \"c871cdcf-6205-4f39-809a-7a72f976e145\") " pod="openshift-ingress/e2e-shared-b-w3-f6d581-openshift-default-6955dfcbb6-szl9x"
Sep 17 03:39:40.767263 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:40.767098    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c871cdcf-6205-4f39-809a-7a72f976e145-credential-socket\") pod \"e2e-shared-b-w3-f6d581-openshift-default-6955dfcbb6-szl9x\" (UID: \"c871cdcf-6205-4f39-809a-7a72f976e145\") " pod="openshift-ingress/e2e-shared-b-w3-f6d581-openshift-default-6955dfcbb6-szl9x"
Sep 17 03:39:40.767263 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:40.767153    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c871cdcf-6205-4f39-809a-7a72f976e145-istio-data\") pod \"e2e-shared-b-w3-f6d581-openshift-default-6955dfcbb6-szl9x\" (UID: \"c871cdcf-6205-4f39-809a-7a72f976e145\") " pod="openshift-ingress/e2e-shared-b-w3-f6d581-openshift-default-6955dfcbb6-szl9x"
Sep 17 03:39:40.767263 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:40.767186    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c871cdcf-6205-4f39-809a-7a72f976e145-istio-podinfo\") pod \"e2e-shared-b-w3-f6d581-openshift-default-6955dfcbb6-szl9x\" (UID: \"c871cdcf-6205-4f39-809a-7a72f976e145\") " pod="openshift-ingress/e2e-shared-b-w3-f6d581-openshift-default-6955dfcbb6-szl9x"
Sep 17 03:39:40.867820 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:40.867785    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c871cdcf-6205-4f39-809a-7a72f976e145-istio-data\") pod \"e2e-shared-b-w3-f6d581-openshift-default-6955dfcbb6-szl9x\" (UID: \"c871cdcf-6205-4f39-809a-7a72f976e145\") " pod="openshift-ingress/e2e-shared-b-w3-f6d581-openshift-default-6955dfcbb6-szl9x"
Sep 17 03:39:40.868038 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:40.867833    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c871cdcf-6205-4f39-809a-7a72f976e145-istio-podinfo\") pod \"e2e-shared-b-w3-f6d581-openshift-default-6955dfcbb6-szl9x\" (UID: \"c871cdcf-6205-4f39-809a-7a72f976e145\") " pod="openshift-ingress/e2e-shared-b-w3-f6d581-openshift-default-6955dfcbb6-szl9x"
Sep 17 03:39:40.868038 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:40.867863    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c871cdcf-6205-4f39-809a-7a72f976e145-istiod-ca-cert\") pod \"e2e-shared-b-w3-f6d581-openshift-default-6955dfcbb6-szl9x\" (UID: \"c871cdcf-6205-4f39-809a-7a72f976e145\") " pod="openshift-ingress/e2e-shared-b-w3-f6d581-openshift-default-6955dfcbb6-szl9x"
Sep 17 03:39:40.868038 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:40.867901    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c871cdcf-6205-4f39-809a-7a72f976e145-workload-socket\") pod \"e2e-shared-b-w3-f6d581-openshift-default-6955dfcbb6-szl9x\" (UID: \"c871cdcf-6205-4f39-809a-7a72f976e145\") " pod="openshift-ingress/e2e-shared-b-w3-f6d581-openshift-default-6955dfcbb6-szl9x"
Sep 17 03:39:40.868038 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:40.867927    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmnxb\" (UniqueName: \"kubernetes.io/projected/c871cdcf-6205-4f39-809a-7a72f976e145-kube-api-access-lmnxb\") pod \"e2e-shared-b-w3-f6d581-openshift-default-6955dfcbb6-szl9x\" (UID: \"c871cdcf-6205-4f39-809a-7a72f976e145\") " pod="openshift-ingress/e2e-shared-b-w3-f6d581-openshift-default-6955dfcbb6-szl9x"
Sep 17 03:39:40.868038 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:40.867960    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c871cdcf-6205-4f39-809a-7a72f976e145-istio-envoy\") pod \"e2e-shared-b-w3-f6d581-openshift-default-6955dfcbb6-szl9x\" (UID: \"c871cdcf-6205-4f39-809a-7a72f976e145\") " pod="openshift-ingress/e2e-shared-b-w3-f6d581-openshift-default-6955dfcbb6-szl9x"
Sep 17 03:39:40.868038 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:40.868001    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c871cdcf-6205-4f39-809a-7a72f976e145-workload-certs\") pod \"e2e-shared-b-w3-f6d581-openshift-default-6955dfcbb6-szl9x\" (UID: \"c871cdcf-6205-4f39-809a-7a72f976e145\") " pod="openshift-ingress/e2e-shared-b-w3-f6d581-openshift-default-6955dfcbb6-szl9x"
Sep 17 03:39:40.868038 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:40.868026    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c871cdcf-6205-4f39-809a-7a72f976e145-istio-token\") pod \"e2e-shared-b-w3-f6d581-openshift-default-6955dfcbb6-szl9x\" (UID: \"c871cdcf-6205-4f39-809a-7a72f976e145\") " pod="openshift-ingress/e2e-shared-b-w3-f6d581-openshift-default-6955dfcbb6-szl9x"
Sep 17 03:39:40.868407 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:40.868141    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c871cdcf-6205-4f39-809a-7a72f976e145-credential-socket\") pod \"e2e-shared-b-w3-f6d581-openshift-default-6955dfcbb6-szl9x\" (UID: \"c871cdcf-6205-4f39-809a-7a72f976e145\") " pod="openshift-ingress/e2e-shared-b-w3-f6d581-openshift-default-6955dfcbb6-szl9x"
Sep 17 03:39:40.868692 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:40.868668    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c871cdcf-6205-4f39-809a-7a72f976e145-istio-data\") pod \"e2e-shared-b-w3-f6d581-openshift-default-6955dfcbb6-szl9x\" (UID: \"c871cdcf-6205-4f39-809a-7a72f976e145\") " pod="openshift-ingress/e2e-shared-b-w3-f6d581-openshift-default-6955dfcbb6-szl9x"
Sep 17 03:39:40.868779 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:40.868689    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c871cdcf-6205-4f39-809a-7a72f976e145-workload-socket\") pod \"e2e-shared-b-w3-f6d581-openshift-default-6955dfcbb6-szl9x\" (UID: \"c871cdcf-6205-4f39-809a-7a72f976e145\") " pod="openshift-ingress/e2e-shared-b-w3-f6d581-openshift-default-6955dfcbb6-szl9x"
Sep 17 03:39:40.868838 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:40.868814    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c871cdcf-6205-4f39-809a-7a72f976e145-workload-certs\") pod \"e2e-shared-b-w3-f6d581-openshift-default-6955dfcbb6-szl9x\" (UID: \"c871cdcf-6205-4f39-809a-7a72f976e145\") " pod="openshift-ingress/e2e-shared-b-w3-f6d581-openshift-default-6955dfcbb6-szl9x"
Sep 17 03:39:40.868909 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:40.868892    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c871cdcf-6205-4f39-809a-7a72f976e145-credential-socket\") pod \"e2e-shared-b-w3-f6d581-openshift-default-6955dfcbb6-szl9x\" (UID: \"c871cdcf-6205-4f39-809a-7a72f976e145\") " pod="openshift-ingress/e2e-shared-b-w3-f6d581-openshift-default-6955dfcbb6-szl9x"
Sep 17 03:39:40.869223 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:40.869207    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c871cdcf-6205-4f39-809a-7a72f976e145-istiod-ca-cert\") pod \"e2e-shared-b-w3-f6d581-openshift-default-6955dfcbb6-szl9x\" (UID: \"c871cdcf-6205-4f39-809a-7a72f976e145\") " pod="openshift-ingress/e2e-shared-b-w3-f6d581-openshift-default-6955dfcbb6-szl9x"
Sep 17 03:39:40.870725 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:40.870700    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c871cdcf-6205-4f39-809a-7a72f976e145-istio-envoy\") pod \"e2e-shared-b-w3-f6d581-openshift-default-6955dfcbb6-szl9x\" (UID: \"c871cdcf-6205-4f39-809a-7a72f976e145\") " pod="openshift-ingress/e2e-shared-b-w3-f6d581-openshift-default-6955dfcbb6-szl9x"
Sep 17 03:39:40.870951 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:40.870936    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c871cdcf-6205-4f39-809a-7a72f976e145-istio-podinfo\") pod \"e2e-shared-b-w3-f6d581-openshift-default-6955dfcbb6-szl9x\" (UID: \"c871cdcf-6205-4f39-809a-7a72f976e145\") " pod="openshift-ingress/e2e-shared-b-w3-f6d581-openshift-default-6955dfcbb6-szl9x"
Sep 17 03:39:40.875896 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:40.875870    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmnxb\" (UniqueName: \"kubernetes.io/projected/c871cdcf-6205-4f39-809a-7a72f976e145-kube-api-access-lmnxb\") pod \"e2e-shared-b-w3-f6d581-openshift-default-6955dfcbb6-szl9x\" (UID: \"c871cdcf-6205-4f39-809a-7a72f976e145\") " pod="openshift-ingress/e2e-shared-b-w3-f6d581-openshift-default-6955dfcbb6-szl9x"
Sep 17 03:39:40.876315 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:40.876299    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c871cdcf-6205-4f39-809a-7a72f976e145-istio-token\") pod \"e2e-shared-b-w3-f6d581-openshift-default-6955dfcbb6-szl9x\" (UID: \"c871cdcf-6205-4f39-809a-7a72f976e145\") " pod="openshift-ingress/e2e-shared-b-w3-f6d581-openshift-default-6955dfcbb6-szl9x"
Sep 17 03:39:40.972456 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:40.972350    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-f6d581-openshift-default-6955dfcbb6-szl9x"
Sep 17 03:39:40.989763 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:40.989735    2552 patch_prober.go:28] interesting pod/e2e-shared-a-w3-22e045-openshift-default-5f598cd8fb-zz7g6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Sep 17 03:39:40.989902 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:40.989876    2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-22e045-openshift-default-5f598cd8fb-zz7g6" podUID="c4d4225c-5283-4143-a89e-9c2fbc0eb172" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Sep 17 03:39:41.270308 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:41.270279    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-f6d581-openshift-default-6955dfcbb6-szl9x"]
Sep 17 03:39:41.271869 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:39:41.271837    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc871cdcf_6205_4f39_809a_7a72f976e145.slice/crio-e642408157451ea3c8dd93aaa03d5dc051b807cb086e9f394f8e6b7fcf0a51c6 WatchSource:0}: Error finding container e642408157451ea3c8dd93aaa03d5dc051b807cb086e9f394f8e6b7fcf0a51c6: Status 404 returned error can't find the container with id e642408157451ea3c8dd93aaa03d5dc051b807cb086e9f394f8e6b7fcf0a51c6
Sep 17 03:39:41.274290 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:41.274251    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 03:39:41.274359 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:41.274339    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 03:39:41.274397 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:41.274385    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 03:39:41.378949 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:41.378908    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-f6d581-openshift-default-6955dfcbb6-szl9x" event={"ID":"c871cdcf-6205-4f39-809a-7a72f976e145","Type":"ContainerStarted","Data":"1018875442a0413e9caf6819eb3790a3bc9d92401b79aa32e7d157f378fca484"}
Sep 17 03:39:41.379129 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:41.378993    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-f6d581-openshift-default-6955dfcbb6-szl9x" event={"ID":"c871cdcf-6205-4f39-809a-7a72f976e145","Type":"ContainerStarted","Data":"e642408157451ea3c8dd93aaa03d5dc051b807cb086e9f394f8e6b7fcf0a51c6"}
Sep 17 03:39:41.406460 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:41.406404    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-f6d581-openshift-default-6955dfcbb6-szl9x" podStartSLOduration=1.406387115 podStartE2EDuration="1.406387115s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 03:39:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:39:41.404779211 +0000 UTC m=+1216.871186742" watchObservedRunningTime="2026-09-17 03:39:41.406387115 +0000 UTC m=+1216.872794642"
Sep 17 03:39:41.972704 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:41.972665    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-f6d581-openshift-default-6955dfcbb6-szl9x"
Sep 17 03:39:41.989570 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:41.989539    2552 patch_prober.go:28] interesting pod/e2e-shared-a-w3-22e045-openshift-default-5f598cd8fb-zz7g6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Sep 17 03:39:41.989752 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:41.989588    2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-22e045-openshift-default-5f598cd8fb-zz7g6" podUID="c4d4225c-5283-4143-a89e-9c2fbc0eb172" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Sep 17 03:39:42.978239 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:42.978208    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-f6d581-openshift-default-6955dfcbb6-szl9x"
Sep 17 03:39:42.989636 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:42.989610    2552 patch_prober.go:28] interesting pod/e2e-shared-a-w3-22e045-openshift-default-5f598cd8fb-zz7g6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Sep 17 03:39:42.989745 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:42.989659    2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-22e045-openshift-default-5f598cd8fb-zz7g6" podUID="c4d4225c-5283-4143-a89e-9c2fbc0eb172" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Sep 17 03:39:43.391224 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:43.391195    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-f6d581-openshift-default-6955dfcbb6-szl9x"
Sep 17 03:39:43.392169 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:43.392152    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-f6d581-openshift-default-6955dfcbb6-szl9x"
Sep 17 03:39:43.993197 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:43.993165    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-22e045-openshift-default-5f598cd8fb-zz7g6"
Sep 17 03:39:43.993828 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:43.993630    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-22e045-openshift-default-5f598cd8fb-zz7g6"
Sep 17 03:39:43.994563 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:43.994542    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-22e045-openshift-default-5f598cd8fb-zz7g6"
Sep 17 03:39:46.074367 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:46.074320    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f6d581-55ccd4fbd4-phzll"]
Sep 17 03:39:46.078941 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:46.078916    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f6d581-55ccd4fbd4-phzll"
Sep 17 03:39:46.081523 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:46.081498    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-f6d581-serving-cert\""
Sep 17 03:39:46.087983 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:46.087904    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f6d581-55ccd4fbd4-phzll"]
Sep 17 03:39:46.125880 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:46.125848    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zg6z\" (UniqueName: \"kubernetes.io/projected/c4851b22-eb50-4f84-aee3-38c7f46ed0c6-kube-api-access-8zg6z\") pod \"maas-api-e2e-shared-b-w3-f6d581-55ccd4fbd4-phzll\" (UID: \"c4851b22-eb50-4f84-aee3-38c7f46ed0c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f6d581-55ccd4fbd4-phzll"
Sep 17 03:39:46.126053 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:46.125962    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4851b22-eb50-4f84-aee3-38c7f46ed0c6-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-f6d581-55ccd4fbd4-phzll\" (UID: \"c4851b22-eb50-4f84-aee3-38c7f46ed0c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f6d581-55ccd4fbd4-phzll"
Sep 17 03:39:46.126053 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:46.125986    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c4851b22-eb50-4f84-aee3-38c7f46ed0c6-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-f6d581-55ccd4fbd4-phzll\" (UID: \"c4851b22-eb50-4f84-aee3-38c7f46ed0c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f6d581-55ccd4fbd4-phzll"
Sep 17 03:39:46.126053 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:46.126032    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c4851b22-eb50-4f84-aee3-38c7f46ed0c6-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-f6d581-55ccd4fbd4-phzll\" (UID: \"c4851b22-eb50-4f84-aee3-38c7f46ed0c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f6d581-55ccd4fbd4-phzll"
Sep 17 03:39:46.227654 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:46.227592    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4851b22-eb50-4f84-aee3-38c7f46ed0c6-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-f6d581-55ccd4fbd4-phzll\" (UID: \"c4851b22-eb50-4f84-aee3-38c7f46ed0c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f6d581-55ccd4fbd4-phzll"
Sep 17 03:39:46.227855 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:46.227665    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c4851b22-eb50-4f84-aee3-38c7f46ed0c6-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-f6d581-55ccd4fbd4-phzll\" (UID: \"c4851b22-eb50-4f84-aee3-38c7f46ed0c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f6d581-55ccd4fbd4-phzll"
Sep 17 03:39:46.227855 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:46.227722    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c4851b22-eb50-4f84-aee3-38c7f46ed0c6-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-f6d581-55ccd4fbd4-phzll\" (UID: \"c4851b22-eb50-4f84-aee3-38c7f46ed0c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f6d581-55ccd4fbd4-phzll"
Sep 17 03:39:46.227855 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:46.227796    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zg6z\" (UniqueName: \"kubernetes.io/projected/c4851b22-eb50-4f84-aee3-38c7f46ed0c6-kube-api-access-8zg6z\") pod \"maas-api-e2e-shared-b-w3-f6d581-55ccd4fbd4-phzll\" (UID: \"c4851b22-eb50-4f84-aee3-38c7f46ed0c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f6d581-55ccd4fbd4-phzll"
Sep 17 03:39:46.227855 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:39:46.227809    2552 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f6d581-serving-cert: secret "maas-api-e2e-shared-b-w3-f6d581-serving-cert" not found
Sep 17 03:39:46.228078 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:39:46.227884    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c4851b22-eb50-4f84-aee3-38c7f46ed0c6-maas-api-tls podName:c4851b22-eb50-4f84-aee3-38c7f46ed0c6 nodeName:}" failed. No retries permitted until 2026-09-17 03:39:46.727866935 +0000 UTC m=+1222.194274439 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c4851b22-eb50-4f84-aee3-38c7f46ed0c6-maas-api-tls") pod "maas-api-e2e-shared-b-w3-f6d581-55ccd4fbd4-phzll" (UID: "c4851b22-eb50-4f84-aee3-38c7f46ed0c6") : secret "maas-api-e2e-shared-b-w3-f6d581-serving-cert" not found
Sep 17 03:39:46.228262 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:46.228243    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4851b22-eb50-4f84-aee3-38c7f46ed0c6-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-f6d581-55ccd4fbd4-phzll\" (UID: \"c4851b22-eb50-4f84-aee3-38c7f46ed0c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f6d581-55ccd4fbd4-phzll"
Sep 17 03:39:46.230213 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:46.230186    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c4851b22-eb50-4f84-aee3-38c7f46ed0c6-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-f6d581-55ccd4fbd4-phzll\" (UID: \"c4851b22-eb50-4f84-aee3-38c7f46ed0c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f6d581-55ccd4fbd4-phzll"
Sep 17 03:39:46.238150 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:46.238127    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zg6z\" (UniqueName: \"kubernetes.io/projected/c4851b22-eb50-4f84-aee3-38c7f46ed0c6-kube-api-access-8zg6z\") pod \"maas-api-e2e-shared-b-w3-f6d581-55ccd4fbd4-phzll\" (UID: \"c4851b22-eb50-4f84-aee3-38c7f46ed0c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f6d581-55ccd4fbd4-phzll"
Sep 17 03:39:46.734625 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:46.734556    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c4851b22-eb50-4f84-aee3-38c7f46ed0c6-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-f6d581-55ccd4fbd4-phzll\" (UID: \"c4851b22-eb50-4f84-aee3-38c7f46ed0c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f6d581-55ccd4fbd4-phzll"
Sep 17 03:39:46.737103 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:46.737073    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c4851b22-eb50-4f84-aee3-38c7f46ed0c6-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-f6d581-55ccd4fbd4-phzll\" (UID: \"c4851b22-eb50-4f84-aee3-38c7f46ed0c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f6d581-55ccd4fbd4-phzll"
Sep 17 03:39:46.992681 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:46.992579    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f6d581-55ccd4fbd4-phzll"
Sep 17 03:39:47.266430 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:47.266403    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f6d581-55ccd4fbd4-phzll"]
Sep 17 03:39:47.269003 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:39:47.268977    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4851b22_eb50_4f84_aee3_38c7f46ed0c6.slice/crio-c0ad593a70c284ee6c16b960b267e2ebfcd903fc58e8433664a44b2f836dc709 WatchSource:0}: Error finding container c0ad593a70c284ee6c16b960b267e2ebfcd903fc58e8433664a44b2f836dc709: Status 404 returned error can't find the container with id c0ad593a70c284ee6c16b960b267e2ebfcd903fc58e8433664a44b2f836dc709
Sep 17 03:39:47.415105 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:47.415071    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f6d581-55ccd4fbd4-phzll" event={"ID":"c4851b22-eb50-4f84-aee3-38c7f46ed0c6","Type":"ContainerStarted","Data":"c0ad593a70c284ee6c16b960b267e2ebfcd903fc58e8433664a44b2f836dc709"}
Sep 17 03:39:48.422069 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:48.422041    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f6d581-55ccd4fbd4-phzll" event={"ID":"c4851b22-eb50-4f84-aee3-38c7f46ed0c6","Type":"ContainerStarted","Data":"5ac2f719658b3e5ebb3b91a1f70ca6ec4214d7be8eba5708de3b6037244d0395"}
Sep 17 03:39:48.422449 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:48.422150    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f6d581-55ccd4fbd4-phzll"
Sep 17 03:39:48.440778 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:48.440734    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f6d581-55ccd4fbd4-phzll" podStartSLOduration=2.1231068730000002 podStartE2EDuration="2.440718449s" totalImagesPullingTime="317.611576ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 03:39:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:39:48.438925766 +0000 UTC m=+1223.905333293" watchObservedRunningTime="2026-09-17 03:39:48.440718449 +0000 UTC m=+1223.907125989"
Sep 17 03:39:48.877330 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:48.877293    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-noten-35cf45/noten-model-35cf45-kserve-64cf784dc5-vlm7n" podUID="f08d64ea-b3e4-439d-bab2-03011a9683b8" containerName="main" containerID="cri-o://de5c2801b174991cfd7d73828fa7eed390bef3f2d090b19c876ce0ca274e9832" gracePeriod=45
Sep 17 03:39:49.127168 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:49.127144    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-35cf45/noten-model-35cf45-kserve-64cf784dc5-vlm7n"
Sep 17 03:39:49.259056 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:49.259034    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f08d64ea-b3e4-439d-bab2-03011a9683b8-model-cache\") pod \"f08d64ea-b3e4-439d-bab2-03011a9683b8\" (UID: \"f08d64ea-b3e4-439d-bab2-03011a9683b8\") "
Sep 17 03:39:49.259230 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:49.259076    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f08d64ea-b3e4-439d-bab2-03011a9683b8-home\") pod \"f08d64ea-b3e4-439d-bab2-03011a9683b8\" (UID: \"f08d64ea-b3e4-439d-bab2-03011a9683b8\") "
Sep 17 03:39:49.259230 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:49.259114    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gvk64\" (UniqueName: \"kubernetes.io/projected/f08d64ea-b3e4-439d-bab2-03011a9683b8-kube-api-access-gvk64\") pod \"f08d64ea-b3e4-439d-bab2-03011a9683b8\" (UID: \"f08d64ea-b3e4-439d-bab2-03011a9683b8\") "
Sep 17 03:39:49.259353 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:49.259232    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f08d64ea-b3e4-439d-bab2-03011a9683b8-tls-certs\") pod \"f08d64ea-b3e4-439d-bab2-03011a9683b8\" (UID: \"f08d64ea-b3e4-439d-bab2-03011a9683b8\") "
Sep 17 03:39:49.259353 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:49.259267    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f08d64ea-b3e4-439d-bab2-03011a9683b8-tmp-dir\") pod \"f08d64ea-b3e4-439d-bab2-03011a9683b8\" (UID: \"f08d64ea-b3e4-439d-bab2-03011a9683b8\") "
Sep 17 03:39:49.259353 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:49.259286    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f08d64ea-b3e4-439d-bab2-03011a9683b8-dshm\") pod \"f08d64ea-b3e4-439d-bab2-03011a9683b8\" (UID: \"f08d64ea-b3e4-439d-bab2-03011a9683b8\") "
Sep 17 03:39:49.259509 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:49.259372    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f08d64ea-b3e4-439d-bab2-03011a9683b8-home" (OuterVolumeSpecName: "home") pod "f08d64ea-b3e4-439d-bab2-03011a9683b8" (UID: "f08d64ea-b3e4-439d-bab2-03011a9683b8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:39:49.259509 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:49.259367    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f08d64ea-b3e4-439d-bab2-03011a9683b8-model-cache" (OuterVolumeSpecName: "model-cache") pod "f08d64ea-b3e4-439d-bab2-03011a9683b8" (UID: "f08d64ea-b3e4-439d-bab2-03011a9683b8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:39:49.259638 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:49.259511    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f08d64ea-b3e4-439d-bab2-03011a9683b8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f08d64ea-b3e4-439d-bab2-03011a9683b8" (UID: "f08d64ea-b3e4-439d-bab2-03011a9683b8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:39:49.259729 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:49.259711    2552 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f08d64ea-b3e4-439d-bab2-03011a9683b8-tmp-dir\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:39:49.259790 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:49.259731    2552 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f08d64ea-b3e4-439d-bab2-03011a9683b8-model-cache\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:39:49.259790 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:49.259746    2552 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f08d64ea-b3e4-439d-bab2-03011a9683b8-home\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:39:49.261467 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:49.261443    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f08d64ea-b3e4-439d-bab2-03011a9683b8-dshm" (OuterVolumeSpecName: "dshm") pod "f08d64ea-b3e4-439d-bab2-03011a9683b8" (UID: "f08d64ea-b3e4-439d-bab2-03011a9683b8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:39:49.261467 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:49.261459    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f08d64ea-b3e4-439d-bab2-03011a9683b8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f08d64ea-b3e4-439d-bab2-03011a9683b8" (UID: "f08d64ea-b3e4-439d-bab2-03011a9683b8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 03:39:49.261622 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:49.261499    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f08d64ea-b3e4-439d-bab2-03011a9683b8-kube-api-access-gvk64" (OuterVolumeSpecName: "kube-api-access-gvk64") pod "f08d64ea-b3e4-439d-bab2-03011a9683b8" (UID: "f08d64ea-b3e4-439d-bab2-03011a9683b8"). InnerVolumeSpecName "kube-api-access-gvk64". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 03:39:49.361319 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:49.361285    2552 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f08d64ea-b3e4-439d-bab2-03011a9683b8-tls-certs\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:39:49.361319 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:49.361315    2552 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f08d64ea-b3e4-439d-bab2-03011a9683b8-dshm\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:39:49.361319 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:49.361324    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvk64\" (UniqueName: \"kubernetes.io/projected/f08d64ea-b3e4-439d-bab2-03011a9683b8-kube-api-access-gvk64\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:39:49.429257 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:49.429221    2552 generic.go:356] "Generic (PLEG): container finished" podID="f08d64ea-b3e4-439d-bab2-03011a9683b8" containerID="de5c2801b174991cfd7d73828fa7eed390bef3f2d090b19c876ce0ca274e9832" exitCode=0
Sep 17 03:39:49.429743 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:49.429263    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-35cf45/noten-model-35cf45-kserve-64cf784dc5-vlm7n"
Sep 17 03:39:49.429743 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:49.429301    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-35cf45/noten-model-35cf45-kserve-64cf784dc5-vlm7n" event={"ID":"f08d64ea-b3e4-439d-bab2-03011a9683b8","Type":"ContainerDied","Data":"de5c2801b174991cfd7d73828fa7eed390bef3f2d090b19c876ce0ca274e9832"}
Sep 17 03:39:49.429743 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:49.429337    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-35cf45/noten-model-35cf45-kserve-64cf784dc5-vlm7n" event={"ID":"f08d64ea-b3e4-439d-bab2-03011a9683b8","Type":"ContainerDied","Data":"ff148db29cefb1d74d3094cdf53bee2161c1267f2cfbbe041e95f743e91a6b0d"}
Sep 17 03:39:49.429743 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:49.429355    2552 scope.go:117] "RemoveContainer" containerID="de5c2801b174991cfd7d73828fa7eed390bef3f2d090b19c876ce0ca274e9832"
Sep 17 03:39:49.445851 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:49.445833    2552 scope.go:117] "RemoveContainer" containerID="de5c2801b174991cfd7d73828fa7eed390bef3f2d090b19c876ce0ca274e9832"
Sep 17 03:39:49.446116 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:39:49.446093    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de5c2801b174991cfd7d73828fa7eed390bef3f2d090b19c876ce0ca274e9832\": container with ID starting with de5c2801b174991cfd7d73828fa7eed390bef3f2d090b19c876ce0ca274e9832 not found: ID does not exist" containerID="de5c2801b174991cfd7d73828fa7eed390bef3f2d090b19c876ce0ca274e9832"
Sep 17 03:39:49.446185 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:49.446122    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de5c2801b174991cfd7d73828fa7eed390bef3f2d090b19c876ce0ca274e9832"} err="failed to get container status \"de5c2801b174991cfd7d73828fa7eed390bef3f2d090b19c876ce0ca274e9832\": rpc error: code = NotFound desc = could not find container \"de5c2801b174991cfd7d73828fa7eed390bef3f2d090b19c876ce0ca274e9832\": container with ID starting with de5c2801b174991cfd7d73828fa7eed390bef3f2d090b19c876ce0ca274e9832 not found: ID does not exist"
Sep 17 03:39:49.455005 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:49.454982    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-35cf45/noten-model-35cf45-kserve-64cf784dc5-vlm7n"]
Sep 17 03:39:49.458565 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:49.458545    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-35cf45/noten-model-35cf45-kserve-64cf784dc5-vlm7n"]
Sep 17 03:39:50.986269 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:50.986231    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f08d64ea-b3e4-439d-bab2-03011a9683b8" path="/var/lib/kubelet/pods/f08d64ea-b3e4-439d-bab2-03011a9683b8/volumes"
Sep 17 03:39:54.434780 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:39:54.434750    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f6d581-55ccd4fbd4-phzll"
Sep 17 03:40:00.375421 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:40:00.375387    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-c10ab8fb/e2e-lifecycle-model-c10ab8fb-kserve-6b95784595-8g2lv"]
Sep 17 03:40:00.376028 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:40:00.375848    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f08d64ea-b3e4-439d-bab2-03011a9683b8" containerName="main"
Sep 17 03:40:00.376028 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:40:00.375861    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="f08d64ea-b3e4-439d-bab2-03011a9683b8" containerName="main"
Sep 17 03:40:00.376028 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:40:00.375950    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="f08d64ea-b3e4-439d-bab2-03011a9683b8" containerName="main"
Sep 17 03:40:00.379146 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:40:00.379128    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-c10ab8fb/e2e-lifecycle-model-c10ab8fb-kserve-6b95784595-8g2lv"
Sep 17 03:40:00.382021 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:40:00.381996    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-c10ab8fb\"/\"kube-root-ca.crt\""
Sep 17 03:40:00.383517 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:40:00.383492    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-c10ab8fb\"/\"openshift-service-ca.crt\""
Sep 17 03:40:00.383653 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:40:00.383587    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-c10ab8fb\"/\"default-dockercfg-zr7pd\""
Sep 17 03:40:00.383808 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:40:00.383786    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-c10ab8fb\"/\"e2e-lifecycle-model-c10ab8fb-kserve-self-signed-certs\""
Sep 17 03:40:00.386559 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:40:00.386535    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-c10ab8fb/e2e-lifecycle-model-c10ab8fb-kserve-6b95784595-8g2lv"]
Sep 17 03:40:00.477918 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:40:00.477883    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcw6d\" (UniqueName: \"kubernetes.io/projected/306e1874-bae0-4260-84f9-7916083e1585-kube-api-access-fcw6d\") pod \"e2e-lifecycle-model-c10ab8fb-kserve-6b95784595-8g2lv\" (UID: \"306e1874-bae0-4260-84f9-7916083e1585\") " pod="ai-tenant-e2e-mt-c10ab8fb/e2e-lifecycle-model-c10ab8fb-kserve-6b95784595-8g2lv"
Sep 17 03:40:00.478084 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:40:00.477931    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/306e1874-bae0-4260-84f9-7916083e1585-dshm\") pod \"e2e-lifecycle-model-c10ab8fb-kserve-6b95784595-8g2lv\" (UID: \"306e1874-bae0-4260-84f9-7916083e1585\") " pod="ai-tenant-e2e-mt-c10ab8fb/e2e-lifecycle-model-c10ab8fb-kserve-6b95784595-8g2lv"
Sep 17 03:40:00.478182 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:40:00.478106    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/306e1874-bae0-4260-84f9-7916083e1585-model-cache\") pod \"e2e-lifecycle-model-c10ab8fb-kserve-6b95784595-8g2lv\" (UID: \"306e1874-bae0-4260-84f9-7916083e1585\") " pod="ai-tenant-e2e-mt-c10ab8fb/e2e-lifecycle-model-c10ab8fb-kserve-6b95784595-8g2lv"
Sep 17 03:40:00.478182 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:40:00.478168    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/306e1874-bae0-4260-84f9-7916083e1585-tmp-dir\") pod \"e2e-lifecycle-model-c10ab8fb-kserve-6b95784595-8g2lv\" (UID: \"306e1874-bae0-4260-84f9-7916083e1585\") " pod="ai-tenant-e2e-mt-c10ab8fb/e2e-lifecycle-model-c10ab8fb-kserve-6b95784595-8g2lv"
Sep 17 03:40:00.478281 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:40:00.478222    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/306e1874-bae0-4260-84f9-7916083e1585-tls-certs\") pod \"e2e-lifecycle-model-c10ab8fb-kserve-6b95784595-8g2lv\" (UID: \"306e1874-bae0-4260-84f9-7916083e1585\") " pod="ai-tenant-e2e-mt-c10ab8fb/e2e-lifecycle-model-c10ab8fb-kserve-6b95784595-8g2lv"
Sep 17 03:40:00.478281 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:40:00.478266    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/306e1874-bae0-4260-84f9-7916083e1585-home\") pod \"e2e-lifecycle-model-c10ab8fb-kserve-6b95784595-8g2lv\" (UID: \"306e1874-bae0-4260-84f9-7916083e1585\") " pod="ai-tenant-e2e-mt-c10ab8fb/e2e-lifecycle-model-c10ab8fb-kserve-6b95784595-8g2lv"
Sep 17 03:40:00.579406 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:40:00.579368    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/306e1874-bae0-4260-84f9-7916083e1585-model-cache\") pod \"e2e-lifecycle-model-c10ab8fb-kserve-6b95784595-8g2lv\" (UID: \"306e1874-bae0-4260-84f9-7916083e1585\") " pod="ai-tenant-e2e-mt-c10ab8fb/e2e-lifecycle-model-c10ab8fb-kserve-6b95784595-8g2lv"
Sep 17 03:40:00.579633 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:40:00.579419    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/306e1874-bae0-4260-84f9-7916083e1585-tmp-dir\") pod \"e2e-lifecycle-model-c10ab8fb-kserve-6b95784595-8g2lv\" (UID: \"306e1874-bae0-4260-84f9-7916083e1585\") " pod="ai-tenant-e2e-mt-c10ab8fb/e2e-lifecycle-model-c10ab8fb-kserve-6b95784595-8g2lv"
Sep 17 03:40:00.579633 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:40:00.579452    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/306e1874-bae0-4260-84f9-7916083e1585-tls-certs\") pod \"e2e-lifecycle-model-c10ab8fb-kserve-6b95784595-8g2lv\" (UID: \"306e1874-bae0-4260-84f9-7916083e1585\") " pod="ai-tenant-e2e-mt-c10ab8fb/e2e-lifecycle-model-c10ab8fb-kserve-6b95784595-8g2lv"
Sep 17 03:40:00.579633 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:40:00.579472    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/306e1874-bae0-4260-84f9-7916083e1585-home\") pod \"e2e-lifecycle-model-c10ab8fb-kserve-6b95784595-8g2lv\" (UID: \"306e1874-bae0-4260-84f9-7916083e1585\") " pod="ai-tenant-e2e-mt-c10ab8fb/e2e-lifecycle-model-c10ab8fb-kserve-6b95784595-8g2lv"
Sep 17 03:40:00.579633 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:40:00.579494    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcw6d\" (UniqueName: \"kubernetes.io/projected/306e1874-bae0-4260-84f9-7916083e1585-kube-api-access-fcw6d\") pod \"e2e-lifecycle-model-c10ab8fb-kserve-6b95784595-8g2lv\" (UID: \"306e1874-bae0-4260-84f9-7916083e1585\") " pod="ai-tenant-e2e-mt-c10ab8fb/e2e-lifecycle-model-c10ab8fb-kserve-6b95784595-8g2lv"
Sep 17 03:40:00.579969 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:40:00.579639    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/306e1874-bae0-4260-84f9-7916083e1585-dshm\") pod \"e2e-lifecycle-model-c10ab8fb-kserve-6b95784595-8g2lv\" (UID: \"306e1874-bae0-4260-84f9-7916083e1585\") " pod="ai-tenant-e2e-mt-c10ab8fb/e2e-lifecycle-model-c10ab8fb-kserve-6b95784595-8g2lv"
Sep 17 03:40:00.579969 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:40:00.579858    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/306e1874-bae0-4260-84f9-7916083e1585-model-cache\") pod \"e2e-lifecycle-model-c10ab8fb-kserve-6b95784595-8g2lv\" (UID: \"306e1874-bae0-4260-84f9-7916083e1585\") " pod="ai-tenant-e2e-mt-c10ab8fb/e2e-lifecycle-model-c10ab8fb-kserve-6b95784595-8g2lv"
Sep 17 03:40:00.579969 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:40:00.579864    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/306e1874-bae0-4260-84f9-7916083e1585-home\") pod \"e2e-lifecycle-model-c10ab8fb-kserve-6b95784595-8g2lv\" (UID: \"306e1874-bae0-4260-84f9-7916083e1585\") " pod="ai-tenant-e2e-mt-c10ab8fb/e2e-lifecycle-model-c10ab8fb-kserve-6b95784595-8g2lv"
Sep 17 03:40:00.579969 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:40:00.579938    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/306e1874-bae0-4260-84f9-7916083e1585-tmp-dir\") pod \"e2e-lifecycle-model-c10ab8fb-kserve-6b95784595-8g2lv\" (UID: \"306e1874-bae0-4260-84f9-7916083e1585\") " pod="ai-tenant-e2e-mt-c10ab8fb/e2e-lifecycle-model-c10ab8fb-kserve-6b95784595-8g2lv"
Sep 17 03:40:00.581789 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:40:00.581770    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/306e1874-bae0-4260-84f9-7916083e1585-dshm\") pod \"e2e-lifecycle-model-c10ab8fb-kserve-6b95784595-8g2lv\" (UID: \"306e1874-bae0-4260-84f9-7916083e1585\") " pod="ai-tenant-e2e-mt-c10ab8fb/e2e-lifecycle-model-c10ab8fb-kserve-6b95784595-8g2lv"
Sep 17 03:40:00.581984 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:40:00.581967    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/306e1874-bae0-4260-84f9-7916083e1585-tls-certs\") pod \"e2e-lifecycle-model-c10ab8fb-kserve-6b95784595-8g2lv\" (UID: \"306e1874-bae0-4260-84f9-7916083e1585\") " pod="ai-tenant-e2e-mt-c10ab8fb/e2e-lifecycle-model-c10ab8fb-kserve-6b95784595-8g2lv"
Sep 17 03:40:00.587310 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:40:00.587290    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcw6d\" (UniqueName: \"kubernetes.io/projected/306e1874-bae0-4260-84f9-7916083e1585-kube-api-access-fcw6d\") pod \"e2e-lifecycle-model-c10ab8fb-kserve-6b95784595-8g2lv\" (UID: \"306e1874-bae0-4260-84f9-7916083e1585\") " pod="ai-tenant-e2e-mt-c10ab8fb/e2e-lifecycle-model-c10ab8fb-kserve-6b95784595-8g2lv"
Sep 17 03:40:00.690700 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:40:00.690610    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-c10ab8fb/e2e-lifecycle-model-c10ab8fb-kserve-6b95784595-8g2lv"
Sep 17 03:40:00.995181 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:40:00.995093    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-c10ab8fb/e2e-lifecycle-model-c10ab8fb-kserve-6b95784595-8g2lv"]
Sep 17 03:40:00.998061 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:40:00.998025    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod306e1874_bae0_4260_84f9_7916083e1585.slice/crio-0efe2a23c80a746b88f986720f0dfb4c6b2daf4115445092c211b61a16c6659c WatchSource:0}: Error finding container 0efe2a23c80a746b88f986720f0dfb4c6b2daf4115445092c211b61a16c6659c: Status 404 returned error can't find the container with id 0efe2a23c80a746b88f986720f0dfb4c6b2daf4115445092c211b61a16c6659c
Sep 17 03:40:01.497624 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:40:01.497571    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-c10ab8fb/e2e-lifecycle-model-c10ab8fb-kserve-6b95784595-8g2lv" event={"ID":"306e1874-bae0-4260-84f9-7916083e1585","Type":"ContainerStarted","Data":"12d8340d363ba87eee32af125ecb834377a5bc6276738ce911778098594d4674"}
Sep 17 03:40:01.497624 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:40:01.497632    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-c10ab8fb/e2e-lifecycle-model-c10ab8fb-kserve-6b95784595-8g2lv" event={"ID":"306e1874-bae0-4260-84f9-7916083e1585","Type":"ContainerStarted","Data":"0efe2a23c80a746b88f986720f0dfb4c6b2daf4115445092c211b61a16c6659c"}
Sep 17 03:40:01.517579 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:40:01.517523    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-c10ab8fb/e2e-lifecycle-model-c10ab8fb-kserve-6b95784595-8g2lv" podStartSLOduration=1.517506797 podStartE2EDuration="1.517506797s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 03:40:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:40:01.514830201 +0000 UTC m=+1236.981237728" watchObservedRunningTime="2026-09-17 03:40:01.517506797 +0000 UTC m=+1236.983914323"
Sep 17 03:40:09.542666 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:40:09.542625    2552 generic.go:356] "Generic (PLEG): container finished" podID="c871cdcf-6205-4f39-809a-7a72f976e145" containerID="1018875442a0413e9caf6819eb3790a3bc9d92401b79aa32e7d157f378fca484" exitCode=0
Sep 17 03:40:09.543081 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:40:09.542690    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-f6d581-openshift-default-6955dfcbb6-szl9x" event={"ID":"c871cdcf-6205-4f39-809a-7a72f976e145","Type":"ContainerDied","Data":"1018875442a0413e9caf6819eb3790a3bc9d92401b79aa32e7d157f378fca484"}
Sep 17 03:40:09.543150 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:40:09.543135    2552 scope.go:117] "RemoveContainer" containerID="1018875442a0413e9caf6819eb3790a3bc9d92401b79aa32e7d157f378fca484"
Sep 17 03:40:09.544315 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:40:09.544290    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 03:40:09.544376 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:40:09.544354    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 03:40:09.544425 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:40:09.544386    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 03:40:10.554799 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:40:10.554766    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-f6d581-openshift-default-6955dfcbb6-szl9x" event={"ID":"c871cdcf-6205-4f39-809a-7a72f976e145","Type":"ContainerStarted","Data":"a58d3fd60bb301db0a8ebf535afceb75b18756fd3293213f286aa504ab7b73fc"}
Sep 17 03:40:10.691032 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:40:10.690987    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-c10ab8fb/e2e-lifecycle-model-c10ab8fb-kserve-6b95784595-8g2lv"
Sep 17 03:40:10.691032 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:40:10.691042    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-c10ab8fb/e2e-lifecycle-model-c10ab8fb-kserve-6b95784595-8g2lv"
Sep 17 03:40:10.704001 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:40:10.703973    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-c10ab8fb/e2e-lifecycle-model-c10ab8fb-kserve-6b95784595-8g2lv"
Sep 17 03:40:10.973588 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:40:10.973486    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-f6d581-openshift-default-6955dfcbb6-szl9x"
Sep 17 03:40:10.974823 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:40:10.974796    2552 patch_prober.go:28] interesting pod/e2e-shared-b-w3-f6d581-openshift-default-6955dfcbb6-szl9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Sep 17 03:40:10.974923 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:40:10.974861    2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-f6d581-openshift-default-6955dfcbb6-szl9x" podUID="c871cdcf-6205-4f39-809a-7a72f976e145" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Sep 17 03:40:11.571877 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:40:11.571845    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-c10ab8fb/e2e-lifecycle-model-c10ab8fb-kserve-6b95784595-8g2lv"
Sep 17 03:40:11.973002 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:40:11.972910    2552 patch_prober.go:28] interesting pod/e2e-shared-b-w3-f6d581-openshift-default-6955dfcbb6-szl9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Sep 17 03:40:11.973002 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:40:11.972966    2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-f6d581-openshift-default-6955dfcbb6-szl9x" podUID="c871cdcf-6205-4f39-809a-7a72f976e145" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Sep 17 03:40:12.972821 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:40:12.972786    2552 patch_prober.go:28] interesting pod/e2e-shared-b-w3-f6d581-openshift-default-6955dfcbb6-szl9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Sep 17 03:40:12.973210 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:40:12.972865    2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-f6d581-openshift-default-6955dfcbb6-szl9x" podUID="c871cdcf-6205-4f39-809a-7a72f976e145" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Sep 17 03:40:13.391910 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:40:13.391880    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-f6d581-openshift-default-6955dfcbb6-szl9x"
Sep 17 03:40:13.972806 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:40:13.972767    2552 patch_prober.go:28] interesting pod/e2e-shared-b-w3-f6d581-openshift-default-6955dfcbb6-szl9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Sep 17 03:40:13.972967 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:40:13.972862    2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-f6d581-openshift-default-6955dfcbb6-szl9x" podUID="c871cdcf-6205-4f39-809a-7a72f976e145" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Sep 17 03:40:14.973226 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:40:14.973188    2552 patch_prober.go:28] interesting pod/e2e-shared-b-w3-f6d581-openshift-default-6955dfcbb6-szl9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Sep 17 03:40:14.973742 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:40:14.973245    2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-f6d581-openshift-default-6955dfcbb6-szl9x" podUID="c871cdcf-6205-4f39-809a-7a72f976e145" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Sep 17 03:40:15.977967 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:40:15.977934    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-f6d581-openshift-default-6955dfcbb6-szl9x"
Sep 17 03:40:15.979002 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:40:15.978983    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-f6d581-openshift-default-6955dfcbb6-szl9x"
Sep 17 03:40:53.349801 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:40:53.349766    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c10ab8fb-znckc"]
Sep 17 03:40:53.354955 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:40:53.354931    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c10ab8fb-znckc"
Sep 17 03:40:53.357808 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:40:53.357788    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8vwnw\""
Sep 17 03:40:53.363055 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:40:53.362959    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c10ab8fb-znckc"]
Sep 17 03:40:53.458042 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:40:53.458002    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7359d1e3-3587-4d7f-8165-eee1731c3f8d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-c10ab8fb-znckc\" (UID: \"7359d1e3-3587-4d7f-8165-eee1731c3f8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c10ab8fb-znckc"
Sep 17 03:40:53.559305 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:40:53.559269    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7359d1e3-3587-4d7f-8165-eee1731c3f8d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-c10ab8fb-znckc\" (UID: \"7359d1e3-3587-4d7f-8165-eee1731c3f8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c10ab8fb-znckc"
Sep 17 03:40:53.559957 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:40:53.559937    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7359d1e3-3587-4d7f-8165-eee1731c3f8d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-c10ab8fb-znckc\" (UID: \"7359d1e3-3587-4d7f-8165-eee1731c3f8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c10ab8fb-znckc"
Sep 17 03:40:53.667132 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:40:53.667036    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c10ab8fb-znckc"
Sep 17 03:40:53.937489 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:40:53.937464    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c10ab8fb-znckc"]
Sep 17 03:40:53.939774 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:40:53.939744    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7359d1e3_3587_4d7f_8165_eee1731c3f8d.slice/crio-901ae9a4da84f17e8ef0e51d07fb06a6cab9851c8ec043e61ce092ac3751627c WatchSource:0}: Error finding container 901ae9a4da84f17e8ef0e51d07fb06a6cab9851c8ec043e61ce092ac3751627c: Status 404 returned error can't find the container with id 901ae9a4da84f17e8ef0e51d07fb06a6cab9851c8ec043e61ce092ac3751627c
Sep 17 03:40:54.803078 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:40:54.803043    2552 generic.go:356] "Generic (PLEG): container finished" podID="7359d1e3-3587-4d7f-8165-eee1731c3f8d" containerID="9403b934d5827143cacd81df24dffd16327fa1990b2a8de2ab4cdc466096ac9a" exitCode=0
Sep 17 03:40:54.803471 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:40:54.803121    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c10ab8fb-znckc" event={"ID":"7359d1e3-3587-4d7f-8165-eee1731c3f8d","Type":"ContainerDied","Data":"9403b934d5827143cacd81df24dffd16327fa1990b2a8de2ab4cdc466096ac9a"}
Sep 17 03:40:54.803471 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:40:54.803178    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c10ab8fb-znckc" event={"ID":"7359d1e3-3587-4d7f-8165-eee1731c3f8d","Type":"ContainerStarted","Data":"901ae9a4da84f17e8ef0e51d07fb06a6cab9851c8ec043e61ce092ac3751627c"}
Sep 17 03:40:55.945405 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:40:55.945380    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c10ab8fb-znckc"
Sep 17 03:40:56.085279 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:40:56.085195    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7359d1e3-3587-4d7f-8165-eee1731c3f8d-maas-api-service-ca\") pod \"7359d1e3-3587-4d7f-8165-eee1731c3f8d\" (UID: \"7359d1e3-3587-4d7f-8165-eee1731c3f8d\") "
Sep 17 03:40:56.085554 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:40:56.085531    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7359d1e3-3587-4d7f-8165-eee1731c3f8d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7359d1e3-3587-4d7f-8165-eee1731c3f8d" (UID: "7359d1e3-3587-4d7f-8165-eee1731c3f8d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 03:40:56.187030 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:40:56.186979    2552 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7359d1e3-3587-4d7f-8165-eee1731c3f8d-maas-api-service-ca\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:40:56.816202 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:40:56.816169    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c10ab8fb-znckc"
Sep 17 03:40:56.816403 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:40:56.816209    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c10ab8fb-znckc" event={"ID":"7359d1e3-3587-4d7f-8165-eee1731c3f8d","Type":"ContainerDied","Data":"901ae9a4da84f17e8ef0e51d07fb06a6cab9851c8ec043e61ce092ac3751627c"}
Sep 17 03:40:56.816403 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:40:56.816263    2552 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="901ae9a4da84f17e8ef0e51d07fb06a6cab9851c8ec043e61ce092ac3751627c"
Sep 17 03:41:09.254434 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:41:09.254398    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c10ab8fb-znckc"]
Sep 17 03:41:09.258189 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:41:09.258162    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c10ab8fb-znckc"]
Sep 17 03:41:10.986234 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:41:10.986197    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7359d1e3-3587-4d7f-8165-eee1731c3f8d" path="/var/lib/kubelet/pods/7359d1e3-3587-4d7f-8165-eee1731c3f8d/volumes"
Sep 17 03:41:17.432853 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:41:17.432809    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-c10ab8fb/e2e-lifecycle-model-c10ab8fb-kserve-6b95784595-8g2lv"]
Sep 17 03:41:32.452487 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:41:32.452419    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-c10ab8fb/e2e-lifecycle-model-c10ab8fb-kserve-6b95784595-8g2lv" podUID="306e1874-bae0-4260-84f9-7916083e1585" containerName="main" containerID="cri-o://12d8340d363ba87eee32af125ecb834377a5bc6276738ce911778098594d4674" gracePeriod=45
Sep 17 03:41:32.707937 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:41:32.707875    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-c10ab8fb/e2e-lifecycle-model-c10ab8fb-kserve-6b95784595-8g2lv"
Sep 17 03:41:32.880874 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:41:32.880845    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/306e1874-bae0-4260-84f9-7916083e1585-dshm\") pod \"306e1874-bae0-4260-84f9-7916083e1585\" (UID: \"306e1874-bae0-4260-84f9-7916083e1585\") "
Sep 17 03:41:32.881073 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:41:32.880905    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/306e1874-bae0-4260-84f9-7916083e1585-tmp-dir\") pod \"306e1874-bae0-4260-84f9-7916083e1585\" (UID: \"306e1874-bae0-4260-84f9-7916083e1585\") "
Sep 17 03:41:32.881073 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:41:32.880936    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/306e1874-bae0-4260-84f9-7916083e1585-tls-certs\") pod \"306e1874-bae0-4260-84f9-7916083e1585\" (UID: \"306e1874-bae0-4260-84f9-7916083e1585\") "
Sep 17 03:41:32.881073 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:41:32.880956    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/306e1874-bae0-4260-84f9-7916083e1585-home\") pod \"306e1874-bae0-4260-84f9-7916083e1585\" (UID: \"306e1874-bae0-4260-84f9-7916083e1585\") "
Sep 17 03:41:32.881073 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:41:32.881007    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/306e1874-bae0-4260-84f9-7916083e1585-model-cache\") pod \"306e1874-bae0-4260-84f9-7916083e1585\" (UID: \"306e1874-bae0-4260-84f9-7916083e1585\") "
Sep 17 03:41:32.881073 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:41:32.881048    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fcw6d\" (UniqueName: \"kubernetes.io/projected/306e1874-bae0-4260-84f9-7916083e1585-kube-api-access-fcw6d\") pod \"306e1874-bae0-4260-84f9-7916083e1585\" (UID: \"306e1874-bae0-4260-84f9-7916083e1585\") "
Sep 17 03:41:32.881355 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:41:32.881258    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/306e1874-bae0-4260-84f9-7916083e1585-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "306e1874-bae0-4260-84f9-7916083e1585" (UID: "306e1874-bae0-4260-84f9-7916083e1585"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:41:32.881355 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:41:32.881275    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/306e1874-bae0-4260-84f9-7916083e1585-home" (OuterVolumeSpecName: "home") pod "306e1874-bae0-4260-84f9-7916083e1585" (UID: "306e1874-bae0-4260-84f9-7916083e1585"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:41:32.881456 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:41:32.881353    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/306e1874-bae0-4260-84f9-7916083e1585-model-cache" (OuterVolumeSpecName: "model-cache") pod "306e1874-bae0-4260-84f9-7916083e1585" (UID: "306e1874-bae0-4260-84f9-7916083e1585"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:41:32.881531 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:41:32.881513    2552 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/306e1874-bae0-4260-84f9-7916083e1585-tmp-dir\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:41:32.881572 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:41:32.881537    2552 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/306e1874-bae0-4260-84f9-7916083e1585-home\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:41:32.881572 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:41:32.881552    2552 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/306e1874-bae0-4260-84f9-7916083e1585-model-cache\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:41:32.883271 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:41:32.883236    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/306e1874-bae0-4260-84f9-7916083e1585-dshm" (OuterVolumeSpecName: "dshm") pod "306e1874-bae0-4260-84f9-7916083e1585" (UID: "306e1874-bae0-4260-84f9-7916083e1585"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:41:32.883388 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:41:32.883298    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/306e1874-bae0-4260-84f9-7916083e1585-kube-api-access-fcw6d" (OuterVolumeSpecName: "kube-api-access-fcw6d") pod "306e1874-bae0-4260-84f9-7916083e1585" (UID: "306e1874-bae0-4260-84f9-7916083e1585"). InnerVolumeSpecName "kube-api-access-fcw6d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 03:41:32.883388 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:41:32.883290    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/306e1874-bae0-4260-84f9-7916083e1585-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "306e1874-bae0-4260-84f9-7916083e1585" (UID: "306e1874-bae0-4260-84f9-7916083e1585"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 03:41:32.982696 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:41:32.982591    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fcw6d\" (UniqueName: \"kubernetes.io/projected/306e1874-bae0-4260-84f9-7916083e1585-kube-api-access-fcw6d\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:41:32.982696 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:41:32.982640    2552 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/306e1874-bae0-4260-84f9-7916083e1585-dshm\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:41:32.982696 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:41:32.982656    2552 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/306e1874-bae0-4260-84f9-7916083e1585-tls-certs\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:41:33.021854 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:41:33.021821    2552 generic.go:356] "Generic (PLEG): container finished" podID="306e1874-bae0-4260-84f9-7916083e1585" containerID="12d8340d363ba87eee32af125ecb834377a5bc6276738ce911778098594d4674" exitCode=0
Sep 17 03:41:33.022029 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:41:33.021868    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-c10ab8fb/e2e-lifecycle-model-c10ab8fb-kserve-6b95784595-8g2lv"
Sep 17 03:41:33.022029 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:41:33.021913    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-c10ab8fb/e2e-lifecycle-model-c10ab8fb-kserve-6b95784595-8g2lv" event={"ID":"306e1874-bae0-4260-84f9-7916083e1585","Type":"ContainerDied","Data":"12d8340d363ba87eee32af125ecb834377a5bc6276738ce911778098594d4674"}
Sep 17 03:41:33.022029 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:41:33.021957    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-c10ab8fb/e2e-lifecycle-model-c10ab8fb-kserve-6b95784595-8g2lv" event={"ID":"306e1874-bae0-4260-84f9-7916083e1585","Type":"ContainerDied","Data":"0efe2a23c80a746b88f986720f0dfb4c6b2daf4115445092c211b61a16c6659c"}
Sep 17 03:41:33.022029 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:41:33.021974    2552 scope.go:117] "RemoveContainer" containerID="12d8340d363ba87eee32af125ecb834377a5bc6276738ce911778098594d4674"
Sep 17 03:41:33.030818 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:41:33.030802    2552 scope.go:117] "RemoveContainer" containerID="12d8340d363ba87eee32af125ecb834377a5bc6276738ce911778098594d4674"
Sep 17 03:41:33.031076 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:41:33.031058    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12d8340d363ba87eee32af125ecb834377a5bc6276738ce911778098594d4674\": container with ID starting with 12d8340d363ba87eee32af125ecb834377a5bc6276738ce911778098594d4674 not found: ID does not exist" containerID="12d8340d363ba87eee32af125ecb834377a5bc6276738ce911778098594d4674"
Sep 17 03:41:33.031129 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:41:33.031086    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12d8340d363ba87eee32af125ecb834377a5bc6276738ce911778098594d4674"} err="failed to get container status \"12d8340d363ba87eee32af125ecb834377a5bc6276738ce911778098594d4674\": rpc error: code = NotFound desc = could not find container \"12d8340d363ba87eee32af125ecb834377a5bc6276738ce911778098594d4674\": container with ID starting with 12d8340d363ba87eee32af125ecb834377a5bc6276738ce911778098594d4674 not found: ID does not exist"
Sep 17 03:41:33.041052 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:41:33.041030    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-c10ab8fb/e2e-lifecycle-model-c10ab8fb-kserve-6b95784595-8g2lv"]
Sep 17 03:41:33.044492 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:41:33.044470    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-c10ab8fb/e2e-lifecycle-model-c10ab8fb-kserve-6b95784595-8g2lv"]
Sep 17 03:41:34.986609 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:41:34.986576    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="306e1874-bae0-4260-84f9-7916083e1585" path="/var/lib/kubelet/pods/306e1874-bae0-4260-84f9-7916083e1585/volumes"
Sep 17 03:41:36.823241 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:41:36.823209    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-f6d581/auth-test-model-f6d581-kserve-5dbd89f68c-lgmzg"]
Sep 17 03:41:36.823835 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:41:36.823814    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="306e1874-bae0-4260-84f9-7916083e1585" containerName="main"
Sep 17 03:41:36.823925 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:41:36.823836    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="306e1874-bae0-4260-84f9-7916083e1585" containerName="main"
Sep 17 03:41:36.823925 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:41:36.823848    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7359d1e3-3587-4d7f-8165-eee1731c3f8d" containerName="revoke-keys"
Sep 17 03:41:36.823925 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:41:36.823856    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="7359d1e3-3587-4d7f-8165-eee1731c3f8d" containerName="revoke-keys"
Sep 17 03:41:36.824082 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:41:36.823972    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="7359d1e3-3587-4d7f-8165-eee1731c3f8d" containerName="revoke-keys"
Sep 17 03:41:36.824082 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:41:36.823991    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="306e1874-bae0-4260-84f9-7916083e1585" containerName="main"
Sep 17 03:41:36.828768 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:41:36.828744    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-f6d581/auth-test-model-f6d581-kserve-5dbd89f68c-lgmzg"
Sep 17 03:41:36.832996 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:41:36.832969    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-f6d581\"/\"kube-root-ca.crt\""
Sep 17 03:41:36.833136 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:41:36.832996    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-f6d581\"/\"auth-test-model-f6d581-kserve-self-signed-certs\""
Sep 17 03:41:36.833136 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:41:36.832972    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-f6d581\"/\"default-dockercfg-msjdf\""
Sep 17 03:41:36.833136 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:41:36.832972    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-f6d581\"/\"openshift-service-ca.crt\""
Sep 17 03:41:36.848331 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:41:36.848306    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-f6d581/auth-test-model-f6d581-kserve-5dbd89f68c-lgmzg"]
Sep 17 03:41:36.921808 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:41:36.921769    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5011121-3390-4619-af40-04e8adaff034-home\") pod \"auth-test-model-f6d581-kserve-5dbd89f68c-lgmzg\" (UID: \"a5011121-3390-4619-af40-04e8adaff034\") " pod="ai-tenant-e2e-shared-b-w3-f6d581/auth-test-model-f6d581-kserve-5dbd89f68c-lgmzg"
Sep 17 03:41:36.921987 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:41:36.921894    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lgfq\" (UniqueName: \"kubernetes.io/projected/a5011121-3390-4619-af40-04e8adaff034-kube-api-access-5lgfq\") pod \"auth-test-model-f6d581-kserve-5dbd89f68c-lgmzg\" (UID: \"a5011121-3390-4619-af40-04e8adaff034\") " pod="ai-tenant-e2e-shared-b-w3-f6d581/auth-test-model-f6d581-kserve-5dbd89f68c-lgmzg"
Sep 17 03:41:36.921987 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:41:36.921957    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5011121-3390-4619-af40-04e8adaff034-tls-certs\") pod \"auth-test-model-f6d581-kserve-5dbd89f68c-lgmzg\" (UID: \"a5011121-3390-4619-af40-04e8adaff034\") " pod="ai-tenant-e2e-shared-b-w3-f6d581/auth-test-model-f6d581-kserve-5dbd89f68c-lgmzg"
Sep 17 03:41:36.922071 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:41:36.922008    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5011121-3390-4619-af40-04e8adaff034-tmp-dir\") pod \"auth-test-model-f6d581-kserve-5dbd89f68c-lgmzg\" (UID: \"a5011121-3390-4619-af40-04e8adaff034\") " pod="ai-tenant-e2e-shared-b-w3-f6d581/auth-test-model-f6d581-kserve-5dbd89f68c-lgmzg"
Sep 17 03:41:36.922071 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:41:36.922052    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5011121-3390-4619-af40-04e8adaff034-dshm\") pod \"auth-test-model-f6d581-kserve-5dbd89f68c-lgmzg\" (UID: \"a5011121-3390-4619-af40-04e8adaff034\") " pod="ai-tenant-e2e-shared-b-w3-f6d581/auth-test-model-f6d581-kserve-5dbd89f68c-lgmzg"
Sep 17 03:41:36.922140 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:41:36.922070    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5011121-3390-4619-af40-04e8adaff034-model-cache\") pod \"auth-test-model-f6d581-kserve-5dbd89f68c-lgmzg\" (UID: \"a5011121-3390-4619-af40-04e8adaff034\") " pod="ai-tenant-e2e-shared-b-w3-f6d581/auth-test-model-f6d581-kserve-5dbd89f68c-lgmzg"
Sep 17 03:41:37.022734 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:41:37.022697    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5011121-3390-4619-af40-04e8adaff034-home\") pod \"auth-test-model-f6d581-kserve-5dbd89f68c-lgmzg\" (UID: \"a5011121-3390-4619-af40-04e8adaff034\") " pod="ai-tenant-e2e-shared-b-w3-f6d581/auth-test-model-f6d581-kserve-5dbd89f68c-lgmzg"
Sep 17 03:41:37.022938 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:41:37.022766    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lgfq\" (UniqueName: \"kubernetes.io/projected/a5011121-3390-4619-af40-04e8adaff034-kube-api-access-5lgfq\") pod \"auth-test-model-f6d581-kserve-5dbd89f68c-lgmzg\" (UID: \"a5011121-3390-4619-af40-04e8adaff034\") " pod="ai-tenant-e2e-shared-b-w3-f6d581/auth-test-model-f6d581-kserve-5dbd89f68c-lgmzg"
Sep 17 03:41:37.022938 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:41:37.022795    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5011121-3390-4619-af40-04e8adaff034-tls-certs\") pod \"auth-test-model-f6d581-kserve-5dbd89f68c-lgmzg\" (UID: \"a5011121-3390-4619-af40-04e8adaff034\") " pod="ai-tenant-e2e-shared-b-w3-f6d581/auth-test-model-f6d581-kserve-5dbd89f68c-lgmzg"
Sep 17 03:41:37.022938 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:41:37.022829    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5011121-3390-4619-af40-04e8adaff034-tmp-dir\") pod \"auth-test-model-f6d581-kserve-5dbd89f68c-lgmzg\" (UID: \"a5011121-3390-4619-af40-04e8adaff034\") " pod="ai-tenant-e2e-shared-b-w3-f6d581/auth-test-model-f6d581-kserve-5dbd89f68c-lgmzg"
Sep 17 03:41:37.022938 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:41:37.022858    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5011121-3390-4619-af40-04e8adaff034-dshm\") pod \"auth-test-model-f6d581-kserve-5dbd89f68c-lgmzg\" (UID: \"a5011121-3390-4619-af40-04e8adaff034\") " pod="ai-tenant-e2e-shared-b-w3-f6d581/auth-test-model-f6d581-kserve-5dbd89f68c-lgmzg"
Sep 17 03:41:37.022938 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:41:37.022876    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5011121-3390-4619-af40-04e8adaff034-model-cache\") pod \"auth-test-model-f6d581-kserve-5dbd89f68c-lgmzg\" (UID: \"a5011121-3390-4619-af40-04e8adaff034\") " pod="ai-tenant-e2e-shared-b-w3-f6d581/auth-test-model-f6d581-kserve-5dbd89f68c-lgmzg"
Sep 17 03:41:37.023309 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:41:37.023138    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5011121-3390-4619-af40-04e8adaff034-home\") pod \"auth-test-model-f6d581-kserve-5dbd89f68c-lgmzg\" (UID: \"a5011121-3390-4619-af40-04e8adaff034\") " pod="ai-tenant-e2e-shared-b-w3-f6d581/auth-test-model-f6d581-kserve-5dbd89f68c-lgmzg"
Sep 17 03:41:37.023309 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:41:37.023206    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5011121-3390-4619-af40-04e8adaff034-model-cache\") pod \"auth-test-model-f6d581-kserve-5dbd89f68c-lgmzg\" (UID: \"a5011121-3390-4619-af40-04e8adaff034\") " pod="ai-tenant-e2e-shared-b-w3-f6d581/auth-test-model-f6d581-kserve-5dbd89f68c-lgmzg"
Sep 17 03:41:37.023309 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:41:37.023231    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5011121-3390-4619-af40-04e8adaff034-tmp-dir\") pod \"auth-test-model-f6d581-kserve-5dbd89f68c-lgmzg\" (UID: \"a5011121-3390-4619-af40-04e8adaff034\") " pod="ai-tenant-e2e-shared-b-w3-f6d581/auth-test-model-f6d581-kserve-5dbd89f68c-lgmzg"
Sep 17 03:41:37.025273 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:41:37.025247    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5011121-3390-4619-af40-04e8adaff034-dshm\") pod \"auth-test-model-f6d581-kserve-5dbd89f68c-lgmzg\" (UID: \"a5011121-3390-4619-af40-04e8adaff034\") " pod="ai-tenant-e2e-shared-b-w3-f6d581/auth-test-model-f6d581-kserve-5dbd89f68c-lgmzg"
Sep 17 03:41:37.025401 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:41:37.025367    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5011121-3390-4619-af40-04e8adaff034-tls-certs\") pod \"auth-test-model-f6d581-kserve-5dbd89f68c-lgmzg\" (UID: \"a5011121-3390-4619-af40-04e8adaff034\") " pod="ai-tenant-e2e-shared-b-w3-f6d581/auth-test-model-f6d581-kserve-5dbd89f68c-lgmzg"
Sep 17 03:41:37.033375 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:41:37.033343    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lgfq\" (UniqueName: \"kubernetes.io/projected/a5011121-3390-4619-af40-04e8adaff034-kube-api-access-5lgfq\") pod \"auth-test-model-f6d581-kserve-5dbd89f68c-lgmzg\" (UID: \"a5011121-3390-4619-af40-04e8adaff034\") " pod="ai-tenant-e2e-shared-b-w3-f6d581/auth-test-model-f6d581-kserve-5dbd89f68c-lgmzg"
Sep 17 03:41:37.137880 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:41:37.137796    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-f6d581/auth-test-model-f6d581-kserve-5dbd89f68c-lgmzg"
Sep 17 03:41:37.420268 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:41:37.420237    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-f6d581/auth-test-model-f6d581-kserve-5dbd89f68c-lgmzg"]
Sep 17 03:41:37.422260 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:41:37.422232    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda5011121_3390_4619_af40_04e8adaff034.slice/crio-f07551cd642859be4f85ae3ef6943305a986388567c904bc34b61b85070ad65c WatchSource:0}: Error finding container f07551cd642859be4f85ae3ef6943305a986388567c904bc34b61b85070ad65c: Status 404 returned error can't find the container with id f07551cd642859be4f85ae3ef6943305a986388567c904bc34b61b85070ad65c
Sep 17 03:41:38.056520 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:41:38.056482    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-f6d581/auth-test-model-f6d581-kserve-5dbd89f68c-lgmzg" event={"ID":"a5011121-3390-4619-af40-04e8adaff034","Type":"ContainerStarted","Data":"43acabaf2872f631b2a35612ecbf343ded78bf421c071e8c9ef932212a4c060f"}
Sep 17 03:41:38.056520 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:41:38.056522    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-f6d581/auth-test-model-f6d581-kserve-5dbd89f68c-lgmzg" event={"ID":"a5011121-3390-4619-af40-04e8adaff034","Type":"ContainerStarted","Data":"f07551cd642859be4f85ae3ef6943305a986388567c904bc34b61b85070ad65c"}
Sep 17 03:41:38.086476 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:41:38.086424    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-f6d581/auth-test-model-f6d581-kserve-5dbd89f68c-lgmzg" podStartSLOduration=2.086410026 podStartE2EDuration="2.086410026s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 03:41:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:41:38.084835525 +0000 UTC m=+1333.551243052" watchObservedRunningTime="2026-09-17 03:41:38.086410026 +0000 UTC m=+1333.552817551"
Sep 17 03:41:47.138198 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:41:47.138120    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-f6d581/auth-test-model-f6d581-kserve-5dbd89f68c-lgmzg"
Sep 17 03:41:47.138198 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:41:47.138164    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-f6d581/auth-test-model-f6d581-kserve-5dbd89f68c-lgmzg"
Sep 17 03:41:47.150400 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:41:47.150375    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-f6d581/auth-test-model-f6d581-kserve-5dbd89f68c-lgmzg"
Sep 17 03:41:48.126959 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:41:48.126934    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-f6d581/auth-test-model-f6d581-kserve-5dbd89f68c-lgmzg"
Sep 17 03:42:15.047925 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:42:15.047887    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-c10ab8fb-openshift-default-8654fcf896-qxlrg"]
Sep 17 03:42:15.048501 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:42:15.048117    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-c10ab8fb-openshift-default-8654fcf896-qxlrg" podUID="78c4208f-673e-4ab0-b439-63c3a10b9cac" containerName="istio-proxy" containerID="cri-o://3850974e33e1c1d1fdc1bf9e64ef484d62b08fffe7f6f65551b28a008d8787c5" gracePeriod=30
Sep 17 03:42:17.548675 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:42:17.548634    2552 patch_prober.go:28] interesting pod/e2e-mt-c10ab8fb-openshift-default-8654fcf896-qxlrg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 17 03:42:17.549068 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:42:17.548703    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-c10ab8fb-openshift-default-8654fcf896-qxlrg" podUID="78c4208f-673e-4ab0-b439-63c3a10b9cac" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 03:42:20.296326 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:42:20.296303    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c10ab8fb-openshift-default-8654fcf896-qxlrg"
Sep 17 03:42:20.298190 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:42:20.298165    2552 generic.go:356] "Generic (PLEG): container finished" podID="78c4208f-673e-4ab0-b439-63c3a10b9cac" containerID="3850974e33e1c1d1fdc1bf9e64ef484d62b08fffe7f6f65551b28a008d8787c5" exitCode=0
Sep 17 03:42:20.298318 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:42:20.298208    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c10ab8fb-openshift-default-8654fcf896-qxlrg"
Sep 17 03:42:20.298318 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:42:20.298237    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c10ab8fb-openshift-default-8654fcf896-qxlrg" event={"ID":"78c4208f-673e-4ab0-b439-63c3a10b9cac","Type":"ContainerDied","Data":"3850974e33e1c1d1fdc1bf9e64ef484d62b08fffe7f6f65551b28a008d8787c5"}
Sep 17 03:42:20.298318 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:42:20.298285    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c10ab8fb-openshift-default-8654fcf896-qxlrg" event={"ID":"78c4208f-673e-4ab0-b439-63c3a10b9cac","Type":"ContainerDied","Data":"c03833cef72702c88129e0d98546e933e47f77b30e5f39bd8b24a5dec8ac7bbf"}
Sep 17 03:42:20.298318 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:42:20.298307    2552 scope.go:117] "RemoveContainer" containerID="3850974e33e1c1d1fdc1bf9e64ef484d62b08fffe7f6f65551b28a008d8787c5"
Sep 17 03:42:20.307078 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:42:20.307063    2552 scope.go:117] "RemoveContainer" containerID="45708f7aaf5a723f7db716c3845aa51e67e1bbea8bf560f2cc1b13c6aa43d061"
Sep 17 03:42:20.315295 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:42:20.315272    2552 scope.go:117] "RemoveContainer" containerID="3850974e33e1c1d1fdc1bf9e64ef484d62b08fffe7f6f65551b28a008d8787c5"
Sep 17 03:42:20.315575 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:42:20.315552    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3850974e33e1c1d1fdc1bf9e64ef484d62b08fffe7f6f65551b28a008d8787c5\": container with ID starting with 3850974e33e1c1d1fdc1bf9e64ef484d62b08fffe7f6f65551b28a008d8787c5 not found: ID does not exist" containerID="3850974e33e1c1d1fdc1bf9e64ef484d62b08fffe7f6f65551b28a008d8787c5"
Sep 17 03:42:20.315659 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:42:20.315588    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3850974e33e1c1d1fdc1bf9e64ef484d62b08fffe7f6f65551b28a008d8787c5"} err="failed to get container status \"3850974e33e1c1d1fdc1bf9e64ef484d62b08fffe7f6f65551b28a008d8787c5\": rpc error: code = NotFound desc = could not find container \"3850974e33e1c1d1fdc1bf9e64ef484d62b08fffe7f6f65551b28a008d8787c5\": container with ID starting with 3850974e33e1c1d1fdc1bf9e64ef484d62b08fffe7f6f65551b28a008d8787c5 not found: ID does not exist"
Sep 17 03:42:20.315659 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:42:20.315630    2552 scope.go:117] "RemoveContainer" containerID="45708f7aaf5a723f7db716c3845aa51e67e1bbea8bf560f2cc1b13c6aa43d061"
Sep 17 03:42:20.315885 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:42:20.315866    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45708f7aaf5a723f7db716c3845aa51e67e1bbea8bf560f2cc1b13c6aa43d061\": container with ID starting with 45708f7aaf5a723f7db716c3845aa51e67e1bbea8bf560f2cc1b13c6aa43d061 not found: ID does not exist" containerID="45708f7aaf5a723f7db716c3845aa51e67e1bbea8bf560f2cc1b13c6aa43d061"
Sep 17 03:42:20.315956 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:42:20.315895    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45708f7aaf5a723f7db716c3845aa51e67e1bbea8bf560f2cc1b13c6aa43d061"} err="failed to get container status \"45708f7aaf5a723f7db716c3845aa51e67e1bbea8bf560f2cc1b13c6aa43d061\": rpc error: code = NotFound desc = could not find container \"45708f7aaf5a723f7db716c3845aa51e67e1bbea8bf560f2cc1b13c6aa43d061\": container with ID starting with 45708f7aaf5a723f7db716c3845aa51e67e1bbea8bf560f2cc1b13c6aa43d061 not found: ID does not exist"
Sep 17 03:42:20.368005 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:42:20.367974    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-58fct\" (UniqueName: \"kubernetes.io/projected/78c4208f-673e-4ab0-b439-63c3a10b9cac-kube-api-access-58fct\") pod \"78c4208f-673e-4ab0-b439-63c3a10b9cac\" (UID: \"78c4208f-673e-4ab0-b439-63c3a10b9cac\") "
Sep 17 03:42:20.368182 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:42:20.368036    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78c4208f-673e-4ab0-b439-63c3a10b9cac-workload-certs\") pod \"78c4208f-673e-4ab0-b439-63c3a10b9cac\" (UID: \"78c4208f-673e-4ab0-b439-63c3a10b9cac\") "
Sep 17 03:42:20.368182 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:42:20.368069    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78c4208f-673e-4ab0-b439-63c3a10b9cac-istiod-ca-cert\") pod \"78c4208f-673e-4ab0-b439-63c3a10b9cac\" (UID: \"78c4208f-673e-4ab0-b439-63c3a10b9cac\") "
Sep 17 03:42:20.368182 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:42:20.368105    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78c4208f-673e-4ab0-b439-63c3a10b9cac-istio-podinfo\") pod \"78c4208f-673e-4ab0-b439-63c3a10b9cac\" (UID: \"78c4208f-673e-4ab0-b439-63c3a10b9cac\") "
Sep 17 03:42:20.368182 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:42:20.368132    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78c4208f-673e-4ab0-b439-63c3a10b9cac-istio-envoy\") pod \"78c4208f-673e-4ab0-b439-63c3a10b9cac\" (UID: \"78c4208f-673e-4ab0-b439-63c3a10b9cac\") "
Sep 17 03:42:20.368182 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:42:20.368169    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78c4208f-673e-4ab0-b439-63c3a10b9cac-istio-token\") pod \"78c4208f-673e-4ab0-b439-63c3a10b9cac\" (UID: \"78c4208f-673e-4ab0-b439-63c3a10b9cac\") "
Sep 17 03:42:20.368462 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:42:20.368225    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78c4208f-673e-4ab0-b439-63c3a10b9cac-workload-socket\") pod \"78c4208f-673e-4ab0-b439-63c3a10b9cac\" (UID: \"78c4208f-673e-4ab0-b439-63c3a10b9cac\") "
Sep 17 03:42:20.368462 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:42:20.368290    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78c4208f-673e-4ab0-b439-63c3a10b9cac-credential-socket\") pod \"78c4208f-673e-4ab0-b439-63c3a10b9cac\" (UID: \"78c4208f-673e-4ab0-b439-63c3a10b9cac\") "
Sep 17 03:42:20.368462 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:42:20.368320    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78c4208f-673e-4ab0-b439-63c3a10b9cac-istio-data\") pod \"78c4208f-673e-4ab0-b439-63c3a10b9cac\" (UID: \"78c4208f-673e-4ab0-b439-63c3a10b9cac\") "
Sep 17 03:42:20.368462 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:42:20.368316    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78c4208f-673e-4ab0-b439-63c3a10b9cac-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "78c4208f-673e-4ab0-b439-63c3a10b9cac" (UID: "78c4208f-673e-4ab0-b439-63c3a10b9cac"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:42:20.368697 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:42:20.368500    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78c4208f-673e-4ab0-b439-63c3a10b9cac-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "78c4208f-673e-4ab0-b439-63c3a10b9cac" (UID: "78c4208f-673e-4ab0-b439-63c3a10b9cac"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:42:20.368697 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:42:20.368556    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78c4208f-673e-4ab0-b439-63c3a10b9cac-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "78c4208f-673e-4ab0-b439-63c3a10b9cac" (UID: "78c4208f-673e-4ab0-b439-63c3a10b9cac"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:42:20.368697 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:42:20.368575    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78c4208f-673e-4ab0-b439-63c3a10b9cac-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "78c4208f-673e-4ab0-b439-63c3a10b9cac" (UID: "78c4208f-673e-4ab0-b439-63c3a10b9cac"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 03:42:20.368827 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:42:20.368765    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78c4208f-673e-4ab0-b439-63c3a10b9cac-istio-data" (OuterVolumeSpecName: "istio-data") pod "78c4208f-673e-4ab0-b439-63c3a10b9cac" (UID: "78c4208f-673e-4ab0-b439-63c3a10b9cac"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:42:20.368893 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:42:20.368863    2552 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78c4208f-673e-4ab0-b439-63c3a10b9cac-credential-socket\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:42:20.368893 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:42:20.368878    2552 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78c4208f-673e-4ab0-b439-63c3a10b9cac-istio-data\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:42:20.368989 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:42:20.368893    2552 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78c4208f-673e-4ab0-b439-63c3a10b9cac-workload-certs\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:42:20.368989 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:42:20.368906    2552 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78c4208f-673e-4ab0-b439-63c3a10b9cac-istiod-ca-cert\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:42:20.368989 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:42:20.368919    2552 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78c4208f-673e-4ab0-b439-63c3a10b9cac-workload-socket\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:42:20.370345 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:42:20.370317    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78c4208f-673e-4ab0-b439-63c3a10b9cac-kube-api-access-58fct" (OuterVolumeSpecName: "kube-api-access-58fct") pod "78c4208f-673e-4ab0-b439-63c3a10b9cac" (UID: "78c4208f-673e-4ab0-b439-63c3a10b9cac"). InnerVolumeSpecName "kube-api-access-58fct". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 03:42:20.370506 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:42:20.370470    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/78c4208f-673e-4ab0-b439-63c3a10b9cac-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "78c4208f-673e-4ab0-b439-63c3a10b9cac" (UID: "78c4208f-673e-4ab0-b439-63c3a10b9cac"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 17 03:42:20.370826 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:42:20.370802    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78c4208f-673e-4ab0-b439-63c3a10b9cac-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "78c4208f-673e-4ab0-b439-63c3a10b9cac" (UID: "78c4208f-673e-4ab0-b439-63c3a10b9cac"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:42:20.370910 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:42:20.370811    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78c4208f-673e-4ab0-b439-63c3a10b9cac-istio-token" (OuterVolumeSpecName: "istio-token") pod "78c4208f-673e-4ab0-b439-63c3a10b9cac" (UID: "78c4208f-673e-4ab0-b439-63c3a10b9cac"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 03:42:20.470400 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:42:20.470372    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58fct\" (UniqueName: \"kubernetes.io/projected/78c4208f-673e-4ab0-b439-63c3a10b9cac-kube-api-access-58fct\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:42:20.470400 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:42:20.470399    2552 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78c4208f-673e-4ab0-b439-63c3a10b9cac-istio-podinfo\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:42:20.470609 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:42:20.470409    2552 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78c4208f-673e-4ab0-b439-63c3a10b9cac-istio-envoy\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:42:20.470609 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:42:20.470418    2552 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78c4208f-673e-4ab0-b439-63c3a10b9cac-istio-token\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:42:20.625372 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:42:20.625336    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-c10ab8fb-openshift-default-8654fcf896-qxlrg"]
Sep 17 03:42:20.629300 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:42:20.629271    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-c10ab8fb-openshift-default-8654fcf896-qxlrg"]
Sep 17 03:42:20.986550 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:42:20.986464    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="78c4208f-673e-4ab0-b439-63c3a10b9cac" path="/var/lib/kubelet/pods/78c4208f-673e-4ab0-b439-63c3a10b9cac/volumes"
Sep 17 03:42:25.307742 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:42:25.307709    2552 scope.go:117] "RemoveContainer" containerID="ea33a6266aa0d4ee0a29121eca11dd48aee436025c713d17d70290f6db2dd1e2"
Sep 17 03:42:32.760075 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:42:32.760041    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-1afd34/embedding-w0-1afd34-kserve-7c79989f85-9k45s"]
Sep 17 03:42:32.760482 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:42:32.760470    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78c4208f-673e-4ab0-b439-63c3a10b9cac" containerName="istio-proxy"
Sep 17 03:42:32.760482 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:42:32.760483    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="78c4208f-673e-4ab0-b439-63c3a10b9cac" containerName="istio-proxy"
Sep 17 03:42:32.760574 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:42:32.760498    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78c4208f-673e-4ab0-b439-63c3a10b9cac" containerName="istio-proxy"
Sep 17 03:42:32.760574 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:42:32.760504    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="78c4208f-673e-4ab0-b439-63c3a10b9cac" containerName="istio-proxy"
Sep 17 03:42:32.760673 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:42:32.760578    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="78c4208f-673e-4ab0-b439-63c3a10b9cac" containerName="istio-proxy"
Sep 17 03:42:32.765366 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:42:32.765347    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1afd34/embedding-w0-1afd34-kserve-7c79989f85-9k45s"
Sep 17 03:42:32.768244 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:42:32.768222    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-1afd34\"/\"embedding-w0-1afd34-kserve-self-signed-certs\""
Sep 17 03:42:32.792540 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:42:32.792518    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-1afd34/embedding-w0-1afd34-kserve-7c79989f85-9k45s"]
Sep 17 03:42:32.905410 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:42:32.905381    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36aa6a53-4557-4431-b02d-cde2e73afd04-home\") pod \"embedding-w0-1afd34-kserve-7c79989f85-9k45s\" (UID: \"36aa6a53-4557-4431-b02d-cde2e73afd04\") " pod="e2e-models-e2e-worker-w0-1afd34/embedding-w0-1afd34-kserve-7c79989f85-9k45s"
Sep 17 03:42:32.905582 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:42:32.905444    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36aa6a53-4557-4431-b02d-cde2e73afd04-dshm\") pod \"embedding-w0-1afd34-kserve-7c79989f85-9k45s\" (UID: \"36aa6a53-4557-4431-b02d-cde2e73afd04\") " pod="e2e-models-e2e-worker-w0-1afd34/embedding-w0-1afd34-kserve-7c79989f85-9k45s"
Sep 17 03:42:32.905582 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:42:32.905552    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36aa6a53-4557-4431-b02d-cde2e73afd04-tmp-dir\") pod \"embedding-w0-1afd34-kserve-7c79989f85-9k45s\" (UID: \"36aa6a53-4557-4431-b02d-cde2e73afd04\") " pod="e2e-models-e2e-worker-w0-1afd34/embedding-w0-1afd34-kserve-7c79989f85-9k45s"
Sep 17 03:42:32.905703 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:42:32.905592    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gk8ww\" (UniqueName: \"kubernetes.io/projected/36aa6a53-4557-4431-b02d-cde2e73afd04-kube-api-access-gk8ww\") pod \"embedding-w0-1afd34-kserve-7c79989f85-9k45s\" (UID: \"36aa6a53-4557-4431-b02d-cde2e73afd04\") " pod="e2e-models-e2e-worker-w0-1afd34/embedding-w0-1afd34-kserve-7c79989f85-9k45s"
Sep 17 03:42:32.905703 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:42:32.905661    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36aa6a53-4557-4431-b02d-cde2e73afd04-model-cache\") pod \"embedding-w0-1afd34-kserve-7c79989f85-9k45s\" (UID: \"36aa6a53-4557-4431-b02d-cde2e73afd04\") " pod="e2e-models-e2e-worker-w0-1afd34/embedding-w0-1afd34-kserve-7c79989f85-9k45s"
Sep 17 03:42:32.905771 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:42:32.905700    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36aa6a53-4557-4431-b02d-cde2e73afd04-tls-certs\") pod \"embedding-w0-1afd34-kserve-7c79989f85-9k45s\" (UID: \"36aa6a53-4557-4431-b02d-cde2e73afd04\") " pod="e2e-models-e2e-worker-w0-1afd34/embedding-w0-1afd34-kserve-7c79989f85-9k45s"
Sep 17 03:42:33.007223 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:42:33.007190    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36aa6a53-4557-4431-b02d-cde2e73afd04-model-cache\") pod \"embedding-w0-1afd34-kserve-7c79989f85-9k45s\" (UID: \"36aa6a53-4557-4431-b02d-cde2e73afd04\") " pod="e2e-models-e2e-worker-w0-1afd34/embedding-w0-1afd34-kserve-7c79989f85-9k45s"
Sep 17 03:42:33.007223 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:42:33.007230    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36aa6a53-4557-4431-b02d-cde2e73afd04-tls-certs\") pod \"embedding-w0-1afd34-kserve-7c79989f85-9k45s\" (UID: \"36aa6a53-4557-4431-b02d-cde2e73afd04\") " pod="e2e-models-e2e-worker-w0-1afd34/embedding-w0-1afd34-kserve-7c79989f85-9k45s"
Sep 17 03:42:33.007425 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:42:33.007257    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36aa6a53-4557-4431-b02d-cde2e73afd04-home\") pod \"embedding-w0-1afd34-kserve-7c79989f85-9k45s\" (UID: \"36aa6a53-4557-4431-b02d-cde2e73afd04\") " pod="e2e-models-e2e-worker-w0-1afd34/embedding-w0-1afd34-kserve-7c79989f85-9k45s"
Sep 17 03:42:33.007425 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:42:33.007310    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36aa6a53-4557-4431-b02d-cde2e73afd04-dshm\") pod \"embedding-w0-1afd34-kserve-7c79989f85-9k45s\" (UID: \"36aa6a53-4557-4431-b02d-cde2e73afd04\") " pod="e2e-models-e2e-worker-w0-1afd34/embedding-w0-1afd34-kserve-7c79989f85-9k45s"
Sep 17 03:42:33.007425 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:42:33.007361    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36aa6a53-4557-4431-b02d-cde2e73afd04-tmp-dir\") pod \"embedding-w0-1afd34-kserve-7c79989f85-9k45s\" (UID: \"36aa6a53-4557-4431-b02d-cde2e73afd04\") " pod="e2e-models-e2e-worker-w0-1afd34/embedding-w0-1afd34-kserve-7c79989f85-9k45s"
Sep 17 03:42:33.007425 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:42:33.007402    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gk8ww\" (UniqueName: \"kubernetes.io/projected/36aa6a53-4557-4431-b02d-cde2e73afd04-kube-api-access-gk8ww\") pod \"embedding-w0-1afd34-kserve-7c79989f85-9k45s\" (UID: \"36aa6a53-4557-4431-b02d-cde2e73afd04\") " pod="e2e-models-e2e-worker-w0-1afd34/embedding-w0-1afd34-kserve-7c79989f85-9k45s"
Sep 17 03:42:33.007717 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:42:33.007691    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36aa6a53-4557-4431-b02d-cde2e73afd04-model-cache\") pod \"embedding-w0-1afd34-kserve-7c79989f85-9k45s\" (UID: \"36aa6a53-4557-4431-b02d-cde2e73afd04\") " pod="e2e-models-e2e-worker-w0-1afd34/embedding-w0-1afd34-kserve-7c79989f85-9k45s"
Sep 17 03:42:33.007800 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:42:33.007728    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36aa6a53-4557-4431-b02d-cde2e73afd04-home\") pod \"embedding-w0-1afd34-kserve-7c79989f85-9k45s\" (UID: \"36aa6a53-4557-4431-b02d-cde2e73afd04\") " pod="e2e-models-e2e-worker-w0-1afd34/embedding-w0-1afd34-kserve-7c79989f85-9k45s"
Sep 17 03:42:33.007856 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:42:33.007806    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36aa6a53-4557-4431-b02d-cde2e73afd04-tmp-dir\") pod \"embedding-w0-1afd34-kserve-7c79989f85-9k45s\" (UID: \"36aa6a53-4557-4431-b02d-cde2e73afd04\") " pod="e2e-models-e2e-worker-w0-1afd34/embedding-w0-1afd34-kserve-7c79989f85-9k45s"
Sep 17 03:42:33.009641 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:42:33.009620    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36aa6a53-4557-4431-b02d-cde2e73afd04-dshm\") pod \"embedding-w0-1afd34-kserve-7c79989f85-9k45s\" (UID: \"36aa6a53-4557-4431-b02d-cde2e73afd04\") " pod="e2e-models-e2e-worker-w0-1afd34/embedding-w0-1afd34-kserve-7c79989f85-9k45s"
Sep 17 03:42:33.009796 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:42:33.009780    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36aa6a53-4557-4431-b02d-cde2e73afd04-tls-certs\") pod \"embedding-w0-1afd34-kserve-7c79989f85-9k45s\" (UID: \"36aa6a53-4557-4431-b02d-cde2e73afd04\") " pod="e2e-models-e2e-worker-w0-1afd34/embedding-w0-1afd34-kserve-7c79989f85-9k45s"
Sep 17 03:42:33.017201 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:42:33.017134    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gk8ww\" (UniqueName: \"kubernetes.io/projected/36aa6a53-4557-4431-b02d-cde2e73afd04-kube-api-access-gk8ww\") pod \"embedding-w0-1afd34-kserve-7c79989f85-9k45s\" (UID: \"36aa6a53-4557-4431-b02d-cde2e73afd04\") " pod="e2e-models-e2e-worker-w0-1afd34/embedding-w0-1afd34-kserve-7c79989f85-9k45s"
Sep 17 03:42:33.074414 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:42:33.074383    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1afd34/embedding-w0-1afd34-kserve-7c79989f85-9k45s"
Sep 17 03:42:33.346015 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:42:33.345989    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-1afd34/embedding-w0-1afd34-kserve-7c79989f85-9k45s"]
Sep 17 03:42:33.348002 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:42:33.347975    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod36aa6a53_4557_4431_b02d_cde2e73afd04.slice/crio-4fd3143e8c15a13a9cf9a61ba8a137013d4cfecb301612b84f06b24410f57760 WatchSource:0}: Error finding container 4fd3143e8c15a13a9cf9a61ba8a137013d4cfecb301612b84f06b24410f57760: Status 404 returned error can't find the container with id 4fd3143e8c15a13a9cf9a61ba8a137013d4cfecb301612b84f06b24410f57760
Sep 17 03:42:33.381360 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:42:33.381327    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1afd34/embedding-w0-1afd34-kserve-7c79989f85-9k45s" event={"ID":"36aa6a53-4557-4431-b02d-cde2e73afd04","Type":"ContainerStarted","Data":"4fd3143e8c15a13a9cf9a61ba8a137013d4cfecb301612b84f06b24410f57760"}
Sep 17 03:42:34.388066 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:42:34.388028    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1afd34/embedding-w0-1afd34-kserve-7c79989f85-9k45s" event={"ID":"36aa6a53-4557-4431-b02d-cde2e73afd04","Type":"ContainerStarted","Data":"6ff98b094f0c969d6d71318e8cd03d4cadff7769418625d8d7b7e93286ad066d"}
Sep 17 03:42:34.406518 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:42:34.406463    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-1afd34/embedding-w0-1afd34-kserve-7c79989f85-9k45s" podStartSLOduration=2.406448516 podStartE2EDuration="2.406448516s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 03:42:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:42:34.404407736 +0000 UTC m=+1389.870815261" watchObservedRunningTime="2026-09-17 03:42:34.406448516 +0000 UTC m=+1389.872856068"
Sep 17 03:42:43.075152 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:42:43.075118    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1afd34/embedding-w0-1afd34-kserve-7c79989f85-9k45s"
Sep 17 03:42:43.075731 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:42:43.075331    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-1afd34/embedding-w0-1afd34-kserve-7c79989f85-9k45s"
Sep 17 03:42:43.087838 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:42:43.087812    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-1afd34/embedding-w0-1afd34-kserve-7c79989f85-9k45s"
Sep 17 03:42:43.448050 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:42:43.447963    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1afd34/embedding-w0-1afd34-kserve-7c79989f85-9k45s"
Sep 17 03:43:42.855478 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:43:42.855443    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-26d5d1-openshift-default-df6dfd77d-w2nq4"]
Sep 17 03:43:42.856193 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:43:42.856171    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="78c4208f-673e-4ab0-b439-63c3a10b9cac" containerName="istio-proxy"
Sep 17 03:43:42.859481 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:43:42.859460    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-26d5d1-openshift-default-df6dfd77d-w2nq4"
Sep 17 03:43:42.862302 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:43:42.862282    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-26d5d1-openshift-default-dockercfg-tpxfz\""
Sep 17 03:43:42.869371 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:43:42.869347    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-26d5d1-openshift-default-df6dfd77d-w2nq4"]
Sep 17 03:43:43.022166 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:43:43.022131    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65a2cb95-0b18-40d3-8076-277905391cd2-workload-certs\") pod \"e2e-api-b-26d5d1-openshift-default-df6dfd77d-w2nq4\" (UID: \"65a2cb95-0b18-40d3-8076-277905391cd2\") " pod="openshift-ingress/e2e-api-b-26d5d1-openshift-default-df6dfd77d-w2nq4"
Sep 17 03:43:43.022166 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:43:43.022168    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8gs5\" (UniqueName: \"kubernetes.io/projected/65a2cb95-0b18-40d3-8076-277905391cd2-kube-api-access-p8gs5\") pod \"e2e-api-b-26d5d1-openshift-default-df6dfd77d-w2nq4\" (UID: \"65a2cb95-0b18-40d3-8076-277905391cd2\") " pod="openshift-ingress/e2e-api-b-26d5d1-openshift-default-df6dfd77d-w2nq4"
Sep 17 03:43:43.022382 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:43:43.022195    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65a2cb95-0b18-40d3-8076-277905391cd2-istiod-ca-cert\") pod \"e2e-api-b-26d5d1-openshift-default-df6dfd77d-w2nq4\" (UID: \"65a2cb95-0b18-40d3-8076-277905391cd2\") " pod="openshift-ingress/e2e-api-b-26d5d1-openshift-default-df6dfd77d-w2nq4"
Sep 17 03:43:43.022382 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:43:43.022292    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65a2cb95-0b18-40d3-8076-277905391cd2-istio-data\") pod \"e2e-api-b-26d5d1-openshift-default-df6dfd77d-w2nq4\" (UID: \"65a2cb95-0b18-40d3-8076-277905391cd2\") " pod="openshift-ingress/e2e-api-b-26d5d1-openshift-default-df6dfd77d-w2nq4"
Sep 17 03:43:43.022382 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:43:43.022355    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65a2cb95-0b18-40d3-8076-277905391cd2-workload-socket\") pod \"e2e-api-b-26d5d1-openshift-default-df6dfd77d-w2nq4\" (UID: \"65a2cb95-0b18-40d3-8076-277905391cd2\") " pod="openshift-ingress/e2e-api-b-26d5d1-openshift-default-df6dfd77d-w2nq4"
Sep 17 03:43:43.022492 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:43:43.022389    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65a2cb95-0b18-40d3-8076-277905391cd2-credential-socket\") pod \"e2e-api-b-26d5d1-openshift-default-df6dfd77d-w2nq4\" (UID: \"65a2cb95-0b18-40d3-8076-277905391cd2\") " pod="openshift-ingress/e2e-api-b-26d5d1-openshift-default-df6dfd77d-w2nq4"
Sep 17 03:43:43.022492 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:43:43.022426    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65a2cb95-0b18-40d3-8076-277905391cd2-istio-token\") pod \"e2e-api-b-26d5d1-openshift-default-df6dfd77d-w2nq4\" (UID: \"65a2cb95-0b18-40d3-8076-277905391cd2\") " pod="openshift-ingress/e2e-api-b-26d5d1-openshift-default-df6dfd77d-w2nq4"
Sep 17 03:43:43.022492 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:43:43.022483    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65a2cb95-0b18-40d3-8076-277905391cd2-istio-podinfo\") pod \"e2e-api-b-26d5d1-openshift-default-df6dfd77d-w2nq4\" (UID: \"65a2cb95-0b18-40d3-8076-277905391cd2\") " pod="openshift-ingress/e2e-api-b-26d5d1-openshift-default-df6dfd77d-w2nq4"
Sep 17 03:43:43.022591 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:43:43.022502    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65a2cb95-0b18-40d3-8076-277905391cd2-istio-envoy\") pod \"e2e-api-b-26d5d1-openshift-default-df6dfd77d-w2nq4\" (UID: \"65a2cb95-0b18-40d3-8076-277905391cd2\") " pod="openshift-ingress/e2e-api-b-26d5d1-openshift-default-df6dfd77d-w2nq4"
Sep 17 03:43:43.123392 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:43:43.123313    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65a2cb95-0b18-40d3-8076-277905391cd2-istio-data\") pod \"e2e-api-b-26d5d1-openshift-default-df6dfd77d-w2nq4\" (UID: \"65a2cb95-0b18-40d3-8076-277905391cd2\") " pod="openshift-ingress/e2e-api-b-26d5d1-openshift-default-df6dfd77d-w2nq4"
Sep 17 03:43:43.123553 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:43:43.123399    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65a2cb95-0b18-40d3-8076-277905391cd2-workload-socket\") pod \"e2e-api-b-26d5d1-openshift-default-df6dfd77d-w2nq4\" (UID: \"65a2cb95-0b18-40d3-8076-277905391cd2\") " pod="openshift-ingress/e2e-api-b-26d5d1-openshift-default-df6dfd77d-w2nq4"
Sep 17 03:43:43.123553 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:43:43.123425    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65a2cb95-0b18-40d3-8076-277905391cd2-credential-socket\") pod \"e2e-api-b-26d5d1-openshift-default-df6dfd77d-w2nq4\" (UID: \"65a2cb95-0b18-40d3-8076-277905391cd2\") " pod="openshift-ingress/e2e-api-b-26d5d1-openshift-default-df6dfd77d-w2nq4"
Sep 17 03:43:43.123553 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:43:43.123458    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65a2cb95-0b18-40d3-8076-277905391cd2-istio-token\") pod \"e2e-api-b-26d5d1-openshift-default-df6dfd77d-w2nq4\" (UID: \"65a2cb95-0b18-40d3-8076-277905391cd2\") " pod="openshift-ingress/e2e-api-b-26d5d1-openshift-default-df6dfd77d-w2nq4"
Sep 17 03:43:43.123553 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:43:43.123494    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65a2cb95-0b18-40d3-8076-277905391cd2-istio-podinfo\") pod \"e2e-api-b-26d5d1-openshift-default-df6dfd77d-w2nq4\" (UID: \"65a2cb95-0b18-40d3-8076-277905391cd2\") " pod="openshift-ingress/e2e-api-b-26d5d1-openshift-default-df6dfd77d-w2nq4"
Sep 17 03:43:43.123553 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:43:43.123513    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65a2cb95-0b18-40d3-8076-277905391cd2-istio-envoy\") pod \"e2e-api-b-26d5d1-openshift-default-df6dfd77d-w2nq4\" (UID: \"65a2cb95-0b18-40d3-8076-277905391cd2\") " pod="openshift-ingress/e2e-api-b-26d5d1-openshift-default-df6dfd77d-w2nq4"
Sep 17 03:43:43.123553 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:43:43.123544    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65a2cb95-0b18-40d3-8076-277905391cd2-workload-certs\") pod \"e2e-api-b-26d5d1-openshift-default-df6dfd77d-w2nq4\" (UID: \"65a2cb95-0b18-40d3-8076-277905391cd2\") " pod="openshift-ingress/e2e-api-b-26d5d1-openshift-default-df6dfd77d-w2nq4"
Sep 17 03:43:43.123927 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:43:43.123569    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8gs5\" (UniqueName: \"kubernetes.io/projected/65a2cb95-0b18-40d3-8076-277905391cd2-kube-api-access-p8gs5\") pod \"e2e-api-b-26d5d1-openshift-default-df6dfd77d-w2nq4\" (UID: \"65a2cb95-0b18-40d3-8076-277905391cd2\") " pod="openshift-ingress/e2e-api-b-26d5d1-openshift-default-df6dfd77d-w2nq4"
Sep 17 03:43:43.123927 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:43:43.123621    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65a2cb95-0b18-40d3-8076-277905391cd2-istiod-ca-cert\") pod \"e2e-api-b-26d5d1-openshift-default-df6dfd77d-w2nq4\" (UID: \"65a2cb95-0b18-40d3-8076-277905391cd2\") " pod="openshift-ingress/e2e-api-b-26d5d1-openshift-default-df6dfd77d-w2nq4"
Sep 17 03:43:43.123927 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:43:43.123796    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65a2cb95-0b18-40d3-8076-277905391cd2-workload-socket\") pod \"e2e-api-b-26d5d1-openshift-default-df6dfd77d-w2nq4\" (UID: \"65a2cb95-0b18-40d3-8076-277905391cd2\") " pod="openshift-ingress/e2e-api-b-26d5d1-openshift-default-df6dfd77d-w2nq4"
Sep 17 03:43:43.123927 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:43:43.123848    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65a2cb95-0b18-40d3-8076-277905391cd2-credential-socket\") pod \"e2e-api-b-26d5d1-openshift-default-df6dfd77d-w2nq4\" (UID: \"65a2cb95-0b18-40d3-8076-277905391cd2\") " pod="openshift-ingress/e2e-api-b-26d5d1-openshift-default-df6dfd77d-w2nq4"
Sep 17 03:43:43.124135 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:43:43.123993    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65a2cb95-0b18-40d3-8076-277905391cd2-istio-data\") pod \"e2e-api-b-26d5d1-openshift-default-df6dfd77d-w2nq4\" (UID: \"65a2cb95-0b18-40d3-8076-277905391cd2\") " pod="openshift-ingress/e2e-api-b-26d5d1-openshift-default-df6dfd77d-w2nq4"
Sep 17 03:43:43.124192 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:43:43.124170    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65a2cb95-0b18-40d3-8076-277905391cd2-workload-certs\") pod \"e2e-api-b-26d5d1-openshift-default-df6dfd77d-w2nq4\" (UID: \"65a2cb95-0b18-40d3-8076-277905391cd2\") " pod="openshift-ingress/e2e-api-b-26d5d1-openshift-default-df6dfd77d-w2nq4"
Sep 17 03:43:43.124545 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:43:43.124522    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65a2cb95-0b18-40d3-8076-277905391cd2-istiod-ca-cert\") pod \"e2e-api-b-26d5d1-openshift-default-df6dfd77d-w2nq4\" (UID: \"65a2cb95-0b18-40d3-8076-277905391cd2\") " pod="openshift-ingress/e2e-api-b-26d5d1-openshift-default-df6dfd77d-w2nq4"
Sep 17 03:43:43.125848 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:43:43.125829    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65a2cb95-0b18-40d3-8076-277905391cd2-istio-envoy\") pod \"e2e-api-b-26d5d1-openshift-default-df6dfd77d-w2nq4\" (UID: \"65a2cb95-0b18-40d3-8076-277905391cd2\") " pod="openshift-ingress/e2e-api-b-26d5d1-openshift-default-df6dfd77d-w2nq4"
Sep 17 03:43:43.126214 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:43:43.126190    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65a2cb95-0b18-40d3-8076-277905391cd2-istio-podinfo\") pod \"e2e-api-b-26d5d1-openshift-default-df6dfd77d-w2nq4\" (UID: \"65a2cb95-0b18-40d3-8076-277905391cd2\") " pod="openshift-ingress/e2e-api-b-26d5d1-openshift-default-df6dfd77d-w2nq4"
Sep 17 03:43:43.135131 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:43:43.135108    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65a2cb95-0b18-40d3-8076-277905391cd2-istio-token\") pod \"e2e-api-b-26d5d1-openshift-default-df6dfd77d-w2nq4\" (UID: \"65a2cb95-0b18-40d3-8076-277905391cd2\") " pod="openshift-ingress/e2e-api-b-26d5d1-openshift-default-df6dfd77d-w2nq4"
Sep 17 03:43:43.135694 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:43:43.135666    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8gs5\" (UniqueName: \"kubernetes.io/projected/65a2cb95-0b18-40d3-8076-277905391cd2-kube-api-access-p8gs5\") pod \"e2e-api-b-26d5d1-openshift-default-df6dfd77d-w2nq4\" (UID: \"65a2cb95-0b18-40d3-8076-277905391cd2\") " pod="openshift-ingress/e2e-api-b-26d5d1-openshift-default-df6dfd77d-w2nq4"
Sep 17 03:43:43.170990 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:43:43.170969    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-26d5d1-openshift-default-df6dfd77d-w2nq4"
Sep 17 03:43:43.476251 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:43:43.476228    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-26d5d1-openshift-default-df6dfd77d-w2nq4"]
Sep 17 03:43:43.478065 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:43:43.478033    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65a2cb95_0b18_40d3_8076_277905391cd2.slice/crio-70390c59cf9709c1dcd3045baa355394219f0c9b651b01748605c50dee379764 WatchSource:0}: Error finding container 70390c59cf9709c1dcd3045baa355394219f0c9b651b01748605c50dee379764: Status 404 returned error can't find the container with id 70390c59cf9709c1dcd3045baa355394219f0c9b651b01748605c50dee379764
Sep 17 03:43:43.480274 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:43:43.480242    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 03:43:43.480360 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:43:43.480305    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 03:43:43.480360 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:43:43.480335    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 03:43:43.767486 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:43:43.767441    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-26d5d1-openshift-default-df6dfd77d-w2nq4" event={"ID":"65a2cb95-0b18-40d3-8076-277905391cd2","Type":"ContainerStarted","Data":"8caf7c76acb33f510d73bd9210d24a8c9d63ec3e611a177d19f475be2e6fc38f"}
Sep 17 03:43:43.767709 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:43:43.767501    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-26d5d1-openshift-default-df6dfd77d-w2nq4" event={"ID":"65a2cb95-0b18-40d3-8076-277905391cd2","Type":"ContainerStarted","Data":"70390c59cf9709c1dcd3045baa355394219f0c9b651b01748605c50dee379764"}
Sep 17 03:43:43.786687 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:43:43.786634    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-26d5d1-openshift-default-df6dfd77d-w2nq4" podStartSLOduration=1.7866152400000002 podStartE2EDuration="1.78661524s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 03:43:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:43:43.785249064 +0000 UTC m=+1459.251656600" watchObservedRunningTime="2026-09-17 03:43:43.78661524 +0000 UTC m=+1459.253022767"
Sep 17 03:43:44.172202 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:43:44.172122    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-26d5d1-openshift-default-df6dfd77d-w2nq4"
Sep 17 03:43:44.177404 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:43:44.177379    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-26d5d1-openshift-default-df6dfd77d-w2nq4"
Sep 17 03:43:44.774274 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:43:44.774242    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-26d5d1-openshift-default-df6dfd77d-w2nq4"
Sep 17 03:43:44.775276 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:43:44.775256    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-26d5d1-openshift-default-df6dfd77d-w2nq4"
Sep 17 03:43:51.581395 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:43:51.581359    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-26d5d1-67664944-mzhwg"]
Sep 17 03:43:51.585445 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:43:51.585425    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-26d5d1-67664944-mzhwg"
Sep 17 03:43:51.587892 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:43:51.587873    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-26d5d1-serving-cert\""
Sep 17 03:43:51.599462 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:43:51.599441    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-26d5d1-67664944-mzhwg"]
Sep 17 03:43:51.608557 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:43:51.608536    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/600316ad-6c07-4f70-ba55-9894d1689735-openshift-service-ca\") pod \"maas-api-e2e-api-b-26d5d1-67664944-mzhwg\" (UID: \"600316ad-6c07-4f70-ba55-9894d1689735\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-26d5d1-67664944-mzhwg"
Sep 17 03:43:51.608669 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:43:51.608580    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/600316ad-6c07-4f70-ba55-9894d1689735-metrics-certs\") pod \"maas-api-e2e-api-b-26d5d1-67664944-mzhwg\" (UID: \"600316ad-6c07-4f70-ba55-9894d1689735\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-26d5d1-67664944-mzhwg"
Sep 17 03:43:51.608669 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:43:51.608651    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/600316ad-6c07-4f70-ba55-9894d1689735-maas-api-tls\") pod \"maas-api-e2e-api-b-26d5d1-67664944-mzhwg\" (UID: \"600316ad-6c07-4f70-ba55-9894d1689735\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-26d5d1-67664944-mzhwg"
Sep 17 03:43:51.608760 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:43:51.608695    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdgdf\" (UniqueName: \"kubernetes.io/projected/600316ad-6c07-4f70-ba55-9894d1689735-kube-api-access-wdgdf\") pod \"maas-api-e2e-api-b-26d5d1-67664944-mzhwg\" (UID: \"600316ad-6c07-4f70-ba55-9894d1689735\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-26d5d1-67664944-mzhwg"
Sep 17 03:43:51.709857 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:43:51.709830    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/600316ad-6c07-4f70-ba55-9894d1689735-maas-api-tls\") pod \"maas-api-e2e-api-b-26d5d1-67664944-mzhwg\" (UID: \"600316ad-6c07-4f70-ba55-9894d1689735\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-26d5d1-67664944-mzhwg"
Sep 17 03:43:51.710027 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:43:51.709872    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdgdf\" (UniqueName: \"kubernetes.io/projected/600316ad-6c07-4f70-ba55-9894d1689735-kube-api-access-wdgdf\") pod \"maas-api-e2e-api-b-26d5d1-67664944-mzhwg\" (UID: \"600316ad-6c07-4f70-ba55-9894d1689735\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-26d5d1-67664944-mzhwg"
Sep 17 03:43:51.710027 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:43:51.709957    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/600316ad-6c07-4f70-ba55-9894d1689735-openshift-service-ca\") pod \"maas-api-e2e-api-b-26d5d1-67664944-mzhwg\" (UID: \"600316ad-6c07-4f70-ba55-9894d1689735\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-26d5d1-67664944-mzhwg"
Sep 17 03:43:51.710027 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:43:51.709968    2552 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-26d5d1-serving-cert: secret "maas-api-e2e-api-b-26d5d1-serving-cert" not found
Sep 17 03:43:51.710027 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:43:51.709987    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/600316ad-6c07-4f70-ba55-9894d1689735-metrics-certs\") pod \"maas-api-e2e-api-b-26d5d1-67664944-mzhwg\" (UID: \"600316ad-6c07-4f70-ba55-9894d1689735\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-26d5d1-67664944-mzhwg"
Sep 17 03:43:51.710240 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:43:51.710046    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/600316ad-6c07-4f70-ba55-9894d1689735-maas-api-tls podName:600316ad-6c07-4f70-ba55-9894d1689735 nodeName:}" failed. No retries permitted until 2026-09-17 03:43:52.210024473 +0000 UTC m=+1467.676431992 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/600316ad-6c07-4f70-ba55-9894d1689735-maas-api-tls") pod "maas-api-e2e-api-b-26d5d1-67664944-mzhwg" (UID: "600316ad-6c07-4f70-ba55-9894d1689735") : secret "maas-api-e2e-api-b-26d5d1-serving-cert" not found
Sep 17 03:43:51.710658 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:43:51.710640    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/600316ad-6c07-4f70-ba55-9894d1689735-openshift-service-ca\") pod \"maas-api-e2e-api-b-26d5d1-67664944-mzhwg\" (UID: \"600316ad-6c07-4f70-ba55-9894d1689735\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-26d5d1-67664944-mzhwg"
Sep 17 03:43:51.712435 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:43:51.712414    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/600316ad-6c07-4f70-ba55-9894d1689735-metrics-certs\") pod \"maas-api-e2e-api-b-26d5d1-67664944-mzhwg\" (UID: \"600316ad-6c07-4f70-ba55-9894d1689735\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-26d5d1-67664944-mzhwg"
Sep 17 03:43:51.720666 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:43:51.720642    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdgdf\" (UniqueName: \"kubernetes.io/projected/600316ad-6c07-4f70-ba55-9894d1689735-kube-api-access-wdgdf\") pod \"maas-api-e2e-api-b-26d5d1-67664944-mzhwg\" (UID: \"600316ad-6c07-4f70-ba55-9894d1689735\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-26d5d1-67664944-mzhwg"
Sep 17 03:43:52.213929 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:43:52.213899    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/600316ad-6c07-4f70-ba55-9894d1689735-maas-api-tls\") pod \"maas-api-e2e-api-b-26d5d1-67664944-mzhwg\" (UID: \"600316ad-6c07-4f70-ba55-9894d1689735\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-26d5d1-67664944-mzhwg"
Sep 17 03:43:52.214113 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:43:52.214023    2552 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-26d5d1-serving-cert: secret "maas-api-e2e-api-b-26d5d1-serving-cert" not found
Sep 17 03:43:52.214113 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:43:52.214083    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/600316ad-6c07-4f70-ba55-9894d1689735-maas-api-tls podName:600316ad-6c07-4f70-ba55-9894d1689735 nodeName:}" failed. No retries permitted until 2026-09-17 03:43:53.214068471 +0000 UTC m=+1468.680475975 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/600316ad-6c07-4f70-ba55-9894d1689735-maas-api-tls") pod "maas-api-e2e-api-b-26d5d1-67664944-mzhwg" (UID: "600316ad-6c07-4f70-ba55-9894d1689735") : secret "maas-api-e2e-api-b-26d5d1-serving-cert" not found
Sep 17 03:43:53.223834 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:43:53.223798    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/600316ad-6c07-4f70-ba55-9894d1689735-maas-api-tls\") pod \"maas-api-e2e-api-b-26d5d1-67664944-mzhwg\" (UID: \"600316ad-6c07-4f70-ba55-9894d1689735\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-26d5d1-67664944-mzhwg"
Sep 17 03:43:53.226187 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:43:53.226167    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/600316ad-6c07-4f70-ba55-9894d1689735-maas-api-tls\") pod \"maas-api-e2e-api-b-26d5d1-67664944-mzhwg\" (UID: \"600316ad-6c07-4f70-ba55-9894d1689735\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-26d5d1-67664944-mzhwg"
Sep 17 03:43:53.398189 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:43:53.398159    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-26d5d1-67664944-mzhwg"
Sep 17 03:43:53.672708 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:43:53.672684    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-26d5d1-67664944-mzhwg"]
Sep 17 03:43:53.674921 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:43:53.674895    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod600316ad_6c07_4f70_ba55_9894d1689735.slice/crio-b5ec9e2fcf2df63890892e87bb888cc8a690b0010d8d96a690c2a508657f4d2c WatchSource:0}: Error finding container b5ec9e2fcf2df63890892e87bb888cc8a690b0010d8d96a690c2a508657f4d2c: Status 404 returned error can't find the container with id b5ec9e2fcf2df63890892e87bb888cc8a690b0010d8d96a690c2a508657f4d2c
Sep 17 03:43:53.676640 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:43:53.676620    2552 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 03:43:53.824407 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:43:53.824370    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-26d5d1-67664944-mzhwg" event={"ID":"600316ad-6c07-4f70-ba55-9894d1689735","Type":"ContainerStarted","Data":"b5ec9e2fcf2df63890892e87bb888cc8a690b0010d8d96a690c2a508657f4d2c"}
Sep 17 03:43:54.831649 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:43:54.831592    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-26d5d1-67664944-mzhwg" event={"ID":"600316ad-6c07-4f70-ba55-9894d1689735","Type":"ContainerStarted","Data":"0db012661b0451d99406d8eac69ea2b306e38416bdc88a9e4014e72f2c08d23c"}
Sep 17 03:43:54.832124 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:43:54.831748    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-26d5d1-67664944-mzhwg"
Sep 17 03:43:54.849664 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:43:54.849623    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-26d5d1-67664944-mzhwg" podStartSLOduration=3.475072791 podStartE2EDuration="3.849585344s" totalImagesPullingTime="374.512553ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 03:43:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:43:54.848897563 +0000 UTC m=+1470.315305090" watchObservedRunningTime="2026-09-17 03:43:54.849585344 +0000 UTC m=+1470.315992870"
Sep 17 03:44:00.842995 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:00.842965    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-26d5d1-67664944-mzhwg"
Sep 17 03:44:06.951553 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:06.951521    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-1afd34/distinct-2-w0-1afd34-kserve-5457449c9-bqjmv"]
Sep 17 03:44:06.987846 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:06.987812    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-1afd34/embedding-w0-1afd34-kserve-7c79989f85-9k45s"]
Sep 17 03:44:07.004313 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:07.004283    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-1afd34/premium-w0-1afd34-kserve-85c998d845-r5zq9"]
Sep 17 03:44:07.017047 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:07.017021    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-1afd34/simulator-w0-1afd34-kserve-6c5ffdd47b-v6h59"]
Sep 17 03:44:20.860446 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:20.860266    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-1afd34-664ddf4d68-7pnj6"]
Sep 17 03:44:20.861073 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:20.860553    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1afd34-664ddf4d68-7pnj6" podUID="6c76c1a2-3cb7-43ae-be88-fe95001caf30" containerName="maas-api" containerID="cri-o://ef100d227e639097f69c90bfb47ed63c548d6253085e5172446f9bc4db772d8a" gracePeriod=30
Sep 17 03:44:21.325406 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:21.325379    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1afd34-664ddf4d68-7pnj6"
Sep 17 03:44:21.520926 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:21.520882    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c76c1a2-3cb7-43ae-be88-fe95001caf30-metrics-certs\") pod \"6c76c1a2-3cb7-43ae-be88-fe95001caf30\" (UID: \"6c76c1a2-3cb7-43ae-be88-fe95001caf30\") "
Sep 17 03:44:21.521116 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:21.520977    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c76c1a2-3cb7-43ae-be88-fe95001caf30-maas-api-tls\") pod \"6c76c1a2-3cb7-43ae-be88-fe95001caf30\" (UID: \"6c76c1a2-3cb7-43ae-be88-fe95001caf30\") "
Sep 17 03:44:21.521116 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:21.521033    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wpbzv\" (UniqueName: \"kubernetes.io/projected/6c76c1a2-3cb7-43ae-be88-fe95001caf30-kube-api-access-wpbzv\") pod \"6c76c1a2-3cb7-43ae-be88-fe95001caf30\" (UID: \"6c76c1a2-3cb7-43ae-be88-fe95001caf30\") "
Sep 17 03:44:21.521116 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:21.521071    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c76c1a2-3cb7-43ae-be88-fe95001caf30-openshift-service-ca\") pod \"6c76c1a2-3cb7-43ae-be88-fe95001caf30\" (UID: \"6c76c1a2-3cb7-43ae-be88-fe95001caf30\") "
Sep 17 03:44:21.521509 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:21.521470    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c76c1a2-3cb7-43ae-be88-fe95001caf30-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6c76c1a2-3cb7-43ae-be88-fe95001caf30" (UID: "6c76c1a2-3cb7-43ae-be88-fe95001caf30"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 03:44:21.523150 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:21.523114    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c76c1a2-3cb7-43ae-be88-fe95001caf30-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6c76c1a2-3cb7-43ae-be88-fe95001caf30" (UID: "6c76c1a2-3cb7-43ae-be88-fe95001caf30"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 03:44:21.523252 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:21.523194    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c76c1a2-3cb7-43ae-be88-fe95001caf30-kube-api-access-wpbzv" (OuterVolumeSpecName: "kube-api-access-wpbzv") pod "6c76c1a2-3cb7-43ae-be88-fe95001caf30" (UID: "6c76c1a2-3cb7-43ae-be88-fe95001caf30"). InnerVolumeSpecName "kube-api-access-wpbzv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 03:44:21.523304 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:21.523270    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c76c1a2-3cb7-43ae-be88-fe95001caf30-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "6c76c1a2-3cb7-43ae-be88-fe95001caf30" (UID: "6c76c1a2-3cb7-43ae-be88-fe95001caf30"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 03:44:21.622171 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:21.622143    2552 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c76c1a2-3cb7-43ae-be88-fe95001caf30-metrics-certs\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:44:21.622171 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:21.622166    2552 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c76c1a2-3cb7-43ae-be88-fe95001caf30-maas-api-tls\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:44:21.622375 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:21.622177    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wpbzv\" (UniqueName: \"kubernetes.io/projected/6c76c1a2-3cb7-43ae-be88-fe95001caf30-kube-api-access-wpbzv\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:44:21.622375 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:21.622198    2552 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c76c1a2-3cb7-43ae-be88-fe95001caf30-openshift-service-ca\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:44:21.970915 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:21.970828    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-1afd34/distinct-2-w0-1afd34-kserve-5457449c9-bqjmv" podUID="2a2ec719-0a31-439c-96af-ee004af31201" containerName="main" containerID="cri-o://13207976dd7484b88ff439e9edcc3cbc78ed36e7f21b5370d004552f1d53e287" gracePeriod=45
Sep 17 03:44:21.982704 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:21.982674    2552 generic.go:356] "Generic (PLEG): container finished" podID="6c76c1a2-3cb7-43ae-be88-fe95001caf30" containerID="ef100d227e639097f69c90bfb47ed63c548d6253085e5172446f9bc4db772d8a" exitCode=0
Sep 17 03:44:21.982805 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:21.982708    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1afd34-664ddf4d68-7pnj6"
Sep 17 03:44:21.982805 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:21.982753    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1afd34-664ddf4d68-7pnj6" event={"ID":"6c76c1a2-3cb7-43ae-be88-fe95001caf30","Type":"ContainerDied","Data":"ef100d227e639097f69c90bfb47ed63c548d6253085e5172446f9bc4db772d8a"}
Sep 17 03:44:21.982805 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:21.982796    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1afd34-664ddf4d68-7pnj6" event={"ID":"6c76c1a2-3cb7-43ae-be88-fe95001caf30","Type":"ContainerDied","Data":"1889e98fcbca3036159993af0d911e0efd3cdc27aad16d9c3968f050dacd79d3"}
Sep 17 03:44:21.982911 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:21.982823    2552 scope.go:117] "RemoveContainer" containerID="ef100d227e639097f69c90bfb47ed63c548d6253085e5172446f9bc4db772d8a"
Sep 17 03:44:22.006557 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:22.006529    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-1afd34/embedding-w0-1afd34-kserve-7c79989f85-9k45s" podUID="36aa6a53-4557-4431-b02d-cde2e73afd04" containerName="main" containerID="cri-o://6ff98b094f0c969d6d71318e8cd03d4cadff7769418625d8d7b7e93286ad066d" gracePeriod=45
Sep 17 03:44:22.023028 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:22.023002    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-1afd34/premium-w0-1afd34-kserve-85c998d845-r5zq9" podUID="6d3098da-c426-4ed1-8932-737fb644755c" containerName="main" containerID="cri-o://2ca1069b5dcba8a8513036fa09f56698a813a07d12d8932fe80da3941b2489f2" gracePeriod=45
Sep 17 03:44:22.051233 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:22.051155    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-1afd34/simulator-w0-1afd34-kserve-6c5ffdd47b-v6h59" podUID="5886143a-fc36-40eb-8818-b2c94767b529" containerName="main" containerID="cri-o://b39473b5ea95e866130295afdb931f04ce58775c9297473637182719e2a79fcb" gracePeriod=45
Sep 17 03:44:22.177189 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:22.177149    2552 scope.go:117] "RemoveContainer" containerID="ef100d227e639097f69c90bfb47ed63c548d6253085e5172446f9bc4db772d8a"
Sep 17 03:44:22.177644 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:44:22.177542    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef100d227e639097f69c90bfb47ed63c548d6253085e5172446f9bc4db772d8a\": container with ID starting with ef100d227e639097f69c90bfb47ed63c548d6253085e5172446f9bc4db772d8a not found: ID does not exist" containerID="ef100d227e639097f69c90bfb47ed63c548d6253085e5172446f9bc4db772d8a"
Sep 17 03:44:22.177644 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:22.177583    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef100d227e639097f69c90bfb47ed63c548d6253085e5172446f9bc4db772d8a"} err="failed to get container status \"ef100d227e639097f69c90bfb47ed63c548d6253085e5172446f9bc4db772d8a\": rpc error: code = NotFound desc = could not find container \"ef100d227e639097f69c90bfb47ed63c548d6253085e5172446f9bc4db772d8a\": container with ID starting with ef100d227e639097f69c90bfb47ed63c548d6253085e5172446f9bc4db772d8a not found: ID does not exist"
Sep 17 03:44:22.187964 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:22.187933    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-1afd34-664ddf4d68-7pnj6"]
Sep 17 03:44:22.194515 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:22.194450    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-1afd34-664ddf4d68-7pnj6"]
Sep 17 03:44:22.379032 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:22.379001    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1afd34/embedding-w0-1afd34-kserve-7c79989f85-9k45s"
Sep 17 03:44:22.455621 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:22.450935    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1afd34/simulator-w0-1afd34-kserve-6c5ffdd47b-v6h59"
Sep 17 03:44:22.455621 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:22.452825    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1afd34/premium-w0-1afd34-kserve-85c998d845-r5zq9"
Sep 17 03:44:22.455859 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:22.455723    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1afd34/distinct-2-w0-1afd34-kserve-5457449c9-bqjmv"
Sep 17 03:44:22.542621 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:22.540865    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36aa6a53-4557-4431-b02d-cde2e73afd04-dshm\") pod \"36aa6a53-4557-4431-b02d-cde2e73afd04\" (UID: \"36aa6a53-4557-4431-b02d-cde2e73afd04\") "
Sep 17 03:44:22.542621 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:22.540907    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36aa6a53-4557-4431-b02d-cde2e73afd04-model-cache\") pod \"36aa6a53-4557-4431-b02d-cde2e73afd04\" (UID: \"36aa6a53-4557-4431-b02d-cde2e73afd04\") "
Sep 17 03:44:22.542621 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:22.540944    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36aa6a53-4557-4431-b02d-cde2e73afd04-tls-certs\") pod \"36aa6a53-4557-4431-b02d-cde2e73afd04\" (UID: \"36aa6a53-4557-4431-b02d-cde2e73afd04\") "
Sep 17 03:44:22.542621 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:22.541049    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36aa6a53-4557-4431-b02d-cde2e73afd04-tmp-dir\") pod \"36aa6a53-4557-4431-b02d-cde2e73afd04\" (UID: \"36aa6a53-4557-4431-b02d-cde2e73afd04\") "
Sep 17 03:44:22.542621 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:22.541129    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36aa6a53-4557-4431-b02d-cde2e73afd04-home\") pod \"36aa6a53-4557-4431-b02d-cde2e73afd04\" (UID: \"36aa6a53-4557-4431-b02d-cde2e73afd04\") "
Sep 17 03:44:22.542621 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:22.541181    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gk8ww\" (UniqueName: \"kubernetes.io/projected/36aa6a53-4557-4431-b02d-cde2e73afd04-kube-api-access-gk8ww\") pod \"36aa6a53-4557-4431-b02d-cde2e73afd04\" (UID: \"36aa6a53-4557-4431-b02d-cde2e73afd04\") "
Sep 17 03:44:22.543100 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:22.542896    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36aa6a53-4557-4431-b02d-cde2e73afd04-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "36aa6a53-4557-4431-b02d-cde2e73afd04" (UID: "36aa6a53-4557-4431-b02d-cde2e73afd04"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:44:22.543100 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:22.542957    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36aa6a53-4557-4431-b02d-cde2e73afd04-model-cache" (OuterVolumeSpecName: "model-cache") pod "36aa6a53-4557-4431-b02d-cde2e73afd04" (UID: "36aa6a53-4557-4431-b02d-cde2e73afd04"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:44:22.543219 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:22.543162    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36aa6a53-4557-4431-b02d-cde2e73afd04-home" (OuterVolumeSpecName: "home") pod "36aa6a53-4557-4431-b02d-cde2e73afd04" (UID: "36aa6a53-4557-4431-b02d-cde2e73afd04"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:44:22.549389 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:22.547922    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36aa6a53-4557-4431-b02d-cde2e73afd04-kube-api-access-gk8ww" (OuterVolumeSpecName: "kube-api-access-gk8ww") pod "36aa6a53-4557-4431-b02d-cde2e73afd04" (UID: "36aa6a53-4557-4431-b02d-cde2e73afd04"). InnerVolumeSpecName "kube-api-access-gk8ww". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 03:44:22.549389 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:22.548028    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36aa6a53-4557-4431-b02d-cde2e73afd04-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "36aa6a53-4557-4431-b02d-cde2e73afd04" (UID: "36aa6a53-4557-4431-b02d-cde2e73afd04"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 03:44:22.549389 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:22.548681    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36aa6a53-4557-4431-b02d-cde2e73afd04-dshm" (OuterVolumeSpecName: "dshm") pod "36aa6a53-4557-4431-b02d-cde2e73afd04" (UID: "36aa6a53-4557-4431-b02d-cde2e73afd04"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:44:22.645238 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:22.642943    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d3098da-c426-4ed1-8932-737fb644755c-home\") pod \"6d3098da-c426-4ed1-8932-737fb644755c\" (UID: \"6d3098da-c426-4ed1-8932-737fb644755c\") "
Sep 17 03:44:22.645238 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:22.643004    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4nwlc\" (UniqueName: \"kubernetes.io/projected/2a2ec719-0a31-439c-96af-ee004af31201-kube-api-access-4nwlc\") pod \"2a2ec719-0a31-439c-96af-ee004af31201\" (UID: \"2a2ec719-0a31-439c-96af-ee004af31201\") "
Sep 17 03:44:22.645238 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:22.643050    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a2ec719-0a31-439c-96af-ee004af31201-dshm\") pod \"2a2ec719-0a31-439c-96af-ee004af31201\" (UID: \"2a2ec719-0a31-439c-96af-ee004af31201\") "
Sep 17 03:44:22.645238 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:22.643082    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5886143a-fc36-40eb-8818-b2c94767b529-home\") pod \"5886143a-fc36-40eb-8818-b2c94767b529\" (UID: \"5886143a-fc36-40eb-8818-b2c94767b529\") "
Sep 17 03:44:22.645238 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:22.643125    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d3098da-c426-4ed1-8932-737fb644755c-tmp-dir\") pod \"6d3098da-c426-4ed1-8932-737fb644755c\" (UID: \"6d3098da-c426-4ed1-8932-737fb644755c\") "
Sep 17 03:44:22.645238 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:22.643162    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8hxq9\" (UniqueName: \"kubernetes.io/projected/5886143a-fc36-40eb-8818-b2c94767b529-kube-api-access-8hxq9\") pod \"5886143a-fc36-40eb-8818-b2c94767b529\" (UID: \"5886143a-fc36-40eb-8818-b2c94767b529\") "
Sep 17 03:44:22.645238 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:22.643185    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5886143a-fc36-40eb-8818-b2c94767b529-tmp-dir\") pod \"5886143a-fc36-40eb-8818-b2c94767b529\" (UID: \"5886143a-fc36-40eb-8818-b2c94767b529\") "
Sep 17 03:44:22.645238 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:22.643228    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5886143a-fc36-40eb-8818-b2c94767b529-dshm\") pod \"5886143a-fc36-40eb-8818-b2c94767b529\" (UID: \"5886143a-fc36-40eb-8818-b2c94767b529\") "
Sep 17 03:44:22.645238 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:22.643252    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d3098da-c426-4ed1-8932-737fb644755c-model-cache\") pod \"6d3098da-c426-4ed1-8932-737fb644755c\" (UID: \"6d3098da-c426-4ed1-8932-737fb644755c\") "
Sep 17 03:44:22.645238 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:22.643300    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a2ec719-0a31-439c-96af-ee004af31201-tls-certs\") pod \"2a2ec719-0a31-439c-96af-ee004af31201\" (UID: \"2a2ec719-0a31-439c-96af-ee004af31201\") "
Sep 17 03:44:22.645238 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:22.643333    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a2ec719-0a31-439c-96af-ee004af31201-model-cache\") pod \"2a2ec719-0a31-439c-96af-ee004af31201\" (UID: \"2a2ec719-0a31-439c-96af-ee004af31201\") "
Sep 17 03:44:22.645238 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:22.643362    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d3098da-c426-4ed1-8932-737fb644755c-dshm\") pod \"6d3098da-c426-4ed1-8932-737fb644755c\" (UID: \"6d3098da-c426-4ed1-8932-737fb644755c\") "
Sep 17 03:44:22.645238 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:22.643399    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a2ec719-0a31-439c-96af-ee004af31201-home\") pod \"2a2ec719-0a31-439c-96af-ee004af31201\" (UID: \"2a2ec719-0a31-439c-96af-ee004af31201\") "
Sep 17 03:44:22.645238 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:22.643454    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d3098da-c426-4ed1-8932-737fb644755c-tls-certs\") pod \"6d3098da-c426-4ed1-8932-737fb644755c\" (UID: \"6d3098da-c426-4ed1-8932-737fb644755c\") "
Sep 17 03:44:22.645238 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:22.643487    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5886143a-fc36-40eb-8818-b2c94767b529-tls-certs\") pod \"5886143a-fc36-40eb-8818-b2c94767b529\" (UID: \"5886143a-fc36-40eb-8818-b2c94767b529\") "
Sep 17 03:44:22.645238 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:22.643542    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5886143a-fc36-40eb-8818-b2c94767b529-model-cache\") pod \"5886143a-fc36-40eb-8818-b2c94767b529\" (UID: \"5886143a-fc36-40eb-8818-b2c94767b529\") "
Sep 17 03:44:22.645238 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:22.643571    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vcncx\" (UniqueName: \"kubernetes.io/projected/6d3098da-c426-4ed1-8932-737fb644755c-kube-api-access-vcncx\") pod \"6d3098da-c426-4ed1-8932-737fb644755c\" (UID: \"6d3098da-c426-4ed1-8932-737fb644755c\") "
Sep 17 03:44:22.645238 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:22.643619    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a2ec719-0a31-439c-96af-ee004af31201-tmp-dir\") pod \"2a2ec719-0a31-439c-96af-ee004af31201\" (UID: \"2a2ec719-0a31-439c-96af-ee004af31201\") "
Sep 17 03:44:22.645238 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:22.644048    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gk8ww\" (UniqueName: \"kubernetes.io/projected/36aa6a53-4557-4431-b02d-cde2e73afd04-kube-api-access-gk8ww\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:44:22.645238 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:22.644067    2552 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36aa6a53-4557-4431-b02d-cde2e73afd04-dshm\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:44:22.646456 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:22.644081    2552 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36aa6a53-4557-4431-b02d-cde2e73afd04-model-cache\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:44:22.646456 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:22.644095    2552 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36aa6a53-4557-4431-b02d-cde2e73afd04-tls-certs\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:44:22.646456 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:22.644107    2552 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36aa6a53-4557-4431-b02d-cde2e73afd04-tmp-dir\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:44:22.646456 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:22.644119    2552 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36aa6a53-4557-4431-b02d-cde2e73afd04-home\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:44:22.646456 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:22.644350    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a2ec719-0a31-439c-96af-ee004af31201-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2a2ec719-0a31-439c-96af-ee004af31201" (UID: "2a2ec719-0a31-439c-96af-ee004af31201"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:44:22.646456 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:22.644503    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d3098da-c426-4ed1-8932-737fb644755c-home" (OuterVolumeSpecName: "home") pod "6d3098da-c426-4ed1-8932-737fb644755c" (UID: "6d3098da-c426-4ed1-8932-737fb644755c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:44:22.647643 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:22.647386    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5886143a-fc36-40eb-8818-b2c94767b529-home" (OuterVolumeSpecName: "home") pod "5886143a-fc36-40eb-8818-b2c94767b529" (UID: "5886143a-fc36-40eb-8818-b2c94767b529"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:44:22.648291 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:22.648268    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5886143a-fc36-40eb-8818-b2c94767b529-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5886143a-fc36-40eb-8818-b2c94767b529" (UID: "5886143a-fc36-40eb-8818-b2c94767b529"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:44:22.648624 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:22.648582    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d3098da-c426-4ed1-8932-737fb644755c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6d3098da-c426-4ed1-8932-737fb644755c" (UID: "6d3098da-c426-4ed1-8932-737fb644755c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:44:22.648921 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:22.648902    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d3098da-c426-4ed1-8932-737fb644755c-model-cache" (OuterVolumeSpecName: "model-cache") pod "6d3098da-c426-4ed1-8932-737fb644755c" (UID: "6d3098da-c426-4ed1-8932-737fb644755c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:44:22.649205 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:22.649187    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a2ec719-0a31-439c-96af-ee004af31201-home" (OuterVolumeSpecName: "home") pod "2a2ec719-0a31-439c-96af-ee004af31201" (UID: "2a2ec719-0a31-439c-96af-ee004af31201"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:44:22.649482 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:22.649463    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5886143a-fc36-40eb-8818-b2c94767b529-model-cache" (OuterVolumeSpecName: "model-cache") pod "5886143a-fc36-40eb-8818-b2c94767b529" (UID: "5886143a-fc36-40eb-8818-b2c94767b529"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:44:22.653627 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:22.652887    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a2ec719-0a31-439c-96af-ee004af31201-model-cache" (OuterVolumeSpecName: "model-cache") pod "2a2ec719-0a31-439c-96af-ee004af31201" (UID: "2a2ec719-0a31-439c-96af-ee004af31201"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:44:22.654821 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:22.654760    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5886143a-fc36-40eb-8818-b2c94767b529-dshm" (OuterVolumeSpecName: "dshm") pod "5886143a-fc36-40eb-8818-b2c94767b529" (UID: "5886143a-fc36-40eb-8818-b2c94767b529"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:44:22.654936 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:22.654821    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a2ec719-0a31-439c-96af-ee004af31201-dshm" (OuterVolumeSpecName: "dshm") pod "2a2ec719-0a31-439c-96af-ee004af31201" (UID: "2a2ec719-0a31-439c-96af-ee004af31201"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:44:22.655455 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:22.655432    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d3098da-c426-4ed1-8932-737fb644755c-dshm" (OuterVolumeSpecName: "dshm") pod "6d3098da-c426-4ed1-8932-737fb644755c" (UID: "6d3098da-c426-4ed1-8932-737fb644755c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:44:22.655983 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:22.655957    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a2ec719-0a31-439c-96af-ee004af31201-kube-api-access-4nwlc" (OuterVolumeSpecName: "kube-api-access-4nwlc") pod "2a2ec719-0a31-439c-96af-ee004af31201" (UID: "2a2ec719-0a31-439c-96af-ee004af31201"). InnerVolumeSpecName "kube-api-access-4nwlc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 03:44:22.656382 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:22.656334    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a2ec719-0a31-439c-96af-ee004af31201-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2a2ec719-0a31-439c-96af-ee004af31201" (UID: "2a2ec719-0a31-439c-96af-ee004af31201"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 03:44:22.656479 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:22.656429    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d3098da-c426-4ed1-8932-737fb644755c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6d3098da-c426-4ed1-8932-737fb644755c" (UID: "6d3098da-c426-4ed1-8932-737fb644755c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 03:44:22.656479 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:22.656465    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5886143a-fc36-40eb-8818-b2c94767b529-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5886143a-fc36-40eb-8818-b2c94767b529" (UID: "5886143a-fc36-40eb-8818-b2c94767b529"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 03:44:22.656717 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:22.656699    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d3098da-c426-4ed1-8932-737fb644755c-kube-api-access-vcncx" (OuterVolumeSpecName: "kube-api-access-vcncx") pod "6d3098da-c426-4ed1-8932-737fb644755c" (UID: "6d3098da-c426-4ed1-8932-737fb644755c"). InnerVolumeSpecName "kube-api-access-vcncx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 03:44:22.656955 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:22.656931    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5886143a-fc36-40eb-8818-b2c94767b529-kube-api-access-8hxq9" (OuterVolumeSpecName: "kube-api-access-8hxq9") pod "5886143a-fc36-40eb-8818-b2c94767b529" (UID: "5886143a-fc36-40eb-8818-b2c94767b529"). InnerVolumeSpecName "kube-api-access-8hxq9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 03:44:22.745981 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:22.745717    2552 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a2ec719-0a31-439c-96af-ee004af31201-model-cache\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:44:22.745981 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:22.745751    2552 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d3098da-c426-4ed1-8932-737fb644755c-dshm\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:44:22.745981 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:22.745766    2552 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a2ec719-0a31-439c-96af-ee004af31201-home\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:44:22.745981 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:22.745780    2552 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d3098da-c426-4ed1-8932-737fb644755c-tls-certs\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:44:22.745981 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:22.745793    2552 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5886143a-fc36-40eb-8818-b2c94767b529-tls-certs\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:44:22.745981 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:22.745805    2552 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5886143a-fc36-40eb-8818-b2c94767b529-model-cache\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:44:22.745981 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:22.745817    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vcncx\" (UniqueName: \"kubernetes.io/projected/6d3098da-c426-4ed1-8932-737fb644755c-kube-api-access-vcncx\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:44:22.745981 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:22.745832    2552 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a2ec719-0a31-439c-96af-ee004af31201-tmp-dir\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:44:22.745981 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:22.745845    2552 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d3098da-c426-4ed1-8932-737fb644755c-home\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:44:22.745981 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:22.745857    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4nwlc\" (UniqueName: \"kubernetes.io/projected/2a2ec719-0a31-439c-96af-ee004af31201-kube-api-access-4nwlc\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:44:22.745981 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:22.745870    2552 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a2ec719-0a31-439c-96af-ee004af31201-dshm\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:44:22.745981 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:22.745882    2552 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5886143a-fc36-40eb-8818-b2c94767b529-home\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:44:22.745981 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:22.745895    2552 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d3098da-c426-4ed1-8932-737fb644755c-tmp-dir\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:44:22.745981 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:22.745907    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8hxq9\" (UniqueName: \"kubernetes.io/projected/5886143a-fc36-40eb-8818-b2c94767b529-kube-api-access-8hxq9\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:44:22.745981 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:22.745919    2552 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5886143a-fc36-40eb-8818-b2c94767b529-tmp-dir\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:44:22.745981 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:22.745933    2552 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5886143a-fc36-40eb-8818-b2c94767b529-dshm\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:44:22.745981 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:22.745944    2552 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d3098da-c426-4ed1-8932-737fb644755c-model-cache\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:44:22.745981 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:22.745956    2552 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a2ec719-0a31-439c-96af-ee004af31201-tls-certs\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:44:22.986111 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:22.986017    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6c76c1a2-3cb7-43ae-be88-fe95001caf30" path="/var/lib/kubelet/pods/6c76c1a2-3cb7-43ae-be88-fe95001caf30/volumes"
Sep 17 03:44:22.989366 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:22.989332    2552 generic.go:356] "Generic (PLEG): container finished" podID="2a2ec719-0a31-439c-96af-ee004af31201" containerID="13207976dd7484b88ff439e9edcc3cbc78ed36e7f21b5370d004552f1d53e287" exitCode=0
Sep 17 03:44:22.989497 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:22.989361    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1afd34/distinct-2-w0-1afd34-kserve-5457449c9-bqjmv" event={"ID":"2a2ec719-0a31-439c-96af-ee004af31201","Type":"ContainerDied","Data":"13207976dd7484b88ff439e9edcc3cbc78ed36e7f21b5370d004552f1d53e287"}
Sep 17 03:44:22.989497 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:22.989387    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1afd34/distinct-2-w0-1afd34-kserve-5457449c9-bqjmv"
Sep 17 03:44:22.989497 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:22.989416    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1afd34/distinct-2-w0-1afd34-kserve-5457449c9-bqjmv" event={"ID":"2a2ec719-0a31-439c-96af-ee004af31201","Type":"ContainerDied","Data":"efb7d5c0cd0eb006278e9cd092ec10884cab6fd87573821675326c555c3eb241"}
Sep 17 03:44:22.989497 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:22.989439    2552 scope.go:117] "RemoveContainer" containerID="13207976dd7484b88ff439e9edcc3cbc78ed36e7f21b5370d004552f1d53e287"
Sep 17 03:44:22.991017 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:22.990997    2552 generic.go:356] "Generic (PLEG): container finished" podID="6d3098da-c426-4ed1-8932-737fb644755c" containerID="2ca1069b5dcba8a8513036fa09f56698a813a07d12d8932fe80da3941b2489f2" exitCode=0
Sep 17 03:44:22.991109 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:22.991071    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1afd34/premium-w0-1afd34-kserve-85c998d845-r5zq9" event={"ID":"6d3098da-c426-4ed1-8932-737fb644755c","Type":"ContainerDied","Data":"2ca1069b5dcba8a8513036fa09f56698a813a07d12d8932fe80da3941b2489f2"}
Sep 17 03:44:22.991109 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:22.991100    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1afd34/premium-w0-1afd34-kserve-85c998d845-r5zq9" event={"ID":"6d3098da-c426-4ed1-8932-737fb644755c","Type":"ContainerDied","Data":"76ee1f0fd34b3962c74eab7838f74f7619c5878e23eda1e6d200addd6b47f80f"}
Sep 17 03:44:22.991332 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:22.991153    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1afd34/premium-w0-1afd34-kserve-85c998d845-r5zq9"
Sep 17 03:44:22.997522 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:22.993371    2552 generic.go:356] "Generic (PLEG): container finished" podID="5886143a-fc36-40eb-8818-b2c94767b529" containerID="b39473b5ea95e866130295afdb931f04ce58775c9297473637182719e2a79fcb" exitCode=0
Sep 17 03:44:22.997522 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:22.993455    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1afd34/simulator-w0-1afd34-kserve-6c5ffdd47b-v6h59" event={"ID":"5886143a-fc36-40eb-8818-b2c94767b529","Type":"ContainerDied","Data":"b39473b5ea95e866130295afdb931f04ce58775c9297473637182719e2a79fcb"}
Sep 17 03:44:22.997522 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:22.993484    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1afd34/simulator-w0-1afd34-kserve-6c5ffdd47b-v6h59" event={"ID":"5886143a-fc36-40eb-8818-b2c94767b529","Type":"ContainerDied","Data":"26544e43367797df736b5f932512c3ba0f5aa954bf34d8a9f7fd3670b3e08dc6"}
Sep 17 03:44:22.997522 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:22.993532    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1afd34/simulator-w0-1afd34-kserve-6c5ffdd47b-v6h59"
Sep 17 03:44:22.999255 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:22.999154    2552 generic.go:356] "Generic (PLEG): container finished" podID="36aa6a53-4557-4431-b02d-cde2e73afd04" containerID="6ff98b094f0c969d6d71318e8cd03d4cadff7769418625d8d7b7e93286ad066d" exitCode=0
Sep 17 03:44:22.999255 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:22.999183    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1afd34/embedding-w0-1afd34-kserve-7c79989f85-9k45s" event={"ID":"36aa6a53-4557-4431-b02d-cde2e73afd04","Type":"ContainerDied","Data":"6ff98b094f0c969d6d71318e8cd03d4cadff7769418625d8d7b7e93286ad066d"}
Sep 17 03:44:22.999255 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:22.999197    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1afd34/embedding-w0-1afd34-kserve-7c79989f85-9k45s"
Sep 17 03:44:22.999255 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:22.999207    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1afd34/embedding-w0-1afd34-kserve-7c79989f85-9k45s" event={"ID":"36aa6a53-4557-4431-b02d-cde2e73afd04","Type":"ContainerDied","Data":"4fd3143e8c15a13a9cf9a61ba8a137013d4cfecb301612b84f06b24410f57760"}
Sep 17 03:44:23.005481 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:23.005457    2552 scope.go:117] "RemoveContainer" containerID="13207976dd7484b88ff439e9edcc3cbc78ed36e7f21b5370d004552f1d53e287"
Sep 17 03:44:23.005784 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:44:23.005760    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13207976dd7484b88ff439e9edcc3cbc78ed36e7f21b5370d004552f1d53e287\": container with ID starting with 13207976dd7484b88ff439e9edcc3cbc78ed36e7f21b5370d004552f1d53e287 not found: ID does not exist" containerID="13207976dd7484b88ff439e9edcc3cbc78ed36e7f21b5370d004552f1d53e287"
Sep 17 03:44:23.005920 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:23.005790    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13207976dd7484b88ff439e9edcc3cbc78ed36e7f21b5370d004552f1d53e287"} err="failed to get container status \"13207976dd7484b88ff439e9edcc3cbc78ed36e7f21b5370d004552f1d53e287\": rpc error: code = NotFound desc = could not find container \"13207976dd7484b88ff439e9edcc3cbc78ed36e7f21b5370d004552f1d53e287\": container with ID starting with 13207976dd7484b88ff439e9edcc3cbc78ed36e7f21b5370d004552f1d53e287 not found: ID does not exist"
Sep 17 03:44:23.005920 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:23.005805    2552 scope.go:117] "RemoveContainer" containerID="2ca1069b5dcba8a8513036fa09f56698a813a07d12d8932fe80da3941b2489f2"
Sep 17 03:44:23.016087 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:23.016069    2552 scope.go:117] "RemoveContainer" containerID="2ca1069b5dcba8a8513036fa09f56698a813a07d12d8932fe80da3941b2489f2"
Sep 17 03:44:23.016390 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:44:23.016371    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ca1069b5dcba8a8513036fa09f56698a813a07d12d8932fe80da3941b2489f2\": container with ID starting with 2ca1069b5dcba8a8513036fa09f56698a813a07d12d8932fe80da3941b2489f2 not found: ID does not exist" containerID="2ca1069b5dcba8a8513036fa09f56698a813a07d12d8932fe80da3941b2489f2"
Sep 17 03:44:23.016466 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:23.016394    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ca1069b5dcba8a8513036fa09f56698a813a07d12d8932fe80da3941b2489f2"} err="failed to get container status \"2ca1069b5dcba8a8513036fa09f56698a813a07d12d8932fe80da3941b2489f2\": rpc error: code = NotFound desc = could not find container \"2ca1069b5dcba8a8513036fa09f56698a813a07d12d8932fe80da3941b2489f2\": container with ID starting with 2ca1069b5dcba8a8513036fa09f56698a813a07d12d8932fe80da3941b2489f2 not found: ID does not exist"
Sep 17 03:44:23.016466 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:23.016410    2552 scope.go:117] "RemoveContainer" containerID="b39473b5ea95e866130295afdb931f04ce58775c9297473637182719e2a79fcb"
Sep 17 03:44:23.024709 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:23.024659    2552 scope.go:117] "RemoveContainer" containerID="b39473b5ea95e866130295afdb931f04ce58775c9297473637182719e2a79fcb"
Sep 17 03:44:23.025150 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:44:23.025132    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b39473b5ea95e866130295afdb931f04ce58775c9297473637182719e2a79fcb\": container with ID starting with b39473b5ea95e866130295afdb931f04ce58775c9297473637182719e2a79fcb not found: ID does not exist" containerID="b39473b5ea95e866130295afdb931f04ce58775c9297473637182719e2a79fcb"
Sep 17 03:44:23.025232 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:23.025156    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b39473b5ea95e866130295afdb931f04ce58775c9297473637182719e2a79fcb"} err="failed to get container status \"b39473b5ea95e866130295afdb931f04ce58775c9297473637182719e2a79fcb\": rpc error: code = NotFound desc = could not find container \"b39473b5ea95e866130295afdb931f04ce58775c9297473637182719e2a79fcb\": container with ID starting with b39473b5ea95e866130295afdb931f04ce58775c9297473637182719e2a79fcb not found: ID does not exist"
Sep 17 03:44:23.025232 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:23.025177    2552 scope.go:117] "RemoveContainer" containerID="6ff98b094f0c969d6d71318e8cd03d4cadff7769418625d8d7b7e93286ad066d"
Sep 17 03:44:23.052656 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:23.048106    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-1afd34/embedding-w0-1afd34-kserve-7c79989f85-9k45s"]
Sep 17 03:44:23.055472 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:23.055443    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-1afd34/embedding-w0-1afd34-kserve-7c79989f85-9k45s"]
Sep 17 03:44:23.058169 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:23.058148    2552 scope.go:117] "RemoveContainer" containerID="6ff98b094f0c969d6d71318e8cd03d4cadff7769418625d8d7b7e93286ad066d"
Sep 17 03:44:23.058492 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:44:23.058464    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ff98b094f0c969d6d71318e8cd03d4cadff7769418625d8d7b7e93286ad066d\": container with ID starting with 6ff98b094f0c969d6d71318e8cd03d4cadff7769418625d8d7b7e93286ad066d not found: ID does not exist" containerID="6ff98b094f0c969d6d71318e8cd03d4cadff7769418625d8d7b7e93286ad066d"
Sep 17 03:44:23.058549 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:23.058501    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ff98b094f0c969d6d71318e8cd03d4cadff7769418625d8d7b7e93286ad066d"} err="failed to get container status \"6ff98b094f0c969d6d71318e8cd03d4cadff7769418625d8d7b7e93286ad066d\": rpc error: code = NotFound desc = could not find container \"6ff98b094f0c969d6d71318e8cd03d4cadff7769418625d8d7b7e93286ad066d\": container with ID starting with 6ff98b094f0c969d6d71318e8cd03d4cadff7769418625d8d7b7e93286ad066d not found: ID does not exist"
Sep 17 03:44:23.070897 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:23.070876    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-1afd34/simulator-w0-1afd34-kserve-6c5ffdd47b-v6h59"]
Sep 17 03:44:23.074716 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:23.074695    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-1afd34/simulator-w0-1afd34-kserve-6c5ffdd47b-v6h59"]
Sep 17 03:44:23.085792 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:23.085774    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-1afd34/premium-w0-1afd34-kserve-85c998d845-r5zq9"]
Sep 17 03:44:23.089492 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:23.089466    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-1afd34/premium-w0-1afd34-kserve-85c998d845-r5zq9"]
Sep 17 03:44:23.104123 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:23.104098    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-1afd34/distinct-2-w0-1afd34-kserve-5457449c9-bqjmv"]
Sep 17 03:44:23.106467 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:23.106448    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-1afd34/distinct-2-w0-1afd34-kserve-5457449c9-bqjmv"]
Sep 17 03:44:24.987550 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:24.987519    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2a2ec719-0a31-439c-96af-ee004af31201" path="/var/lib/kubelet/pods/2a2ec719-0a31-439c-96af-ee004af31201/volumes"
Sep 17 03:44:24.987960 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:24.987945    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="36aa6a53-4557-4431-b02d-cde2e73afd04" path="/var/lib/kubelet/pods/36aa6a53-4557-4431-b02d-cde2e73afd04/volumes"
Sep 17 03:44:24.988310 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:24.988299    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5886143a-fc36-40eb-8818-b2c94767b529" path="/var/lib/kubelet/pods/5886143a-fc36-40eb-8818-b2c94767b529/volumes"
Sep 17 03:44:24.988965 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:24.988950    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6d3098da-c426-4ed1-8932-737fb644755c" path="/var/lib/kubelet/pods/6d3098da-c426-4ed1-8932-737fb644755c/volumes"
Sep 17 03:44:25.354583 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:25.354551    2552 scope.go:117] "RemoveContainer" containerID="d8122c179bb57a4714b93f4e2e17f768ef9a84194e63090c10b0939e7dc57e86"
Sep 17 03:44:25.365638 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:25.365614    2552 scope.go:117] "RemoveContainer" containerID="383741806db3a8b09734d42bc46507ba82fe319e08889d73fd63370f2712974b"
Sep 17 03:44:27.323206 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:27.323155    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1afd34/distinct-2-w0-1afd34-kserve-5457449c9-bqjmv" podUID="2a2ec719-0a31-439c-96af-ee004af31201" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Sep 17 03:44:28.033955 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:28.033929    2552 generic.go:356] "Generic (PLEG): container finished" podID="65a2cb95-0b18-40d3-8076-277905391cd2" containerID="8caf7c76acb33f510d73bd9210d24a8c9d63ec3e611a177d19f475be2e6fc38f" exitCode=0
Sep 17 03:44:28.034120 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:28.033998    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-26d5d1-openshift-default-df6dfd77d-w2nq4" event={"ID":"65a2cb95-0b18-40d3-8076-277905391cd2","Type":"ContainerDied","Data":"8caf7c76acb33f510d73bd9210d24a8c9d63ec3e611a177d19f475be2e6fc38f"}
Sep 17 03:44:28.034461 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:28.034446    2552 scope.go:117] "RemoveContainer" containerID="8caf7c76acb33f510d73bd9210d24a8c9d63ec3e611a177d19f475be2e6fc38f"
Sep 17 03:44:28.042083 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:28.042031    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 03:44:28.042483 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:28.042449    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 03:44:28.042584 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:28.042518    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 03:44:29.040989 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:29.040958    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-26d5d1-openshift-default-df6dfd77d-w2nq4" event={"ID":"65a2cb95-0b18-40d3-8076-277905391cd2","Type":"ContainerStarted","Data":"dee9c3d3e5ee52d844e834b608c35cadfd244aa8b89bec3eabe5d8dd734601b7"}
Sep 17 03:44:29.171246 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:29.171215    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-26d5d1-openshift-default-df6dfd77d-w2nq4"
Sep 17 03:44:29.172197 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:29.172170    2552 patch_prober.go:28] interesting pod/e2e-api-b-26d5d1-openshift-default-df6dfd77d-w2nq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Sep 17 03:44:29.172319 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:29.172219    2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-26d5d1-openshift-default-df6dfd77d-w2nq4" podUID="65a2cb95-0b18-40d3-8076-277905391cd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Sep 17 03:44:29.774450 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:29.774419    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-26d5d1-openshift-default-df6dfd77d-w2nq4"
Sep 17 03:44:30.172355 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:30.172270    2552 patch_prober.go:28] interesting pod/e2e-api-b-26d5d1-openshift-default-df6dfd77d-w2nq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Sep 17 03:44:30.172355 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:30.172333    2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-26d5d1-openshift-default-df6dfd77d-w2nq4" podUID="65a2cb95-0b18-40d3-8076-277905391cd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Sep 17 03:44:31.171319 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:31.171283    2552 patch_prober.go:28] interesting pod/e2e-api-b-26d5d1-openshift-default-df6dfd77d-w2nq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Sep 17 03:44:31.171502 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:31.171353    2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-26d5d1-openshift-default-df6dfd77d-w2nq4" podUID="65a2cb95-0b18-40d3-8076-277905391cd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Sep 17 03:44:32.172274 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:32.172240    2552 patch_prober.go:28] interesting pod/e2e-api-b-26d5d1-openshift-default-df6dfd77d-w2nq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Sep 17 03:44:32.172761 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:32.172318    2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-26d5d1-openshift-default-df6dfd77d-w2nq4" podUID="65a2cb95-0b18-40d3-8076-277905391cd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Sep 17 03:44:33.171565 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:33.171529    2552 patch_prober.go:28] interesting pod/e2e-api-b-26d5d1-openshift-default-df6dfd77d-w2nq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Sep 17 03:44:33.171759 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:33.171634    2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-26d5d1-openshift-default-df6dfd77d-w2nq4" podUID="65a2cb95-0b18-40d3-8076-277905391cd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Sep 17 03:44:34.171868 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:34.171830    2552 patch_prober.go:28] interesting pod/e2e-api-b-26d5d1-openshift-default-df6dfd77d-w2nq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Sep 17 03:44:34.172256 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:34.171891    2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-26d5d1-openshift-default-df6dfd77d-w2nq4" podUID="65a2cb95-0b18-40d3-8076-277905391cd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Sep 17 03:44:35.171451 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:35.171412    2552 patch_prober.go:28] interesting pod/e2e-api-b-26d5d1-openshift-default-df6dfd77d-w2nq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Sep 17 03:44:35.171636 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:35.171489    2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-26d5d1-openshift-default-df6dfd77d-w2nq4" podUID="65a2cb95-0b18-40d3-8076-277905391cd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Sep 17 03:44:36.176181 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:36.176149    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-26d5d1-openshift-default-df6dfd77d-w2nq4"
Sep 17 03:44:36.177116 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:36.177099    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-26d5d1-openshift-default-df6dfd77d-w2nq4"
Sep 17 03:44:54.410936 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:54.410844    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-26d5d1-67664944-mzhwg"]
Sep 17 03:44:54.411503 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:54.411140    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-26d5d1-67664944-mzhwg" podUID="600316ad-6c07-4f70-ba55-9894d1689735" containerName="maas-api" containerID="cri-o://0db012661b0451d99406d8eac69ea2b306e38416bdc88a9e4014e72f2c08d23c" gracePeriod=30
Sep 17 03:44:54.662688 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:54.662633    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-26d5d1-67664944-mzhwg"
Sep 17 03:44:54.773683 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:54.773656    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/600316ad-6c07-4f70-ba55-9894d1689735-maas-api-tls\") pod \"600316ad-6c07-4f70-ba55-9894d1689735\" (UID: \"600316ad-6c07-4f70-ba55-9894d1689735\") "
Sep 17 03:44:54.773863 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:54.773700    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/600316ad-6c07-4f70-ba55-9894d1689735-openshift-service-ca\") pod \"600316ad-6c07-4f70-ba55-9894d1689735\" (UID: \"600316ad-6c07-4f70-ba55-9894d1689735\") "
Sep 17 03:44:54.773863 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:54.773718    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wdgdf\" (UniqueName: \"kubernetes.io/projected/600316ad-6c07-4f70-ba55-9894d1689735-kube-api-access-wdgdf\") pod \"600316ad-6c07-4f70-ba55-9894d1689735\" (UID: \"600316ad-6c07-4f70-ba55-9894d1689735\") "
Sep 17 03:44:54.773863 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:54.773742    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/600316ad-6c07-4f70-ba55-9894d1689735-metrics-certs\") pod \"600316ad-6c07-4f70-ba55-9894d1689735\" (UID: \"600316ad-6c07-4f70-ba55-9894d1689735\") "
Sep 17 03:44:54.774091 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:54.774066    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/600316ad-6c07-4f70-ba55-9894d1689735-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "600316ad-6c07-4f70-ba55-9894d1689735" (UID: "600316ad-6c07-4f70-ba55-9894d1689735"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 03:44:54.775949 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:54.775917    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/600316ad-6c07-4f70-ba55-9894d1689735-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "600316ad-6c07-4f70-ba55-9894d1689735" (UID: "600316ad-6c07-4f70-ba55-9894d1689735"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 03:44:54.776057 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:54.775964    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/600316ad-6c07-4f70-ba55-9894d1689735-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "600316ad-6c07-4f70-ba55-9894d1689735" (UID: "600316ad-6c07-4f70-ba55-9894d1689735"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 03:44:54.776057 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:54.776017    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/600316ad-6c07-4f70-ba55-9894d1689735-kube-api-access-wdgdf" (OuterVolumeSpecName: "kube-api-access-wdgdf") pod "600316ad-6c07-4f70-ba55-9894d1689735" (UID: "600316ad-6c07-4f70-ba55-9894d1689735"). InnerVolumeSpecName "kube-api-access-wdgdf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 03:44:54.875221 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:54.875182    2552 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/600316ad-6c07-4f70-ba55-9894d1689735-maas-api-tls\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:44:54.875221 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:54.875215    2552 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/600316ad-6c07-4f70-ba55-9894d1689735-openshift-service-ca\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:44:54.875221 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:54.875226    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wdgdf\" (UniqueName: \"kubernetes.io/projected/600316ad-6c07-4f70-ba55-9894d1689735-kube-api-access-wdgdf\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:44:54.875470 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:54.875234    2552 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/600316ad-6c07-4f70-ba55-9894d1689735-metrics-certs\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:44:55.201364 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:55.201329    2552 generic.go:356] "Generic (PLEG): container finished" podID="600316ad-6c07-4f70-ba55-9894d1689735" containerID="0db012661b0451d99406d8eac69ea2b306e38416bdc88a9e4014e72f2c08d23c" exitCode=0
Sep 17 03:44:55.201566 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:55.201376    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-26d5d1-67664944-mzhwg"
Sep 17 03:44:55.201566 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:55.201398    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-26d5d1-67664944-mzhwg" event={"ID":"600316ad-6c07-4f70-ba55-9894d1689735","Type":"ContainerDied","Data":"0db012661b0451d99406d8eac69ea2b306e38416bdc88a9e4014e72f2c08d23c"}
Sep 17 03:44:55.201566 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:55.201443    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-26d5d1-67664944-mzhwg" event={"ID":"600316ad-6c07-4f70-ba55-9894d1689735","Type":"ContainerDied","Data":"b5ec9e2fcf2df63890892e87bb888cc8a690b0010d8d96a690c2a508657f4d2c"}
Sep 17 03:44:55.201566 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:55.201466    2552 scope.go:117] "RemoveContainer" containerID="0db012661b0451d99406d8eac69ea2b306e38416bdc88a9e4014e72f2c08d23c"
Sep 17 03:44:55.210670 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:55.210648    2552 scope.go:117] "RemoveContainer" containerID="0db012661b0451d99406d8eac69ea2b306e38416bdc88a9e4014e72f2c08d23c"
Sep 17 03:44:55.210959 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:44:55.210939    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0db012661b0451d99406d8eac69ea2b306e38416bdc88a9e4014e72f2c08d23c\": container with ID starting with 0db012661b0451d99406d8eac69ea2b306e38416bdc88a9e4014e72f2c08d23c not found: ID does not exist" containerID="0db012661b0451d99406d8eac69ea2b306e38416bdc88a9e4014e72f2c08d23c"
Sep 17 03:44:55.211033 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:55.210968    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0db012661b0451d99406d8eac69ea2b306e38416bdc88a9e4014e72f2c08d23c"} err="failed to get container status \"0db012661b0451d99406d8eac69ea2b306e38416bdc88a9e4014e72f2c08d23c\": rpc error: code = NotFound desc = could not find container \"0db012661b0451d99406d8eac69ea2b306e38416bdc88a9e4014e72f2c08d23c\": container with ID starting with 0db012661b0451d99406d8eac69ea2b306e38416bdc88a9e4014e72f2c08d23c not found: ID does not exist"
Sep 17 03:44:55.228867 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:55.228840    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-26d5d1-67664944-mzhwg"]
Sep 17 03:44:55.233501 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:55.233482    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-26d5d1-67664944-mzhwg"]
Sep 17 03:44:56.985973 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:56.985937    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="600316ad-6c07-4f70-ba55-9894d1689735" path="/var/lib/kubelet/pods/600316ad-6c07-4f70-ba55-9894d1689735/volumes"
Sep 17 03:44:58.351803 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:58.351768    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-214fea-7gt8k"]
Sep 17 03:44:58.352162 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:58.352149    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="600316ad-6c07-4f70-ba55-9894d1689735" containerName="maas-api"
Sep 17 03:44:58.352218 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:58.352164    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="600316ad-6c07-4f70-ba55-9894d1689735" containerName="maas-api"
Sep 17 03:44:58.352218 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:58.352187    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5886143a-fc36-40eb-8818-b2c94767b529" containerName="main"
Sep 17 03:44:58.352218 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:58.352194    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="5886143a-fc36-40eb-8818-b2c94767b529" containerName="main"
Sep 17 03:44:58.352218 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:58.352201    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c76c1a2-3cb7-43ae-be88-fe95001caf30" containerName="maas-api"
Sep 17 03:44:58.352218 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:58.352206    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c76c1a2-3cb7-43ae-be88-fe95001caf30" containerName="maas-api"
Sep 17 03:44:58.352218 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:58.352214    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d3098da-c426-4ed1-8932-737fb644755c" containerName="main"
Sep 17 03:44:58.352218 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:58.352219    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d3098da-c426-4ed1-8932-737fb644755c" containerName="main"
Sep 17 03:44:58.352432 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:58.352229    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="36aa6a53-4557-4431-b02d-cde2e73afd04" containerName="main"
Sep 17 03:44:58.352432 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:58.352234    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="36aa6a53-4557-4431-b02d-cde2e73afd04" containerName="main"
Sep 17 03:44:58.352432 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:58.352239    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a2ec719-0a31-439c-96af-ee004af31201" containerName="main"
Sep 17 03:44:58.352432 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:58.352244    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a2ec719-0a31-439c-96af-ee004af31201" containerName="main"
Sep 17 03:44:58.352432 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:58.352306    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a2ec719-0a31-439c-96af-ee004af31201" containerName="main"
Sep 17 03:44:58.352432 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:58.352314    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="5886143a-fc36-40eb-8818-b2c94767b529" containerName="main"
Sep 17 03:44:58.352432 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:58.352319    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d3098da-c426-4ed1-8932-737fb644755c" containerName="main"
Sep 17 03:44:58.352432 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:58.352325    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="600316ad-6c07-4f70-ba55-9894d1689735" containerName="maas-api"
Sep 17 03:44:58.352432 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:58.352334    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="36aa6a53-4557-4431-b02d-cde2e73afd04" containerName="main"
Sep 17 03:44:58.352432 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:58.352339    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c76c1a2-3cb7-43ae-be88-fe95001caf30" containerName="maas-api"
Sep 17 03:44:58.355486 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:58.355471    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-214fea-7gt8k"
Sep 17 03:44:58.358333 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:58.358314    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8vwnw\""
Sep 17 03:44:58.364440 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:58.364415    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-214fea-7gt8k"]
Sep 17 03:44:58.412010 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:58.411979    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/10a7a359-b8aa-438c-896f-7d1ae989a550-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-214fea-7gt8k\" (UID: \"10a7a359-b8aa-438c-896f-7d1ae989a550\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-214fea-7gt8k"
Sep 17 03:44:58.513348 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:58.513323    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/10a7a359-b8aa-438c-896f-7d1ae989a550-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-214fea-7gt8k\" (UID: \"10a7a359-b8aa-438c-896f-7d1ae989a550\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-214fea-7gt8k"
Sep 17 03:44:58.513973 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:58.513953    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/10a7a359-b8aa-438c-896f-7d1ae989a550-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-214fea-7gt8k\" (UID: \"10a7a359-b8aa-438c-896f-7d1ae989a550\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-214fea-7gt8k"
Sep 17 03:44:58.666964 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:58.666893    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-214fea-7gt8k"
Sep 17 03:44:58.936540 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:58.936519    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-214fea-7gt8k"]
Sep 17 03:44:58.938588 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:44:58.938555    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10a7a359_b8aa_438c_896f_7d1ae989a550.slice/crio-da161ac4c979c5e966ee1b061013c6ab8307de0c3bc9a1fbe4efdc4fee819876 WatchSource:0}: Error finding container da161ac4c979c5e966ee1b061013c6ab8307de0c3bc9a1fbe4efdc4fee819876: Status 404 returned error can't find the container with id da161ac4c979c5e966ee1b061013c6ab8307de0c3bc9a1fbe4efdc4fee819876
Sep 17 03:44:59.224654 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:44:59.224553    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-214fea-7gt8k" event={"ID":"10a7a359-b8aa-438c-896f-7d1ae989a550","Type":"ContainerStarted","Data":"da161ac4c979c5e966ee1b061013c6ab8307de0c3bc9a1fbe4efdc4fee819876"}
Sep 17 03:45:00.136665 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:00.136632    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-214fea-29826945-kjjgv"]
Sep 17 03:45:00.140130 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:00.140113    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-214fea-29826945-kjjgv"
Sep 17 03:45:00.146591 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:00.146561    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f6d581-29826945-5w7g2"]
Sep 17 03:45:00.153959 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:00.153720    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-214fea-29826945-kjjgv"
Sep 17 03:45:00.153959 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:00.153743    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f6d581-29826945-5w7g2"
Sep 17 03:45:00.154160 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:00.154146    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-214fea-29826945-kjjgv"]
Sep 17 03:45:00.157408 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:00.156922    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f6d581-29826945-5w7g2"]
Sep 17 03:45:00.163336 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:00.163005    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29826945-s5d4q"]
Sep 17 03:45:00.167289 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:00.167263    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f6d581-29826945-5w7g2"
Sep 17 03:45:00.167611 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:00.167564    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826945-s5d4q"
Sep 17 03:45:00.178813 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:00.178778    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29826945-s5d4q"]
Sep 17 03:45:00.217897 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:00.217864    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826945-s5d4q"
Sep 17 03:45:00.247744 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:00.247704    2552 generic.go:356] "Generic (PLEG): container finished" podID="10a7a359-b8aa-438c-896f-7d1ae989a550" containerID="aa80281c6e1a2c9e90b30b8f8f1595edd9422fc2ab5309359893af80c50040b8" exitCode=0
Sep 17 03:45:00.247871 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:00.247802    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-214fea-7gt8k" event={"ID":"10a7a359-b8aa-438c-896f-7d1ae989a550","Type":"ContainerDied","Data":"aa80281c6e1a2c9e90b30b8f8f1595edd9422fc2ab5309359893af80c50040b8"}
Sep 17 03:45:00.446592 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:00.446562    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-214fea-29826945-kjjgv"]
Sep 17 03:45:00.447555 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:45:00.447532    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda69b8fbf_bd67_44b0_84e4_5726c21d6fce.slice/crio-cc2af81646a7d97040456d528051b7fbc5b01f43ed82c6910c1fe03ed70bd51b WatchSource:0}: Error finding container cc2af81646a7d97040456d528051b7fbc5b01f43ed82c6910c1fe03ed70bd51b: Status 404 returned error can't find the container with id cc2af81646a7d97040456d528051b7fbc5b01f43ed82c6910c1fe03ed70bd51b
Sep 17 03:45:00.464034 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:00.464012    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f6d581-29826945-5w7g2"]
Sep 17 03:45:00.465487 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:45:00.465464    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2338e9b_a64c_47d6_8b38_d47b0d148690.slice/crio-3ce5505f40f5b8c9f5a0006d187bd223611dbe91e1e3824bc764d6229b78f553 WatchSource:0}: Error finding container 3ce5505f40f5b8c9f5a0006d187bd223611dbe91e1e3824bc764d6229b78f553: Status 404 returned error can't find the container with id 3ce5505f40f5b8c9f5a0006d187bd223611dbe91e1e3824bc764d6229b78f553
Sep 17 03:45:00.517807 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:00.517779    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29826945-s5d4q"]
Sep 17 03:45:00.519675 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:45:00.519648    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab9688c7_a95b_4430_8075_a0008c6595fc.slice/crio-904c02fb8e17b96321f01dcc3e19d01d72e85fc856f9b31e3f3fb86179820ada WatchSource:0}: Error finding container 904c02fb8e17b96321f01dcc3e19d01d72e85fc856f9b31e3f3fb86179820ada: Status 404 returned error can't find the container with id 904c02fb8e17b96321f01dcc3e19d01d72e85fc856f9b31e3f3fb86179820ada
Sep 17 03:45:01.254463 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:01.254430    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-214fea-29826945-kjjgv" event={"ID":"a69b8fbf-bd67-44b0-84e4-5726c21d6fce","Type":"ContainerStarted","Data":"8ef4202747f881eef14d29a0cfad8a59ca782bbe33af4d69b0b54d042368d5b4"}
Sep 17 03:45:01.254932 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:01.254500    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-214fea-29826945-kjjgv" event={"ID":"a69b8fbf-bd67-44b0-84e4-5726c21d6fce","Type":"ContainerStarted","Data":"cc2af81646a7d97040456d528051b7fbc5b01f43ed82c6910c1fe03ed70bd51b"}
Sep 17 03:45:01.256055 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:01.256027    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f6d581-29826945-5w7g2" event={"ID":"c2338e9b-a64c-47d6-8b38-d47b0d148690","Type":"ContainerStarted","Data":"eea2344cff1f1fc900dc92ff258ccf5d57b77615c5f1713f1bdda12a2434398d"}
Sep 17 03:45:01.256176 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:01.256081    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f6d581-29826945-5w7g2" event={"ID":"c2338e9b-a64c-47d6-8b38-d47b0d148690","Type":"ContainerStarted","Data":"3ce5505f40f5b8c9f5a0006d187bd223611dbe91e1e3824bc764d6229b78f553"}
Sep 17 03:45:01.257493 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:01.257471    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826945-s5d4q" event={"ID":"ab9688c7-a95b-4430-8075-a0008c6595fc","Type":"ContainerStarted","Data":"e03bac6800988c13b7b895bb008647fcbddc6ede1242e5af6800676ebae77c98"}
Sep 17 03:45:01.257590 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:01.257511    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826945-s5d4q" event={"ID":"ab9688c7-a95b-4430-8075-a0008c6595fc","Type":"ContainerStarted","Data":"904c02fb8e17b96321f01dcc3e19d01d72e85fc856f9b31e3f3fb86179820ada"}
Sep 17 03:45:01.271728 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:01.271634    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-214fea-29826945-kjjgv" podStartSLOduration=1.271619177 podStartE2EDuration="1.271619177s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 03:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:45:01.271062279 +0000 UTC m=+1536.737469804" watchObservedRunningTime="2026-09-17 03:45:01.271619177 +0000 UTC m=+1536.738026703"
Sep 17 03:45:01.287946 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:01.287896    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f6d581-29826945-5w7g2" podStartSLOduration=1.287885776 podStartE2EDuration="1.287885776s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 03:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:45:01.285628231 +0000 UTC m=+1536.752035758" watchObservedRunningTime="2026-09-17 03:45:01.287885776 +0000 UTC m=+1536.754293357"
Sep 17 03:45:01.300405 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:01.300356    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826945-s5d4q" podStartSLOduration=1.3003421419999999 podStartE2EDuration="1.300342142s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 03:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:45:01.298707417 +0000 UTC m=+1536.765114943" watchObservedRunningTime="2026-09-17 03:45:01.300342142 +0000 UTC m=+1536.766749669"
Sep 17 03:45:01.407003 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:01.406981    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-214fea-7gt8k"
Sep 17 03:45:01.445979 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:01.445952    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/10a7a359-b8aa-438c-896f-7d1ae989a550-maas-api-service-ca\") pod \"10a7a359-b8aa-438c-896f-7d1ae989a550\" (UID: \"10a7a359-b8aa-438c-896f-7d1ae989a550\") "
Sep 17 03:45:01.446340 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:01.446318    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10a7a359-b8aa-438c-896f-7d1ae989a550-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "10a7a359-b8aa-438c-896f-7d1ae989a550" (UID: "10a7a359-b8aa-438c-896f-7d1ae989a550"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 03:45:01.547856 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:01.547808    2552 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/10a7a359-b8aa-438c-896f-7d1ae989a550-maas-api-service-ca\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:45:01.704460 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:01.704424    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-26d5d1-openshift-default-df6dfd77d-w2nq4"]
Sep 17 03:45:01.704712 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:01.704685    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-26d5d1-openshift-default-df6dfd77d-w2nq4" podUID="65a2cb95-0b18-40d3-8076-277905391cd2" containerName="istio-proxy" containerID="cri-o://dee9c3d3e5ee52d844e834b608c35cadfd244aa8b89bec3eabe5d8dd734601b7" gracePeriod=30
Sep 17 03:45:02.060795 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:02.060763    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-57958f-p9vwh"]
Sep 17 03:45:02.061191 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:02.061179    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10a7a359-b8aa-438c-896f-7d1ae989a550" containerName="revoke-keys"
Sep 17 03:45:02.061242 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:02.061193    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="10a7a359-b8aa-438c-896f-7d1ae989a550" containerName="revoke-keys"
Sep 17 03:45:02.061276 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:02.061267    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="10a7a359-b8aa-438c-896f-7d1ae989a550" containerName="revoke-keys"
Sep 17 03:45:02.064383 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:02.064367    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-57958f-p9vwh"
Sep 17 03:45:02.072118 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:02.072060    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-57958f-p9vwh"]
Sep 17 03:45:02.153571 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:02.153539    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a276646-b9a0-4a4e-9893-7a7c825c9256-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-57958f-p9vwh\" (UID: \"4a276646-b9a0-4a4e-9893-7a7c825c9256\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-57958f-p9vwh"
Sep 17 03:45:02.254496 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:02.254444    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a276646-b9a0-4a4e-9893-7a7c825c9256-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-57958f-p9vwh\" (UID: \"4a276646-b9a0-4a4e-9893-7a7c825c9256\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-57958f-p9vwh"
Sep 17 03:45:02.255263 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:02.255236    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a276646-b9a0-4a4e-9893-7a7c825c9256-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-57958f-p9vwh\" (UID: \"4a276646-b9a0-4a4e-9893-7a7c825c9256\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-57958f-p9vwh"
Sep 17 03:45:02.263563 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:02.263537    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-214fea-7gt8k"
Sep 17 03:45:02.263700 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:02.263554    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-214fea-7gt8k" event={"ID":"10a7a359-b8aa-438c-896f-7d1ae989a550","Type":"ContainerDied","Data":"da161ac4c979c5e966ee1b061013c6ab8307de0c3bc9a1fbe4efdc4fee819876"}
Sep 17 03:45:02.263700 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:02.263620    2552 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="da161ac4c979c5e966ee1b061013c6ab8307de0c3bc9a1fbe4efdc4fee819876"
Sep 17 03:45:02.265076 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:02.265049    2552 generic.go:356] "Generic (PLEG): container finished" podID="ab9688c7-a95b-4430-8075-a0008c6595fc" containerID="e03bac6800988c13b7b895bb008647fcbddc6ede1242e5af6800676ebae77c98" exitCode=0
Sep 17 03:45:02.265192 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:02.265123    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826945-s5d4q" event={"ID":"ab9688c7-a95b-4430-8075-a0008c6595fc","Type":"ContainerDied","Data":"e03bac6800988c13b7b895bb008647fcbddc6ede1242e5af6800676ebae77c98"}
Sep 17 03:45:02.266849 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:02.266826    2552 generic.go:356] "Generic (PLEG): container finished" podID="a69b8fbf-bd67-44b0-84e4-5726c21d6fce" containerID="8ef4202747f881eef14d29a0cfad8a59ca782bbe33af4d69b0b54d042368d5b4" exitCode=0
Sep 17 03:45:02.266947 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:02.266914    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-214fea-29826945-kjjgv" event={"ID":"a69b8fbf-bd67-44b0-84e4-5726c21d6fce","Type":"ContainerDied","Data":"8ef4202747f881eef14d29a0cfad8a59ca782bbe33af4d69b0b54d042368d5b4"}
Sep 17 03:45:02.268809 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:02.268792    2552 generic.go:356] "Generic (PLEG): container finished" podID="c2338e9b-a64c-47d6-8b38-d47b0d148690" containerID="eea2344cff1f1fc900dc92ff258ccf5d57b77615c5f1713f1bdda12a2434398d" exitCode=0
Sep 17 03:45:02.268894 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:02.268820    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f6d581-29826945-5w7g2" event={"ID":"c2338e9b-a64c-47d6-8b38-d47b0d148690","Type":"ContainerDied","Data":"eea2344cff1f1fc900dc92ff258ccf5d57b77615c5f1713f1bdda12a2434398d"}
Sep 17 03:45:02.376416 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:02.376334    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-57958f-p9vwh"
Sep 17 03:45:02.652375 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:02.652352    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-57958f-p9vwh"]
Sep 17 03:45:02.654396 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:45:02.654367    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a276646_b9a0_4a4e_9893_7a7c825c9256.slice/crio-2e1f991ebff04385725b5116347c6ca6a4d882d062a671bfd7abd7c898eb5b3f WatchSource:0}: Error finding container 2e1f991ebff04385725b5116347c6ca6a4d882d062a671bfd7abd7c898eb5b3f: Status 404 returned error can't find the container with id 2e1f991ebff04385725b5116347c6ca6a4d882d062a671bfd7abd7c898eb5b3f
Sep 17 03:45:03.198724 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:03.198642    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-214fea/premium-w4-214fea-kserve-589dcd865f-hkkwm"]
Sep 17 03:45:03.207752 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:03.207728    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-214fea/simulator-w4-214fea-kserve-5965bff56b-mkmt9"]
Sep 17 03:45:03.276058 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:03.276021    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-57958f-p9vwh" event={"ID":"4a276646-b9a0-4a4e-9893-7a7c825c9256","Type":"ContainerStarted","Data":"45633a0fe5a648090bff8efc059be5df69826dd672974a6bea66a5d5d79564c8"}
Sep 17 03:45:03.276487 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:03.276081    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-57958f-p9vwh" event={"ID":"4a276646-b9a0-4a4e-9893-7a7c825c9256","Type":"ContainerStarted","Data":"2e1f991ebff04385725b5116347c6ca6a4d882d062a671bfd7abd7c898eb5b3f"}
Sep 17 03:45:03.290912 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:03.290861    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-57958f-p9vwh" podStartSLOduration=1.290843909 podStartE2EDuration="1.290843909s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 03:45:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:45:03.29015649 +0000 UTC m=+1538.756564016" watchObservedRunningTime="2026-09-17 03:45:03.290843909 +0000 UTC m=+1538.757251436"
Sep 17 03:45:03.433778 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:03.433039    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826945-s5d4q"
Sep 17 03:45:03.503810 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:03.503783    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-214fea-29826945-kjjgv"
Sep 17 03:45:03.506461 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:03.506446    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f6d581-29826945-5w7g2"
Sep 17 03:45:04.282984 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:04.282959    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f6d581-29826945-5w7g2"
Sep 17 03:45:04.283424 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:04.282999    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f6d581-29826945-5w7g2" event={"ID":"c2338e9b-a64c-47d6-8b38-d47b0d148690","Type":"ContainerDied","Data":"3ce5505f40f5b8c9f5a0006d187bd223611dbe91e1e3824bc764d6229b78f553"}
Sep 17 03:45:04.283424 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:04.283045    2552 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3ce5505f40f5b8c9f5a0006d187bd223611dbe91e1e3824bc764d6229b78f553"
Sep 17 03:45:04.284404 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:04.284384    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826945-s5d4q"
Sep 17 03:45:04.284404 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:04.284398    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826945-s5d4q" event={"ID":"ab9688c7-a95b-4430-8075-a0008c6595fc","Type":"ContainerDied","Data":"904c02fb8e17b96321f01dcc3e19d01d72e85fc856f9b31e3f3fb86179820ada"}
Sep 17 03:45:04.284631 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:04.284437    2552 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="904c02fb8e17b96321f01dcc3e19d01d72e85fc856f9b31e3f3fb86179820ada"
Sep 17 03:45:04.286113 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:04.286090    2552 generic.go:356] "Generic (PLEG): container finished" podID="4a276646-b9a0-4a4e-9893-7a7c825c9256" containerID="45633a0fe5a648090bff8efc059be5df69826dd672974a6bea66a5d5d79564c8" exitCode=0
Sep 17 03:45:04.286229 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:04.286168    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-57958f-p9vwh" event={"ID":"4a276646-b9a0-4a4e-9893-7a7c825c9256","Type":"ContainerDied","Data":"45633a0fe5a648090bff8efc059be5df69826dd672974a6bea66a5d5d79564c8"}
Sep 17 03:45:04.287567 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:04.287550    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-214fea-29826945-kjjgv"
Sep 17 03:45:04.287567 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:04.287556    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-214fea-29826945-kjjgv" event={"ID":"a69b8fbf-bd67-44b0-84e4-5726c21d6fce","Type":"ContainerDied","Data":"cc2af81646a7d97040456d528051b7fbc5b01f43ed82c6910c1fe03ed70bd51b"}
Sep 17 03:45:04.287715 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:04.287584    2552 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cc2af81646a7d97040456d528051b7fbc5b01f43ed82c6910c1fe03ed70bd51b"
Sep 17 03:45:05.435432 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:05.435411    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-57958f-p9vwh"
Sep 17 03:45:05.484483 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:05.484452    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a276646-b9a0-4a4e-9893-7a7c825c9256-maas-api-service-ca\") pod \"4a276646-b9a0-4a4e-9893-7a7c825c9256\" (UID: \"4a276646-b9a0-4a4e-9893-7a7c825c9256\") "
Sep 17 03:45:05.484840 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:05.484816    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a276646-b9a0-4a4e-9893-7a7c825c9256-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4a276646-b9a0-4a4e-9893-7a7c825c9256" (UID: "4a276646-b9a0-4a4e-9893-7a7c825c9256"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 03:45:05.586126 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:05.586034    2552 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a276646-b9a0-4a4e-9893-7a7c825c9256-maas-api-service-ca\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:45:06.179109 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:06.179074    2552 patch_prober.go:28] interesting pod/e2e-api-b-26d5d1-openshift-default-df6dfd77d-w2nq4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 17 03:45:06.179286 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:06.179133    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-26d5d1-openshift-default-df6dfd77d-w2nq4" podUID="65a2cb95-0b18-40d3-8076-277905391cd2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 03:45:06.300175 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:06.300151    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-57958f-p9vwh"
Sep 17 03:45:06.300347 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:06.300195    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-57958f-p9vwh" event={"ID":"4a276646-b9a0-4a4e-9893-7a7c825c9256","Type":"ContainerDied","Data":"2e1f991ebff04385725b5116347c6ca6a4d882d062a671bfd7abd7c898eb5b3f"}
Sep 17 03:45:06.300347 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:06.300244    2552 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2e1f991ebff04385725b5116347c6ca6a4d882d062a671bfd7abd7c898eb5b3f"
Sep 17 03:45:06.953622 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:06.953578    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-26d5d1-openshift-default-df6dfd77d-w2nq4"
Sep 17 03:45:07.000280 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:07.000257    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65a2cb95-0b18-40d3-8076-277905391cd2-istiod-ca-cert\") pod \"65a2cb95-0b18-40d3-8076-277905391cd2\" (UID: \"65a2cb95-0b18-40d3-8076-277905391cd2\") "
Sep 17 03:45:07.000433 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:07.000319    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65a2cb95-0b18-40d3-8076-277905391cd2-istio-token\") pod \"65a2cb95-0b18-40d3-8076-277905391cd2\" (UID: \"65a2cb95-0b18-40d3-8076-277905391cd2\") "
Sep 17 03:45:07.000433 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:07.000347    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65a2cb95-0b18-40d3-8076-277905391cd2-workload-certs\") pod \"65a2cb95-0b18-40d3-8076-277905391cd2\" (UID: \"65a2cb95-0b18-40d3-8076-277905391cd2\") "
Sep 17 03:45:07.000433 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:07.000365    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65a2cb95-0b18-40d3-8076-277905391cd2-istio-data\") pod \"65a2cb95-0b18-40d3-8076-277905391cd2\" (UID: \"65a2cb95-0b18-40d3-8076-277905391cd2\") "
Sep 17 03:45:07.000433 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:07.000392    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65a2cb95-0b18-40d3-8076-277905391cd2-workload-socket\") pod \"65a2cb95-0b18-40d3-8076-277905391cd2\" (UID: \"65a2cb95-0b18-40d3-8076-277905391cd2\") "
Sep 17 03:45:07.000433 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:07.000426    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65a2cb95-0b18-40d3-8076-277905391cd2-istio-envoy\") pod \"65a2cb95-0b18-40d3-8076-277905391cd2\" (UID: \"65a2cb95-0b18-40d3-8076-277905391cd2\") "
Sep 17 03:45:07.000886 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:07.000463    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65a2cb95-0b18-40d3-8076-277905391cd2-credential-socket\") pod \"65a2cb95-0b18-40d3-8076-277905391cd2\" (UID: \"65a2cb95-0b18-40d3-8076-277905391cd2\") "
Sep 17 03:45:07.000886 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:07.000520    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65a2cb95-0b18-40d3-8076-277905391cd2-istio-podinfo\") pod \"65a2cb95-0b18-40d3-8076-277905391cd2\" (UID: \"65a2cb95-0b18-40d3-8076-277905391cd2\") "
Sep 17 03:45:07.000886 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:07.000567    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p8gs5\" (UniqueName: \"kubernetes.io/projected/65a2cb95-0b18-40d3-8076-277905391cd2-kube-api-access-p8gs5\") pod \"65a2cb95-0b18-40d3-8076-277905391cd2\" (UID: \"65a2cb95-0b18-40d3-8076-277905391cd2\") "
Sep 17 03:45:07.000886 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:07.000714    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65a2cb95-0b18-40d3-8076-277905391cd2-istio-data" (OuterVolumeSpecName: "istio-data") pod "65a2cb95-0b18-40d3-8076-277905391cd2" (UID: "65a2cb95-0b18-40d3-8076-277905391cd2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:45:07.000886 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:07.000730    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65a2cb95-0b18-40d3-8076-277905391cd2-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "65a2cb95-0b18-40d3-8076-277905391cd2" (UID: "65a2cb95-0b18-40d3-8076-277905391cd2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:45:07.000886 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:07.000748    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65a2cb95-0b18-40d3-8076-277905391cd2-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "65a2cb95-0b18-40d3-8076-277905391cd2" (UID: "65a2cb95-0b18-40d3-8076-277905391cd2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 03:45:07.001201 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:07.001059    2552 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65a2cb95-0b18-40d3-8076-277905391cd2-istio-data\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:45:07.001201 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:07.001083    2552 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65a2cb95-0b18-40d3-8076-277905391cd2-workload-socket\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:45:07.001201 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:07.001099    2552 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65a2cb95-0b18-40d3-8076-277905391cd2-istiod-ca-cert\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:45:07.001201 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:07.001103    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65a2cb95-0b18-40d3-8076-277905391cd2-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "65a2cb95-0b18-40d3-8076-277905391cd2" (UID: "65a2cb95-0b18-40d3-8076-277905391cd2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:45:07.001201 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:07.001119    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65a2cb95-0b18-40d3-8076-277905391cd2-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "65a2cb95-0b18-40d3-8076-277905391cd2" (UID: "65a2cb95-0b18-40d3-8076-277905391cd2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:45:07.002877 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:07.002842    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65a2cb95-0b18-40d3-8076-277905391cd2-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "65a2cb95-0b18-40d3-8076-277905391cd2" (UID: "65a2cb95-0b18-40d3-8076-277905391cd2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:45:07.002995 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:07.002906    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/65a2cb95-0b18-40d3-8076-277905391cd2-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "65a2cb95-0b18-40d3-8076-277905391cd2" (UID: "65a2cb95-0b18-40d3-8076-277905391cd2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 17 03:45:07.003103 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:07.003086    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65a2cb95-0b18-40d3-8076-277905391cd2-kube-api-access-p8gs5" (OuterVolumeSpecName: "kube-api-access-p8gs5") pod "65a2cb95-0b18-40d3-8076-277905391cd2" (UID: "65a2cb95-0b18-40d3-8076-277905391cd2"). InnerVolumeSpecName "kube-api-access-p8gs5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 03:45:07.003163 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:07.003147    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65a2cb95-0b18-40d3-8076-277905391cd2-istio-token" (OuterVolumeSpecName: "istio-token") pod "65a2cb95-0b18-40d3-8076-277905391cd2" (UID: "65a2cb95-0b18-40d3-8076-277905391cd2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 03:45:07.102498 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:07.102458    2552 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65a2cb95-0b18-40d3-8076-277905391cd2-workload-certs\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:45:07.102498 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:07.102492    2552 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65a2cb95-0b18-40d3-8076-277905391cd2-istio-envoy\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:45:07.102498 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:07.102501    2552 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65a2cb95-0b18-40d3-8076-277905391cd2-credential-socket\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:45:07.102762 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:07.102509    2552 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65a2cb95-0b18-40d3-8076-277905391cd2-istio-podinfo\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:45:07.102762 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:07.102518    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8gs5\" (UniqueName: \"kubernetes.io/projected/65a2cb95-0b18-40d3-8076-277905391cd2-kube-api-access-p8gs5\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:45:07.102762 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:07.102529    2552 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65a2cb95-0b18-40d3-8076-277905391cd2-istio-token\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:45:07.307866 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:07.307773    2552 generic.go:356] "Generic (PLEG): container finished" podID="65a2cb95-0b18-40d3-8076-277905391cd2" containerID="dee9c3d3e5ee52d844e834b608c35cadfd244aa8b89bec3eabe5d8dd734601b7" exitCode=0
Sep 17 03:45:07.307866 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:07.307826    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-26d5d1-openshift-default-df6dfd77d-w2nq4"
Sep 17 03:45:07.308086 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:07.307863    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-26d5d1-openshift-default-df6dfd77d-w2nq4" event={"ID":"65a2cb95-0b18-40d3-8076-277905391cd2","Type":"ContainerDied","Data":"dee9c3d3e5ee52d844e834b608c35cadfd244aa8b89bec3eabe5d8dd734601b7"}
Sep 17 03:45:07.308086 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:07.307923    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-26d5d1-openshift-default-df6dfd77d-w2nq4" event={"ID":"65a2cb95-0b18-40d3-8076-277905391cd2","Type":"ContainerDied","Data":"70390c59cf9709c1dcd3045baa355394219f0c9b651b01748605c50dee379764"}
Sep 17 03:45:07.308086 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:07.307946    2552 scope.go:117] "RemoveContainer" containerID="dee9c3d3e5ee52d844e834b608c35cadfd244aa8b89bec3eabe5d8dd734601b7"
Sep 17 03:45:07.317465 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:07.317448    2552 scope.go:117] "RemoveContainer" containerID="8caf7c76acb33f510d73bd9210d24a8c9d63ec3e611a177d19f475be2e6fc38f"
Sep 17 03:45:07.325544 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:07.325525    2552 scope.go:117] "RemoveContainer" containerID="dee9c3d3e5ee52d844e834b608c35cadfd244aa8b89bec3eabe5d8dd734601b7"
Sep 17 03:45:07.325824 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:45:07.325807    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dee9c3d3e5ee52d844e834b608c35cadfd244aa8b89bec3eabe5d8dd734601b7\": container with ID starting with dee9c3d3e5ee52d844e834b608c35cadfd244aa8b89bec3eabe5d8dd734601b7 not found: ID does not exist" containerID="dee9c3d3e5ee52d844e834b608c35cadfd244aa8b89bec3eabe5d8dd734601b7"
Sep 17 03:45:07.325876 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:07.325831    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dee9c3d3e5ee52d844e834b608c35cadfd244aa8b89bec3eabe5d8dd734601b7"} err="failed to get container status \"dee9c3d3e5ee52d844e834b608c35cadfd244aa8b89bec3eabe5d8dd734601b7\": rpc error: code = NotFound desc = could not find container \"dee9c3d3e5ee52d844e834b608c35cadfd244aa8b89bec3eabe5d8dd734601b7\": container with ID starting with dee9c3d3e5ee52d844e834b608c35cadfd244aa8b89bec3eabe5d8dd734601b7 not found: ID does not exist"
Sep 17 03:45:07.325876 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:07.325846    2552 scope.go:117] "RemoveContainer" containerID="8caf7c76acb33f510d73bd9210d24a8c9d63ec3e611a177d19f475be2e6fc38f"
Sep 17 03:45:07.326049 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:45:07.326034    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8caf7c76acb33f510d73bd9210d24a8c9d63ec3e611a177d19f475be2e6fc38f\": container with ID starting with 8caf7c76acb33f510d73bd9210d24a8c9d63ec3e611a177d19f475be2e6fc38f not found: ID does not exist" containerID="8caf7c76acb33f510d73bd9210d24a8c9d63ec3e611a177d19f475be2e6fc38f"
Sep 17 03:45:07.326087 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:07.326053    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8caf7c76acb33f510d73bd9210d24a8c9d63ec3e611a177d19f475be2e6fc38f"} err="failed to get container status \"8caf7c76acb33f510d73bd9210d24a8c9d63ec3e611a177d19f475be2e6fc38f\": rpc error: code = NotFound desc = could not find container \"8caf7c76acb33f510d73bd9210d24a8c9d63ec3e611a177d19f475be2e6fc38f\": container with ID starting with 8caf7c76acb33f510d73bd9210d24a8c9d63ec3e611a177d19f475be2e6fc38f not found: ID does not exist"
Sep 17 03:45:07.330088 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:07.330065    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-26d5d1-openshift-default-df6dfd77d-w2nq4"]
Sep 17 03:45:07.333211 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:07.333191    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-26d5d1-openshift-default-df6dfd77d-w2nq4"]
Sep 17 03:45:08.986699 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:08.986660    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="65a2cb95-0b18-40d3-8076-277905391cd2" path="/var/lib/kubelet/pods/65a2cb95-0b18-40d3-8076-277905391cd2/volumes"
Sep 17 03:45:12.917150 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:12.917115    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-57958f-p9vwh"]
Sep 17 03:45:12.922553 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:12.922510    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-57958f-p9vwh"]
Sep 17 03:45:12.986092 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:12.986065    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4a276646-b9a0-4a4e-9893-7a7c825c9256" path="/var/lib/kubelet/pods/4a276646-b9a0-4a4e-9893-7a7c825c9256/volumes"
Sep 17 03:45:15.992824 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:15.992793    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-214fea-29826945-kjjgv"]
Sep 17 03:45:15.996161 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:15.996139    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-214fea-29826945-kjjgv"]
Sep 17 03:45:16.029120 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:16.029090    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-214fea-57f567459d-hq9js"]
Sep 17 03:45:16.029334 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:16.029296    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-214fea-57f567459d-hq9js" podUID="e9a9f472-6a40-4fe0-b9ab-5cfd765aae46" containerName="payload-pre-processing" containerID="cri-o://66eaea4bcd4bd06ce0622b2f69a8fd0c42156cb8ac72cd5b45fd30dc58c26464" gracePeriod=30
Sep 17 03:45:16.256102 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:16.256070    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-214fea-7gt8k"]
Sep 17 03:45:16.263569 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:16.263540    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-214fea-7gt8k"]
Sep 17 03:45:16.294243 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:16.294216    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-214fea-57f567459d-hq9js"
Sep 17 03:45:16.358908 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:16.358876    2552 generic.go:356] "Generic (PLEG): container finished" podID="e9a9f472-6a40-4fe0-b9ab-5cfd765aae46" containerID="66eaea4bcd4bd06ce0622b2f69a8fd0c42156cb8ac72cd5b45fd30dc58c26464" exitCode=0
Sep 17 03:45:16.359092 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:16.358925    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-214fea-57f567459d-hq9js"
Sep 17 03:45:16.359092 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:16.358952    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-214fea-57f567459d-hq9js" event={"ID":"e9a9f472-6a40-4fe0-b9ab-5cfd765aae46","Type":"ContainerDied","Data":"66eaea4bcd4bd06ce0622b2f69a8fd0c42156cb8ac72cd5b45fd30dc58c26464"}
Sep 17 03:45:16.359092 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:16.359004    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-214fea-57f567459d-hq9js" event={"ID":"e9a9f472-6a40-4fe0-b9ab-5cfd765aae46","Type":"ContainerDied","Data":"dd5fd123632484c968725d7d168c1efe32a7fb8ed71e90ceecc8b9386360eec5"}
Sep 17 03:45:16.359092 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:16.359029    2552 scope.go:117] "RemoveContainer" containerID="66eaea4bcd4bd06ce0622b2f69a8fd0c42156cb8ac72cd5b45fd30dc58c26464"
Sep 17 03:45:16.367682 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:16.367657    2552 scope.go:117] "RemoveContainer" containerID="66eaea4bcd4bd06ce0622b2f69a8fd0c42156cb8ac72cd5b45fd30dc58c26464"
Sep 17 03:45:16.367955 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:45:16.367937    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66eaea4bcd4bd06ce0622b2f69a8fd0c42156cb8ac72cd5b45fd30dc58c26464\": container with ID starting with 66eaea4bcd4bd06ce0622b2f69a8fd0c42156cb8ac72cd5b45fd30dc58c26464 not found: ID does not exist" containerID="66eaea4bcd4bd06ce0622b2f69a8fd0c42156cb8ac72cd5b45fd30dc58c26464"
Sep 17 03:45:16.368010 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:16.367964    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66eaea4bcd4bd06ce0622b2f69a8fd0c42156cb8ac72cd5b45fd30dc58c26464"} err="failed to get container status \"66eaea4bcd4bd06ce0622b2f69a8fd0c42156cb8ac72cd5b45fd30dc58c26464\": rpc error: code = NotFound desc = could not find container \"66eaea4bcd4bd06ce0622b2f69a8fd0c42156cb8ac72cd5b45fd30dc58c26464\": container with ID starting with 66eaea4bcd4bd06ce0622b2f69a8fd0c42156cb8ac72cd5b45fd30dc58c26464 not found: ID does not exist"
Sep 17 03:45:16.396413 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:16.396382    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qxdvm\" (UniqueName: \"kubernetes.io/projected/e9a9f472-6a40-4fe0-b9ab-5cfd765aae46-kube-api-access-qxdvm\") pod \"e9a9f472-6a40-4fe0-b9ab-5cfd765aae46\" (UID: \"e9a9f472-6a40-4fe0-b9ab-5cfd765aae46\") "
Sep 17 03:45:16.396567 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:16.396434    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9a9f472-6a40-4fe0-b9ab-5cfd765aae46-plugins-config-volume\") pod \"e9a9f472-6a40-4fe0-b9ab-5cfd765aae46\" (UID: \"e9a9f472-6a40-4fe0-b9ab-5cfd765aae46\") "
Sep 17 03:45:16.396871 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:16.396850    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9a9f472-6a40-4fe0-b9ab-5cfd765aae46-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e9a9f472-6a40-4fe0-b9ab-5cfd765aae46" (UID: "e9a9f472-6a40-4fe0-b9ab-5cfd765aae46"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 03:45:16.398498 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:16.398482    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9a9f472-6a40-4fe0-b9ab-5cfd765aae46-kube-api-access-qxdvm" (OuterVolumeSpecName: "kube-api-access-qxdvm") pod "e9a9f472-6a40-4fe0-b9ab-5cfd765aae46" (UID: "e9a9f472-6a40-4fe0-b9ab-5cfd765aae46"). InnerVolumeSpecName "kube-api-access-qxdvm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 03:45:16.497792 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:16.497760    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxdvm\" (UniqueName: \"kubernetes.io/projected/e9a9f472-6a40-4fe0-b9ab-5cfd765aae46-kube-api-access-qxdvm\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:45:16.497792 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:16.497788    2552 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9a9f472-6a40-4fe0-b9ab-5cfd765aae46-plugins-config-volume\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:45:16.682027 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:16.681994    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-214fea-57f567459d-hq9js"]
Sep 17 03:45:16.686268 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:16.686244    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-214fea-57f567459d-hq9js"]
Sep 17 03:45:16.992666 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:16.986410    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="10a7a359-b8aa-438c-896f-7d1ae989a550" path="/var/lib/kubelet/pods/10a7a359-b8aa-438c-896f-7d1ae989a550/volumes"
Sep 17 03:45:16.992666 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:16.986945    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a69b8fbf-bd67-44b0-84e4-5726c21d6fce" path="/var/lib/kubelet/pods/a69b8fbf-bd67-44b0-84e4-5726c21d6fce/volumes"
Sep 17 03:45:16.992666 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:16.987416    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e9a9f472-6a40-4fe0-b9ab-5cfd765aae46" path="/var/lib/kubelet/pods/e9a9f472-6a40-4fe0-b9ab-5cfd765aae46/volumes"
Sep 17 03:45:18.218280 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:18.218241    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-214fea/premium-w4-214fea-kserve-589dcd865f-hkkwm" podUID="469467a8-4b70-4019-a870-e8e600f9f840" containerName="main" containerID="cri-o://51ba05a0267aa4f6465a1201ce577023b024740d86548a5e7b76a0fbc2863af3" gracePeriod=45
Sep 17 03:45:18.233014 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:18.232979    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-214fea/simulator-w4-214fea-kserve-5965bff56b-mkmt9" podUID="f29ebfeb-d96e-41de-b4e5-c5f1d6fbe785" containerName="main" containerID="cri-o://8fb54ab60850e74ec523ec3ada66a1c496063a36c6c973a6605df2a9b8f3e852" gracePeriod=45
Sep 17 03:45:18.373202 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:18.373153    2552 generic.go:356] "Generic (PLEG): container finished" podID="f29ebfeb-d96e-41de-b4e5-c5f1d6fbe785" containerID="8fb54ab60850e74ec523ec3ada66a1c496063a36c6c973a6605df2a9b8f3e852" exitCode=0
Sep 17 03:45:18.373473 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:18.373222    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-214fea/simulator-w4-214fea-kserve-5965bff56b-mkmt9" event={"ID":"f29ebfeb-d96e-41de-b4e5-c5f1d6fbe785","Type":"ContainerDied","Data":"8fb54ab60850e74ec523ec3ada66a1c496063a36c6c973a6605df2a9b8f3e852"}
Sep 17 03:45:18.375149 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:18.375122    2552 generic.go:356] "Generic (PLEG): container finished" podID="469467a8-4b70-4019-a870-e8e600f9f840" containerID="51ba05a0267aa4f6465a1201ce577023b024740d86548a5e7b76a0fbc2863af3" exitCode=0
Sep 17 03:45:18.375295 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:18.375203    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-214fea/premium-w4-214fea-kserve-589dcd865f-hkkwm" event={"ID":"469467a8-4b70-4019-a870-e8e600f9f840","Type":"ContainerDied","Data":"51ba05a0267aa4f6465a1201ce577023b024740d86548a5e7b76a0fbc2863af3"}
Sep 17 03:45:18.475055 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:18.474993    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-214fea/premium-w4-214fea-kserve-589dcd865f-hkkwm"
Sep 17 03:45:18.505426 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:18.505403    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-214fea/simulator-w4-214fea-kserve-5965bff56b-mkmt9"
Sep 17 03:45:18.616444 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:18.616412    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f29ebfeb-d96e-41de-b4e5-c5f1d6fbe785-tmp-dir\") pod \"f29ebfeb-d96e-41de-b4e5-c5f1d6fbe785\" (UID: \"f29ebfeb-d96e-41de-b4e5-c5f1d6fbe785\") "
Sep 17 03:45:18.616638 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:18.616456    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f29ebfeb-d96e-41de-b4e5-c5f1d6fbe785-dshm\") pod \"f29ebfeb-d96e-41de-b4e5-c5f1d6fbe785\" (UID: \"f29ebfeb-d96e-41de-b4e5-c5f1d6fbe785\") "
Sep 17 03:45:18.616638 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:18.616486    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2jpz8\" (UniqueName: \"kubernetes.io/projected/f29ebfeb-d96e-41de-b4e5-c5f1d6fbe785-kube-api-access-2jpz8\") pod \"f29ebfeb-d96e-41de-b4e5-c5f1d6fbe785\" (UID: \"f29ebfeb-d96e-41de-b4e5-c5f1d6fbe785\") "
Sep 17 03:45:18.616638 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:18.616533    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/469467a8-4b70-4019-a870-e8e600f9f840-model-cache\") pod \"469467a8-4b70-4019-a870-e8e600f9f840\" (UID: \"469467a8-4b70-4019-a870-e8e600f9f840\") "
Sep 17 03:45:18.616638 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:18.616565    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f29ebfeb-d96e-41de-b4e5-c5f1d6fbe785-tls-certs\") pod \"f29ebfeb-d96e-41de-b4e5-c5f1d6fbe785\" (UID: \"f29ebfeb-d96e-41de-b4e5-c5f1d6fbe785\") "
Sep 17 03:45:18.616638 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:18.616607    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f29ebfeb-d96e-41de-b4e5-c5f1d6fbe785-model-cache\") pod \"f29ebfeb-d96e-41de-b4e5-c5f1d6fbe785\" (UID: \"f29ebfeb-d96e-41de-b4e5-c5f1d6fbe785\") "
Sep 17 03:45:18.616638 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:18.616635    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f29ebfeb-d96e-41de-b4e5-c5f1d6fbe785-home\") pod \"f29ebfeb-d96e-41de-b4e5-c5f1d6fbe785\" (UID: \"f29ebfeb-d96e-41de-b4e5-c5f1d6fbe785\") "
Sep 17 03:45:18.616965 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:18.616664    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tklb9\" (UniqueName: \"kubernetes.io/projected/469467a8-4b70-4019-a870-e8e600f9f840-kube-api-access-tklb9\") pod \"469467a8-4b70-4019-a870-e8e600f9f840\" (UID: \"469467a8-4b70-4019-a870-e8e600f9f840\") "
Sep 17 03:45:18.616965 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:18.616692    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/469467a8-4b70-4019-a870-e8e600f9f840-tmp-dir\") pod \"469467a8-4b70-4019-a870-e8e600f9f840\" (UID: \"469467a8-4b70-4019-a870-e8e600f9f840\") "
Sep 17 03:45:18.616965 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:18.616734    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/469467a8-4b70-4019-a870-e8e600f9f840-home\") pod \"469467a8-4b70-4019-a870-e8e600f9f840\" (UID: \"469467a8-4b70-4019-a870-e8e600f9f840\") "
Sep 17 03:45:18.616965 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:18.616759    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/469467a8-4b70-4019-a870-e8e600f9f840-tls-certs\") pod \"469467a8-4b70-4019-a870-e8e600f9f840\" (UID: \"469467a8-4b70-4019-a870-e8e600f9f840\") "
Sep 17 03:45:18.616965 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:18.616790    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f29ebfeb-d96e-41de-b4e5-c5f1d6fbe785-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f29ebfeb-d96e-41de-b4e5-c5f1d6fbe785" (UID: "f29ebfeb-d96e-41de-b4e5-c5f1d6fbe785"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:45:18.616965 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:18.616810    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/469467a8-4b70-4019-a870-e8e600f9f840-dshm\") pod \"469467a8-4b70-4019-a870-e8e600f9f840\" (UID: \"469467a8-4b70-4019-a870-e8e600f9f840\") "
Sep 17 03:45:18.616965 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:18.616845    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/469467a8-4b70-4019-a870-e8e600f9f840-model-cache" (OuterVolumeSpecName: "model-cache") pod "469467a8-4b70-4019-a870-e8e600f9f840" (UID: "469467a8-4b70-4019-a870-e8e600f9f840"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:45:18.617332 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:18.616972    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/469467a8-4b70-4019-a870-e8e600f9f840-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "469467a8-4b70-4019-a870-e8e600f9f840" (UID: "469467a8-4b70-4019-a870-e8e600f9f840"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:45:18.617332 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:18.616991    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f29ebfeb-d96e-41de-b4e5-c5f1d6fbe785-model-cache" (OuterVolumeSpecName: "model-cache") pod "f29ebfeb-d96e-41de-b4e5-c5f1d6fbe785" (UID: "f29ebfeb-d96e-41de-b4e5-c5f1d6fbe785"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:45:18.617332 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:18.617089    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f29ebfeb-d96e-41de-b4e5-c5f1d6fbe785-home" (OuterVolumeSpecName: "home") pod "f29ebfeb-d96e-41de-b4e5-c5f1d6fbe785" (UID: "f29ebfeb-d96e-41de-b4e5-c5f1d6fbe785"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:45:18.617332 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:18.617192    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/469467a8-4b70-4019-a870-e8e600f9f840-home" (OuterVolumeSpecName: "home") pod "469467a8-4b70-4019-a870-e8e600f9f840" (UID: "469467a8-4b70-4019-a870-e8e600f9f840"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:45:18.617332 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:18.617288    2552 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/469467a8-4b70-4019-a870-e8e600f9f840-model-cache\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:45:18.617332 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:18.617308    2552 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f29ebfeb-d96e-41de-b4e5-c5f1d6fbe785-model-cache\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:45:18.617332 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:18.617322    2552 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f29ebfeb-d96e-41de-b4e5-c5f1d6fbe785-home\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:45:18.617332 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:18.617335    2552 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/469467a8-4b70-4019-a870-e8e600f9f840-tmp-dir\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:45:18.617828 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:18.617348    2552 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/469467a8-4b70-4019-a870-e8e600f9f840-home\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:45:18.617828 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:18.617358    2552 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f29ebfeb-d96e-41de-b4e5-c5f1d6fbe785-tmp-dir\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:45:18.619244 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:18.619215    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/469467a8-4b70-4019-a870-e8e600f9f840-dshm" (OuterVolumeSpecName: "dshm") pod "469467a8-4b70-4019-a870-e8e600f9f840" (UID: "469467a8-4b70-4019-a870-e8e600f9f840"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:45:18.619364 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:18.619265    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/469467a8-4b70-4019-a870-e8e600f9f840-kube-api-access-tklb9" (OuterVolumeSpecName: "kube-api-access-tklb9") pod "469467a8-4b70-4019-a870-e8e600f9f840" (UID: "469467a8-4b70-4019-a870-e8e600f9f840"). InnerVolumeSpecName "kube-api-access-tklb9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 03:45:18.619364 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:18.619273    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f29ebfeb-d96e-41de-b4e5-c5f1d6fbe785-dshm" (OuterVolumeSpecName: "dshm") pod "f29ebfeb-d96e-41de-b4e5-c5f1d6fbe785" (UID: "f29ebfeb-d96e-41de-b4e5-c5f1d6fbe785"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:45:18.619364 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:18.619345    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f29ebfeb-d96e-41de-b4e5-c5f1d6fbe785-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f29ebfeb-d96e-41de-b4e5-c5f1d6fbe785" (UID: "f29ebfeb-d96e-41de-b4e5-c5f1d6fbe785"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 03:45:18.619484 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:18.619345    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/469467a8-4b70-4019-a870-e8e600f9f840-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "469467a8-4b70-4019-a870-e8e600f9f840" (UID: "469467a8-4b70-4019-a870-e8e600f9f840"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 03:45:18.619569 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:18.619554    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f29ebfeb-d96e-41de-b4e5-c5f1d6fbe785-kube-api-access-2jpz8" (OuterVolumeSpecName: "kube-api-access-2jpz8") pod "f29ebfeb-d96e-41de-b4e5-c5f1d6fbe785" (UID: "f29ebfeb-d96e-41de-b4e5-c5f1d6fbe785"). InnerVolumeSpecName "kube-api-access-2jpz8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 03:45:18.717792 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:18.717765    2552 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f29ebfeb-d96e-41de-b4e5-c5f1d6fbe785-dshm\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:45:18.717792 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:18.717786    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2jpz8\" (UniqueName: \"kubernetes.io/projected/f29ebfeb-d96e-41de-b4e5-c5f1d6fbe785-kube-api-access-2jpz8\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:45:18.717951 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:18.717797    2552 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f29ebfeb-d96e-41de-b4e5-c5f1d6fbe785-tls-certs\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:45:18.717951 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:18.717807    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tklb9\" (UniqueName: \"kubernetes.io/projected/469467a8-4b70-4019-a870-e8e600f9f840-kube-api-access-tklb9\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:45:18.717951 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:18.717815    2552 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/469467a8-4b70-4019-a870-e8e600f9f840-tls-certs\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:45:18.717951 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:18.717823    2552 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/469467a8-4b70-4019-a870-e8e600f9f840-dshm\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:45:19.382283 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:19.382253    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-214fea/simulator-w4-214fea-kserve-5965bff56b-mkmt9"
Sep 17 03:45:19.382758 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:19.382278    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-214fea/simulator-w4-214fea-kserve-5965bff56b-mkmt9" event={"ID":"f29ebfeb-d96e-41de-b4e5-c5f1d6fbe785","Type":"ContainerDied","Data":"c0cb5fe07f472fbba59c2bd5816dbcfafb7db4c2f1ef786d5a099183c11922e4"}
Sep 17 03:45:19.382758 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:19.382335    2552 scope.go:117] "RemoveContainer" containerID="8fb54ab60850e74ec523ec3ada66a1c496063a36c6c973a6605df2a9b8f3e852"
Sep 17 03:45:19.383707 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:19.383690    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-214fea/premium-w4-214fea-kserve-589dcd865f-hkkwm"
Sep 17 03:45:19.383788 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:19.383691    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-214fea/premium-w4-214fea-kserve-589dcd865f-hkkwm" event={"ID":"469467a8-4b70-4019-a870-e8e600f9f840","Type":"ContainerDied","Data":"e75aeb83fe2cc95c1359dff83526eaceb3a949a0d5809f7dcc15205ee3f0f64f"}
Sep 17 03:45:19.393744 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:19.393729    2552 scope.go:117] "RemoveContainer" containerID="51ba05a0267aa4f6465a1201ce577023b024740d86548a5e7b76a0fbc2863af3"
Sep 17 03:45:19.400566 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:19.400539    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-214fea/premium-w4-214fea-kserve-589dcd865f-hkkwm"]
Sep 17 03:45:19.403644 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:19.403625    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-214fea/premium-w4-214fea-kserve-589dcd865f-hkkwm"]
Sep 17 03:45:19.413938 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:19.413916    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-214fea/simulator-w4-214fea-kserve-5965bff56b-mkmt9"]
Sep 17 03:45:19.417505 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:19.417474    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-214fea/simulator-w4-214fea-kserve-5965bff56b-mkmt9"]
Sep 17 03:45:20.276813 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:20.276778    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-fc820e-gqsng"]
Sep 17 03:45:20.277214 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:20.277195    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a276646-b9a0-4a4e-9893-7a7c825c9256" containerName="revoke-keys"
Sep 17 03:45:20.277294 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:20.277215    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a276646-b9a0-4a4e-9893-7a7c825c9256" containerName="revoke-keys"
Sep 17 03:45:20.277294 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:20.277234    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65a2cb95-0b18-40d3-8076-277905391cd2" containerName="istio-proxy"
Sep 17 03:45:20.277294 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:20.277242    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="65a2cb95-0b18-40d3-8076-277905391cd2" containerName="istio-proxy"
Sep 17 03:45:20.277294 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:20.277257    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab9688c7-a95b-4430-8075-a0008c6595fc" containerName="cleanup"
Sep 17 03:45:20.277294 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:20.277266    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab9688c7-a95b-4430-8075-a0008c6595fc" containerName="cleanup"
Sep 17 03:45:20.277294 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:20.277285    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2338e9b-a64c-47d6-8b38-d47b0d148690" containerName="cleanup"
Sep 17 03:45:20.277294 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:20.277293    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2338e9b-a64c-47d6-8b38-d47b0d148690" containerName="cleanup"
Sep 17 03:45:20.277671 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:20.277305    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a69b8fbf-bd67-44b0-84e4-5726c21d6fce" containerName="cleanup"
Sep 17 03:45:20.277671 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:20.277335    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="a69b8fbf-bd67-44b0-84e4-5726c21d6fce" containerName="cleanup"
Sep 17 03:45:20.277671 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:20.277349    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="469467a8-4b70-4019-a870-e8e600f9f840" containerName="main"
Sep 17 03:45:20.277671 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:20.277357    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="469467a8-4b70-4019-a870-e8e600f9f840" containerName="main"
Sep 17 03:45:20.277671 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:20.277367    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65a2cb95-0b18-40d3-8076-277905391cd2" containerName="istio-proxy"
Sep 17 03:45:20.277671 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:20.277376    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="65a2cb95-0b18-40d3-8076-277905391cd2" containerName="istio-proxy"
Sep 17 03:45:20.277671 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:20.277394    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9a9f472-6a40-4fe0-b9ab-5cfd765aae46" containerName="payload-pre-processing"
Sep 17 03:45:20.277671 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:20.277402    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9a9f472-6a40-4fe0-b9ab-5cfd765aae46" containerName="payload-pre-processing"
Sep 17 03:45:20.277671 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:20.277424    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f29ebfeb-d96e-41de-b4e5-c5f1d6fbe785" containerName="main"
Sep 17 03:45:20.277671 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:20.277432    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="f29ebfeb-d96e-41de-b4e5-c5f1d6fbe785" containerName="main"
Sep 17 03:45:20.277671 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:20.277526    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="e9a9f472-6a40-4fe0-b9ab-5cfd765aae46" containerName="payload-pre-processing"
Sep 17 03:45:20.277671 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:20.277536    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a276646-b9a0-4a4e-9893-7a7c825c9256" containerName="revoke-keys"
Sep 17 03:45:20.277671 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:20.277546    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="f29ebfeb-d96e-41de-b4e5-c5f1d6fbe785" containerName="main"
Sep 17 03:45:20.277671 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:20.277556    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="65a2cb95-0b18-40d3-8076-277905391cd2" containerName="istio-proxy"
Sep 17 03:45:20.277671 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:20.277567    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab9688c7-a95b-4430-8075-a0008c6595fc" containerName="cleanup"
Sep 17 03:45:20.277671 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:20.277576    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="c2338e9b-a64c-47d6-8b38-d47b0d148690" containerName="cleanup"
Sep 17 03:45:20.277671 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:20.277586    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="469467a8-4b70-4019-a870-e8e600f9f840" containerName="main"
Sep 17 03:45:20.277671 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:20.277611    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="a69b8fbf-bd67-44b0-84e4-5726c21d6fce" containerName="cleanup"
Sep 17 03:45:20.282187 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:20.282167    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-fc820e-gqsng"
Sep 17 03:45:20.284551 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:20.284531    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8vwnw\""
Sep 17 03:45:20.288570 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:20.288552    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-fc820e-gqsng"]
Sep 17 03:45:20.434990 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:20.434959    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ef74694-cfd8-4bdd-8803-eed95b2cf6b4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-fc820e-gqsng\" (UID: \"2ef74694-cfd8-4bdd-8803-eed95b2cf6b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-fc820e-gqsng"
Sep 17 03:45:20.536426 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:20.536344    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ef74694-cfd8-4bdd-8803-eed95b2cf6b4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-fc820e-gqsng\" (UID: \"2ef74694-cfd8-4bdd-8803-eed95b2cf6b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-fc820e-gqsng"
Sep 17 03:45:20.536985 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:20.536964    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ef74694-cfd8-4bdd-8803-eed95b2cf6b4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-fc820e-gqsng\" (UID: \"2ef74694-cfd8-4bdd-8803-eed95b2cf6b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-fc820e-gqsng"
Sep 17 03:45:20.596335 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:20.596308    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-fc820e-gqsng"
Sep 17 03:45:20.866005 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:20.865980    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-fc820e-gqsng"]
Sep 17 03:45:20.868284 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:45:20.868259    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ef74694_cfd8_4bdd_8803_eed95b2cf6b4.slice/crio-f6a95dc02fb1735b94cc3e8a2ea25102464645b71d2a36515d5a14ae452b2ba1 WatchSource:0}: Error finding container f6a95dc02fb1735b94cc3e8a2ea25102464645b71d2a36515d5a14ae452b2ba1: Status 404 returned error can't find the container with id f6a95dc02fb1735b94cc3e8a2ea25102464645b71d2a36515d5a14ae452b2ba1
Sep 17 03:45:20.985640 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:20.985587    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="469467a8-4b70-4019-a870-e8e600f9f840" path="/var/lib/kubelet/pods/469467a8-4b70-4019-a870-e8e600f9f840/volumes"
Sep 17 03:45:20.986035 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:20.986022    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f29ebfeb-d96e-41de-b4e5-c5f1d6fbe785" path="/var/lib/kubelet/pods/f29ebfeb-d96e-41de-b4e5-c5f1d6fbe785/volumes"
Sep 17 03:45:21.397920 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:21.397835    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-fc820e-gqsng" event={"ID":"2ef74694-cfd8-4bdd-8803-eed95b2cf6b4","Type":"ContainerStarted","Data":"33123885a34b4b19e3989cc924465ea977034ceb152861c7f97c102fdf3b5b7e"}
Sep 17 03:45:21.397920 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:21.397885    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-fc820e-gqsng" event={"ID":"2ef74694-cfd8-4bdd-8803-eed95b2cf6b4","Type":"ContainerStarted","Data":"f6a95dc02fb1735b94cc3e8a2ea25102464645b71d2a36515d5a14ae452b2ba1"}
Sep 17 03:45:21.413231 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:21.413184    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-fc820e-gqsng" podStartSLOduration=1.41316987 podStartE2EDuration="1.41316987s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 03:45:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:45:21.411638623 +0000 UTC m=+1556.878046150" watchObservedRunningTime="2026-09-17 03:45:21.41316987 +0000 UTC m=+1556.879577397"
Sep 17 03:45:22.404724 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:22.404685    2552 generic.go:356] "Generic (PLEG): container finished" podID="2ef74694-cfd8-4bdd-8803-eed95b2cf6b4" containerID="33123885a34b4b19e3989cc924465ea977034ceb152861c7f97c102fdf3b5b7e" exitCode=0
Sep 17 03:45:22.405079 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:22.404764    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-fc820e-gqsng" event={"ID":"2ef74694-cfd8-4bdd-8803-eed95b2cf6b4","Type":"ContainerDied","Data":"33123885a34b4b19e3989cc924465ea977034ceb152861c7f97c102fdf3b5b7e"}
Sep 17 03:45:23.531727 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:23.531700    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-fc820e-gqsng"
Sep 17 03:45:23.668813 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:23.668724    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ef74694-cfd8-4bdd-8803-eed95b2cf6b4-maas-api-service-ca\") pod \"2ef74694-cfd8-4bdd-8803-eed95b2cf6b4\" (UID: \"2ef74694-cfd8-4bdd-8803-eed95b2cf6b4\") "
Sep 17 03:45:23.669049 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:23.669029    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ef74694-cfd8-4bdd-8803-eed95b2cf6b4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2ef74694-cfd8-4bdd-8803-eed95b2cf6b4" (UID: "2ef74694-cfd8-4bdd-8803-eed95b2cf6b4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 03:45:23.770156 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:23.770120    2552 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ef74694-cfd8-4bdd-8803-eed95b2cf6b4-maas-api-service-ca\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:45:24.415820 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:24.415791    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-fc820e-gqsng"
Sep 17 03:45:24.416019 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:24.415791    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-fc820e-gqsng" event={"ID":"2ef74694-cfd8-4bdd-8803-eed95b2cf6b4","Type":"ContainerDied","Data":"f6a95dc02fb1735b94cc3e8a2ea25102464645b71d2a36515d5a14ae452b2ba1"}
Sep 17 03:45:24.416019 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:24.415915    2552 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f6a95dc02fb1735b94cc3e8a2ea25102464645b71d2a36515d5a14ae452b2ba1"
Sep 17 03:45:31.090687 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:31.090651    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-fc820e-gqsng"]
Sep 17 03:45:31.096396 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:31.096370    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-fc820e-gqsng"]
Sep 17 03:45:32.985685 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:32.985652    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2ef74694-cfd8-4bdd-8803-eed95b2cf6b4" path="/var/lib/kubelet/pods/2ef74694-cfd8-4bdd-8803-eed95b2cf6b4/volumes"
Sep 17 03:45:53.189676 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:53.189638    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7746-543cf5"]
Sep 17 03:45:53.190082 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:53.190062    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ef74694-cfd8-4bdd-8803-eed95b2cf6b4" containerName="revoke-keys"
Sep 17 03:45:53.190082 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:53.190075    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ef74694-cfd8-4bdd-8803-eed95b2cf6b4" containerName="revoke-keys"
Sep 17 03:45:53.190172 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:53.190162    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ef74694-cfd8-4bdd-8803-eed95b2cf6b4" containerName="revoke-keys"
Sep 17 03:45:53.190211 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:53.190173    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="65a2cb95-0b18-40d3-8076-277905391cd2" containerName="istio-proxy"
Sep 17 03:45:53.194615 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:53.194583    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7746-543cf5"
Sep 17 03:45:53.197558 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:53.197529    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-s27lz\""
Sep 17 03:45:53.199120 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:53.199097    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7746-543cf5"]
Sep 17 03:45:53.348035 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:53.348000    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7pmw\" (UniqueName: \"kubernetes.io/projected/715ae696-d23e-4a80-98df-b327cfcfe601-kube-api-access-c7pmw\") pod \"test-curl-7746-543cf5\" (UID: \"715ae696-d23e-4a80-98df-b327cfcfe601\") " pod="opendatahub/test-curl-7746-543cf5"
Sep 17 03:45:53.449093 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:53.449005    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7pmw\" (UniqueName: \"kubernetes.io/projected/715ae696-d23e-4a80-98df-b327cfcfe601-kube-api-access-c7pmw\") pod \"test-curl-7746-543cf5\" (UID: \"715ae696-d23e-4a80-98df-b327cfcfe601\") " pod="opendatahub/test-curl-7746-543cf5"
Sep 17 03:45:53.457093 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:53.457071    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7pmw\" (UniqueName: \"kubernetes.io/projected/715ae696-d23e-4a80-98df-b327cfcfe601-kube-api-access-c7pmw\") pod \"test-curl-7746-543cf5\" (UID: \"715ae696-d23e-4a80-98df-b327cfcfe601\") " pod="opendatahub/test-curl-7746-543cf5"
Sep 17 03:45:53.505611 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:53.505562    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7746-543cf5"
Sep 17 03:45:53.772359 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:53.772334    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7746-543cf5"]
Sep 17 03:45:53.774233 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:45:53.774198    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod715ae696_d23e_4a80_98df_b327cfcfe601.slice/crio-dec76d60d7129af0e371a981fe7feb79e703a7ef2fc7a8456eba1c4fac9f6a14 WatchSource:0}: Error finding container dec76d60d7129af0e371a981fe7feb79e703a7ef2fc7a8456eba1c4fac9f6a14: Status 404 returned error can't find the container with id dec76d60d7129af0e371a981fe7feb79e703a7ef2fc7a8456eba1c4fac9f6a14
Sep 17 03:45:54.580014 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:54.579976    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7746-543cf5" event={"ID":"715ae696-d23e-4a80-98df-b327cfcfe601","Type":"ContainerStarted","Data":"ffa4ad4978cd2c1767244490d84bb4e1ba96e5705f1d0d169badcd5500b6a6b3"}
Sep 17 03:45:54.580014 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:54.580022    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7746-543cf5" event={"ID":"715ae696-d23e-4a80-98df-b327cfcfe601","Type":"ContainerStarted","Data":"dec76d60d7129af0e371a981fe7feb79e703a7ef2fc7a8456eba1c4fac9f6a14"}
Sep 17 03:45:54.596102 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:54.596055    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7746-543cf5" podStartSLOduration=1.5960393320000001 podStartE2EDuration="1.596039332s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 03:45:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:45:54.594028627 +0000 UTC m=+1590.060436153" watchObservedRunningTime="2026-09-17 03:45:54.596039332 +0000 UTC m=+1590.062446858"
Sep 17 03:45:55.171722 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:55.171688    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7746-543cf5"]
Sep 17 03:45:55.176572 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:55.176537    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7746-543cf5"]
Sep 17 03:45:55.585548 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:45:55.585516    2552 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7746-543cf5" container="test-curl-7746-543cf5"
Sep 17 03:45:56.590335 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:56.590299    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7746-543cf5" podUID="715ae696-d23e-4a80-98df-b327cfcfe601" containerName="test-curl-7746-543cf5" containerID="cri-o://ffa4ad4978cd2c1767244490d84bb4e1ba96e5705f1d0d169badcd5500b6a6b3" gracePeriod=2
Sep 17 03:45:56.592636 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:45:56.592592    2552 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7746-543cf5\" is forbidden: User \"system:node:ip-10-0-139-53.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-53.ec2.internal' and this object" podUID="715ae696-d23e-4a80-98df-b327cfcfe601" pod="opendatahub/test-curl-7746-543cf5"
Sep 17 03:45:57.480935 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:57.480902    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-f6d581/auth-test-model-f6d581-kserve-5dbd89f68c-lgmzg"]
Sep 17 03:45:58.767058 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:58.767029    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7746-543cf5"
Sep 17 03:45:58.902547 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:58.902459    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c7pmw\" (UniqueName: \"kubernetes.io/projected/715ae696-d23e-4a80-98df-b327cfcfe601-kube-api-access-c7pmw\") pod \"715ae696-d23e-4a80-98df-b327cfcfe601\" (UID: \"715ae696-d23e-4a80-98df-b327cfcfe601\") "
Sep 17 03:45:58.904648 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:58.904612    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/715ae696-d23e-4a80-98df-b327cfcfe601-kube-api-access-c7pmw" (OuterVolumeSpecName: "kube-api-access-c7pmw") pod "715ae696-d23e-4a80-98df-b327cfcfe601" (UID: "715ae696-d23e-4a80-98df-b327cfcfe601"). InnerVolumeSpecName "kube-api-access-c7pmw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 03:45:58.985021 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:58.984987    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="715ae696-d23e-4a80-98df-b327cfcfe601" path="/var/lib/kubelet/pods/715ae696-d23e-4a80-98df-b327cfcfe601/volumes"
Sep 17 03:45:59.004226 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:59.004201    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c7pmw\" (UniqueName: \"kubernetes.io/projected/715ae696-d23e-4a80-98df-b327cfcfe601-kube-api-access-c7pmw\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:45:59.608076 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:59.608035    2552 generic.go:356] "Generic (PLEG): container finished" podID="715ae696-d23e-4a80-98df-b327cfcfe601" containerID="ffa4ad4978cd2c1767244490d84bb4e1ba96e5705f1d0d169badcd5500b6a6b3" exitCode=137
Sep 17 03:45:59.608076 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:59.608071    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7746-543cf5"
Sep 17 03:45:59.608303 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:59.608121    2552 scope.go:117] "RemoveContainer" containerID="ffa4ad4978cd2c1767244490d84bb4e1ba96e5705f1d0d169badcd5500b6a6b3"
Sep 17 03:45:59.616817 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:59.616803    2552 scope.go:117] "RemoveContainer" containerID="ffa4ad4978cd2c1767244490d84bb4e1ba96e5705f1d0d169badcd5500b6a6b3"
Sep 17 03:45:59.617081 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:45:59.617063    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ffa4ad4978cd2c1767244490d84bb4e1ba96e5705f1d0d169badcd5500b6a6b3\": container with ID starting with ffa4ad4978cd2c1767244490d84bb4e1ba96e5705f1d0d169badcd5500b6a6b3 not found: ID does not exist" containerID="ffa4ad4978cd2c1767244490d84bb4e1ba96e5705f1d0d169badcd5500b6a6b3"
Sep 17 03:45:59.617157 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:45:59.617091    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ffa4ad4978cd2c1767244490d84bb4e1ba96e5705f1d0d169badcd5500b6a6b3"} err="failed to get container status \"ffa4ad4978cd2c1767244490d84bb4e1ba96e5705f1d0d169badcd5500b6a6b3\": rpc error: code = NotFound desc = could not find container \"ffa4ad4978cd2c1767244490d84bb4e1ba96e5705f1d0d169badcd5500b6a6b3\": container with ID starting with ffa4ad4978cd2c1767244490d84bb4e1ba96e5705f1d0d169badcd5500b6a6b3 not found: ID does not exist"
Sep 17 03:46:05.211684 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:46:05.211637    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-22e045/sub-test-model-22e045-kserve-c6645c55f-6t7mz"]
Sep 17 03:46:05.212109 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:46:05.212036    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="715ae696-d23e-4a80-98df-b327cfcfe601" containerName="test-curl-7746-543cf5"
Sep 17 03:46:05.212109 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:46:05.212049    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="715ae696-d23e-4a80-98df-b327cfcfe601" containerName="test-curl-7746-543cf5"
Sep 17 03:46:05.212189 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:46:05.212127    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="715ae696-d23e-4a80-98df-b327cfcfe601" containerName="test-curl-7746-543cf5"
Sep 17 03:46:05.216697 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:46:05.216674    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-22e045/sub-test-model-22e045-kserve-c6645c55f-6t7mz"
Sep 17 03:46:05.224632 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:46:05.224264    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-22e045\"/\"sub-test-model-22e045-kserve-self-signed-certs\""
Sep 17 03:46:05.224632 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:46:05.224336    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-22e045\"/\"kube-root-ca.crt\""
Sep 17 03:46:05.224783 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:46:05.224658    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-22e045\"/\"default-dockercfg-8dd44\""
Sep 17 03:46:05.224783 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:46:05.224670    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-22e045\"/\"openshift-service-ca.crt\""
Sep 17 03:46:05.227384 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:46:05.227363    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-22e045/sub-test-model-22e045-kserve-c6645c55f-6t7mz"]
Sep 17 03:46:05.362895 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:46:05.362864    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91cec756-29a9-4e15-a7f0-fbb88cb7fa88-model-cache\") pod \"sub-test-model-22e045-kserve-c6645c55f-6t7mz\" (UID: \"91cec756-29a9-4e15-a7f0-fbb88cb7fa88\") " pod="ai-tenant-e2e-shared-a-w3-22e045/sub-test-model-22e045-kserve-c6645c55f-6t7mz"
Sep 17 03:46:05.363086 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:46:05.362929    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91cec756-29a9-4e15-a7f0-fbb88cb7fa88-tmp-dir\") pod \"sub-test-model-22e045-kserve-c6645c55f-6t7mz\" (UID: \"91cec756-29a9-4e15-a7f0-fbb88cb7fa88\") " pod="ai-tenant-e2e-shared-a-w3-22e045/sub-test-model-22e045-kserve-c6645c55f-6t7mz"
Sep 17 03:46:05.363086 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:46:05.362975    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91cec756-29a9-4e15-a7f0-fbb88cb7fa88-dshm\") pod \"sub-test-model-22e045-kserve-c6645c55f-6t7mz\" (UID: \"91cec756-29a9-4e15-a7f0-fbb88cb7fa88\") " pod="ai-tenant-e2e-shared-a-w3-22e045/sub-test-model-22e045-kserve-c6645c55f-6t7mz"
Sep 17 03:46:05.363086 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:46:05.363003    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66s6w\" (UniqueName: \"kubernetes.io/projected/91cec756-29a9-4e15-a7f0-fbb88cb7fa88-kube-api-access-66s6w\") pod \"sub-test-model-22e045-kserve-c6645c55f-6t7mz\" (UID: \"91cec756-29a9-4e15-a7f0-fbb88cb7fa88\") " pod="ai-tenant-e2e-shared-a-w3-22e045/sub-test-model-22e045-kserve-c6645c55f-6t7mz"
Sep 17 03:46:05.363206 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:46:05.363086    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91cec756-29a9-4e15-a7f0-fbb88cb7fa88-tls-certs\") pod \"sub-test-model-22e045-kserve-c6645c55f-6t7mz\" (UID: \"91cec756-29a9-4e15-a7f0-fbb88cb7fa88\") " pod="ai-tenant-e2e-shared-a-w3-22e045/sub-test-model-22e045-kserve-c6645c55f-6t7mz"
Sep 17 03:46:05.363206 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:46:05.363157    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91cec756-29a9-4e15-a7f0-fbb88cb7fa88-home\") pod \"sub-test-model-22e045-kserve-c6645c55f-6t7mz\" (UID: \"91cec756-29a9-4e15-a7f0-fbb88cb7fa88\") " pod="ai-tenant-e2e-shared-a-w3-22e045/sub-test-model-22e045-kserve-c6645c55f-6t7mz"
Sep 17 03:46:05.464334 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:46:05.464239    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91cec756-29a9-4e15-a7f0-fbb88cb7fa88-home\") pod \"sub-test-model-22e045-kserve-c6645c55f-6t7mz\" (UID: \"91cec756-29a9-4e15-a7f0-fbb88cb7fa88\") " pod="ai-tenant-e2e-shared-a-w3-22e045/sub-test-model-22e045-kserve-c6645c55f-6t7mz"
Sep 17 03:46:05.464334 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:46:05.464322    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91cec756-29a9-4e15-a7f0-fbb88cb7fa88-model-cache\") pod \"sub-test-model-22e045-kserve-c6645c55f-6t7mz\" (UID: \"91cec756-29a9-4e15-a7f0-fbb88cb7fa88\") " pod="ai-tenant-e2e-shared-a-w3-22e045/sub-test-model-22e045-kserve-c6645c55f-6t7mz"
Sep 17 03:46:05.464573 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:46:05.464391    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91cec756-29a9-4e15-a7f0-fbb88cb7fa88-tmp-dir\") pod \"sub-test-model-22e045-kserve-c6645c55f-6t7mz\" (UID: \"91cec756-29a9-4e15-a7f0-fbb88cb7fa88\") " pod="ai-tenant-e2e-shared-a-w3-22e045/sub-test-model-22e045-kserve-c6645c55f-6t7mz"
Sep 17 03:46:05.464573 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:46:05.464424    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91cec756-29a9-4e15-a7f0-fbb88cb7fa88-dshm\") pod \"sub-test-model-22e045-kserve-c6645c55f-6t7mz\" (UID: \"91cec756-29a9-4e15-a7f0-fbb88cb7fa88\") " pod="ai-tenant-e2e-shared-a-w3-22e045/sub-test-model-22e045-kserve-c6645c55f-6t7mz"
Sep 17 03:46:05.464573 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:46:05.464462    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66s6w\" (UniqueName: \"kubernetes.io/projected/91cec756-29a9-4e15-a7f0-fbb88cb7fa88-kube-api-access-66s6w\") pod \"sub-test-model-22e045-kserve-c6645c55f-6t7mz\" (UID: \"91cec756-29a9-4e15-a7f0-fbb88cb7fa88\") " pod="ai-tenant-e2e-shared-a-w3-22e045/sub-test-model-22e045-kserve-c6645c55f-6t7mz"
Sep 17 03:46:05.464573 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:46:05.464498    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91cec756-29a9-4e15-a7f0-fbb88cb7fa88-tls-certs\") pod \"sub-test-model-22e045-kserve-c6645c55f-6t7mz\" (UID: \"91cec756-29a9-4e15-a7f0-fbb88cb7fa88\") " pod="ai-tenant-e2e-shared-a-w3-22e045/sub-test-model-22e045-kserve-c6645c55f-6t7mz"
Sep 17 03:46:05.464835 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:46:05.464645    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91cec756-29a9-4e15-a7f0-fbb88cb7fa88-home\") pod \"sub-test-model-22e045-kserve-c6645c55f-6t7mz\" (UID: \"91cec756-29a9-4e15-a7f0-fbb88cb7fa88\") " pod="ai-tenant-e2e-shared-a-w3-22e045/sub-test-model-22e045-kserve-c6645c55f-6t7mz"
Sep 17 03:46:05.464835 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:46:05.464659    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91cec756-29a9-4e15-a7f0-fbb88cb7fa88-model-cache\") pod \"sub-test-model-22e045-kserve-c6645c55f-6t7mz\" (UID: \"91cec756-29a9-4e15-a7f0-fbb88cb7fa88\") " pod="ai-tenant-e2e-shared-a-w3-22e045/sub-test-model-22e045-kserve-c6645c55f-6t7mz"
Sep 17 03:46:05.464835 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:46:05.464811    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91cec756-29a9-4e15-a7f0-fbb88cb7fa88-tmp-dir\") pod \"sub-test-model-22e045-kserve-c6645c55f-6t7mz\" (UID: \"91cec756-29a9-4e15-a7f0-fbb88cb7fa88\") " pod="ai-tenant-e2e-shared-a-w3-22e045/sub-test-model-22e045-kserve-c6645c55f-6t7mz"
Sep 17 03:46:05.466587 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:46:05.466564    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91cec756-29a9-4e15-a7f0-fbb88cb7fa88-dshm\") pod \"sub-test-model-22e045-kserve-c6645c55f-6t7mz\" (UID: \"91cec756-29a9-4e15-a7f0-fbb88cb7fa88\") " pod="ai-tenant-e2e-shared-a-w3-22e045/sub-test-model-22e045-kserve-c6645c55f-6t7mz"
Sep 17 03:46:05.466868 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:46:05.466851    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91cec756-29a9-4e15-a7f0-fbb88cb7fa88-tls-certs\") pod \"sub-test-model-22e045-kserve-c6645c55f-6t7mz\" (UID: \"91cec756-29a9-4e15-a7f0-fbb88cb7fa88\") " pod="ai-tenant-e2e-shared-a-w3-22e045/sub-test-model-22e045-kserve-c6645c55f-6t7mz"
Sep 17 03:46:05.473624 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:46:05.473571    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-66s6w\" (UniqueName: \"kubernetes.io/projected/91cec756-29a9-4e15-a7f0-fbb88cb7fa88-kube-api-access-66s6w\") pod \"sub-test-model-22e045-kserve-c6645c55f-6t7mz\" (UID: \"91cec756-29a9-4e15-a7f0-fbb88cb7fa88\") " pod="ai-tenant-e2e-shared-a-w3-22e045/sub-test-model-22e045-kserve-c6645c55f-6t7mz"
Sep 17 03:46:05.527586 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:46:05.527560    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-22e045/sub-test-model-22e045-kserve-c6645c55f-6t7mz"
Sep 17 03:46:05.800523 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:46:05.800498    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-22e045/sub-test-model-22e045-kserve-c6645c55f-6t7mz"]
Sep 17 03:46:05.802682 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:46:05.802653    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod91cec756_29a9_4e15_a7f0_fbb88cb7fa88.slice/crio-e0ecf8edba32c63fe871df6406f0aba2b990bedd86bd68750f90fc4200eff274 WatchSource:0}: Error finding container e0ecf8edba32c63fe871df6406f0aba2b990bedd86bd68750f90fc4200eff274: Status 404 returned error can't find the container with id e0ecf8edba32c63fe871df6406f0aba2b990bedd86bd68750f90fc4200eff274
Sep 17 03:46:06.651329 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:46:06.651297    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-22e045/sub-test-model-22e045-kserve-c6645c55f-6t7mz" event={"ID":"91cec756-29a9-4e15-a7f0-fbb88cb7fa88","Type":"ContainerStarted","Data":"9e6938b3225295417d4d7c6951a38a2d9cb7eac1e8a0254f0b2834739b270c34"}
Sep 17 03:46:06.651732 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:46:06.651338    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-22e045/sub-test-model-22e045-kserve-c6645c55f-6t7mz" event={"ID":"91cec756-29a9-4e15-a7f0-fbb88cb7fa88","Type":"ContainerStarted","Data":"e0ecf8edba32c63fe871df6406f0aba2b990bedd86bd68750f90fc4200eff274"}
Sep 17 03:46:06.668192 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:46:06.668144    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-22e045/sub-test-model-22e045-kserve-c6645c55f-6t7mz" podStartSLOduration=1.668130525 podStartE2EDuration="1.668130525s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 03:46:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:46:06.667443462 +0000 UTC m=+1602.133851001" watchObservedRunningTime="2026-09-17 03:46:06.668130525 +0000 UTC m=+1602.134538051"
Sep 17 03:46:12.500404 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:46:12.500368    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-f6d581/auth-test-model-f6d581-kserve-5dbd89f68c-lgmzg" podUID="a5011121-3390-4619-af40-04e8adaff034" containerName="main" containerID="cri-o://43acabaf2872f631b2a35612ecbf343ded78bf421c071e8c9ef932212a4c060f" gracePeriod=45
Sep 17 03:46:12.688113 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:46:12.688083    2552 generic.go:356] "Generic (PLEG): container finished" podID="a5011121-3390-4619-af40-04e8adaff034" containerID="43acabaf2872f631b2a35612ecbf343ded78bf421c071e8c9ef932212a4c060f" exitCode=0
Sep 17 03:46:12.688295 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:46:12.688150    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-f6d581/auth-test-model-f6d581-kserve-5dbd89f68c-lgmzg" event={"ID":"a5011121-3390-4619-af40-04e8adaff034","Type":"ContainerDied","Data":"43acabaf2872f631b2a35612ecbf343ded78bf421c071e8c9ef932212a4c060f"}
Sep 17 03:46:12.739611 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:46:12.739577    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-f6d581/auth-test-model-f6d581-kserve-5dbd89f68c-lgmzg"
Sep 17 03:46:12.834075 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:46:12.833979    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5011121-3390-4619-af40-04e8adaff034-dshm\") pod \"a5011121-3390-4619-af40-04e8adaff034\" (UID: \"a5011121-3390-4619-af40-04e8adaff034\") "
Sep 17 03:46:12.834075 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:46:12.834024    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5lgfq\" (UniqueName: \"kubernetes.io/projected/a5011121-3390-4619-af40-04e8adaff034-kube-api-access-5lgfq\") pod \"a5011121-3390-4619-af40-04e8adaff034\" (UID: \"a5011121-3390-4619-af40-04e8adaff034\") "
Sep 17 03:46:12.834075 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:46:12.834046    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5011121-3390-4619-af40-04e8adaff034-tmp-dir\") pod \"a5011121-3390-4619-af40-04e8adaff034\" (UID: \"a5011121-3390-4619-af40-04e8adaff034\") "
Sep 17 03:46:12.834075 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:46:12.834079    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5011121-3390-4619-af40-04e8adaff034-home\") pod \"a5011121-3390-4619-af40-04e8adaff034\" (UID: \"a5011121-3390-4619-af40-04e8adaff034\") "
Sep 17 03:46:12.834416 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:46:12.834116    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5011121-3390-4619-af40-04e8adaff034-model-cache\") pod \"a5011121-3390-4619-af40-04e8adaff034\" (UID: \"a5011121-3390-4619-af40-04e8adaff034\") "
Sep 17 03:46:12.834416 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:46:12.834157    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5011121-3390-4619-af40-04e8adaff034-tls-certs\") pod \"a5011121-3390-4619-af40-04e8adaff034\" (UID: \"a5011121-3390-4619-af40-04e8adaff034\") "
Sep 17 03:46:12.834416 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:46:12.834364    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5011121-3390-4619-af40-04e8adaff034-home" (OuterVolumeSpecName: "home") pod "a5011121-3390-4619-af40-04e8adaff034" (UID: "a5011121-3390-4619-af40-04e8adaff034"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:46:12.834416 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:46:12.834391    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5011121-3390-4619-af40-04e8adaff034-model-cache" (OuterVolumeSpecName: "model-cache") pod "a5011121-3390-4619-af40-04e8adaff034" (UID: "a5011121-3390-4619-af40-04e8adaff034"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:46:12.834416 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:46:12.834408    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5011121-3390-4619-af40-04e8adaff034-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a5011121-3390-4619-af40-04e8adaff034" (UID: "a5011121-3390-4619-af40-04e8adaff034"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:46:12.834664 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:46:12.834480    2552 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5011121-3390-4619-af40-04e8adaff034-home\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:46:12.836458 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:46:12.836433    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5011121-3390-4619-af40-04e8adaff034-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a5011121-3390-4619-af40-04e8adaff034" (UID: "a5011121-3390-4619-af40-04e8adaff034"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 03:46:12.836458 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:46:12.836447    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5011121-3390-4619-af40-04e8adaff034-dshm" (OuterVolumeSpecName: "dshm") pod "a5011121-3390-4619-af40-04e8adaff034" (UID: "a5011121-3390-4619-af40-04e8adaff034"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:46:12.836576 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:46:12.836489    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5011121-3390-4619-af40-04e8adaff034-kube-api-access-5lgfq" (OuterVolumeSpecName: "kube-api-access-5lgfq") pod "a5011121-3390-4619-af40-04e8adaff034" (UID: "a5011121-3390-4619-af40-04e8adaff034"). InnerVolumeSpecName "kube-api-access-5lgfq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 03:46:12.935292 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:46:12.935259    2552 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5011121-3390-4619-af40-04e8adaff034-model-cache\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:46:12.935292 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:46:12.935285    2552 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5011121-3390-4619-af40-04e8adaff034-tls-certs\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:46:12.935292 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:46:12.935293    2552 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5011121-3390-4619-af40-04e8adaff034-dshm\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:46:12.935510 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:46:12.935303    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5lgfq\" (UniqueName: \"kubernetes.io/projected/a5011121-3390-4619-af40-04e8adaff034-kube-api-access-5lgfq\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:46:12.935510 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:46:12.935314    2552 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5011121-3390-4619-af40-04e8adaff034-tmp-dir\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:46:13.694933 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:46:13.694896    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-f6d581/auth-test-model-f6d581-kserve-5dbd89f68c-lgmzg" event={"ID":"a5011121-3390-4619-af40-04e8adaff034","Type":"ContainerDied","Data":"f07551cd642859be4f85ae3ef6943305a986388567c904bc34b61b85070ad65c"}
Sep 17 03:46:13.695421 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:46:13.694952    2552 scope.go:117] "RemoveContainer" containerID="43acabaf2872f631b2a35612ecbf343ded78bf421c071e8c9ef932212a4c060f"
Sep 17 03:46:13.695421 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:46:13.694910    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-f6d581/auth-test-model-f6d581-kserve-5dbd89f68c-lgmzg"
Sep 17 03:46:13.711414 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:46:13.711390    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-f6d581/auth-test-model-f6d581-kserve-5dbd89f68c-lgmzg"]
Sep 17 03:46:13.715575 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:46:13.715551    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-f6d581/auth-test-model-f6d581-kserve-5dbd89f68c-lgmzg"]
Sep 17 03:46:14.985574 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:46:14.985534    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a5011121-3390-4619-af40-04e8adaff034" path="/var/lib/kubelet/pods/a5011121-3390-4619-af40-04e8adaff034/volumes"
Sep 17 03:46:15.528663 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:46:15.528621    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-22e045/sub-test-model-22e045-kserve-c6645c55f-6t7mz"
Sep 17 03:46:15.528848 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:46:15.528677    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-22e045/sub-test-model-22e045-kserve-c6645c55f-6t7mz"
Sep 17 03:46:15.541507 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:46:15.541477    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-22e045/sub-test-model-22e045-kserve-c6645c55f-6t7mz"
Sep 17 03:46:15.723203 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:46:15.723174    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-22e045/sub-test-model-22e045-kserve-c6645c55f-6t7mz"
Sep 17 03:46:45.987129 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:46:45.987097    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-f6d581/sub-test-model-f6d581-kserve-7f48c5bb9f-hmqk2"]
Sep 17 03:46:45.987580 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:46:45.987506    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5011121-3390-4619-af40-04e8adaff034" containerName="main"
Sep 17 03:46:45.987580 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:46:45.987518    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5011121-3390-4619-af40-04e8adaff034" containerName="main"
Sep 17 03:46:45.987676 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:46:45.987611    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="a5011121-3390-4619-af40-04e8adaff034" containerName="main"
Sep 17 03:46:45.993588 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:46:45.993568    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-f6d581/sub-test-model-f6d581-kserve-7f48c5bb9f-hmqk2"
Sep 17 03:46:45.996644 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:46:45.996623    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-f6d581\"/\"default-dockercfg-msjdf\""
Sep 17 03:46:45.997828 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:46:45.997806    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-f6d581\"/\"openshift-service-ca.crt\""
Sep 17 03:46:45.997958 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:46:45.997807    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-f6d581\"/\"kube-root-ca.crt\""
Sep 17 03:46:46.002868 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:46:46.002852    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-f6d581\"/\"sub-test-model-f6d581-kserve-self-signed-certs\""
Sep 17 03:46:46.009996 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:46:46.009973    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-f6d581/sub-test-model-f6d581-kserve-7f48c5bb9f-hmqk2"]
Sep 17 03:46:46.055237 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:46:46.055205    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44ffe51e-c929-45a6-9165-7e6072dfcd76-model-cache\") pod \"sub-test-model-f6d581-kserve-7f48c5bb9f-hmqk2\" (UID: \"44ffe51e-c929-45a6-9165-7e6072dfcd76\") " pod="ai-tenant-e2e-shared-b-w3-f6d581/sub-test-model-f6d581-kserve-7f48c5bb9f-hmqk2"
Sep 17 03:46:46.055237 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:46:46.055241    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44ffe51e-c929-45a6-9165-7e6072dfcd76-tmp-dir\") pod \"sub-test-model-f6d581-kserve-7f48c5bb9f-hmqk2\" (UID: \"44ffe51e-c929-45a6-9165-7e6072dfcd76\") " pod="ai-tenant-e2e-shared-b-w3-f6d581/sub-test-model-f6d581-kserve-7f48c5bb9f-hmqk2"
Sep 17 03:46:46.055436 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:46:46.055329    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44ffe51e-c929-45a6-9165-7e6072dfcd76-dshm\") pod \"sub-test-model-f6d581-kserve-7f48c5bb9f-hmqk2\" (UID: \"44ffe51e-c929-45a6-9165-7e6072dfcd76\") " pod="ai-tenant-e2e-shared-b-w3-f6d581/sub-test-model-f6d581-kserve-7f48c5bb9f-hmqk2"
Sep 17 03:46:46.055436 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:46:46.055353    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44ffe51e-c929-45a6-9165-7e6072dfcd76-home\") pod \"sub-test-model-f6d581-kserve-7f48c5bb9f-hmqk2\" (UID: \"44ffe51e-c929-45a6-9165-7e6072dfcd76\") " pod="ai-tenant-e2e-shared-b-w3-f6d581/sub-test-model-f6d581-kserve-7f48c5bb9f-hmqk2"
Sep 17 03:46:46.055436 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:46:46.055392    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44ffe51e-c929-45a6-9165-7e6072dfcd76-tls-certs\") pod \"sub-test-model-f6d581-kserve-7f48c5bb9f-hmqk2\" (UID: \"44ffe51e-c929-45a6-9165-7e6072dfcd76\") " pod="ai-tenant-e2e-shared-b-w3-f6d581/sub-test-model-f6d581-kserve-7f48c5bb9f-hmqk2"
Sep 17 03:46:46.055620 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:46:46.055454    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9rfq\" (UniqueName: \"kubernetes.io/projected/44ffe51e-c929-45a6-9165-7e6072dfcd76-kube-api-access-l9rfq\") pod \"sub-test-model-f6d581-kserve-7f48c5bb9f-hmqk2\" (UID: \"44ffe51e-c929-45a6-9165-7e6072dfcd76\") " pod="ai-tenant-e2e-shared-b-w3-f6d581/sub-test-model-f6d581-kserve-7f48c5bb9f-hmqk2"
Sep 17 03:46:46.156125 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:46:46.156086    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44ffe51e-c929-45a6-9165-7e6072dfcd76-model-cache\") pod \"sub-test-model-f6d581-kserve-7f48c5bb9f-hmqk2\" (UID: \"44ffe51e-c929-45a6-9165-7e6072dfcd76\") " pod="ai-tenant-e2e-shared-b-w3-f6d581/sub-test-model-f6d581-kserve-7f48c5bb9f-hmqk2"
Sep 17 03:46:46.156313 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:46:46.156142    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44ffe51e-c929-45a6-9165-7e6072dfcd76-tmp-dir\") pod \"sub-test-model-f6d581-kserve-7f48c5bb9f-hmqk2\" (UID: \"44ffe51e-c929-45a6-9165-7e6072dfcd76\") " pod="ai-tenant-e2e-shared-b-w3-f6d581/sub-test-model-f6d581-kserve-7f48c5bb9f-hmqk2"
Sep 17 03:46:46.156313 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:46:46.156214    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44ffe51e-c929-45a6-9165-7e6072dfcd76-dshm\") pod \"sub-test-model-f6d581-kserve-7f48c5bb9f-hmqk2\" (UID: \"44ffe51e-c929-45a6-9165-7e6072dfcd76\") " pod="ai-tenant-e2e-shared-b-w3-f6d581/sub-test-model-f6d581-kserve-7f48c5bb9f-hmqk2"
Sep 17 03:46:46.156313 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:46:46.156240    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44ffe51e-c929-45a6-9165-7e6072dfcd76-home\") pod \"sub-test-model-f6d581-kserve-7f48c5bb9f-hmqk2\" (UID: \"44ffe51e-c929-45a6-9165-7e6072dfcd76\") " pod="ai-tenant-e2e-shared-b-w3-f6d581/sub-test-model-f6d581-kserve-7f48c5bb9f-hmqk2"
Sep 17 03:46:46.156313 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:46:46.156300    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44ffe51e-c929-45a6-9165-7e6072dfcd76-tls-certs\") pod \"sub-test-model-f6d581-kserve-7f48c5bb9f-hmqk2\" (UID: \"44ffe51e-c929-45a6-9165-7e6072dfcd76\") " pod="ai-tenant-e2e-shared-b-w3-f6d581/sub-test-model-f6d581-kserve-7f48c5bb9f-hmqk2"
Sep 17 03:46:46.156537 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:46:46.156332    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9rfq\" (UniqueName: \"kubernetes.io/projected/44ffe51e-c929-45a6-9165-7e6072dfcd76-kube-api-access-l9rfq\") pod \"sub-test-model-f6d581-kserve-7f48c5bb9f-hmqk2\" (UID: \"44ffe51e-c929-45a6-9165-7e6072dfcd76\") " pod="ai-tenant-e2e-shared-b-w3-f6d581/sub-test-model-f6d581-kserve-7f48c5bb9f-hmqk2"
Sep 17 03:46:46.156638 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:46:46.156579    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44ffe51e-c929-45a6-9165-7e6072dfcd76-model-cache\") pod \"sub-test-model-f6d581-kserve-7f48c5bb9f-hmqk2\" (UID: \"44ffe51e-c929-45a6-9165-7e6072dfcd76\") " pod="ai-tenant-e2e-shared-b-w3-f6d581/sub-test-model-f6d581-kserve-7f48c5bb9f-hmqk2"
Sep 17 03:46:46.156738 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:46:46.156706    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44ffe51e-c929-45a6-9165-7e6072dfcd76-tmp-dir\") pod \"sub-test-model-f6d581-kserve-7f48c5bb9f-hmqk2\" (UID: \"44ffe51e-c929-45a6-9165-7e6072dfcd76\") " pod="ai-tenant-e2e-shared-b-w3-f6d581/sub-test-model-f6d581-kserve-7f48c5bb9f-hmqk2"
Sep 17 03:46:46.157201 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:46:46.157179    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44ffe51e-c929-45a6-9165-7e6072dfcd76-home\") pod \"sub-test-model-f6d581-kserve-7f48c5bb9f-hmqk2\" (UID: \"44ffe51e-c929-45a6-9165-7e6072dfcd76\") " pod="ai-tenant-e2e-shared-b-w3-f6d581/sub-test-model-f6d581-kserve-7f48c5bb9f-hmqk2"
Sep 17 03:46:46.158948 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:46:46.158926    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44ffe51e-c929-45a6-9165-7e6072dfcd76-dshm\") pod \"sub-test-model-f6d581-kserve-7f48c5bb9f-hmqk2\" (UID: \"44ffe51e-c929-45a6-9165-7e6072dfcd76\") " pod="ai-tenant-e2e-shared-b-w3-f6d581/sub-test-model-f6d581-kserve-7f48c5bb9f-hmqk2"
Sep 17 03:46:46.159178 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:46:46.159157    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44ffe51e-c929-45a6-9165-7e6072dfcd76-tls-certs\") pod \"sub-test-model-f6d581-kserve-7f48c5bb9f-hmqk2\" (UID: \"44ffe51e-c929-45a6-9165-7e6072dfcd76\") " pod="ai-tenant-e2e-shared-b-w3-f6d581/sub-test-model-f6d581-kserve-7f48c5bb9f-hmqk2"
Sep 17 03:46:46.173353 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:46:46.173325    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9rfq\" (UniqueName: \"kubernetes.io/projected/44ffe51e-c929-45a6-9165-7e6072dfcd76-kube-api-access-l9rfq\") pod \"sub-test-model-f6d581-kserve-7f48c5bb9f-hmqk2\" (UID: \"44ffe51e-c929-45a6-9165-7e6072dfcd76\") " pod="ai-tenant-e2e-shared-b-w3-f6d581/sub-test-model-f6d581-kserve-7f48c5bb9f-hmqk2"
Sep 17 03:46:46.303553 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:46:46.303522    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-f6d581/sub-test-model-f6d581-kserve-7f48c5bb9f-hmqk2"
Sep 17 03:46:46.575905 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:46:46.575754    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-f6d581/sub-test-model-f6d581-kserve-7f48c5bb9f-hmqk2"]
Sep 17 03:46:46.578278 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:46:46.578252    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod44ffe51e_c929_45a6_9165_7e6072dfcd76.slice/crio-4d13830a76737ef02f830f3dbb5ba2420714fb6ee321522a53390f9abe830295 WatchSource:0}: Error finding container 4d13830a76737ef02f830f3dbb5ba2420714fb6ee321522a53390f9abe830295: Status 404 returned error can't find the container with id 4d13830a76737ef02f830f3dbb5ba2420714fb6ee321522a53390f9abe830295
Sep 17 03:46:46.887017 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:46:46.886923    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-f6d581/sub-test-model-f6d581-kserve-7f48c5bb9f-hmqk2" event={"ID":"44ffe51e-c929-45a6-9165-7e6072dfcd76","Type":"ContainerStarted","Data":"f781bcc0a02b6121e8a15aedc6cd386cb7afa2030ecf04e1de07ba5363ab8507"}
Sep 17 03:46:46.887017 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:46:46.886963    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-f6d581/sub-test-model-f6d581-kserve-7f48c5bb9f-hmqk2" event={"ID":"44ffe51e-c929-45a6-9165-7e6072dfcd76","Type":"ContainerStarted","Data":"4d13830a76737ef02f830f3dbb5ba2420714fb6ee321522a53390f9abe830295"}
Sep 17 03:46:46.907495 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:46:46.907439    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-f6d581/sub-test-model-f6d581-kserve-7f48c5bb9f-hmqk2" podStartSLOduration=1.9074247199999999 podStartE2EDuration="1.90742472s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 03:46:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:46:46.905675932 +0000 UTC m=+1642.372083460" watchObservedRunningTime="2026-09-17 03:46:46.90742472 +0000 UTC m=+1642.373832246"
Sep 17 03:46:56.304343 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:46:56.304309    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-f6d581/sub-test-model-f6d581-kserve-7f48c5bb9f-hmqk2"
Sep 17 03:46:56.304792 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:46:56.304567    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-f6d581/sub-test-model-f6d581-kserve-7f48c5bb9f-hmqk2"
Sep 17 03:46:56.317200 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:46:56.317174    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-f6d581/sub-test-model-f6d581-kserve-7f48c5bb9f-hmqk2"
Sep 17 03:46:56.957487 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:46:56.957458    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-f6d581/sub-test-model-f6d581-kserve-7f48c5bb9f-hmqk2"
Sep 17 03:47:25.460126 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:47:25.460095    2552 scope.go:117] "RemoveContainer" containerID="9403b934d5827143cacd81df24dffd16327fa1990b2a8de2ab4cdc466096ac9a"
Sep 17 03:47:51.362443 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:47:51.362363    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-22e045/sub-test-model-22e045-kserve-c6645c55f-6t7mz"]
Sep 17 03:47:52.203846 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:47:52.203812    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-f6d581/sub-test-model-f6d581-kserve-7f48c5bb9f-hmqk2"]
Sep 17 03:48:06.381842 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:48:06.381767    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-22e045/sub-test-model-22e045-kserve-c6645c55f-6t7mz" podUID="91cec756-29a9-4e15-a7f0-fbb88cb7fa88" containerName="main" containerID="cri-o://9e6938b3225295417d4d7c6951a38a2d9cb7eac1e8a0254f0b2834739b270c34" gracePeriod=45
Sep 17 03:48:06.628933 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:48:06.628906    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-22e045/sub-test-model-22e045-kserve-c6645c55f-6t7mz"
Sep 17 03:48:06.640095 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:48:06.640012    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91cec756-29a9-4e15-a7f0-fbb88cb7fa88-tmp-dir\") pod \"91cec756-29a9-4e15-a7f0-fbb88cb7fa88\" (UID: \"91cec756-29a9-4e15-a7f0-fbb88cb7fa88\") "
Sep 17 03:48:06.640095 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:48:06.640077    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-66s6w\" (UniqueName: \"kubernetes.io/projected/91cec756-29a9-4e15-a7f0-fbb88cb7fa88-kube-api-access-66s6w\") pod \"91cec756-29a9-4e15-a7f0-fbb88cb7fa88\" (UID: \"91cec756-29a9-4e15-a7f0-fbb88cb7fa88\") "
Sep 17 03:48:06.640297 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:48:06.640106    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91cec756-29a9-4e15-a7f0-fbb88cb7fa88-home\") pod \"91cec756-29a9-4e15-a7f0-fbb88cb7fa88\" (UID: \"91cec756-29a9-4e15-a7f0-fbb88cb7fa88\") "
Sep 17 03:48:06.640297 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:48:06.640161    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91cec756-29a9-4e15-a7f0-fbb88cb7fa88-dshm\") pod \"91cec756-29a9-4e15-a7f0-fbb88cb7fa88\" (UID: \"91cec756-29a9-4e15-a7f0-fbb88cb7fa88\") "
Sep 17 03:48:06.640297 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:48:06.640199    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91cec756-29a9-4e15-a7f0-fbb88cb7fa88-tls-certs\") pod \"91cec756-29a9-4e15-a7f0-fbb88cb7fa88\" (UID: \"91cec756-29a9-4e15-a7f0-fbb88cb7fa88\") "
Sep 17 03:48:06.640297 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:48:06.640283    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91cec756-29a9-4e15-a7f0-fbb88cb7fa88-model-cache\") pod \"91cec756-29a9-4e15-a7f0-fbb88cb7fa88\" (UID: \"91cec756-29a9-4e15-a7f0-fbb88cb7fa88\") "
Sep 17 03:48:06.640502 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:48:06.640315    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91cec756-29a9-4e15-a7f0-fbb88cb7fa88-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "91cec756-29a9-4e15-a7f0-fbb88cb7fa88" (UID: "91cec756-29a9-4e15-a7f0-fbb88cb7fa88"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:48:06.640502 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:48:06.640350    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91cec756-29a9-4e15-a7f0-fbb88cb7fa88-home" (OuterVolumeSpecName: "home") pod "91cec756-29a9-4e15-a7f0-fbb88cb7fa88" (UID: "91cec756-29a9-4e15-a7f0-fbb88cb7fa88"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:48:06.640630 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:48:06.640567    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91cec756-29a9-4e15-a7f0-fbb88cb7fa88-model-cache" (OuterVolumeSpecName: "model-cache") pod "91cec756-29a9-4e15-a7f0-fbb88cb7fa88" (UID: "91cec756-29a9-4e15-a7f0-fbb88cb7fa88"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:48:06.640691 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:48:06.640653    2552 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91cec756-29a9-4e15-a7f0-fbb88cb7fa88-home\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:48:06.640691 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:48:06.640671    2552 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91cec756-29a9-4e15-a7f0-fbb88cb7fa88-tmp-dir\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:48:06.642341 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:48:06.642313    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91cec756-29a9-4e15-a7f0-fbb88cb7fa88-dshm" (OuterVolumeSpecName: "dshm") pod "91cec756-29a9-4e15-a7f0-fbb88cb7fa88" (UID: "91cec756-29a9-4e15-a7f0-fbb88cb7fa88"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:48:06.642501 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:48:06.642468    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91cec756-29a9-4e15-a7f0-fbb88cb7fa88-kube-api-access-66s6w" (OuterVolumeSpecName: "kube-api-access-66s6w") pod "91cec756-29a9-4e15-a7f0-fbb88cb7fa88" (UID: "91cec756-29a9-4e15-a7f0-fbb88cb7fa88"). InnerVolumeSpecName "kube-api-access-66s6w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 03:48:06.642719 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:48:06.642701    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91cec756-29a9-4e15-a7f0-fbb88cb7fa88-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "91cec756-29a9-4e15-a7f0-fbb88cb7fa88" (UID: "91cec756-29a9-4e15-a7f0-fbb88cb7fa88"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 03:48:06.741327 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:48:06.741298    2552 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91cec756-29a9-4e15-a7f0-fbb88cb7fa88-dshm\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:48:06.741327 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:48:06.741321    2552 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91cec756-29a9-4e15-a7f0-fbb88cb7fa88-tls-certs\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:48:06.741327 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:48:06.741330    2552 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91cec756-29a9-4e15-a7f0-fbb88cb7fa88-model-cache\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:48:06.741568 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:48:06.741339    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-66s6w\" (UniqueName: \"kubernetes.io/projected/91cec756-29a9-4e15-a7f0-fbb88cb7fa88-kube-api-access-66s6w\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:48:07.223430 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:48:07.223386    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-f6d581/sub-test-model-f6d581-kserve-7f48c5bb9f-hmqk2" podUID="44ffe51e-c929-45a6-9165-7e6072dfcd76" containerName="main" containerID="cri-o://f781bcc0a02b6121e8a15aedc6cd386cb7afa2030ecf04e1de07ba5363ab8507" gracePeriod=45
Sep 17 03:48:07.329098 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:48:07.329062    2552 generic.go:356] "Generic (PLEG): container finished" podID="91cec756-29a9-4e15-a7f0-fbb88cb7fa88" containerID="9e6938b3225295417d4d7c6951a38a2d9cb7eac1e8a0254f0b2834739b270c34" exitCode=0
Sep 17 03:48:07.329289 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:48:07.329126    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-22e045/sub-test-model-22e045-kserve-c6645c55f-6t7mz"
Sep 17 03:48:07.329289 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:48:07.329138    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-22e045/sub-test-model-22e045-kserve-c6645c55f-6t7mz" event={"ID":"91cec756-29a9-4e15-a7f0-fbb88cb7fa88","Type":"ContainerDied","Data":"9e6938b3225295417d4d7c6951a38a2d9cb7eac1e8a0254f0b2834739b270c34"}
Sep 17 03:48:07.329289 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:48:07.329180    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-22e045/sub-test-model-22e045-kserve-c6645c55f-6t7mz" event={"ID":"91cec756-29a9-4e15-a7f0-fbb88cb7fa88","Type":"ContainerDied","Data":"e0ecf8edba32c63fe871df6406f0aba2b990bedd86bd68750f90fc4200eff274"}
Sep 17 03:48:07.329289 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:48:07.329200    2552 scope.go:117] "RemoveContainer" containerID="9e6938b3225295417d4d7c6951a38a2d9cb7eac1e8a0254f0b2834739b270c34"
Sep 17 03:48:07.337793 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:48:07.337760    2552 scope.go:117] "RemoveContainer" containerID="9e6938b3225295417d4d7c6951a38a2d9cb7eac1e8a0254f0b2834739b270c34"
Sep 17 03:48:07.338141 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:48:07.338110    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e6938b3225295417d4d7c6951a38a2d9cb7eac1e8a0254f0b2834739b270c34\": container with ID starting with 9e6938b3225295417d4d7c6951a38a2d9cb7eac1e8a0254f0b2834739b270c34 not found: ID does not exist" containerID="9e6938b3225295417d4d7c6951a38a2d9cb7eac1e8a0254f0b2834739b270c34"
Sep 17 03:48:07.338283 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:48:07.338149    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e6938b3225295417d4d7c6951a38a2d9cb7eac1e8a0254f0b2834739b270c34"} err="failed to get container status \"9e6938b3225295417d4d7c6951a38a2d9cb7eac1e8a0254f0b2834739b270c34\": rpc error: code = NotFound desc = could not find container \"9e6938b3225295417d4d7c6951a38a2d9cb7eac1e8a0254f0b2834739b270c34\": container with ID starting with 9e6938b3225295417d4d7c6951a38a2d9cb7eac1e8a0254f0b2834739b270c34 not found: ID does not exist"
Sep 17 03:48:07.347480 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:48:07.347455    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-22e045/sub-test-model-22e045-kserve-c6645c55f-6t7mz"]
Sep 17 03:48:07.353948 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:48:07.353925    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-22e045/sub-test-model-22e045-kserve-c6645c55f-6t7mz"]
Sep 17 03:48:07.470834 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:48:07.470807    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-f6d581/sub-test-model-f6d581-kserve-7f48c5bb9f-hmqk2"
Sep 17 03:48:07.549270 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:48:07.549242    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l9rfq\" (UniqueName: \"kubernetes.io/projected/44ffe51e-c929-45a6-9165-7e6072dfcd76-kube-api-access-l9rfq\") pod \"44ffe51e-c929-45a6-9165-7e6072dfcd76\" (UID: \"44ffe51e-c929-45a6-9165-7e6072dfcd76\") "
Sep 17 03:48:07.549437 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:48:07.549299    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44ffe51e-c929-45a6-9165-7e6072dfcd76-home\") pod \"44ffe51e-c929-45a6-9165-7e6072dfcd76\" (UID: \"44ffe51e-c929-45a6-9165-7e6072dfcd76\") "
Sep 17 03:48:07.549437 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:48:07.549323    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44ffe51e-c929-45a6-9165-7e6072dfcd76-tmp-dir\") pod \"44ffe51e-c929-45a6-9165-7e6072dfcd76\" (UID: \"44ffe51e-c929-45a6-9165-7e6072dfcd76\") "
Sep 17 03:48:07.549437 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:48:07.549345    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44ffe51e-c929-45a6-9165-7e6072dfcd76-dshm\") pod \"44ffe51e-c929-45a6-9165-7e6072dfcd76\" (UID: \"44ffe51e-c929-45a6-9165-7e6072dfcd76\") "
Sep 17 03:48:07.549437 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:48:07.549374    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44ffe51e-c929-45a6-9165-7e6072dfcd76-model-cache\") pod \"44ffe51e-c929-45a6-9165-7e6072dfcd76\" (UID: \"44ffe51e-c929-45a6-9165-7e6072dfcd76\") "
Sep 17 03:48:07.549437 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:48:07.549420    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44ffe51e-c929-45a6-9165-7e6072dfcd76-tls-certs\") pod \"44ffe51e-c929-45a6-9165-7e6072dfcd76\" (UID: \"44ffe51e-c929-45a6-9165-7e6072dfcd76\") "
Sep 17 03:48:07.549746 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:48:07.549578    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44ffe51e-c929-45a6-9165-7e6072dfcd76-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "44ffe51e-c929-45a6-9165-7e6072dfcd76" (UID: "44ffe51e-c929-45a6-9165-7e6072dfcd76"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:48:07.549746 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:48:07.549709    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44ffe51e-c929-45a6-9165-7e6072dfcd76-home" (OuterVolumeSpecName: "home") pod "44ffe51e-c929-45a6-9165-7e6072dfcd76" (UID: "44ffe51e-c929-45a6-9165-7e6072dfcd76"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:48:07.549746 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:48:07.549728    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44ffe51e-c929-45a6-9165-7e6072dfcd76-model-cache" (OuterVolumeSpecName: "model-cache") pod "44ffe51e-c929-45a6-9165-7e6072dfcd76" (UID: "44ffe51e-c929-45a6-9165-7e6072dfcd76"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:48:07.549893 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:48:07.549800    2552 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44ffe51e-c929-45a6-9165-7e6072dfcd76-home\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:48:07.549893 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:48:07.549844    2552 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44ffe51e-c929-45a6-9165-7e6072dfcd76-tmp-dir\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:48:07.551526 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:48:07.551496    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44ffe51e-c929-45a6-9165-7e6072dfcd76-dshm" (OuterVolumeSpecName: "dshm") pod "44ffe51e-c929-45a6-9165-7e6072dfcd76" (UID: "44ffe51e-c929-45a6-9165-7e6072dfcd76"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:48:07.551677 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:48:07.551537    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44ffe51e-c929-45a6-9165-7e6072dfcd76-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "44ffe51e-c929-45a6-9165-7e6072dfcd76" (UID: "44ffe51e-c929-45a6-9165-7e6072dfcd76"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 03:48:07.551677 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:48:07.551563    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44ffe51e-c929-45a6-9165-7e6072dfcd76-kube-api-access-l9rfq" (OuterVolumeSpecName: "kube-api-access-l9rfq") pod "44ffe51e-c929-45a6-9165-7e6072dfcd76" (UID: "44ffe51e-c929-45a6-9165-7e6072dfcd76"). InnerVolumeSpecName "kube-api-access-l9rfq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 03:48:07.651301 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:48:07.651213    2552 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44ffe51e-c929-45a6-9165-7e6072dfcd76-tls-certs\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:48:07.651301 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:48:07.651244    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l9rfq\" (UniqueName: \"kubernetes.io/projected/44ffe51e-c929-45a6-9165-7e6072dfcd76-kube-api-access-l9rfq\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:48:07.651301 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:48:07.651254    2552 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44ffe51e-c929-45a6-9165-7e6072dfcd76-dshm\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:48:07.651301 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:48:07.651262    2552 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44ffe51e-c929-45a6-9165-7e6072dfcd76-model-cache\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:48:08.336233 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:48:08.336200    2552 generic.go:356] "Generic (PLEG): container finished" podID="44ffe51e-c929-45a6-9165-7e6072dfcd76" containerID="f781bcc0a02b6121e8a15aedc6cd386cb7afa2030ecf04e1de07ba5363ab8507" exitCode=0
Sep 17 03:48:08.336427 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:48:08.336249    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-f6d581/sub-test-model-f6d581-kserve-7f48c5bb9f-hmqk2"
Sep 17 03:48:08.336427 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:48:08.336296    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-f6d581/sub-test-model-f6d581-kserve-7f48c5bb9f-hmqk2" event={"ID":"44ffe51e-c929-45a6-9165-7e6072dfcd76","Type":"ContainerDied","Data":"f781bcc0a02b6121e8a15aedc6cd386cb7afa2030ecf04e1de07ba5363ab8507"}
Sep 17 03:48:08.336427 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:48:08.336333    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-f6d581/sub-test-model-f6d581-kserve-7f48c5bb9f-hmqk2" event={"ID":"44ffe51e-c929-45a6-9165-7e6072dfcd76","Type":"ContainerDied","Data":"4d13830a76737ef02f830f3dbb5ba2420714fb6ee321522a53390f9abe830295"}
Sep 17 03:48:08.336427 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:48:08.336355    2552 scope.go:117] "RemoveContainer" containerID="f781bcc0a02b6121e8a15aedc6cd386cb7afa2030ecf04e1de07ba5363ab8507"
Sep 17 03:48:08.345430 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:48:08.345414    2552 scope.go:117] "RemoveContainer" containerID="f781bcc0a02b6121e8a15aedc6cd386cb7afa2030ecf04e1de07ba5363ab8507"
Sep 17 03:48:08.345730 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:48:08.345712    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f781bcc0a02b6121e8a15aedc6cd386cb7afa2030ecf04e1de07ba5363ab8507\": container with ID starting with f781bcc0a02b6121e8a15aedc6cd386cb7afa2030ecf04e1de07ba5363ab8507 not found: ID does not exist" containerID="f781bcc0a02b6121e8a15aedc6cd386cb7afa2030ecf04e1de07ba5363ab8507"
Sep 17 03:48:08.345796 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:48:08.345738    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f781bcc0a02b6121e8a15aedc6cd386cb7afa2030ecf04e1de07ba5363ab8507"} err="failed to get container status \"f781bcc0a02b6121e8a15aedc6cd386cb7afa2030ecf04e1de07ba5363ab8507\": rpc error: code = NotFound desc = could not find container \"f781bcc0a02b6121e8a15aedc6cd386cb7afa2030ecf04e1de07ba5363ab8507\": container with ID starting with f781bcc0a02b6121e8a15aedc6cd386cb7afa2030ecf04e1de07ba5363ab8507 not found: ID does not exist"
Sep 17 03:48:08.357471 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:48:08.357325    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-f6d581/sub-test-model-f6d581-kserve-7f48c5bb9f-hmqk2"]
Sep 17 03:48:08.359453 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:48:08.359432    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-f6d581/sub-test-model-f6d581-kserve-7f48c5bb9f-hmqk2"]
Sep 17 03:48:08.986293 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:48:08.986248    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="44ffe51e-c929-45a6-9165-7e6072dfcd76" path="/var/lib/kubelet/pods/44ffe51e-c929-45a6-9165-7e6072dfcd76/volumes"
Sep 17 03:48:08.986861 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:48:08.986839    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="91cec756-29a9-4e15-a7f0-fbb88cb7fa88" path="/var/lib/kubelet/pods/91cec756-29a9-4e15-a7f0-fbb88cb7fa88/volumes"
Sep 17 03:48:32.795579 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:48:32.795547    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-f6d581/rate-test-model-f6d581-kserve-68f4ddb45f-cgxkp"]
Sep 17 03:48:32.795961 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:48:32.795944    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91cec756-29a9-4e15-a7f0-fbb88cb7fa88" containerName="main"
Sep 17 03:48:32.795961 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:48:32.795956    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="91cec756-29a9-4e15-a7f0-fbb88cb7fa88" containerName="main"
Sep 17 03:48:32.796036 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:48:32.795967    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44ffe51e-c929-45a6-9165-7e6072dfcd76" containerName="main"
Sep 17 03:48:32.796036 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:48:32.795973    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="44ffe51e-c929-45a6-9165-7e6072dfcd76" containerName="main"
Sep 17 03:48:32.796098 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:48:32.796063    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="91cec756-29a9-4e15-a7f0-fbb88cb7fa88" containerName="main"
Sep 17 03:48:32.796098 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:48:32.796074    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="44ffe51e-c929-45a6-9165-7e6072dfcd76" containerName="main"
Sep 17 03:48:32.802620 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:48:32.802580    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-f6d581/rate-test-model-f6d581-kserve-68f4ddb45f-cgxkp"
Sep 17 03:48:32.805479 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:48:32.805463    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-f6d581\"/\"kube-root-ca.crt\""
Sep 17 03:48:32.805852 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:48:32.805836    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-f6d581\"/\"openshift-service-ca.crt\""
Sep 17 03:48:32.806997 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:48:32.806978    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-f6d581\"/\"rate-test-model-f6d581-kserve-self-signed-certs\""
Sep 17 03:48:32.807058 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:48:32.807040    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-f6d581\"/\"default-dockercfg-msjdf\""
Sep 17 03:48:32.814775 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:48:32.814749    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-f6d581/rate-test-model-f6d581-kserve-68f4ddb45f-cgxkp"]
Sep 17 03:48:32.883798 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:48:32.883765    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4150b67b-7f33-435e-ac25-6d6557244278-model-cache\") pod \"rate-test-model-f6d581-kserve-68f4ddb45f-cgxkp\" (UID: \"4150b67b-7f33-435e-ac25-6d6557244278\") " pod="ai-tenant-e2e-shared-b-w3-f6d581/rate-test-model-f6d581-kserve-68f4ddb45f-cgxkp"
Sep 17 03:48:32.883798 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:48:32.883800    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4150b67b-7f33-435e-ac25-6d6557244278-tls-certs\") pod \"rate-test-model-f6d581-kserve-68f4ddb45f-cgxkp\" (UID: \"4150b67b-7f33-435e-ac25-6d6557244278\") " pod="ai-tenant-e2e-shared-b-w3-f6d581/rate-test-model-f6d581-kserve-68f4ddb45f-cgxkp"
Sep 17 03:48:32.884000 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:48:32.883873    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqpfs\" (UniqueName: \"kubernetes.io/projected/4150b67b-7f33-435e-ac25-6d6557244278-kube-api-access-cqpfs\") pod \"rate-test-model-f6d581-kserve-68f4ddb45f-cgxkp\" (UID: \"4150b67b-7f33-435e-ac25-6d6557244278\") " pod="ai-tenant-e2e-shared-b-w3-f6d581/rate-test-model-f6d581-kserve-68f4ddb45f-cgxkp"
Sep 17 03:48:32.884000 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:48:32.883895    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4150b67b-7f33-435e-ac25-6d6557244278-home\") pod \"rate-test-model-f6d581-kserve-68f4ddb45f-cgxkp\" (UID: \"4150b67b-7f33-435e-ac25-6d6557244278\") " pod="ai-tenant-e2e-shared-b-w3-f6d581/rate-test-model-f6d581-kserve-68f4ddb45f-cgxkp"
Sep 17 03:48:32.884000 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:48:32.883948    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4150b67b-7f33-435e-ac25-6d6557244278-tmp-dir\") pod \"rate-test-model-f6d581-kserve-68f4ddb45f-cgxkp\" (UID: \"4150b67b-7f33-435e-ac25-6d6557244278\") " pod="ai-tenant-e2e-shared-b-w3-f6d581/rate-test-model-f6d581-kserve-68f4ddb45f-cgxkp"
Sep 17 03:48:32.884000 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:48:32.883986    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4150b67b-7f33-435e-ac25-6d6557244278-dshm\") pod \"rate-test-model-f6d581-kserve-68f4ddb45f-cgxkp\" (UID: \"4150b67b-7f33-435e-ac25-6d6557244278\") " pod="ai-tenant-e2e-shared-b-w3-f6d581/rate-test-model-f6d581-kserve-68f4ddb45f-cgxkp"
Sep 17 03:48:32.984431 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:48:32.984398    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqpfs\" (UniqueName: \"kubernetes.io/projected/4150b67b-7f33-435e-ac25-6d6557244278-kube-api-access-cqpfs\") pod \"rate-test-model-f6d581-kserve-68f4ddb45f-cgxkp\" (UID: \"4150b67b-7f33-435e-ac25-6d6557244278\") " pod="ai-tenant-e2e-shared-b-w3-f6d581/rate-test-model-f6d581-kserve-68f4ddb45f-cgxkp"
Sep 17 03:48:32.984431 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:48:32.984434    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4150b67b-7f33-435e-ac25-6d6557244278-home\") pod \"rate-test-model-f6d581-kserve-68f4ddb45f-cgxkp\" (UID: \"4150b67b-7f33-435e-ac25-6d6557244278\") " pod="ai-tenant-e2e-shared-b-w3-f6d581/rate-test-model-f6d581-kserve-68f4ddb45f-cgxkp"
Sep 17 03:48:32.984821 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:48:32.984475    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4150b67b-7f33-435e-ac25-6d6557244278-tmp-dir\") pod \"rate-test-model-f6d581-kserve-68f4ddb45f-cgxkp\" (UID: \"4150b67b-7f33-435e-ac25-6d6557244278\") " pod="ai-tenant-e2e-shared-b-w3-f6d581/rate-test-model-f6d581-kserve-68f4ddb45f-cgxkp"
Sep 17 03:48:32.984821 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:48:32.984503    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4150b67b-7f33-435e-ac25-6d6557244278-dshm\") pod \"rate-test-model-f6d581-kserve-68f4ddb45f-cgxkp\" (UID: \"4150b67b-7f33-435e-ac25-6d6557244278\") " pod="ai-tenant-e2e-shared-b-w3-f6d581/rate-test-model-f6d581-kserve-68f4ddb45f-cgxkp"
Sep 17 03:48:32.984821 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:48:32.984551    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4150b67b-7f33-435e-ac25-6d6557244278-model-cache\") pod \"rate-test-model-f6d581-kserve-68f4ddb45f-cgxkp\" (UID: \"4150b67b-7f33-435e-ac25-6d6557244278\") " pod="ai-tenant-e2e-shared-b-w3-f6d581/rate-test-model-f6d581-kserve-68f4ddb45f-cgxkp"
Sep 17 03:48:32.984821 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:48:32.984573    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4150b67b-7f33-435e-ac25-6d6557244278-tls-certs\") pod \"rate-test-model-f6d581-kserve-68f4ddb45f-cgxkp\" (UID: \"4150b67b-7f33-435e-ac25-6d6557244278\") " pod="ai-tenant-e2e-shared-b-w3-f6d581/rate-test-model-f6d581-kserve-68f4ddb45f-cgxkp"
Sep 17 03:48:32.985038 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:48:32.985009    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4150b67b-7f33-435e-ac25-6d6557244278-tmp-dir\") pod \"rate-test-model-f6d581-kserve-68f4ddb45f-cgxkp\" (UID: \"4150b67b-7f33-435e-ac25-6d6557244278\") " pod="ai-tenant-e2e-shared-b-w3-f6d581/rate-test-model-f6d581-kserve-68f4ddb45f-cgxkp"
Sep 17 03:48:32.985091 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:48:32.985075    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4150b67b-7f33-435e-ac25-6d6557244278-home\") pod \"rate-test-model-f6d581-kserve-68f4ddb45f-cgxkp\" (UID: \"4150b67b-7f33-435e-ac25-6d6557244278\") " pod="ai-tenant-e2e-shared-b-w3-f6d581/rate-test-model-f6d581-kserve-68f4ddb45f-cgxkp"
Sep 17 03:48:32.985212 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:48:32.985159    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4150b67b-7f33-435e-ac25-6d6557244278-model-cache\") pod \"rate-test-model-f6d581-kserve-68f4ddb45f-cgxkp\" (UID: \"4150b67b-7f33-435e-ac25-6d6557244278\") " pod="ai-tenant-e2e-shared-b-w3-f6d581/rate-test-model-f6d581-kserve-68f4ddb45f-cgxkp"
Sep 17 03:48:32.987374 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:48:32.987342    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4150b67b-7f33-435e-ac25-6d6557244278-dshm\") pod \"rate-test-model-f6d581-kserve-68f4ddb45f-cgxkp\" (UID: \"4150b67b-7f33-435e-ac25-6d6557244278\") " pod="ai-tenant-e2e-shared-b-w3-f6d581/rate-test-model-f6d581-kserve-68f4ddb45f-cgxkp"
Sep 17 03:48:32.987857 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:48:32.987837    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4150b67b-7f33-435e-ac25-6d6557244278-tls-certs\") pod \"rate-test-model-f6d581-kserve-68f4ddb45f-cgxkp\" (UID: \"4150b67b-7f33-435e-ac25-6d6557244278\") " pod="ai-tenant-e2e-shared-b-w3-f6d581/rate-test-model-f6d581-kserve-68f4ddb45f-cgxkp"
Sep 17 03:48:32.993357 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:48:32.993326    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqpfs\" (UniqueName: \"kubernetes.io/projected/4150b67b-7f33-435e-ac25-6d6557244278-kube-api-access-cqpfs\") pod \"rate-test-model-f6d581-kserve-68f4ddb45f-cgxkp\" (UID: \"4150b67b-7f33-435e-ac25-6d6557244278\") " pod="ai-tenant-e2e-shared-b-w3-f6d581/rate-test-model-f6d581-kserve-68f4ddb45f-cgxkp"
Sep 17 03:48:33.120323 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:48:33.120234    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-f6d581/rate-test-model-f6d581-kserve-68f4ddb45f-cgxkp"
Sep 17 03:48:33.392756 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:48:33.392731    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-f6d581/rate-test-model-f6d581-kserve-68f4ddb45f-cgxkp"]
Sep 17 03:48:33.394261 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:48:33.394226    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4150b67b_7f33_435e_ac25_6d6557244278.slice/crio-61bf55a7b7d23d7f15e6bb8029b538e24f9abbca2402e4f09bcaa0f077cee152 WatchSource:0}: Error finding container 61bf55a7b7d23d7f15e6bb8029b538e24f9abbca2402e4f09bcaa0f077cee152: Status 404 returned error can't find the container with id 61bf55a7b7d23d7f15e6bb8029b538e24f9abbca2402e4f09bcaa0f077cee152
Sep 17 03:48:33.469552 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:48:33.469523    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-f6d581/rate-test-model-f6d581-kserve-68f4ddb45f-cgxkp" event={"ID":"4150b67b-7f33-435e-ac25-6d6557244278","Type":"ContainerStarted","Data":"92d49d7228e1d944c907ba9ef162da3c9c52cf412c6941598c2b3a3f89081f6b"}
Sep 17 03:48:33.469711 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:48:33.469560    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-f6d581/rate-test-model-f6d581-kserve-68f4ddb45f-cgxkp" event={"ID":"4150b67b-7f33-435e-ac25-6d6557244278","Type":"ContainerStarted","Data":"61bf55a7b7d23d7f15e6bb8029b538e24f9abbca2402e4f09bcaa0f077cee152"}
Sep 17 03:48:33.488045 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:48:33.487993    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-f6d581/rate-test-model-f6d581-kserve-68f4ddb45f-cgxkp" podStartSLOduration=1.487971733 podStartE2EDuration="1.487971733s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 03:48:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:48:33.487384895 +0000 UTC m=+1748.953792422" watchObservedRunningTime="2026-09-17 03:48:33.487971733 +0000 UTC m=+1748.954379259"
Sep 17 03:48:43.120724 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:48:43.120687    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-f6d581/rate-test-model-f6d581-kserve-68f4ddb45f-cgxkp"
Sep 17 03:48:43.120724 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:48:43.120734    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-f6d581/rate-test-model-f6d581-kserve-68f4ddb45f-cgxkp"
Sep 17 03:48:43.133635 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:48:43.133585    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-f6d581/rate-test-model-f6d581-kserve-68f4ddb45f-cgxkp"
Sep 17 03:48:43.542176 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:48:43.542144    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-f6d581/rate-test-model-f6d581-kserve-68f4ddb45f-cgxkp"
Sep 17 03:49:06.554949 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:49:06.554910    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-f6d581/rate-test-model-f6d581-kserve-68f4ddb45f-cgxkp"]
Sep 17 03:49:21.575794 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:49:21.575728    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-f6d581/rate-test-model-f6d581-kserve-68f4ddb45f-cgxkp" podUID="4150b67b-7f33-435e-ac25-6d6557244278" containerName="main" containerID="cri-o://92d49d7228e1d944c907ba9ef162da3c9c52cf412c6941598c2b3a3f89081f6b" gracePeriod=45
Sep 17 03:49:21.745363 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:49:21.745337    2552 generic.go:356] "Generic (PLEG): container finished" podID="4150b67b-7f33-435e-ac25-6d6557244278" containerID="92d49d7228e1d944c907ba9ef162da3c9c52cf412c6941598c2b3a3f89081f6b" exitCode=0
Sep 17 03:49:21.745477 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:49:21.745404    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-f6d581/rate-test-model-f6d581-kserve-68f4ddb45f-cgxkp" event={"ID":"4150b67b-7f33-435e-ac25-6d6557244278","Type":"ContainerDied","Data":"92d49d7228e1d944c907ba9ef162da3c9c52cf412c6941598c2b3a3f89081f6b"}
Sep 17 03:49:21.816299 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:49:21.816275    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-f6d581/rate-test-model-f6d581-kserve-68f4ddb45f-cgxkp"
Sep 17 03:49:21.958301 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:49:21.958211    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4150b67b-7f33-435e-ac25-6d6557244278-home\") pod \"4150b67b-7f33-435e-ac25-6d6557244278\" (UID: \"4150b67b-7f33-435e-ac25-6d6557244278\") "
Sep 17 03:49:21.958301 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:49:21.958268    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4150b67b-7f33-435e-ac25-6d6557244278-tmp-dir\") pod \"4150b67b-7f33-435e-ac25-6d6557244278\" (UID: \"4150b67b-7f33-435e-ac25-6d6557244278\") "
Sep 17 03:49:21.958553 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:49:21.958307    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cqpfs\" (UniqueName: \"kubernetes.io/projected/4150b67b-7f33-435e-ac25-6d6557244278-kube-api-access-cqpfs\") pod \"4150b67b-7f33-435e-ac25-6d6557244278\" (UID: \"4150b67b-7f33-435e-ac25-6d6557244278\") "
Sep 17 03:49:21.958553 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:49:21.958331    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4150b67b-7f33-435e-ac25-6d6557244278-dshm\") pod \"4150b67b-7f33-435e-ac25-6d6557244278\" (UID: \"4150b67b-7f33-435e-ac25-6d6557244278\") "
Sep 17 03:49:21.958553 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:49:21.958373    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4150b67b-7f33-435e-ac25-6d6557244278-tls-certs\") pod \"4150b67b-7f33-435e-ac25-6d6557244278\" (UID: \"4150b67b-7f33-435e-ac25-6d6557244278\") "
Sep 17 03:49:21.958759 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:49:21.958548    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4150b67b-7f33-435e-ac25-6d6557244278-home" (OuterVolumeSpecName: "home") pod "4150b67b-7f33-435e-ac25-6d6557244278" (UID: "4150b67b-7f33-435e-ac25-6d6557244278"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:49:21.958759 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:49:21.958565    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4150b67b-7f33-435e-ac25-6d6557244278-model-cache\") pod \"4150b67b-7f33-435e-ac25-6d6557244278\" (UID: \"4150b67b-7f33-435e-ac25-6d6557244278\") "
Sep 17 03:49:21.958759 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:49:21.958590    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4150b67b-7f33-435e-ac25-6d6557244278-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4150b67b-7f33-435e-ac25-6d6557244278" (UID: "4150b67b-7f33-435e-ac25-6d6557244278"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:49:21.958915 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:49:21.958811    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4150b67b-7f33-435e-ac25-6d6557244278-model-cache" (OuterVolumeSpecName: "model-cache") pod "4150b67b-7f33-435e-ac25-6d6557244278" (UID: "4150b67b-7f33-435e-ac25-6d6557244278"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:49:21.959042 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:49:21.959003    2552 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4150b67b-7f33-435e-ac25-6d6557244278-home\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:49:21.959042 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:49:21.959019    2552 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4150b67b-7f33-435e-ac25-6d6557244278-tmp-dir\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:49:21.959042 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:49:21.959033    2552 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4150b67b-7f33-435e-ac25-6d6557244278-model-cache\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:49:21.960528 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:49:21.960508    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4150b67b-7f33-435e-ac25-6d6557244278-dshm" (OuterVolumeSpecName: "dshm") pod "4150b67b-7f33-435e-ac25-6d6557244278" (UID: "4150b67b-7f33-435e-ac25-6d6557244278"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:49:21.960587 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:49:21.960527    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4150b67b-7f33-435e-ac25-6d6557244278-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4150b67b-7f33-435e-ac25-6d6557244278" (UID: "4150b67b-7f33-435e-ac25-6d6557244278"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 03:49:21.960644 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:49:21.960633    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4150b67b-7f33-435e-ac25-6d6557244278-kube-api-access-cqpfs" (OuterVolumeSpecName: "kube-api-access-cqpfs") pod "4150b67b-7f33-435e-ac25-6d6557244278" (UID: "4150b67b-7f33-435e-ac25-6d6557244278"). InnerVolumeSpecName "kube-api-access-cqpfs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 03:49:22.060422 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:49:22.060385    2552 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4150b67b-7f33-435e-ac25-6d6557244278-tls-certs\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:49:22.060422 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:49:22.060413    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cqpfs\" (UniqueName: \"kubernetes.io/projected/4150b67b-7f33-435e-ac25-6d6557244278-kube-api-access-cqpfs\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:49:22.060422 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:49:22.060421    2552 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4150b67b-7f33-435e-ac25-6d6557244278-dshm\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:49:22.752798 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:49:22.752765    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-f6d581/rate-test-model-f6d581-kserve-68f4ddb45f-cgxkp" event={"ID":"4150b67b-7f33-435e-ac25-6d6557244278","Type":"ContainerDied","Data":"61bf55a7b7d23d7f15e6bb8029b538e24f9abbca2402e4f09bcaa0f077cee152"}
Sep 17 03:49:22.753197 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:49:22.752810    2552 scope.go:117] "RemoveContainer" containerID="92d49d7228e1d944c907ba9ef162da3c9c52cf412c6941598c2b3a3f89081f6b"
Sep 17 03:49:22.753197 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:49:22.752775    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-f6d581/rate-test-model-f6d581-kserve-68f4ddb45f-cgxkp"
Sep 17 03:49:22.774451 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:49:22.774428    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-f6d581/rate-test-model-f6d581-kserve-68f4ddb45f-cgxkp"]
Sep 17 03:49:22.776300 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:49:22.776277    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-f6d581/rate-test-model-f6d581-kserve-68f4ddb45f-cgxkp"]
Sep 17 03:49:22.992688 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:49:22.992652    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4150b67b-7f33-435e-ac25-6d6557244278" path="/var/lib/kubelet/pods/4150b67b-7f33-435e-ac25-6d6557244278/volumes"
Sep 17 03:49:35.950259 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:49:35.950225    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-84cd5d-openshift-default-66478d5f7b-dgwrz"]
Sep 17 03:49:35.950920 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:49:35.950902    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4150b67b-7f33-435e-ac25-6d6557244278" containerName="main"
Sep 17 03:49:35.950998 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:49:35.950922    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="4150b67b-7f33-435e-ac25-6d6557244278" containerName="main"
Sep 17 03:49:35.951055 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:49:35.951024    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="4150b67b-7f33-435e-ac25-6d6557244278" containerName="main"
Sep 17 03:49:35.954330 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:49:35.954309    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-84cd5d-openshift-default-66478d5f7b-dgwrz"
Sep 17 03:49:35.957051 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:49:35.957033    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-84cd5d-openshift-default-dockercfg-jxwsg\""
Sep 17 03:49:35.964221 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:49:35.964199    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-84cd5d-openshift-default-66478d5f7b-dgwrz"]
Sep 17 03:49:35.993352 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:49:35.993328    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ccecfb8-e85c-4656-b88d-0551baf776dd-istio-token\") pod \"e2e-ipp-b-84cd5d-openshift-default-66478d5f7b-dgwrz\" (UID: \"8ccecfb8-e85c-4656-b88d-0551baf776dd\") " pod="openshift-ingress/e2e-ipp-b-84cd5d-openshift-default-66478d5f7b-dgwrz"
Sep 17 03:49:35.993474 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:49:35.993360    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ccecfb8-e85c-4656-b88d-0551baf776dd-istio-data\") pod \"e2e-ipp-b-84cd5d-openshift-default-66478d5f7b-dgwrz\" (UID: \"8ccecfb8-e85c-4656-b88d-0551baf776dd\") " pod="openshift-ingress/e2e-ipp-b-84cd5d-openshift-default-66478d5f7b-dgwrz"
Sep 17 03:49:35.993474 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:49:35.993379    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ccecfb8-e85c-4656-b88d-0551baf776dd-workload-certs\") pod \"e2e-ipp-b-84cd5d-openshift-default-66478d5f7b-dgwrz\" (UID: \"8ccecfb8-e85c-4656-b88d-0551baf776dd\") " pod="openshift-ingress/e2e-ipp-b-84cd5d-openshift-default-66478d5f7b-dgwrz"
Sep 17 03:49:35.993474 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:49:35.993407    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxkgf\" (UniqueName: \"kubernetes.io/projected/8ccecfb8-e85c-4656-b88d-0551baf776dd-kube-api-access-dxkgf\") pod \"e2e-ipp-b-84cd5d-openshift-default-66478d5f7b-dgwrz\" (UID: \"8ccecfb8-e85c-4656-b88d-0551baf776dd\") " pod="openshift-ingress/e2e-ipp-b-84cd5d-openshift-default-66478d5f7b-dgwrz"
Sep 17 03:49:35.993474 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:49:35.993454    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ccecfb8-e85c-4656-b88d-0551baf776dd-istiod-ca-cert\") pod \"e2e-ipp-b-84cd5d-openshift-default-66478d5f7b-dgwrz\" (UID: \"8ccecfb8-e85c-4656-b88d-0551baf776dd\") " pod="openshift-ingress/e2e-ipp-b-84cd5d-openshift-default-66478d5f7b-dgwrz"
Sep 17 03:49:35.993642 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:49:35.993506    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ccecfb8-e85c-4656-b88d-0551baf776dd-istio-envoy\") pod \"e2e-ipp-b-84cd5d-openshift-default-66478d5f7b-dgwrz\" (UID: \"8ccecfb8-e85c-4656-b88d-0551baf776dd\") " pod="openshift-ingress/e2e-ipp-b-84cd5d-openshift-default-66478d5f7b-dgwrz"
Sep 17 03:49:35.993642 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:49:35.993547    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ccecfb8-e85c-4656-b88d-0551baf776dd-workload-socket\") pod \"e2e-ipp-b-84cd5d-openshift-default-66478d5f7b-dgwrz\" (UID: \"8ccecfb8-e85c-4656-b88d-0551baf776dd\") " pod="openshift-ingress/e2e-ipp-b-84cd5d-openshift-default-66478d5f7b-dgwrz"
Sep 17 03:49:35.993642 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:49:35.993624    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ccecfb8-e85c-4656-b88d-0551baf776dd-istio-podinfo\") pod \"e2e-ipp-b-84cd5d-openshift-default-66478d5f7b-dgwrz\" (UID: \"8ccecfb8-e85c-4656-b88d-0551baf776dd\") " pod="openshift-ingress/e2e-ipp-b-84cd5d-openshift-default-66478d5f7b-dgwrz"
Sep 17 03:49:35.993749 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:49:35.993678    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ccecfb8-e85c-4656-b88d-0551baf776dd-credential-socket\") pod \"e2e-ipp-b-84cd5d-openshift-default-66478d5f7b-dgwrz\" (UID: \"8ccecfb8-e85c-4656-b88d-0551baf776dd\") " pod="openshift-ingress/e2e-ipp-b-84cd5d-openshift-default-66478d5f7b-dgwrz"
Sep 17 03:49:36.094435 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:49:36.094398    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ccecfb8-e85c-4656-b88d-0551baf776dd-istio-podinfo\") pod \"e2e-ipp-b-84cd5d-openshift-default-66478d5f7b-dgwrz\" (UID: \"8ccecfb8-e85c-4656-b88d-0551baf776dd\") " pod="openshift-ingress/e2e-ipp-b-84cd5d-openshift-default-66478d5f7b-dgwrz"
Sep 17 03:49:36.094677 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:49:36.094456    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ccecfb8-e85c-4656-b88d-0551baf776dd-credential-socket\") pod \"e2e-ipp-b-84cd5d-openshift-default-66478d5f7b-dgwrz\" (UID: \"8ccecfb8-e85c-4656-b88d-0551baf776dd\") " pod="openshift-ingress/e2e-ipp-b-84cd5d-openshift-default-66478d5f7b-dgwrz"
Sep 17 03:49:36.094677 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:49:36.094492    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ccecfb8-e85c-4656-b88d-0551baf776dd-istio-token\") pod \"e2e-ipp-b-84cd5d-openshift-default-66478d5f7b-dgwrz\" (UID: \"8ccecfb8-e85c-4656-b88d-0551baf776dd\") " pod="openshift-ingress/e2e-ipp-b-84cd5d-openshift-default-66478d5f7b-dgwrz"
Sep 17 03:49:36.094677 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:49:36.094518    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ccecfb8-e85c-4656-b88d-0551baf776dd-istio-data\") pod \"e2e-ipp-b-84cd5d-openshift-default-66478d5f7b-dgwrz\" (UID: \"8ccecfb8-e85c-4656-b88d-0551baf776dd\") " pod="openshift-ingress/e2e-ipp-b-84cd5d-openshift-default-66478d5f7b-dgwrz"
Sep 17 03:49:36.094677 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:49:36.094541    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ccecfb8-e85c-4656-b88d-0551baf776dd-workload-certs\") pod \"e2e-ipp-b-84cd5d-openshift-default-66478d5f7b-dgwrz\" (UID: \"8ccecfb8-e85c-4656-b88d-0551baf776dd\") " pod="openshift-ingress/e2e-ipp-b-84cd5d-openshift-default-66478d5f7b-dgwrz"
Sep 17 03:49:36.094677 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:49:36.094585    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxkgf\" (UniqueName: \"kubernetes.io/projected/8ccecfb8-e85c-4656-b88d-0551baf776dd-kube-api-access-dxkgf\") pod \"e2e-ipp-b-84cd5d-openshift-default-66478d5f7b-dgwrz\" (UID: \"8ccecfb8-e85c-4656-b88d-0551baf776dd\") " pod="openshift-ingress/e2e-ipp-b-84cd5d-openshift-default-66478d5f7b-dgwrz"
Sep 17 03:49:36.094677 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:49:36.094635    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ccecfb8-e85c-4656-b88d-0551baf776dd-istiod-ca-cert\") pod \"e2e-ipp-b-84cd5d-openshift-default-66478d5f7b-dgwrz\" (UID: \"8ccecfb8-e85c-4656-b88d-0551baf776dd\") " pod="openshift-ingress/e2e-ipp-b-84cd5d-openshift-default-66478d5f7b-dgwrz"
Sep 17 03:49:36.094677 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:49:36.094661    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ccecfb8-e85c-4656-b88d-0551baf776dd-istio-envoy\") pod \"e2e-ipp-b-84cd5d-openshift-default-66478d5f7b-dgwrz\" (UID: \"8ccecfb8-e85c-4656-b88d-0551baf776dd\") " pod="openshift-ingress/e2e-ipp-b-84cd5d-openshift-default-66478d5f7b-dgwrz"
Sep 17 03:49:36.095075 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:49:36.094709    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ccecfb8-e85c-4656-b88d-0551baf776dd-workload-socket\") pod \"e2e-ipp-b-84cd5d-openshift-default-66478d5f7b-dgwrz\" (UID: \"8ccecfb8-e85c-4656-b88d-0551baf776dd\") " pod="openshift-ingress/e2e-ipp-b-84cd5d-openshift-default-66478d5f7b-dgwrz"
Sep 17 03:49:36.095075 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:49:36.094968    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ccecfb8-e85c-4656-b88d-0551baf776dd-credential-socket\") pod \"e2e-ipp-b-84cd5d-openshift-default-66478d5f7b-dgwrz\" (UID: \"8ccecfb8-e85c-4656-b88d-0551baf776dd\") " pod="openshift-ingress/e2e-ipp-b-84cd5d-openshift-default-66478d5f7b-dgwrz"
Sep 17 03:49:36.095075 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:49:36.095001    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ccecfb8-e85c-4656-b88d-0551baf776dd-workload-certs\") pod \"e2e-ipp-b-84cd5d-openshift-default-66478d5f7b-dgwrz\" (UID: \"8ccecfb8-e85c-4656-b88d-0551baf776dd\") " pod="openshift-ingress/e2e-ipp-b-84cd5d-openshift-default-66478d5f7b-dgwrz"
Sep 17 03:49:36.095075 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:49:36.095049    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ccecfb8-e85c-4656-b88d-0551baf776dd-workload-socket\") pod \"e2e-ipp-b-84cd5d-openshift-default-66478d5f7b-dgwrz\" (UID: \"8ccecfb8-e85c-4656-b88d-0551baf776dd\") " pod="openshift-ingress/e2e-ipp-b-84cd5d-openshift-default-66478d5f7b-dgwrz"
Sep 17 03:49:36.095238 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:49:36.095116    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ccecfb8-e85c-4656-b88d-0551baf776dd-istio-data\") pod \"e2e-ipp-b-84cd5d-openshift-default-66478d5f7b-dgwrz\" (UID: \"8ccecfb8-e85c-4656-b88d-0551baf776dd\") " pod="openshift-ingress/e2e-ipp-b-84cd5d-openshift-default-66478d5f7b-dgwrz"
Sep 17 03:49:36.095448 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:49:36.095427    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ccecfb8-e85c-4656-b88d-0551baf776dd-istiod-ca-cert\") pod \"e2e-ipp-b-84cd5d-openshift-default-66478d5f7b-dgwrz\" (UID: \"8ccecfb8-e85c-4656-b88d-0551baf776dd\") " pod="openshift-ingress/e2e-ipp-b-84cd5d-openshift-default-66478d5f7b-dgwrz"
Sep 17 03:49:36.097055 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:49:36.097026    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ccecfb8-e85c-4656-b88d-0551baf776dd-istio-envoy\") pod \"e2e-ipp-b-84cd5d-openshift-default-66478d5f7b-dgwrz\" (UID: \"8ccecfb8-e85c-4656-b88d-0551baf776dd\") " pod="openshift-ingress/e2e-ipp-b-84cd5d-openshift-default-66478d5f7b-dgwrz"
Sep 17 03:49:36.097282 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:49:36.097264    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ccecfb8-e85c-4656-b88d-0551baf776dd-istio-podinfo\") pod \"e2e-ipp-b-84cd5d-openshift-default-66478d5f7b-dgwrz\" (UID: \"8ccecfb8-e85c-4656-b88d-0551baf776dd\") " pod="openshift-ingress/e2e-ipp-b-84cd5d-openshift-default-66478d5f7b-dgwrz"
Sep 17 03:49:36.105997 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:49:36.105968    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ccecfb8-e85c-4656-b88d-0551baf776dd-istio-token\") pod \"e2e-ipp-b-84cd5d-openshift-default-66478d5f7b-dgwrz\" (UID: \"8ccecfb8-e85c-4656-b88d-0551baf776dd\") " pod="openshift-ingress/e2e-ipp-b-84cd5d-openshift-default-66478d5f7b-dgwrz"
Sep 17 03:49:36.106150 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:49:36.106078    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxkgf\" (UniqueName: \"kubernetes.io/projected/8ccecfb8-e85c-4656-b88d-0551baf776dd-kube-api-access-dxkgf\") pod \"e2e-ipp-b-84cd5d-openshift-default-66478d5f7b-dgwrz\" (UID: \"8ccecfb8-e85c-4656-b88d-0551baf776dd\") " pod="openshift-ingress/e2e-ipp-b-84cd5d-openshift-default-66478d5f7b-dgwrz"
Sep 17 03:49:36.266755 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:49:36.266716    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-84cd5d-openshift-default-66478d5f7b-dgwrz"
Sep 17 03:49:36.547900 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:49:36.547872    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-84cd5d-openshift-default-66478d5f7b-dgwrz"]
Sep 17 03:49:36.549578 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:49:36.549556    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ccecfb8_e85c_4656_b88d_0551baf776dd.slice/crio-10a0c93fbd9ebb0c5bbb0aa1bb819dab93dc81a8e86a5d1b94ba4755ab050904 WatchSource:0}: Error finding container 10a0c93fbd9ebb0c5bbb0aa1bb819dab93dc81a8e86a5d1b94ba4755ab050904: Status 404 returned error can't find the container with id 10a0c93fbd9ebb0c5bbb0aa1bb819dab93dc81a8e86a5d1b94ba4755ab050904
Sep 17 03:49:36.551805 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:49:36.551773    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 03:49:36.551905 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:49:36.551855    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 03:49:36.551905 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:49:36.551900    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 03:49:36.834099 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:49:36.834011    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-84cd5d-openshift-default-66478d5f7b-dgwrz" event={"ID":"8ccecfb8-e85c-4656-b88d-0551baf776dd","Type":"ContainerStarted","Data":"bfc2000ad9136e1cf9fe59c1d9c86a12aeb3a9c8e5d258572eeb352a1fc2755d"}
Sep 17 03:49:36.834099 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:49:36.834070    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-84cd5d-openshift-default-66478d5f7b-dgwrz" event={"ID":"8ccecfb8-e85c-4656-b88d-0551baf776dd","Type":"ContainerStarted","Data":"10a0c93fbd9ebb0c5bbb0aa1bb819dab93dc81a8e86a5d1b94ba4755ab050904"}
Sep 17 03:49:36.854384 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:49:36.854333    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-84cd5d-openshift-default-66478d5f7b-dgwrz" podStartSLOduration=1.854318921 podStartE2EDuration="1.854318921s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 03:49:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:49:36.85194462 +0000 UTC m=+1812.318352147" watchObservedRunningTime="2026-09-17 03:49:36.854318921 +0000 UTC m=+1812.320726447"
Sep 17 03:49:37.267060 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:49:37.267021    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-84cd5d-openshift-default-66478d5f7b-dgwrz"
Sep 17 03:49:37.272200 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:49:37.272178    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-84cd5d-openshift-default-66478d5f7b-dgwrz"
Sep 17 03:49:37.839572 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:49:37.839541    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-84cd5d-openshift-default-66478d5f7b-dgwrz"
Sep 17 03:49:37.840592 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:49:37.840571    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-84cd5d-openshift-default-66478d5f7b-dgwrz"
Sep 17 03:49:53.880801 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:49:53.880746    2552 patch_prober.go:28] interesting pod/e2e-ipp-b-84cd5d-openshift-default-66478d5f7b-dgwrz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 17 03:49:53.881222 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:49:53.880832    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-84cd5d-openshift-default-66478d5f7b-dgwrz" podUID="8ccecfb8-e85c-4656-b88d-0551baf776dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 17 03:49:55.937513 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:49:55.937426    2552 generic.go:356] "Generic (PLEG): container finished" podID="8ccecfb8-e85c-4656-b88d-0551baf776dd" containerID="bfc2000ad9136e1cf9fe59c1d9c86a12aeb3a9c8e5d258572eeb352a1fc2755d" exitCode=0
Sep 17 03:49:55.938052 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:49:55.937500    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-84cd5d-openshift-default-66478d5f7b-dgwrz" event={"ID":"8ccecfb8-e85c-4656-b88d-0551baf776dd","Type":"ContainerDied","Data":"bfc2000ad9136e1cf9fe59c1d9c86a12aeb3a9c8e5d258572eeb352a1fc2755d"}
Sep 17 03:49:55.938121 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:49:55.938051    2552 scope.go:117] "RemoveContainer" containerID="bfc2000ad9136e1cf9fe59c1d9c86a12aeb3a9c8e5d258572eeb352a1fc2755d"
Sep 17 03:49:55.939238 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:49:55.939210    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 03:49:55.939352 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:49:55.939295    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 03:49:55.939352 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:49:55.939339    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 03:49:56.950847 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:49:56.950812    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-84cd5d-openshift-default-66478d5f7b-dgwrz" event={"ID":"8ccecfb8-e85c-4656-b88d-0551baf776dd","Type":"ContainerStarted","Data":"296354264a4e70a0b5465e04208236deaab5beafc80d7cce9554e0fd8e0cc8cf"}
Sep 17 03:49:57.266916 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:49:57.266875    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-84cd5d-openshift-default-66478d5f7b-dgwrz"
Sep 17 03:49:57.267141 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:49:57.267118    2552 patch_prober.go:28] interesting pod/e2e-ipp-b-84cd5d-openshift-default-66478d5f7b-dgwrz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Sep 17 03:49:57.267196 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:49:57.267179    2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-84cd5d-openshift-default-66478d5f7b-dgwrz" podUID="8ccecfb8-e85c-4656-b88d-0551baf776dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Sep 17 03:49:58.267952 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:49:58.267915    2552 patch_prober.go:28] interesting pod/e2e-ipp-b-84cd5d-openshift-default-66478d5f7b-dgwrz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Sep 17 03:49:58.268362 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:49:58.267970    2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-84cd5d-openshift-default-66478d5f7b-dgwrz" podUID="8ccecfb8-e85c-4656-b88d-0551baf776dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Sep 17 03:49:59.268079 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:49:59.268042    2552 patch_prober.go:28] interesting pod/e2e-ipp-b-84cd5d-openshift-default-66478d5f7b-dgwrz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Sep 17 03:49:59.268485 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:49:59.268097    2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-84cd5d-openshift-default-66478d5f7b-dgwrz" podUID="8ccecfb8-e85c-4656-b88d-0551baf776dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Sep 17 03:50:00.267418 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:50:00.267383    2552 patch_prober.go:28] interesting pod/e2e-ipp-b-84cd5d-openshift-default-66478d5f7b-dgwrz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Sep 17 03:50:00.267695 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:50:00.267665    2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-84cd5d-openshift-default-66478d5f7b-dgwrz" podUID="8ccecfb8-e85c-4656-b88d-0551baf776dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Sep 17 03:50:01.267552 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:50:01.267514    2552 patch_prober.go:28] interesting pod/e2e-ipp-b-84cd5d-openshift-default-66478d5f7b-dgwrz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Sep 17 03:50:01.267968 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:50:01.267572    2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-84cd5d-openshift-default-66478d5f7b-dgwrz" podUID="8ccecfb8-e85c-4656-b88d-0551baf776dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Sep 17 03:50:02.270616 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:50:02.270578    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-84cd5d-openshift-default-66478d5f7b-dgwrz"
Sep 17 03:50:02.271134 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:50:02.271108    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-84cd5d-openshift-default-66478d5f7b-dgwrz"
Sep 17 03:50:02.271930 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:50:02.271911    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-84cd5d-openshift-default-66478d5f7b-dgwrz"
Sep 17 03:50:22.577702 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:50:22.577663    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-d0ab1a/ipp-route-d0ab1a-kserve-545c9b4f8c-p74wv"]
Sep 17 03:50:22.581305 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:50:22.581282    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-d0ab1a/ipp-route-d0ab1a-kserve-545c9b4f8c-p74wv"
Sep 17 03:50:22.585375 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:50:22.585352    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-d0ab1a\"/\"default-dockercfg-cm6fv\""
Sep 17 03:50:22.585375 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:50:22.585352    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-d0ab1a\"/\"openshift-service-ca.crt\""
Sep 17 03:50:22.585549 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:50:22.585378    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-d0ab1a\"/\"kube-root-ca.crt\""
Sep 17 03:50:22.585549 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:50:22.585385    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-d0ab1a\"/\"ipp-route-d0ab1a-kserve-self-signed-certs\""
Sep 17 03:50:22.590128 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:50:22.590108    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-d0ab1a/ipp-route-d0ab1a-kserve-545c9b4f8c-p74wv"]
Sep 17 03:50:22.644173 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:50:22.644147    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28c32ecd-31c3-4069-8701-0997e5146b32-tls-certs\") pod \"ipp-route-d0ab1a-kserve-545c9b4f8c-p74wv\" (UID: \"28c32ecd-31c3-4069-8701-0997e5146b32\") " pod="ai-tenant-e2e-ipp-a-d0ab1a/ipp-route-d0ab1a-kserve-545c9b4f8c-p74wv"
Sep 17 03:50:22.644323 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:50:22.644197    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28c32ecd-31c3-4069-8701-0997e5146b32-home\") pod \"ipp-route-d0ab1a-kserve-545c9b4f8c-p74wv\" (UID: \"28c32ecd-31c3-4069-8701-0997e5146b32\") " pod="ai-tenant-e2e-ipp-a-d0ab1a/ipp-route-d0ab1a-kserve-545c9b4f8c-p74wv"
Sep 17 03:50:22.644323 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:50:22.644283    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbp9x\" (UniqueName: \"kubernetes.io/projected/28c32ecd-31c3-4069-8701-0997e5146b32-kube-api-access-mbp9x\") pod \"ipp-route-d0ab1a-kserve-545c9b4f8c-p74wv\" (UID: \"28c32ecd-31c3-4069-8701-0997e5146b32\") " pod="ai-tenant-e2e-ipp-a-d0ab1a/ipp-route-d0ab1a-kserve-545c9b4f8c-p74wv"
Sep 17 03:50:22.644323 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:50:22.644313    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28c32ecd-31c3-4069-8701-0997e5146b32-model-cache\") pod \"ipp-route-d0ab1a-kserve-545c9b4f8c-p74wv\" (UID: \"28c32ecd-31c3-4069-8701-0997e5146b32\") " pod="ai-tenant-e2e-ipp-a-d0ab1a/ipp-route-d0ab1a-kserve-545c9b4f8c-p74wv"
Sep 17 03:50:22.644429 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:50:22.644341    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28c32ecd-31c3-4069-8701-0997e5146b32-dshm\") pod \"ipp-route-d0ab1a-kserve-545c9b4f8c-p74wv\" (UID: \"28c32ecd-31c3-4069-8701-0997e5146b32\") " pod="ai-tenant-e2e-ipp-a-d0ab1a/ipp-route-d0ab1a-kserve-545c9b4f8c-p74wv"
Sep 17 03:50:22.644429 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:50:22.644358    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28c32ecd-31c3-4069-8701-0997e5146b32-tmp-dir\") pod \"ipp-route-d0ab1a-kserve-545c9b4f8c-p74wv\" (UID: \"28c32ecd-31c3-4069-8701-0997e5146b32\") " pod="ai-tenant-e2e-ipp-a-d0ab1a/ipp-route-d0ab1a-kserve-545c9b4f8c-p74wv"
Sep 17 03:50:22.745612 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:50:22.745575    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28c32ecd-31c3-4069-8701-0997e5146b32-tmp-dir\") pod \"ipp-route-d0ab1a-kserve-545c9b4f8c-p74wv\" (UID: \"28c32ecd-31c3-4069-8701-0997e5146b32\") " pod="ai-tenant-e2e-ipp-a-d0ab1a/ipp-route-d0ab1a-kserve-545c9b4f8c-p74wv"
Sep 17 03:50:22.745777 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:50:22.745658    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28c32ecd-31c3-4069-8701-0997e5146b32-tls-certs\") pod \"ipp-route-d0ab1a-kserve-545c9b4f8c-p74wv\" (UID: \"28c32ecd-31c3-4069-8701-0997e5146b32\") " pod="ai-tenant-e2e-ipp-a-d0ab1a/ipp-route-d0ab1a-kserve-545c9b4f8c-p74wv"
Sep 17 03:50:22.745777 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:50:22.745721    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28c32ecd-31c3-4069-8701-0997e5146b32-home\") pod \"ipp-route-d0ab1a-kserve-545c9b4f8c-p74wv\" (UID: \"28c32ecd-31c3-4069-8701-0997e5146b32\") " pod="ai-tenant-e2e-ipp-a-d0ab1a/ipp-route-d0ab1a-kserve-545c9b4f8c-p74wv"
Sep 17 03:50:22.745777 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:50:22.745766    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbp9x\" (UniqueName: \"kubernetes.io/projected/28c32ecd-31c3-4069-8701-0997e5146b32-kube-api-access-mbp9x\") pod \"ipp-route-d0ab1a-kserve-545c9b4f8c-p74wv\" (UID: \"28c32ecd-31c3-4069-8701-0997e5146b32\") " pod="ai-tenant-e2e-ipp-a-d0ab1a/ipp-route-d0ab1a-kserve-545c9b4f8c-p74wv"
Sep 17 03:50:22.745946 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:50:22.745784    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28c32ecd-31c3-4069-8701-0997e5146b32-model-cache\") pod \"ipp-route-d0ab1a-kserve-545c9b4f8c-p74wv\" (UID: \"28c32ecd-31c3-4069-8701-0997e5146b32\") " pod="ai-tenant-e2e-ipp-a-d0ab1a/ipp-route-d0ab1a-kserve-545c9b4f8c-p74wv"
Sep 17 03:50:22.745946 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:50:22.745900    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28c32ecd-31c3-4069-8701-0997e5146b32-dshm\") pod \"ipp-route-d0ab1a-kserve-545c9b4f8c-p74wv\" (UID: \"28c32ecd-31c3-4069-8701-0997e5146b32\") " pod="ai-tenant-e2e-ipp-a-d0ab1a/ipp-route-d0ab1a-kserve-545c9b4f8c-p74wv"
Sep 17 03:50:22.746075 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:50:22.746050    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28c32ecd-31c3-4069-8701-0997e5146b32-tmp-dir\") pod \"ipp-route-d0ab1a-kserve-545c9b4f8c-p74wv\" (UID: \"28c32ecd-31c3-4069-8701-0997e5146b32\") " pod="ai-tenant-e2e-ipp-a-d0ab1a/ipp-route-d0ab1a-kserve-545c9b4f8c-p74wv"
Sep 17 03:50:22.746143 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:50:22.746098    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28c32ecd-31c3-4069-8701-0997e5146b32-model-cache\") pod \"ipp-route-d0ab1a-kserve-545c9b4f8c-p74wv\" (UID: \"28c32ecd-31c3-4069-8701-0997e5146b32\") " pod="ai-tenant-e2e-ipp-a-d0ab1a/ipp-route-d0ab1a-kserve-545c9b4f8c-p74wv"
Sep 17 03:50:22.746197 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:50:22.746141    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28c32ecd-31c3-4069-8701-0997e5146b32-home\") pod \"ipp-route-d0ab1a-kserve-545c9b4f8c-p74wv\" (UID: \"28c32ecd-31c3-4069-8701-0997e5146b32\") " pod="ai-tenant-e2e-ipp-a-d0ab1a/ipp-route-d0ab1a-kserve-545c9b4f8c-p74wv"
Sep 17 03:50:22.748093 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:50:22.748058    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28c32ecd-31c3-4069-8701-0997e5146b32-dshm\") pod \"ipp-route-d0ab1a-kserve-545c9b4f8c-p74wv\" (UID: \"28c32ecd-31c3-4069-8701-0997e5146b32\") " pod="ai-tenant-e2e-ipp-a-d0ab1a/ipp-route-d0ab1a-kserve-545c9b4f8c-p74wv"
Sep 17 03:50:22.748223 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:50:22.748210    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28c32ecd-31c3-4069-8701-0997e5146b32-tls-certs\") pod \"ipp-route-d0ab1a-kserve-545c9b4f8c-p74wv\" (UID: \"28c32ecd-31c3-4069-8701-0997e5146b32\") " pod="ai-tenant-e2e-ipp-a-d0ab1a/ipp-route-d0ab1a-kserve-545c9b4f8c-p74wv"
Sep 17 03:50:22.753915 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:50:22.753890    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbp9x\" (UniqueName: \"kubernetes.io/projected/28c32ecd-31c3-4069-8701-0997e5146b32-kube-api-access-mbp9x\") pod \"ipp-route-d0ab1a-kserve-545c9b4f8c-p74wv\" (UID: \"28c32ecd-31c3-4069-8701-0997e5146b32\") " pod="ai-tenant-e2e-ipp-a-d0ab1a/ipp-route-d0ab1a-kserve-545c9b4f8c-p74wv"
Sep 17 03:50:22.892307 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:50:22.892206    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-d0ab1a/ipp-route-d0ab1a-kserve-545c9b4f8c-p74wv"
Sep 17 03:50:23.168535 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:50:23.168510    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-d0ab1a/ipp-route-d0ab1a-kserve-545c9b4f8c-p74wv"]
Sep 17 03:50:23.170272 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:50:23.170242    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod28c32ecd_31c3_4069_8701_0997e5146b32.slice/crio-58b8c58d5fcb2d475e8495010afec0f11642bff25d7af4af7c3f79d99737b7bf WatchSource:0}: Error finding container 58b8c58d5fcb2d475e8495010afec0f11642bff25d7af4af7c3f79d99737b7bf: Status 404 returned error can't find the container with id 58b8c58d5fcb2d475e8495010afec0f11642bff25d7af4af7c3f79d99737b7bf
Sep 17 03:50:24.099270 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:50:24.099225    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-d0ab1a/ipp-route-d0ab1a-kserve-545c9b4f8c-p74wv" event={"ID":"28c32ecd-31c3-4069-8701-0997e5146b32","Type":"ContainerStarted","Data":"f254820143bd00dbde88814a170aae2dced437522380fbfd18ac121abf6b675b"}
Sep 17 03:50:24.099270 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:50:24.099273    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-d0ab1a/ipp-route-d0ab1a-kserve-545c9b4f8c-p74wv" event={"ID":"28c32ecd-31c3-4069-8701-0997e5146b32","Type":"ContainerStarted","Data":"58b8c58d5fcb2d475e8495010afec0f11642bff25d7af4af7c3f79d99737b7bf"}
Sep 17 03:50:24.118326 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:50:24.118279    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-d0ab1a/ipp-route-d0ab1a-kserve-545c9b4f8c-p74wv" podStartSLOduration=2.118263318 podStartE2EDuration="2.118263318s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 03:50:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:50:24.116748822 +0000 UTC m=+1859.583156347" watchObservedRunningTime="2026-09-17 03:50:24.118263318 +0000 UTC m=+1859.584670843"
Sep 17 03:50:32.893056 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:50:32.893015    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-d0ab1a/ipp-route-d0ab1a-kserve-545c9b4f8c-p74wv"
Sep 17 03:50:32.893536 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:50:32.893070    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-d0ab1a/ipp-route-d0ab1a-kserve-545c9b4f8c-p74wv"
Sep 17 03:50:32.905752 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:50:32.905730    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-d0ab1a/ipp-route-d0ab1a-kserve-545c9b4f8c-p74wv"
Sep 17 03:50:33.156007 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:50:33.155915    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-d0ab1a/ipp-route-d0ab1a-kserve-545c9b4f8c-p74wv"
Sep 17 03:50:51.112075 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:50:51.111996    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-0ccfd0-openshift-default-7bff59778d-bt5rk"]
Sep 17 03:50:51.116427 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:50:51.116404    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-0ccfd0-openshift-default-7bff59778d-bt5rk"
Sep 17 03:50:51.122222 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:50:51.122196    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-cleanup-0ccfd0-openshift-default-dockercfg-zlx45\""
Sep 17 03:50:51.129407 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:50:51.129381    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-0ccfd0-openshift-default-7bff59778d-bt5rk"]
Sep 17 03:50:51.222722 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:50:51.222689    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a78851c0-74a9-4e0c-9864-71b65331fcdb-credential-socket\") pod \"e2e-ipp-cleanup-0ccfd0-openshift-default-7bff59778d-bt5rk\" (UID: \"a78851c0-74a9-4e0c-9864-71b65331fcdb\") " pod="openshift-ingress/e2e-ipp-cleanup-0ccfd0-openshift-default-7bff59778d-bt5rk"
Sep 17 03:50:51.222912 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:50:51.222728    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a78851c0-74a9-4e0c-9864-71b65331fcdb-istio-envoy\") pod \"e2e-ipp-cleanup-0ccfd0-openshift-default-7bff59778d-bt5rk\" (UID: \"a78851c0-74a9-4e0c-9864-71b65331fcdb\") " pod="openshift-ingress/e2e-ipp-cleanup-0ccfd0-openshift-default-7bff59778d-bt5rk"
Sep 17 03:50:51.222912 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:50:51.222771    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a78851c0-74a9-4e0c-9864-71b65331fcdb-istio-podinfo\") pod \"e2e-ipp-cleanup-0ccfd0-openshift-default-7bff59778d-bt5rk\" (UID: \"a78851c0-74a9-4e0c-9864-71b65331fcdb\") " pod="openshift-ingress/e2e-ipp-cleanup-0ccfd0-openshift-default-7bff59778d-bt5rk"
Sep 17 03:50:51.222912 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:50:51.222847    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59m4d\" (UniqueName: \"kubernetes.io/projected/a78851c0-74a9-4e0c-9864-71b65331fcdb-kube-api-access-59m4d\") pod \"e2e-ipp-cleanup-0ccfd0-openshift-default-7bff59778d-bt5rk\" (UID: \"a78851c0-74a9-4e0c-9864-71b65331fcdb\") " pod="openshift-ingress/e2e-ipp-cleanup-0ccfd0-openshift-default-7bff59778d-bt5rk"
Sep 17 03:50:51.222912 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:50:51.222873    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a78851c0-74a9-4e0c-9864-71b65331fcdb-istio-token\") pod \"e2e-ipp-cleanup-0ccfd0-openshift-default-7bff59778d-bt5rk\" (UID: \"a78851c0-74a9-4e0c-9864-71b65331fcdb\") " pod="openshift-ingress/e2e-ipp-cleanup-0ccfd0-openshift-default-7bff59778d-bt5rk"
Sep 17 03:50:51.222912 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:50:51.222911    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a78851c0-74a9-4e0c-9864-71b65331fcdb-workload-certs\") pod \"e2e-ipp-cleanup-0ccfd0-openshift-default-7bff59778d-bt5rk\" (UID: \"a78851c0-74a9-4e0c-9864-71b65331fcdb\") " pod="openshift-ingress/e2e-ipp-cleanup-0ccfd0-openshift-default-7bff59778d-bt5rk"
Sep 17 03:50:51.223097 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:50:51.222938    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a78851c0-74a9-4e0c-9864-71b65331fcdb-workload-socket\") pod \"e2e-ipp-cleanup-0ccfd0-openshift-default-7bff59778d-bt5rk\" (UID: \"a78851c0-74a9-4e0c-9864-71b65331fcdb\") " pod="openshift-ingress/e2e-ipp-cleanup-0ccfd0-openshift-default-7bff59778d-bt5rk"
Sep 17 03:50:51.223097 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:50:51.222953    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a78851c0-74a9-4e0c-9864-71b65331fcdb-istio-data\") pod \"e2e-ipp-cleanup-0ccfd0-openshift-default-7bff59778d-bt5rk\" (UID: \"a78851c0-74a9-4e0c-9864-71b65331fcdb\") " pod="openshift-ingress/e2e-ipp-cleanup-0ccfd0-openshift-default-7bff59778d-bt5rk"
Sep 17 03:50:51.223097 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:50:51.222994    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a78851c0-74a9-4e0c-9864-71b65331fcdb-istiod-ca-cert\") pod \"e2e-ipp-cleanup-0ccfd0-openshift-default-7bff59778d-bt5rk\" (UID: \"a78851c0-74a9-4e0c-9864-71b65331fcdb\") " pod="openshift-ingress/e2e-ipp-cleanup-0ccfd0-openshift-default-7bff59778d-bt5rk"
Sep 17 03:50:51.323938 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:50:51.323900    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a78851c0-74a9-4e0c-9864-71b65331fcdb-credential-socket\") pod \"e2e-ipp-cleanup-0ccfd0-openshift-default-7bff59778d-bt5rk\" (UID: \"a78851c0-74a9-4e0c-9864-71b65331fcdb\") " pod="openshift-ingress/e2e-ipp-cleanup-0ccfd0-openshift-default-7bff59778d-bt5rk"
Sep 17 03:50:51.323938 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:50:51.323942    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a78851c0-74a9-4e0c-9864-71b65331fcdb-istio-envoy\") pod \"e2e-ipp-cleanup-0ccfd0-openshift-default-7bff59778d-bt5rk\" (UID: \"a78851c0-74a9-4e0c-9864-71b65331fcdb\") " pod="openshift-ingress/e2e-ipp-cleanup-0ccfd0-openshift-default-7bff59778d-bt5rk"
Sep 17 03:50:51.324201 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:50:51.323961    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a78851c0-74a9-4e0c-9864-71b65331fcdb-istio-podinfo\") pod \"e2e-ipp-cleanup-0ccfd0-openshift-default-7bff59778d-bt5rk\" (UID: \"a78851c0-74a9-4e0c-9864-71b65331fcdb\") " pod="openshift-ingress/e2e-ipp-cleanup-0ccfd0-openshift-default-7bff59778d-bt5rk"
Sep 17 03:50:51.324201 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:50:51.323993    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59m4d\" (UniqueName: \"kubernetes.io/projected/a78851c0-74a9-4e0c-9864-71b65331fcdb-kube-api-access-59m4d\") pod \"e2e-ipp-cleanup-0ccfd0-openshift-default-7bff59778d-bt5rk\" (UID: \"a78851c0-74a9-4e0c-9864-71b65331fcdb\") " pod="openshift-ingress/e2e-ipp-cleanup-0ccfd0-openshift-default-7bff59778d-bt5rk"
Sep 17 03:50:51.324201 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:50:51.324015    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a78851c0-74a9-4e0c-9864-71b65331fcdb-istio-token\") pod \"e2e-ipp-cleanup-0ccfd0-openshift-default-7bff59778d-bt5rk\" (UID: \"a78851c0-74a9-4e0c-9864-71b65331fcdb\") " pod="openshift-ingress/e2e-ipp-cleanup-0ccfd0-openshift-default-7bff59778d-bt5rk"
Sep 17 03:50:51.324201 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:50:51.324042    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a78851c0-74a9-4e0c-9864-71b65331fcdb-workload-certs\") pod \"e2e-ipp-cleanup-0ccfd0-openshift-default-7bff59778d-bt5rk\" (UID: \"a78851c0-74a9-4e0c-9864-71b65331fcdb\") " pod="openshift-ingress/e2e-ipp-cleanup-0ccfd0-openshift-default-7bff59778d-bt5rk"
Sep 17 03:50:51.324201 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:50:51.324062    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a78851c0-74a9-4e0c-9864-71b65331fcdb-workload-socket\") pod \"e2e-ipp-cleanup-0ccfd0-openshift-default-7bff59778d-bt5rk\" (UID: \"a78851c0-74a9-4e0c-9864-71b65331fcdb\") " pod="openshift-ingress/e2e-ipp-cleanup-0ccfd0-openshift-default-7bff59778d-bt5rk"
Sep 17 03:50:51.324458 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:50:51.324299    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a78851c0-74a9-4e0c-9864-71b65331fcdb-istio-data\") pod \"e2e-ipp-cleanup-0ccfd0-openshift-default-7bff59778d-bt5rk\" (UID: \"a78851c0-74a9-4e0c-9864-71b65331fcdb\") " pod="openshift-ingress/e2e-ipp-cleanup-0ccfd0-openshift-default-7bff59778d-bt5rk"
Sep 17 03:50:51.324458 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:50:51.324357    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a78851c0-74a9-4e0c-9864-71b65331fcdb-istiod-ca-cert\") pod \"e2e-ipp-cleanup-0ccfd0-openshift-default-7bff59778d-bt5rk\" (UID: \"a78851c0-74a9-4e0c-9864-71b65331fcdb\") " pod="openshift-ingress/e2e-ipp-cleanup-0ccfd0-openshift-default-7bff59778d-bt5rk"
Sep 17 03:50:51.324547 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:50:51.324482    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a78851c0-74a9-4e0c-9864-71b65331fcdb-workload-certs\") pod \"e2e-ipp-cleanup-0ccfd0-openshift-default-7bff59778d-bt5rk\" (UID: \"a78851c0-74a9-4e0c-9864-71b65331fcdb\") " pod="openshift-ingress/e2e-ipp-cleanup-0ccfd0-openshift-default-7bff59778d-bt5rk"
Sep 17 03:50:51.324547 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:50:51.324525    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a78851c0-74a9-4e0c-9864-71b65331fcdb-workload-socket\") pod \"e2e-ipp-cleanup-0ccfd0-openshift-default-7bff59778d-bt5rk\" (UID: \"a78851c0-74a9-4e0c-9864-71b65331fcdb\") " pod="openshift-ingress/e2e-ipp-cleanup-0ccfd0-openshift-default-7bff59778d-bt5rk"
Sep 17 03:50:51.324655 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:50:51.324590    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a78851c0-74a9-4e0c-9864-71b65331fcdb-credential-socket\") pod \"e2e-ipp-cleanup-0ccfd0-openshift-default-7bff59778d-bt5rk\" (UID: \"a78851c0-74a9-4e0c-9864-71b65331fcdb\") " pod="openshift-ingress/e2e-ipp-cleanup-0ccfd0-openshift-default-7bff59778d-bt5rk"
Sep 17 03:50:51.324710 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:50:51.324689    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a78851c0-74a9-4e0c-9864-71b65331fcdb-istio-data\") pod \"e2e-ipp-cleanup-0ccfd0-openshift-default-7bff59778d-bt5rk\" (UID: \"a78851c0-74a9-4e0c-9864-71b65331fcdb\") " pod="openshift-ingress/e2e-ipp-cleanup-0ccfd0-openshift-default-7bff59778d-bt5rk"
Sep 17 03:50:51.324994 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:50:51.324974    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a78851c0-74a9-4e0c-9864-71b65331fcdb-istiod-ca-cert\") pod \"e2e-ipp-cleanup-0ccfd0-openshift-default-7bff59778d-bt5rk\" (UID: \"a78851c0-74a9-4e0c-9864-71b65331fcdb\") " pod="openshift-ingress/e2e-ipp-cleanup-0ccfd0-openshift-default-7bff59778d-bt5rk"
Sep 17 03:50:51.326322 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:50:51.326299    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a78851c0-74a9-4e0c-9864-71b65331fcdb-istio-envoy\") pod \"e2e-ipp-cleanup-0ccfd0-openshift-default-7bff59778d-bt5rk\" (UID: \"a78851c0-74a9-4e0c-9864-71b65331fcdb\") " pod="openshift-ingress/e2e-ipp-cleanup-0ccfd0-openshift-default-7bff59778d-bt5rk"
Sep 17 03:50:51.326789 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:50:51.326767    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a78851c0-74a9-4e0c-9864-71b65331fcdb-istio-podinfo\") pod \"e2e-ipp-cleanup-0ccfd0-openshift-default-7bff59778d-bt5rk\" (UID: \"a78851c0-74a9-4e0c-9864-71b65331fcdb\") " pod="openshift-ingress/e2e-ipp-cleanup-0ccfd0-openshift-default-7bff59778d-bt5rk"
Sep 17 03:50:51.331906 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:50:51.331880    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a78851c0-74a9-4e0c-9864-71b65331fcdb-istio-token\") pod \"e2e-ipp-cleanup-0ccfd0-openshift-default-7bff59778d-bt5rk\" (UID: \"a78851c0-74a9-4e0c-9864-71b65331fcdb\") " pod="openshift-ingress/e2e-ipp-cleanup-0ccfd0-openshift-default-7bff59778d-bt5rk"
Sep 17 03:50:51.332005 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:50:51.331988    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-59m4d\" (UniqueName: \"kubernetes.io/projected/a78851c0-74a9-4e0c-9864-71b65331fcdb-kube-api-access-59m4d\") pod \"e2e-ipp-cleanup-0ccfd0-openshift-default-7bff59778d-bt5rk\" (UID: \"a78851c0-74a9-4e0c-9864-71b65331fcdb\") " pod="openshift-ingress/e2e-ipp-cleanup-0ccfd0-openshift-default-7bff59778d-bt5rk"
Sep 17 03:50:51.428010 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:50:51.427917    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-0ccfd0-openshift-default-7bff59778d-bt5rk"
Sep 17 03:50:51.711399 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:50:51.711375    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-0ccfd0-openshift-default-7bff59778d-bt5rk"]
Sep 17 03:50:51.713438 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:50:51.713403    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda78851c0_74a9_4e0c_9864_71b65331fcdb.slice/crio-93e3b237774f15a47656d0016b051d5aacd74260ea46e2637d38fd818e8840e0 WatchSource:0}: Error finding container 93e3b237774f15a47656d0016b051d5aacd74260ea46e2637d38fd818e8840e0: Status 404 returned error can't find the container with id 93e3b237774f15a47656d0016b051d5aacd74260ea46e2637d38fd818e8840e0
Sep 17 03:50:51.715907 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:50:51.715874    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 03:50:51.715996 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:50:51.715940    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 03:50:51.715996 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:50:51.715970    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 03:50:52.249999 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:50:52.249959    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-0ccfd0-openshift-default-7bff59778d-bt5rk" event={"ID":"a78851c0-74a9-4e0c-9864-71b65331fcdb","Type":"ContainerStarted","Data":"83717da3f5f55c94c8425d5323fa36076aeb00026e2e8b5f4cdeb73bc05e0171"}
Sep 17 03:50:52.250420 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:50:52.250017    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-0ccfd0-openshift-default-7bff59778d-bt5rk" event={"ID":"a78851c0-74a9-4e0c-9864-71b65331fcdb","Type":"ContainerStarted","Data":"93e3b237774f15a47656d0016b051d5aacd74260ea46e2637d38fd818e8840e0"}
Sep 17 03:50:52.269482 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:50:52.269429    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-cleanup-0ccfd0-openshift-default-7bff59778d-bt5rk" podStartSLOduration=1.269415029 podStartE2EDuration="1.269415029s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 03:50:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:50:52.268105551 +0000 UTC m=+1887.734513077" watchObservedRunningTime="2026-09-17 03:50:52.269415029 +0000 UTC m=+1887.735822555"
Sep 17 03:50:52.428828 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:50:52.428793    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-0ccfd0-openshift-default-7bff59778d-bt5rk"
Sep 17 03:50:52.434058 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:50:52.434033    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-0ccfd0-openshift-default-7bff59778d-bt5rk"
Sep 17 03:50:53.255116 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:50:53.255082    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-0ccfd0-openshift-default-7bff59778d-bt5rk"
Sep 17 03:50:53.256125 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:50:53.256109    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-0ccfd0-openshift-default-7bff59778d-bt5rk"
Sep 17 03:51:14.382227 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:51:14.382189    2552 generic.go:356] "Generic (PLEG): container finished" podID="a78851c0-74a9-4e0c-9864-71b65331fcdb" containerID="83717da3f5f55c94c8425d5323fa36076aeb00026e2e8b5f4cdeb73bc05e0171" exitCode=0
Sep 17 03:51:14.382696 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:51:14.382264    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-0ccfd0-openshift-default-7bff59778d-bt5rk" event={"ID":"a78851c0-74a9-4e0c-9864-71b65331fcdb","Type":"ContainerDied","Data":"83717da3f5f55c94c8425d5323fa36076aeb00026e2e8b5f4cdeb73bc05e0171"}
Sep 17 03:51:14.382762 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:51:14.382748    2552 scope.go:117] "RemoveContainer" containerID="83717da3f5f55c94c8425d5323fa36076aeb00026e2e8b5f4cdeb73bc05e0171"
Sep 17 03:51:14.383864 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:51:14.383838    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 03:51:14.383934 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:51:14.383920    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 03:51:14.383972 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:51:14.383956    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 03:51:15.389582 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:51:15.389545    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-0ccfd0-openshift-default-7bff59778d-bt5rk" event={"ID":"a78851c0-74a9-4e0c-9864-71b65331fcdb","Type":"ContainerStarted","Data":"609a74a4bcd6e08a317d9a8ca0e2290a47ed323ddf474453b4a8d4909c8dd96f"}
Sep 17 03:51:15.429014 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:51:15.428980    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-0ccfd0-openshift-default-7bff59778d-bt5rk"
Sep 17 03:51:15.429344 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:51:15.429313    2552 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-0ccfd0-openshift-default-7bff59778d-bt5rk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Sep 17 03:51:15.429550 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:51:15.429524    2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-0ccfd0-openshift-default-7bff59778d-bt5rk" podUID="a78851c0-74a9-4e0c-9864-71b65331fcdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Sep 17 03:51:16.429074 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:51:16.428589    2552 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-0ccfd0-openshift-default-7bff59778d-bt5rk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Sep 17 03:51:16.429074 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:51:16.428688    2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-0ccfd0-openshift-default-7bff59778d-bt5rk" podUID="a78851c0-74a9-4e0c-9864-71b65331fcdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Sep 17 03:51:17.428378 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:51:17.428344    2552 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-0ccfd0-openshift-default-7bff59778d-bt5rk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Sep 17 03:51:17.428706 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:51:17.428671    2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-0ccfd0-openshift-default-7bff59778d-bt5rk" podUID="a78851c0-74a9-4e0c-9864-71b65331fcdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Sep 17 03:51:18.434098 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:51:18.434058    2552 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-0ccfd0-openshift-default-7bff59778d-bt5rk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Sep 17 03:51:18.434718 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:51:18.434676    2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-0ccfd0-openshift-default-7bff59778d-bt5rk" podUID="a78851c0-74a9-4e0c-9864-71b65331fcdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Sep 17 03:51:19.428880 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:51:19.428842    2552 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-0ccfd0-openshift-default-7bff59778d-bt5rk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Sep 17 03:51:19.429098 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:51:19.428919    2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-0ccfd0-openshift-default-7bff59778d-bt5rk" podUID="a78851c0-74a9-4e0c-9864-71b65331fcdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Sep 17 03:51:19.976699 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:51:19.976662    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-0ccfd0-7cwgx"]
Sep 17 03:51:19.980453 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:51:19.980428    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-0ccfd0-7cwgx"
Sep 17 03:51:19.983517 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:51:19.983457    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8vwnw\""
Sep 17 03:51:19.988197 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:51:19.988167    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-0ccfd0-7cwgx"]
Sep 17 03:51:20.113014 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:51:20.112962    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ecfeb30-4a72-44e8-ab67-21e3a9be8ecb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-0ccfd0-7cwgx\" (UID: \"4ecfeb30-4a72-44e8-ab67-21e3a9be8ecb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-0ccfd0-7cwgx"
Sep 17 03:51:20.214316 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:51:20.214279    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ecfeb30-4a72-44e8-ab67-21e3a9be8ecb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-0ccfd0-7cwgx\" (UID: \"4ecfeb30-4a72-44e8-ab67-21e3a9be8ecb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-0ccfd0-7cwgx"
Sep 17 03:51:20.215032 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:51:20.215003    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ecfeb30-4a72-44e8-ab67-21e3a9be8ecb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-0ccfd0-7cwgx\" (UID: \"4ecfeb30-4a72-44e8-ab67-21e3a9be8ecb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-0ccfd0-7cwgx"
Sep 17 03:51:20.292993 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:51:20.292959    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-0ccfd0-7cwgx"
Sep 17 03:51:20.429246 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:51:20.429212    2552 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-0ccfd0-openshift-default-7bff59778d-bt5rk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Sep 17 03:51:20.429417 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:51:20.429269    2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-0ccfd0-openshift-default-7bff59778d-bt5rk" podUID="a78851c0-74a9-4e0c-9864-71b65331fcdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Sep 17 03:51:20.589075 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:51:20.589050    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-0ccfd0-7cwgx"]
Sep 17 03:51:20.592284 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:51:20.592255    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ecfeb30_4a72_44e8_ab67_21e3a9be8ecb.slice/crio-44a43632a8b59d476aa870642b3970920986afdeefb836f1c5743e0630a7c640 WatchSource:0}: Error finding container 44a43632a8b59d476aa870642b3970920986afdeefb836f1c5743e0630a7c640: Status 404 returned error can't find the container with id 44a43632a8b59d476aa870642b3970920986afdeefb836f1c5743e0630a7c640
Sep 17 03:51:21.429740 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:51:21.429708    2552 generic.go:356] "Generic (PLEG): container finished" podID="4ecfeb30-4a72-44e8-ab67-21e3a9be8ecb" containerID="c891888b7fba48001e6bd5114ada486ce62ec217efce7d164db37615d3e0d6e3" exitCode=0
Sep 17 03:51:21.430200 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:51:21.429790    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-0ccfd0-7cwgx" event={"ID":"4ecfeb30-4a72-44e8-ab67-21e3a9be8ecb","Type":"ContainerDied","Data":"c891888b7fba48001e6bd5114ada486ce62ec217efce7d164db37615d3e0d6e3"}
Sep 17 03:51:21.430200 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:51:21.429845    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-0ccfd0-7cwgx" event={"ID":"4ecfeb30-4a72-44e8-ab67-21e3a9be8ecb","Type":"ContainerStarted","Data":"44a43632a8b59d476aa870642b3970920986afdeefb836f1c5743e0630a7c640"}
Sep 17 03:51:21.432208 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:51:21.432185    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-0ccfd0-openshift-default-7bff59778d-bt5rk"
Sep 17 03:51:21.432449 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:51:21.432435    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-0ccfd0-openshift-default-7bff59778d-bt5rk"
Sep 17 03:51:21.433129 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:51:21.433111    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-0ccfd0-openshift-default-7bff59778d-bt5rk"
Sep 17 03:51:22.574351 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:51:22.574330    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-0ccfd0-7cwgx"
Sep 17 03:51:22.739974 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:51:22.739891    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ecfeb30-4a72-44e8-ab67-21e3a9be8ecb-maas-api-service-ca\") pod \"4ecfeb30-4a72-44e8-ab67-21e3a9be8ecb\" (UID: \"4ecfeb30-4a72-44e8-ab67-21e3a9be8ecb\") "
Sep 17 03:51:22.740309 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:51:22.740278    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ecfeb30-4a72-44e8-ab67-21e3a9be8ecb-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4ecfeb30-4a72-44e8-ab67-21e3a9be8ecb" (UID: "4ecfeb30-4a72-44e8-ab67-21e3a9be8ecb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 03:51:22.840859 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:51:22.840829    2552 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ecfeb30-4a72-44e8-ab67-21e3a9be8ecb-maas-api-service-ca\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:51:23.442192 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:51:23.442159    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-0ccfd0-7cwgx"
Sep 17 03:51:23.442428 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:51:23.442194    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-0ccfd0-7cwgx" event={"ID":"4ecfeb30-4a72-44e8-ab67-21e3a9be8ecb","Type":"ContainerDied","Data":"44a43632a8b59d476aa870642b3970920986afdeefb836f1c5743e0630a7c640"}
Sep 17 03:51:23.442428 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:51:23.442255    2552 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="44a43632a8b59d476aa870642b3970920986afdeefb836f1c5743e0630a7c640"
Sep 17 03:51:25.519090 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:51:25.519058    2552 scope.go:117] "RemoveContainer" containerID="aa80281c6e1a2c9e90b30b8f8f1595edd9422fc2ab5309359893af80c50040b8"
Sep 17 03:51:25.527841 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:51:25.527822    2552 scope.go:117] "RemoveContainer" containerID="8ef4202747f881eef14d29a0cfad8a59ca782bbe33af4d69b0b54d042368d5b4"
Sep 17 03:51:25.535452 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:51:25.535432    2552 scope.go:117] "RemoveContainer" containerID="33123885a34b4b19e3989cc924465ea977034ceb152861c7f97c102fdf3b5b7e"
Sep 17 03:51:25.542792 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:51:25.542775    2552 scope.go:117] "RemoveContainer" containerID="45633a0fe5a648090bff8efc059be5df69826dd672974a6bea66a5d5d79564c8"
Sep 17 03:51:26.194071 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:51:26.194037    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-0ccfd0-7cwgx"]
Sep 17 03:51:26.196855 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:51:26.196829    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-0ccfd0-7cwgx"]
Sep 17 03:51:26.986062 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:51:26.986028    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4ecfeb30-4a72-44e8-ab67-21e3a9be8ecb" path="/var/lib/kubelet/pods/4ecfeb30-4a72-44e8-ab67-21e3a9be8ecb/volumes"
Sep 17 03:51:32.539070 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:51:32.539039    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-0ccfd0-openshift-default-7bff59778d-bt5rk"]
Sep 17 03:51:32.539910 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:51:32.539224    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-cleanup-0ccfd0-openshift-default-7bff59778d-bt5rk" podUID="a78851c0-74a9-4e0c-9864-71b65331fcdb" containerName="istio-proxy" containerID="cri-o://609a74a4bcd6e08a317d9a8ca0e2290a47ed323ddf474453b4a8d4909c8dd96f" gracePeriod=30
Sep 17 03:51:36.434798 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:51:36.434756    2552 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-0ccfd0-openshift-default-7bff59778d-bt5rk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 17 03:51:36.435179 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:51:36.434842    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-cleanup-0ccfd0-openshift-default-7bff59778d-bt5rk" podUID="a78851c0-74a9-4e0c-9864-71b65331fcdb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 03:51:36.469815 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:51:36.469785    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d0ab1a-ngp76"]
Sep 17 03:51:36.470199 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:51:36.470187    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ecfeb30-4a72-44e8-ab67-21e3a9be8ecb" containerName="revoke-keys"
Sep 17 03:51:36.470243 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:51:36.470201    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ecfeb30-4a72-44e8-ab67-21e3a9be8ecb" containerName="revoke-keys"
Sep 17 03:51:36.470278 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:51:36.470266    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ecfeb30-4a72-44e8-ab67-21e3a9be8ecb" containerName="revoke-keys"
Sep 17 03:51:36.473262 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:51:36.473246    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d0ab1a-ngp76"
Sep 17 03:51:36.476311 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:51:36.476293    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8vwnw\""
Sep 17 03:51:36.489066 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:51:36.489045    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d0ab1a-ngp76"]
Sep 17 03:51:36.575320 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:51:36.575287    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1afd61b-bf80-4b36-993a-433f7356c372-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-d0ab1a-ngp76\" (UID: \"d1afd61b-bf80-4b36-993a-433f7356c372\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d0ab1a-ngp76"
Sep 17 03:51:36.676918 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:51:36.676875    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1afd61b-bf80-4b36-993a-433f7356c372-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-d0ab1a-ngp76\" (UID: \"d1afd61b-bf80-4b36-993a-433f7356c372\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d0ab1a-ngp76"
Sep 17 03:51:36.677534 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:51:36.677511    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1afd61b-bf80-4b36-993a-433f7356c372-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-d0ab1a-ngp76\" (UID: \"d1afd61b-bf80-4b36-993a-433f7356c372\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d0ab1a-ngp76"
Sep 17 03:51:36.783856 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:51:36.783822    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d0ab1a-ngp76"
Sep 17 03:51:37.058118 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:51:37.058093    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d0ab1a-ngp76"]
Sep 17 03:51:37.059901 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:51:37.059870    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1afd61b_bf80_4b36_993a_433f7356c372.slice/crio-be2bf38c8cb3ff0655445ae2f5ea021db84edfd02cd48260fcfd78b85fb67907 WatchSource:0}: Error finding container be2bf38c8cb3ff0655445ae2f5ea021db84edfd02cd48260fcfd78b85fb67907: Status 404 returned error can't find the container with id be2bf38c8cb3ff0655445ae2f5ea021db84edfd02cd48260fcfd78b85fb67907
Sep 17 03:51:37.520858 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:51:37.520822    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d0ab1a-ngp76" event={"ID":"d1afd61b-bf80-4b36-993a-433f7356c372","Type":"ContainerStarted","Data":"a27aa5ff3eede4ca9ad46b51c97772b93fc9011b4d74af28247867b5df86737e"}
Sep 17 03:51:37.521333 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:51:37.520882    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d0ab1a-ngp76" event={"ID":"d1afd61b-bf80-4b36-993a-433f7356c372","Type":"ContainerStarted","Data":"be2bf38c8cb3ff0655445ae2f5ea021db84edfd02cd48260fcfd78b85fb67907"}
Sep 17 03:51:37.537268 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:51:37.537225    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d0ab1a-ngp76" podStartSLOduration=1.5372107289999999 podStartE2EDuration="1.537210729s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 03:51:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:51:37.53571519 +0000 UTC m=+1933.002122709" watchObservedRunningTime="2026-09-17 03:51:37.537210729 +0000 UTC m=+1933.003618255"
Sep 17 03:51:37.768052 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:51:37.768024    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-0ccfd0-openshift-default-7bff59778d-bt5rk"
Sep 17 03:51:37.786997 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:51:37.786869    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a78851c0-74a9-4e0c-9864-71b65331fcdb-workload-certs\") pod \"a78851c0-74a9-4e0c-9864-71b65331fcdb\" (UID: \"a78851c0-74a9-4e0c-9864-71b65331fcdb\") "
Sep 17 03:51:37.786997 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:51:37.786942    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a78851c0-74a9-4e0c-9864-71b65331fcdb-istio-token\") pod \"a78851c0-74a9-4e0c-9864-71b65331fcdb\" (UID: \"a78851c0-74a9-4e0c-9864-71b65331fcdb\") "
Sep 17 03:51:37.786997 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:51:37.786975    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-59m4d\" (UniqueName: \"kubernetes.io/projected/a78851c0-74a9-4e0c-9864-71b65331fcdb-kube-api-access-59m4d\") pod \"a78851c0-74a9-4e0c-9864-71b65331fcdb\" (UID: \"a78851c0-74a9-4e0c-9864-71b65331fcdb\") "
Sep 17 03:51:37.787279 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:51:37.787001    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a78851c0-74a9-4e0c-9864-71b65331fcdb-istiod-ca-cert\") pod \"a78851c0-74a9-4e0c-9864-71b65331fcdb\" (UID: \"a78851c0-74a9-4e0c-9864-71b65331fcdb\") "
Sep 17 03:51:37.787279 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:51:37.787026    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a78851c0-74a9-4e0c-9864-71b65331fcdb-istio-data\") pod \"a78851c0-74a9-4e0c-9864-71b65331fcdb\" (UID: \"a78851c0-74a9-4e0c-9864-71b65331fcdb\") "
Sep 17 03:51:37.787279 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:51:37.787064    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a78851c0-74a9-4e0c-9864-71b65331fcdb-credential-socket\") pod \"a78851c0-74a9-4e0c-9864-71b65331fcdb\" (UID: \"a78851c0-74a9-4e0c-9864-71b65331fcdb\") "
Sep 17 03:51:37.787279 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:51:37.787086    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a78851c0-74a9-4e0c-9864-71b65331fcdb-istio-podinfo\") pod \"a78851c0-74a9-4e0c-9864-71b65331fcdb\" (UID: \"a78851c0-74a9-4e0c-9864-71b65331fcdb\") "
Sep 17 03:51:37.787279 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:51:37.787136    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a78851c0-74a9-4e0c-9864-71b65331fcdb-workload-socket\") pod \"a78851c0-74a9-4e0c-9864-71b65331fcdb\" (UID: \"a78851c0-74a9-4e0c-9864-71b65331fcdb\") "
Sep 17 03:51:37.787527 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:51:37.787417    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a78851c0-74a9-4e0c-9864-71b65331fcdb-istio-envoy\") pod \"a78851c0-74a9-4e0c-9864-71b65331fcdb\" (UID: \"a78851c0-74a9-4e0c-9864-71b65331fcdb\") "
Sep 17 03:51:37.787527 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:51:37.787471    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a78851c0-74a9-4e0c-9864-71b65331fcdb-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a78851c0-74a9-4e0c-9864-71b65331fcdb" (UID: "a78851c0-74a9-4e0c-9864-71b65331fcdb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 03:51:37.788004 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:51:37.787811    2552 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a78851c0-74a9-4e0c-9864-71b65331fcdb-istiod-ca-cert\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:51:37.788004 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:51:37.787838    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a78851c0-74a9-4e0c-9864-71b65331fcdb-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a78851c0-74a9-4e0c-9864-71b65331fcdb" (UID: "a78851c0-74a9-4e0c-9864-71b65331fcdb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:51:37.788004 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:51:37.787901    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a78851c0-74a9-4e0c-9864-71b65331fcdb-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a78851c0-74a9-4e0c-9864-71b65331fcdb" (UID: "a78851c0-74a9-4e0c-9864-71b65331fcdb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:51:37.789709 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:51:37.788443    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a78851c0-74a9-4e0c-9864-71b65331fcdb-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a78851c0-74a9-4e0c-9864-71b65331fcdb" (UID: "a78851c0-74a9-4e0c-9864-71b65331fcdb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:51:37.789709 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:51:37.788517    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a78851c0-74a9-4e0c-9864-71b65331fcdb-istio-data" (OuterVolumeSpecName: "istio-data") pod "a78851c0-74a9-4e0c-9864-71b65331fcdb" (UID: "a78851c0-74a9-4e0c-9864-71b65331fcdb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:51:37.790366 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:51:37.790332    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a78851c0-74a9-4e0c-9864-71b65331fcdb-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a78851c0-74a9-4e0c-9864-71b65331fcdb" (UID: "a78851c0-74a9-4e0c-9864-71b65331fcdb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:51:37.790467 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:51:37.790404    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a78851c0-74a9-4e0c-9864-71b65331fcdb-istio-token" (OuterVolumeSpecName: "istio-token") pod "a78851c0-74a9-4e0c-9864-71b65331fcdb" (UID: "a78851c0-74a9-4e0c-9864-71b65331fcdb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 03:51:37.790873 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:51:37.790837    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a78851c0-74a9-4e0c-9864-71b65331fcdb-kube-api-access-59m4d" (OuterVolumeSpecName: "kube-api-access-59m4d") pod "a78851c0-74a9-4e0c-9864-71b65331fcdb" (UID: "a78851c0-74a9-4e0c-9864-71b65331fcdb"). InnerVolumeSpecName "kube-api-access-59m4d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 03:51:37.791050 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:51:37.791019    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a78851c0-74a9-4e0c-9864-71b65331fcdb-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a78851c0-74a9-4e0c-9864-71b65331fcdb" (UID: "a78851c0-74a9-4e0c-9864-71b65331fcdb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 17 03:51:37.888313 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:51:37.888283    2552 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a78851c0-74a9-4e0c-9864-71b65331fcdb-workload-certs\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:51:37.888313 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:51:37.888311    2552 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a78851c0-74a9-4e0c-9864-71b65331fcdb-istio-token\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:51:37.888480 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:51:37.888320    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-59m4d\" (UniqueName: \"kubernetes.io/projected/a78851c0-74a9-4e0c-9864-71b65331fcdb-kube-api-access-59m4d\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:51:37.888480 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:51:37.888329    2552 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a78851c0-74a9-4e0c-9864-71b65331fcdb-istio-data\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:51:37.888480 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:51:37.888338    2552 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a78851c0-74a9-4e0c-9864-71b65331fcdb-credential-socket\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:51:37.888480 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:51:37.888345    2552 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a78851c0-74a9-4e0c-9864-71b65331fcdb-istio-podinfo\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:51:37.888480 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:51:37.888353    2552 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a78851c0-74a9-4e0c-9864-71b65331fcdb-workload-socket\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:51:37.888480 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:51:37.888361    2552 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a78851c0-74a9-4e0c-9864-71b65331fcdb-istio-envoy\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:51:38.527399 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:51:38.527362    2552 generic.go:356] "Generic (PLEG): container finished" podID="a78851c0-74a9-4e0c-9864-71b65331fcdb" containerID="609a74a4bcd6e08a317d9a8ca0e2290a47ed323ddf474453b4a8d4909c8dd96f" exitCode=0
Sep 17 03:51:38.527901 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:51:38.527414    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-0ccfd0-openshift-default-7bff59778d-bt5rk"
Sep 17 03:51:38.527901 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:51:38.527443    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-0ccfd0-openshift-default-7bff59778d-bt5rk" event={"ID":"a78851c0-74a9-4e0c-9864-71b65331fcdb","Type":"ContainerDied","Data":"609a74a4bcd6e08a317d9a8ca0e2290a47ed323ddf474453b4a8d4909c8dd96f"}
Sep 17 03:51:38.527901 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:51:38.527495    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-0ccfd0-openshift-default-7bff59778d-bt5rk" event={"ID":"a78851c0-74a9-4e0c-9864-71b65331fcdb","Type":"ContainerDied","Data":"93e3b237774f15a47656d0016b051d5aacd74260ea46e2637d38fd818e8840e0"}
Sep 17 03:51:38.527901 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:51:38.527525    2552 scope.go:117] "RemoveContainer" containerID="609a74a4bcd6e08a317d9a8ca0e2290a47ed323ddf474453b4a8d4909c8dd96f"
Sep 17 03:51:38.529083 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:51:38.529057    2552 generic.go:356] "Generic (PLEG): container finished" podID="d1afd61b-bf80-4b36-993a-433f7356c372" containerID="a27aa5ff3eede4ca9ad46b51c97772b93fc9011b4d74af28247867b5df86737e" exitCode=0
Sep 17 03:51:38.529223 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:51:38.529133    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d0ab1a-ngp76" event={"ID":"d1afd61b-bf80-4b36-993a-433f7356c372","Type":"ContainerDied","Data":"a27aa5ff3eede4ca9ad46b51c97772b93fc9011b4d74af28247867b5df86737e"}
Sep 17 03:51:38.537013 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:51:38.536999    2552 scope.go:117] "RemoveContainer" containerID="83717da3f5f55c94c8425d5323fa36076aeb00026e2e8b5f4cdeb73bc05e0171"
Sep 17 03:51:38.544984 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:51:38.544962    2552 scope.go:117] "RemoveContainer" containerID="609a74a4bcd6e08a317d9a8ca0e2290a47ed323ddf474453b4a8d4909c8dd96f"
Sep 17 03:51:38.545236 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:51:38.545214    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"609a74a4bcd6e08a317d9a8ca0e2290a47ed323ddf474453b4a8d4909c8dd96f\": container with ID starting with 609a74a4bcd6e08a317d9a8ca0e2290a47ed323ddf474453b4a8d4909c8dd96f not found: ID does not exist" containerID="609a74a4bcd6e08a317d9a8ca0e2290a47ed323ddf474453b4a8d4909c8dd96f"
Sep 17 03:51:38.545286 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:51:38.545244    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"609a74a4bcd6e08a317d9a8ca0e2290a47ed323ddf474453b4a8d4909c8dd96f"} err="failed to get container status \"609a74a4bcd6e08a317d9a8ca0e2290a47ed323ddf474453b4a8d4909c8dd96f\": rpc error: code = NotFound desc = could not find container \"609a74a4bcd6e08a317d9a8ca0e2290a47ed323ddf474453b4a8d4909c8dd96f\": container with ID starting with 609a74a4bcd6e08a317d9a8ca0e2290a47ed323ddf474453b4a8d4909c8dd96f not found: ID does not exist"
Sep 17 03:51:38.545286 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:51:38.545261    2552 scope.go:117] "RemoveContainer" containerID="83717da3f5f55c94c8425d5323fa36076aeb00026e2e8b5f4cdeb73bc05e0171"
Sep 17 03:51:38.545507 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:51:38.545490    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83717da3f5f55c94c8425d5323fa36076aeb00026e2e8b5f4cdeb73bc05e0171\": container with ID starting with 83717da3f5f55c94c8425d5323fa36076aeb00026e2e8b5f4cdeb73bc05e0171 not found: ID does not exist" containerID="83717da3f5f55c94c8425d5323fa36076aeb00026e2e8b5f4cdeb73bc05e0171"
Sep 17 03:51:38.545566 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:51:38.545516    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83717da3f5f55c94c8425d5323fa36076aeb00026e2e8b5f4cdeb73bc05e0171"} err="failed to get container status \"83717da3f5f55c94c8425d5323fa36076aeb00026e2e8b5f4cdeb73bc05e0171\": rpc error: code = NotFound desc = could not find container \"83717da3f5f55c94c8425d5323fa36076aeb00026e2e8b5f4cdeb73bc05e0171\": container with ID starting with 83717da3f5f55c94c8425d5323fa36076aeb00026e2e8b5f4cdeb73bc05e0171 not found: ID does not exist"
Sep 17 03:51:38.560614 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:51:38.560583    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-0ccfd0-openshift-default-7bff59778d-bt5rk"]
Sep 17 03:51:38.563834 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:51:38.563809    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-0ccfd0-openshift-default-7bff59778d-bt5rk"]
Sep 17 03:51:38.985797 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:51:38.985697    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a78851c0-74a9-4e0c-9864-71b65331fcdb" path="/var/lib/kubelet/pods/a78851c0-74a9-4e0c-9864-71b65331fcdb/volumes"
Sep 17 03:51:39.668968 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:51:39.668942    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d0ab1a-ngp76"
Sep 17 03:51:39.707145 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:51:39.707109    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1afd61b-bf80-4b36-993a-433f7356c372-maas-api-service-ca\") pod \"d1afd61b-bf80-4b36-993a-433f7356c372\" (UID: \"d1afd61b-bf80-4b36-993a-433f7356c372\") "
Sep 17 03:51:39.707534 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:51:39.707503    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1afd61b-bf80-4b36-993a-433f7356c372-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d1afd61b-bf80-4b36-993a-433f7356c372" (UID: "d1afd61b-bf80-4b36-993a-433f7356c372"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 03:51:39.808401 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:51:39.808320    2552 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1afd61b-bf80-4b36-993a-433f7356c372-maas-api-service-ca\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:51:40.542469 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:51:40.542440    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d0ab1a-ngp76"
Sep 17 03:51:40.542678 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:51:40.542467    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d0ab1a-ngp76" event={"ID":"d1afd61b-bf80-4b36-993a-433f7356c372","Type":"ContainerDied","Data":"be2bf38c8cb3ff0655445ae2f5ea021db84edfd02cd48260fcfd78b85fb67907"}
Sep 17 03:51:40.542678 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:51:40.542519    2552 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="be2bf38c8cb3ff0655445ae2f5ea021db84edfd02cd48260fcfd78b85fb67907"
Sep 17 03:51:52.220481 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:51:52.220436    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d0ab1a-ngp76"]
Sep 17 03:51:52.222977 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:51:52.222951    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d0ab1a-ngp76"]
Sep 17 03:51:52.985823 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:51:52.985779    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d1afd61b-bf80-4b36-993a-433f7356c372" path="/var/lib/kubelet/pods/d1afd61b-bf80-4b36-993a-433f7356c372/volumes"
Sep 17 03:51:57.609774 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:51:57.609739    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-d0ab1a/ipp-route-d0ab1a-kserve-545c9b4f8c-p74wv"]
Sep 17 03:52:12.629381 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:52:12.629333    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-d0ab1a/ipp-route-d0ab1a-kserve-545c9b4f8c-p74wv" podUID="28c32ecd-31c3-4069-8701-0997e5146b32" containerName="main" containerID="cri-o://f254820143bd00dbde88814a170aae2dced437522380fbfd18ac121abf6b675b" gracePeriod=45
Sep 17 03:52:12.871434 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:52:12.871411    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-d0ab1a/ipp-route-d0ab1a-kserve-545c9b4f8c-p74wv"
Sep 17 03:52:12.932143 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:52:12.932054    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28c32ecd-31c3-4069-8701-0997e5146b32-dshm\") pod \"28c32ecd-31c3-4069-8701-0997e5146b32\" (UID: \"28c32ecd-31c3-4069-8701-0997e5146b32\") "
Sep 17 03:52:12.932143 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:52:12.932110    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28c32ecd-31c3-4069-8701-0997e5146b32-tls-certs\") pod \"28c32ecd-31c3-4069-8701-0997e5146b32\" (UID: \"28c32ecd-31c3-4069-8701-0997e5146b32\") "
Sep 17 03:52:12.932143 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:52:12.932143    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mbp9x\" (UniqueName: \"kubernetes.io/projected/28c32ecd-31c3-4069-8701-0997e5146b32-kube-api-access-mbp9x\") pod \"28c32ecd-31c3-4069-8701-0997e5146b32\" (UID: \"28c32ecd-31c3-4069-8701-0997e5146b32\") "
Sep 17 03:52:12.932423 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:52:12.932169    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28c32ecd-31c3-4069-8701-0997e5146b32-model-cache\") pod \"28c32ecd-31c3-4069-8701-0997e5146b32\" (UID: \"28c32ecd-31c3-4069-8701-0997e5146b32\") "
Sep 17 03:52:12.932423 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:52:12.932206    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28c32ecd-31c3-4069-8701-0997e5146b32-home\") pod \"28c32ecd-31c3-4069-8701-0997e5146b32\" (UID: \"28c32ecd-31c3-4069-8701-0997e5146b32\") "
Sep 17 03:52:12.932423 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:52:12.932286    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28c32ecd-31c3-4069-8701-0997e5146b32-tmp-dir\") pod \"28c32ecd-31c3-4069-8701-0997e5146b32\" (UID: \"28c32ecd-31c3-4069-8701-0997e5146b32\") "
Sep 17 03:52:12.932617 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:52:12.932486    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28c32ecd-31c3-4069-8701-0997e5146b32-model-cache" (OuterVolumeSpecName: "model-cache") pod "28c32ecd-31c3-4069-8701-0997e5146b32" (UID: "28c32ecd-31c3-4069-8701-0997e5146b32"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:52:12.932617 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:52:12.932534    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28c32ecd-31c3-4069-8701-0997e5146b32-home" (OuterVolumeSpecName: "home") pod "28c32ecd-31c3-4069-8701-0997e5146b32" (UID: "28c32ecd-31c3-4069-8701-0997e5146b32"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:52:12.932728 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:52:12.932657    2552 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28c32ecd-31c3-4069-8701-0997e5146b32-model-cache\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:52:12.932728 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:52:12.932675    2552 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28c32ecd-31c3-4069-8701-0997e5146b32-home\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:52:12.932728 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:52:12.932706    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28c32ecd-31c3-4069-8701-0997e5146b32-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "28c32ecd-31c3-4069-8701-0997e5146b32" (UID: "28c32ecd-31c3-4069-8701-0997e5146b32"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:52:12.934384 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:52:12.934349    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28c32ecd-31c3-4069-8701-0997e5146b32-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "28c32ecd-31c3-4069-8701-0997e5146b32" (UID: "28c32ecd-31c3-4069-8701-0997e5146b32"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 03:52:12.934498 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:52:12.934398    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28c32ecd-31c3-4069-8701-0997e5146b32-dshm" (OuterVolumeSpecName: "dshm") pod "28c32ecd-31c3-4069-8701-0997e5146b32" (UID: "28c32ecd-31c3-4069-8701-0997e5146b32"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:52:12.934580 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:52:12.934564    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28c32ecd-31c3-4069-8701-0997e5146b32-kube-api-access-mbp9x" (OuterVolumeSpecName: "kube-api-access-mbp9x") pod "28c32ecd-31c3-4069-8701-0997e5146b32" (UID: "28c32ecd-31c3-4069-8701-0997e5146b32"). InnerVolumeSpecName "kube-api-access-mbp9x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 03:52:13.033823 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:52:13.033785    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mbp9x\" (UniqueName: \"kubernetes.io/projected/28c32ecd-31c3-4069-8701-0997e5146b32-kube-api-access-mbp9x\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:52:13.033823 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:52:13.033811    2552 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28c32ecd-31c3-4069-8701-0997e5146b32-tmp-dir\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:52:13.033823 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:52:13.033821    2552 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28c32ecd-31c3-4069-8701-0997e5146b32-dshm\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:52:13.033823 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:52:13.033830    2552 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28c32ecd-31c3-4069-8701-0997e5146b32-tls-certs\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:52:13.712770 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:52:13.712729    2552 generic.go:356] "Generic (PLEG): container finished" podID="28c32ecd-31c3-4069-8701-0997e5146b32" containerID="f254820143bd00dbde88814a170aae2dced437522380fbfd18ac121abf6b675b" exitCode=0
Sep 17 03:52:13.713175 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:52:13.712779    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-d0ab1a/ipp-route-d0ab1a-kserve-545c9b4f8c-p74wv"
Sep 17 03:52:13.713175 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:52:13.712813    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-d0ab1a/ipp-route-d0ab1a-kserve-545c9b4f8c-p74wv" event={"ID":"28c32ecd-31c3-4069-8701-0997e5146b32","Type":"ContainerDied","Data":"f254820143bd00dbde88814a170aae2dced437522380fbfd18ac121abf6b675b"}
Sep 17 03:52:13.713175 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:52:13.712859    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-d0ab1a/ipp-route-d0ab1a-kserve-545c9b4f8c-p74wv" event={"ID":"28c32ecd-31c3-4069-8701-0997e5146b32","Type":"ContainerDied","Data":"58b8c58d5fcb2d475e8495010afec0f11642bff25d7af4af7c3f79d99737b7bf"}
Sep 17 03:52:13.713175 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:52:13.712879    2552 scope.go:117] "RemoveContainer" containerID="f254820143bd00dbde88814a170aae2dced437522380fbfd18ac121abf6b675b"
Sep 17 03:52:13.725177 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:52:13.725153    2552 scope.go:117] "RemoveContainer" containerID="f254820143bd00dbde88814a170aae2dced437522380fbfd18ac121abf6b675b"
Sep 17 03:52:13.725419 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:52:13.725395    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f254820143bd00dbde88814a170aae2dced437522380fbfd18ac121abf6b675b\": container with ID starting with f254820143bd00dbde88814a170aae2dced437522380fbfd18ac121abf6b675b not found: ID does not exist" containerID="f254820143bd00dbde88814a170aae2dced437522380fbfd18ac121abf6b675b"
Sep 17 03:52:13.725496 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:52:13.725429    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f254820143bd00dbde88814a170aae2dced437522380fbfd18ac121abf6b675b"} err="failed to get container status \"f254820143bd00dbde88814a170aae2dced437522380fbfd18ac121abf6b675b\": rpc error: code = NotFound desc = could not find container \"f254820143bd00dbde88814a170aae2dced437522380fbfd18ac121abf6b675b\": container with ID starting with f254820143bd00dbde88814a170aae2dced437522380fbfd18ac121abf6b675b not found: ID does not exist"
Sep 17 03:52:13.729201 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:52:13.729178    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-d0ab1a/ipp-route-d0ab1a-kserve-545c9b4f8c-p74wv"]
Sep 17 03:52:13.733181 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:52:13.733161    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-d0ab1a/ipp-route-d0ab1a-kserve-545c9b4f8c-p74wv"]
Sep 17 03:52:14.986001 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:52:14.985967    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="28c32ecd-31c3-4069-8701-0997e5146b32" path="/var/lib/kubelet/pods/28c32ecd-31c3-4069-8701-0997e5146b32/volumes"
Sep 17 03:53:21.097887 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:53:21.097851    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-84cd5d-openshift-default-66478d5f7b-dgwrz"]
Sep 17 03:53:21.098478 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:53:21.098071    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-84cd5d-openshift-default-66478d5f7b-dgwrz" podUID="8ccecfb8-e85c-4656-b88d-0551baf776dd" containerName="istio-proxy" containerID="cri-o://296354264a4e70a0b5465e04208236deaab5beafc80d7cce9554e0fd8e0cc8cf" gracePeriod=30
Sep 17 03:53:26.346817 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:53:26.346792    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-84cd5d-openshift-default-66478d5f7b-dgwrz"
Sep 17 03:53:26.416388 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:53:26.416303    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ccecfb8-e85c-4656-b88d-0551baf776dd-istio-envoy\") pod \"8ccecfb8-e85c-4656-b88d-0551baf776dd\" (UID: \"8ccecfb8-e85c-4656-b88d-0551baf776dd\") "
Sep 17 03:53:26.416388 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:53:26.416365    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ccecfb8-e85c-4656-b88d-0551baf776dd-workload-socket\") pod \"8ccecfb8-e85c-4656-b88d-0551baf776dd\" (UID: \"8ccecfb8-e85c-4656-b88d-0551baf776dd\") "
Sep 17 03:53:26.416642 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:53:26.416392    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ccecfb8-e85c-4656-b88d-0551baf776dd-istiod-ca-cert\") pod \"8ccecfb8-e85c-4656-b88d-0551baf776dd\" (UID: \"8ccecfb8-e85c-4656-b88d-0551baf776dd\") "
Sep 17 03:53:26.416642 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:53:26.416421    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ccecfb8-e85c-4656-b88d-0551baf776dd-istio-data\") pod \"8ccecfb8-e85c-4656-b88d-0551baf776dd\" (UID: \"8ccecfb8-e85c-4656-b88d-0551baf776dd\") "
Sep 17 03:53:26.416642 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:53:26.416439    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ccecfb8-e85c-4656-b88d-0551baf776dd-workload-certs\") pod \"8ccecfb8-e85c-4656-b88d-0551baf776dd\" (UID: \"8ccecfb8-e85c-4656-b88d-0551baf776dd\") "
Sep 17 03:53:26.416642 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:53:26.416459    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ccecfb8-e85c-4656-b88d-0551baf776dd-istio-podinfo\") pod \"8ccecfb8-e85c-4656-b88d-0551baf776dd\" (UID: \"8ccecfb8-e85c-4656-b88d-0551baf776dd\") "
Sep 17 03:53:26.416642 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:53:26.416549    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ccecfb8-e85c-4656-b88d-0551baf776dd-istio-token\") pod \"8ccecfb8-e85c-4656-b88d-0551baf776dd\" (UID: \"8ccecfb8-e85c-4656-b88d-0551baf776dd\") "
Sep 17 03:53:26.416642 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:53:26.416621    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ccecfb8-e85c-4656-b88d-0551baf776dd-credential-socket\") pod \"8ccecfb8-e85c-4656-b88d-0551baf776dd\" (UID: \"8ccecfb8-e85c-4656-b88d-0551baf776dd\") "
Sep 17 03:53:26.416968 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:53:26.416675    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dxkgf\" (UniqueName: \"kubernetes.io/projected/8ccecfb8-e85c-4656-b88d-0551baf776dd-kube-api-access-dxkgf\") pod \"8ccecfb8-e85c-4656-b88d-0551baf776dd\" (UID: \"8ccecfb8-e85c-4656-b88d-0551baf776dd\") "
Sep 17 03:53:26.416968 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:53:26.416707    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ccecfb8-e85c-4656-b88d-0551baf776dd-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8ccecfb8-e85c-4656-b88d-0551baf776dd" (UID: "8ccecfb8-e85c-4656-b88d-0551baf776dd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:53:26.416968 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:53:26.416930    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ccecfb8-e85c-4656-b88d-0551baf776dd-istio-data" (OuterVolumeSpecName: "istio-data") pod "8ccecfb8-e85c-4656-b88d-0551baf776dd" (UID: "8ccecfb8-e85c-4656-b88d-0551baf776dd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:53:26.416968 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:53:26.416940    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ccecfb8-e85c-4656-b88d-0551baf776dd-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8ccecfb8-e85c-4656-b88d-0551baf776dd" (UID: "8ccecfb8-e85c-4656-b88d-0551baf776dd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 03:53:26.416968 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:53:26.416942    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ccecfb8-e85c-4656-b88d-0551baf776dd-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8ccecfb8-e85c-4656-b88d-0551baf776dd" (UID: "8ccecfb8-e85c-4656-b88d-0551baf776dd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:53:26.417224 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:53:26.417099    2552 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ccecfb8-e85c-4656-b88d-0551baf776dd-workload-socket\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:53:26.417224 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:53:26.417119    2552 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ccecfb8-e85c-4656-b88d-0551baf776dd-istiod-ca-cert\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:53:26.417224 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:53:26.417133    2552 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ccecfb8-e85c-4656-b88d-0551baf776dd-istio-data\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:53:26.417224 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:53:26.417145    2552 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ccecfb8-e85c-4656-b88d-0551baf776dd-credential-socket\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:53:26.417224 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:53:26.417204    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ccecfb8-e85c-4656-b88d-0551baf776dd-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8ccecfb8-e85c-4656-b88d-0551baf776dd" (UID: "8ccecfb8-e85c-4656-b88d-0551baf776dd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:53:26.418695 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:53:26.418659    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ccecfb8-e85c-4656-b88d-0551baf776dd-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8ccecfb8-e85c-4656-b88d-0551baf776dd" (UID: "8ccecfb8-e85c-4656-b88d-0551baf776dd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:53:26.418822 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:53:26.418790    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ccecfb8-e85c-4656-b88d-0551baf776dd-istio-token" (OuterVolumeSpecName: "istio-token") pod "8ccecfb8-e85c-4656-b88d-0551baf776dd" (UID: "8ccecfb8-e85c-4656-b88d-0551baf776dd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 03:53:26.418961 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:53:26.418942    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8ccecfb8-e85c-4656-b88d-0551baf776dd-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8ccecfb8-e85c-4656-b88d-0551baf776dd" (UID: "8ccecfb8-e85c-4656-b88d-0551baf776dd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 17 03:53:26.419222 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:53:26.419200    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ccecfb8-e85c-4656-b88d-0551baf776dd-kube-api-access-dxkgf" (OuterVolumeSpecName: "kube-api-access-dxkgf") pod "8ccecfb8-e85c-4656-b88d-0551baf776dd" (UID: "8ccecfb8-e85c-4656-b88d-0551baf776dd"). InnerVolumeSpecName "kube-api-access-dxkgf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 03:53:26.517758 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:53:26.517728    2552 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ccecfb8-e85c-4656-b88d-0551baf776dd-istio-envoy\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:53:26.517758 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:53:26.517751    2552 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ccecfb8-e85c-4656-b88d-0551baf776dd-workload-certs\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:53:26.517758 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:53:26.517761    2552 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ccecfb8-e85c-4656-b88d-0551baf776dd-istio-podinfo\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:53:26.517957 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:53:26.517769    2552 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ccecfb8-e85c-4656-b88d-0551baf776dd-istio-token\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:53:26.517957 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:53:26.517777    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dxkgf\" (UniqueName: \"kubernetes.io/projected/8ccecfb8-e85c-4656-b88d-0551baf776dd-kube-api-access-dxkgf\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:53:27.084955 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:53:27.084920    2552 generic.go:356] "Generic (PLEG): container finished" podID="8ccecfb8-e85c-4656-b88d-0551baf776dd" containerID="296354264a4e70a0b5465e04208236deaab5beafc80d7cce9554e0fd8e0cc8cf" exitCode=0
Sep 17 03:53:27.085113 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:53:27.084966    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-84cd5d-openshift-default-66478d5f7b-dgwrz"
Sep 17 03:53:27.085113 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:53:27.085002    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-84cd5d-openshift-default-66478d5f7b-dgwrz" event={"ID":"8ccecfb8-e85c-4656-b88d-0551baf776dd","Type":"ContainerDied","Data":"296354264a4e70a0b5465e04208236deaab5beafc80d7cce9554e0fd8e0cc8cf"}
Sep 17 03:53:27.085113 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:53:27.085058    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-84cd5d-openshift-default-66478d5f7b-dgwrz" event={"ID":"8ccecfb8-e85c-4656-b88d-0551baf776dd","Type":"ContainerDied","Data":"10a0c93fbd9ebb0c5bbb0aa1bb819dab93dc81a8e86a5d1b94ba4755ab050904"}
Sep 17 03:53:27.085113 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:53:27.085080    2552 scope.go:117] "RemoveContainer" containerID="296354264a4e70a0b5465e04208236deaab5beafc80d7cce9554e0fd8e0cc8cf"
Sep 17 03:53:27.093901 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:53:27.093869    2552 scope.go:117] "RemoveContainer" containerID="bfc2000ad9136e1cf9fe59c1d9c86a12aeb3a9c8e5d258572eeb352a1fc2755d"
Sep 17 03:53:27.101849 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:53:27.101822    2552 scope.go:117] "RemoveContainer" containerID="296354264a4e70a0b5465e04208236deaab5beafc80d7cce9554e0fd8e0cc8cf"
Sep 17 03:53:27.102102 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:53:27.102082    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"296354264a4e70a0b5465e04208236deaab5beafc80d7cce9554e0fd8e0cc8cf\": container with ID starting with 296354264a4e70a0b5465e04208236deaab5beafc80d7cce9554e0fd8e0cc8cf not found: ID does not exist" containerID="296354264a4e70a0b5465e04208236deaab5beafc80d7cce9554e0fd8e0cc8cf"
Sep 17 03:53:27.102157 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:53:27.102110    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"296354264a4e70a0b5465e04208236deaab5beafc80d7cce9554e0fd8e0cc8cf"} err="failed to get container status \"296354264a4e70a0b5465e04208236deaab5beafc80d7cce9554e0fd8e0cc8cf\": rpc error: code = NotFound desc = could not find container \"296354264a4e70a0b5465e04208236deaab5beafc80d7cce9554e0fd8e0cc8cf\": container with ID starting with 296354264a4e70a0b5465e04208236deaab5beafc80d7cce9554e0fd8e0cc8cf not found: ID does not exist"
Sep 17 03:53:27.102157 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:53:27.102128    2552 scope.go:117] "RemoveContainer" containerID="bfc2000ad9136e1cf9fe59c1d9c86a12aeb3a9c8e5d258572eeb352a1fc2755d"
Sep 17 03:53:27.102390 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:53:27.102373    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bfc2000ad9136e1cf9fe59c1d9c86a12aeb3a9c8e5d258572eeb352a1fc2755d\": container with ID starting with bfc2000ad9136e1cf9fe59c1d9c86a12aeb3a9c8e5d258572eeb352a1fc2755d not found: ID does not exist" containerID="bfc2000ad9136e1cf9fe59c1d9c86a12aeb3a9c8e5d258572eeb352a1fc2755d"
Sep 17 03:53:27.102450 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:53:27.102399    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfc2000ad9136e1cf9fe59c1d9c86a12aeb3a9c8e5d258572eeb352a1fc2755d"} err="failed to get container status \"bfc2000ad9136e1cf9fe59c1d9c86a12aeb3a9c8e5d258572eeb352a1fc2755d\": rpc error: code = NotFound desc = could not find container \"bfc2000ad9136e1cf9fe59c1d9c86a12aeb3a9c8e5d258572eeb352a1fc2755d\": container with ID starting with bfc2000ad9136e1cf9fe59c1d9c86a12aeb3a9c8e5d258572eeb352a1fc2755d not found: ID does not exist"
Sep 17 03:53:27.109102 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:53:27.109081    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-84cd5d-openshift-default-66478d5f7b-dgwrz"]
Sep 17 03:53:27.113071 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:53:27.113052    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-84cd5d-openshift-default-66478d5f7b-dgwrz"]
Sep 17 03:53:28.986125 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:53:28.986087    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8ccecfb8-e85c-4656-b88d-0551baf776dd" path="/var/lib/kubelet/pods/8ccecfb8-e85c-4656-b88d-0551baf776dd/volumes"
Sep 17 03:53:38.581493 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:53:38.581464    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-22e045-openshift-default-5f598cd8fb-zz7g6"]
Sep 17 03:53:38.581948 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:53:38.581678    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-22e045-openshift-default-5f598cd8fb-zz7g6" podUID="c4d4225c-5283-4143-a89e-9c2fbc0eb172" containerName="istio-proxy" containerID="cri-o://44455b80e068b92576da9d313a0f7409fff0b057cae9bb6c6a045561bd1e5588" gracePeriod=30
Sep 17 03:53:43.828103 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:53:43.828080    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-22e045-openshift-default-5f598cd8fb-zz7g6"
Sep 17 03:53:43.875490 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:53:43.875422    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4d4225c-5283-4143-a89e-9c2fbc0eb172-credential-socket\") pod \"c4d4225c-5283-4143-a89e-9c2fbc0eb172\" (UID: \"c4d4225c-5283-4143-a89e-9c2fbc0eb172\") "
Sep 17 03:53:43.875490 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:53:43.875455    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4d4225c-5283-4143-a89e-9c2fbc0eb172-istio-envoy\") pod \"c4d4225c-5283-4143-a89e-9c2fbc0eb172\" (UID: \"c4d4225c-5283-4143-a89e-9c2fbc0eb172\") "
Sep 17 03:53:43.875490 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:53:43.875472    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4d4225c-5283-4143-a89e-9c2fbc0eb172-workload-certs\") pod \"c4d4225c-5283-4143-a89e-9c2fbc0eb172\" (UID: \"c4d4225c-5283-4143-a89e-9c2fbc0eb172\") "
Sep 17 03:53:43.875490 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:53:43.875489    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4d4225c-5283-4143-a89e-9c2fbc0eb172-istio-podinfo\") pod \"c4d4225c-5283-4143-a89e-9c2fbc0eb172\" (UID: \"c4d4225c-5283-4143-a89e-9c2fbc0eb172\") "
Sep 17 03:53:43.875810 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:53:43.875515    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4d4225c-5283-4143-a89e-9c2fbc0eb172-istio-token\") pod \"c4d4225c-5283-4143-a89e-9c2fbc0eb172\" (UID: \"c4d4225c-5283-4143-a89e-9c2fbc0eb172\") "
Sep 17 03:53:43.875810 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:53:43.875560    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-67x7d\" (UniqueName: \"kubernetes.io/projected/c4d4225c-5283-4143-a89e-9c2fbc0eb172-kube-api-access-67x7d\") pod \"c4d4225c-5283-4143-a89e-9c2fbc0eb172\" (UID: \"c4d4225c-5283-4143-a89e-9c2fbc0eb172\") "
Sep 17 03:53:43.875810 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:53:43.875631    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4d4225c-5283-4143-a89e-9c2fbc0eb172-istiod-ca-cert\") pod \"c4d4225c-5283-4143-a89e-9c2fbc0eb172\" (UID: \"c4d4225c-5283-4143-a89e-9c2fbc0eb172\") "
Sep 17 03:53:43.875810 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:53:43.875709    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4d4225c-5283-4143-a89e-9c2fbc0eb172-istio-data\") pod \"c4d4225c-5283-4143-a89e-9c2fbc0eb172\" (UID: \"c4d4225c-5283-4143-a89e-9c2fbc0eb172\") "
Sep 17 03:53:43.875810 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:53:43.875743    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4d4225c-5283-4143-a89e-9c2fbc0eb172-workload-socket\") pod \"c4d4225c-5283-4143-a89e-9c2fbc0eb172\" (UID: \"c4d4225c-5283-4143-a89e-9c2fbc0eb172\") "
Sep 17 03:53:43.875810 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:53:43.875777    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4d4225c-5283-4143-a89e-9c2fbc0eb172-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c4d4225c-5283-4143-a89e-9c2fbc0eb172" (UID: "c4d4225c-5283-4143-a89e-9c2fbc0eb172"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:53:43.876142 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:53:43.875838    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4d4225c-5283-4143-a89e-9c2fbc0eb172-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c4d4225c-5283-4143-a89e-9c2fbc0eb172" (UID: "c4d4225c-5283-4143-a89e-9c2fbc0eb172"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:53:43.876142 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:53:43.876055    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4d4225c-5283-4143-a89e-9c2fbc0eb172-istio-data" (OuterVolumeSpecName: "istio-data") pod "c4d4225c-5283-4143-a89e-9c2fbc0eb172" (UID: "c4d4225c-5283-4143-a89e-9c2fbc0eb172"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:53:43.876142 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:53:43.876109    2552 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4d4225c-5283-4143-a89e-9c2fbc0eb172-credential-socket\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:53:43.876142 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:53:43.876130    2552 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4d4225c-5283-4143-a89e-9c2fbc0eb172-workload-certs\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:53:43.876353 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:53:43.876140    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4d4225c-5283-4143-a89e-9c2fbc0eb172-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c4d4225c-5283-4143-a89e-9c2fbc0eb172" (UID: "c4d4225c-5283-4143-a89e-9c2fbc0eb172"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 03:53:43.876353 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:53:43.876323    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4d4225c-5283-4143-a89e-9c2fbc0eb172-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c4d4225c-5283-4143-a89e-9c2fbc0eb172" (UID: "c4d4225c-5283-4143-a89e-9c2fbc0eb172"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:53:43.877916 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:53:43.877869    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4d4225c-5283-4143-a89e-9c2fbc0eb172-kube-api-access-67x7d" (OuterVolumeSpecName: "kube-api-access-67x7d") pod "c4d4225c-5283-4143-a89e-9c2fbc0eb172" (UID: "c4d4225c-5283-4143-a89e-9c2fbc0eb172"). InnerVolumeSpecName "kube-api-access-67x7d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 03:53:43.877916 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:53:43.877888    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4d4225c-5283-4143-a89e-9c2fbc0eb172-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c4d4225c-5283-4143-a89e-9c2fbc0eb172" (UID: "c4d4225c-5283-4143-a89e-9c2fbc0eb172"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:53:43.878233 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:53:43.878206    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4d4225c-5283-4143-a89e-9c2fbc0eb172-istio-token" (OuterVolumeSpecName: "istio-token") pod "c4d4225c-5283-4143-a89e-9c2fbc0eb172" (UID: "c4d4225c-5283-4143-a89e-9c2fbc0eb172"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 03:53:43.878328 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:53:43.878290    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c4d4225c-5283-4143-a89e-9c2fbc0eb172-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c4d4225c-5283-4143-a89e-9c2fbc0eb172" (UID: "c4d4225c-5283-4143-a89e-9c2fbc0eb172"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 17 03:53:43.977559 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:53:43.977527    2552 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4d4225c-5283-4143-a89e-9c2fbc0eb172-istiod-ca-cert\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:53:43.977559 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:53:43.977553    2552 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4d4225c-5283-4143-a89e-9c2fbc0eb172-istio-data\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:53:43.977559 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:53:43.977562    2552 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4d4225c-5283-4143-a89e-9c2fbc0eb172-workload-socket\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:53:43.977559 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:53:43.977570    2552 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4d4225c-5283-4143-a89e-9c2fbc0eb172-istio-envoy\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:53:43.977831 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:53:43.977579    2552 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4d4225c-5283-4143-a89e-9c2fbc0eb172-istio-podinfo\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:53:43.977831 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:53:43.977587    2552 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4d4225c-5283-4143-a89e-9c2fbc0eb172-istio-token\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:53:43.977831 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:53:43.977612    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67x7d\" (UniqueName: \"kubernetes.io/projected/c4d4225c-5283-4143-a89e-9c2fbc0eb172-kube-api-access-67x7d\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:53:44.174099 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:53:44.174005    2552 generic.go:356] "Generic (PLEG): container finished" podID="c4d4225c-5283-4143-a89e-9c2fbc0eb172" containerID="44455b80e068b92576da9d313a0f7409fff0b057cae9bb6c6a045561bd1e5588" exitCode=0
Sep 17 03:53:44.174099 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:53:44.174051    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-22e045-openshift-default-5f598cd8fb-zz7g6"
Sep 17 03:53:44.174099 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:53:44.174091    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-22e045-openshift-default-5f598cd8fb-zz7g6" event={"ID":"c4d4225c-5283-4143-a89e-9c2fbc0eb172","Type":"ContainerDied","Data":"44455b80e068b92576da9d313a0f7409fff0b057cae9bb6c6a045561bd1e5588"}
Sep 17 03:53:44.174326 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:53:44.174141    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-22e045-openshift-default-5f598cd8fb-zz7g6" event={"ID":"c4d4225c-5283-4143-a89e-9c2fbc0eb172","Type":"ContainerDied","Data":"1736feeec2b6eeb24a6d1fde1a81775f37bed8ed84566f9fe2c856327cf1fba1"}
Sep 17 03:53:44.174326 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:53:44.174162    2552 scope.go:117] "RemoveContainer" containerID="44455b80e068b92576da9d313a0f7409fff0b057cae9bb6c6a045561bd1e5588"
Sep 17 03:53:44.187515 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:53:44.187496    2552 scope.go:117] "RemoveContainer" containerID="e1b6f851237769b107bbdca0c10b962d54e00b828de8f82dbfbc38d4138b2d4f"
Sep 17 03:53:44.195697 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:53:44.195678    2552 scope.go:117] "RemoveContainer" containerID="44455b80e068b92576da9d313a0f7409fff0b057cae9bb6c6a045561bd1e5588"
Sep 17 03:53:44.195948 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:53:44.195930    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44455b80e068b92576da9d313a0f7409fff0b057cae9bb6c6a045561bd1e5588\": container with ID starting with 44455b80e068b92576da9d313a0f7409fff0b057cae9bb6c6a045561bd1e5588 not found: ID does not exist" containerID="44455b80e068b92576da9d313a0f7409fff0b057cae9bb6c6a045561bd1e5588"
Sep 17 03:53:44.196010 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:53:44.195954    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44455b80e068b92576da9d313a0f7409fff0b057cae9bb6c6a045561bd1e5588"} err="failed to get container status \"44455b80e068b92576da9d313a0f7409fff0b057cae9bb6c6a045561bd1e5588\": rpc error: code = NotFound desc = could not find container \"44455b80e068b92576da9d313a0f7409fff0b057cae9bb6c6a045561bd1e5588\": container with ID starting with 44455b80e068b92576da9d313a0f7409fff0b057cae9bb6c6a045561bd1e5588 not found: ID does not exist"
Sep 17 03:53:44.196010 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:53:44.195969    2552 scope.go:117] "RemoveContainer" containerID="e1b6f851237769b107bbdca0c10b962d54e00b828de8f82dbfbc38d4138b2d4f"
Sep 17 03:53:44.196208 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:53:44.196190    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1b6f851237769b107bbdca0c10b962d54e00b828de8f82dbfbc38d4138b2d4f\": container with ID starting with e1b6f851237769b107bbdca0c10b962d54e00b828de8f82dbfbc38d4138b2d4f not found: ID does not exist" containerID="e1b6f851237769b107bbdca0c10b962d54e00b828de8f82dbfbc38d4138b2d4f"
Sep 17 03:53:44.196253 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:53:44.196213    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1b6f851237769b107bbdca0c10b962d54e00b828de8f82dbfbc38d4138b2d4f"} err="failed to get container status \"e1b6f851237769b107bbdca0c10b962d54e00b828de8f82dbfbc38d4138b2d4f\": rpc error: code = NotFound desc = could not find container \"e1b6f851237769b107bbdca0c10b962d54e00b828de8f82dbfbc38d4138b2d4f\": container with ID starting with e1b6f851237769b107bbdca0c10b962d54e00b828de8f82dbfbc38d4138b2d4f not found: ID does not exist"
Sep 17 03:53:44.200024 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:53:44.200003    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-22e045-openshift-default-5f598cd8fb-zz7g6"]
Sep 17 03:53:44.203649 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:53:44.203627    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-22e045-openshift-default-5f598cd8fb-zz7g6"]
Sep 17 03:53:44.985800 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:53:44.985761    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c4d4225c-5283-4143-a89e-9c2fbc0eb172" path="/var/lib/kubelet/pods/c4d4225c-5283-4143-a89e-9c2fbc0eb172/volumes"
Sep 17 03:53:49.398783 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:53:49.398695    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f6d581-55ccd4fbd4-phzll"]
Sep 17 03:53:49.399218 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:53:49.398942    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f6d581-55ccd4fbd4-phzll" podUID="c4851b22-eb50-4f84-aee3-38c7f46ed0c6" containerName="maas-api" containerID="cri-o://5ac2f719658b3e5ebb3b91a1f70ca6ec4214d7be8eba5708de3b6037244d0395" gracePeriod=30
Sep 17 03:53:49.426019 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:53:49.425961    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f6d581-29826945-5w7g2"]
Sep 17 03:53:49.428866 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:53:49.428843    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f6d581-29826945-5w7g2"]
Sep 17 03:53:49.429924 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:53:49.429899    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f6d581-55ccd4fbd4-phzll" podUID="c4851b22-eb50-4f84-aee3-38c7f46ed0c6" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.76:8443/health\": dial tcp 10.133.0.76:8443: connect: connection refused"
Sep 17 03:53:49.641876 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:53:49.641852    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f6d581-55ccd4fbd4-phzll"
Sep 17 03:53:49.729308 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:53:49.729233    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4851b22-eb50-4f84-aee3-38c7f46ed0c6-openshift-service-ca\") pod \"c4851b22-eb50-4f84-aee3-38c7f46ed0c6\" (UID: \"c4851b22-eb50-4f84-aee3-38c7f46ed0c6\") "
Sep 17 03:53:49.729308 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:53:49.729278    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8zg6z\" (UniqueName: \"kubernetes.io/projected/c4851b22-eb50-4f84-aee3-38c7f46ed0c6-kube-api-access-8zg6z\") pod \"c4851b22-eb50-4f84-aee3-38c7f46ed0c6\" (UID: \"c4851b22-eb50-4f84-aee3-38c7f46ed0c6\") "
Sep 17 03:53:49.729532 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:53:49.729312    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c4851b22-eb50-4f84-aee3-38c7f46ed0c6-metrics-certs\") pod \"c4851b22-eb50-4f84-aee3-38c7f46ed0c6\" (UID: \"c4851b22-eb50-4f84-aee3-38c7f46ed0c6\") "
Sep 17 03:53:49.729532 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:53:49.729347    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c4851b22-eb50-4f84-aee3-38c7f46ed0c6-maas-api-tls\") pod \"c4851b22-eb50-4f84-aee3-38c7f46ed0c6\" (UID: \"c4851b22-eb50-4f84-aee3-38c7f46ed0c6\") "
Sep 17 03:53:49.729681 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:53:49.729651    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4851b22-eb50-4f84-aee3-38c7f46ed0c6-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c4851b22-eb50-4f84-aee3-38c7f46ed0c6" (UID: "c4851b22-eb50-4f84-aee3-38c7f46ed0c6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 03:53:49.731465 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:53:49.731442    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4851b22-eb50-4f84-aee3-38c7f46ed0c6-kube-api-access-8zg6z" (OuterVolumeSpecName: "kube-api-access-8zg6z") pod "c4851b22-eb50-4f84-aee3-38c7f46ed0c6" (UID: "c4851b22-eb50-4f84-aee3-38c7f46ed0c6"). InnerVolumeSpecName "kube-api-access-8zg6z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 03:53:49.731566 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:53:49.731496    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4851b22-eb50-4f84-aee3-38c7f46ed0c6-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "c4851b22-eb50-4f84-aee3-38c7f46ed0c6" (UID: "c4851b22-eb50-4f84-aee3-38c7f46ed0c6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 03:53:49.731566 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:53:49.731515    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4851b22-eb50-4f84-aee3-38c7f46ed0c6-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c4851b22-eb50-4f84-aee3-38c7f46ed0c6" (UID: "c4851b22-eb50-4f84-aee3-38c7f46ed0c6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 03:53:49.830546 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:53:49.830509    2552 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4851b22-eb50-4f84-aee3-38c7f46ed0c6-openshift-service-ca\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:53:49.830546 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:53:49.830540    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zg6z\" (UniqueName: \"kubernetes.io/projected/c4851b22-eb50-4f84-aee3-38c7f46ed0c6-kube-api-access-8zg6z\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:53:49.830770 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:53:49.830553    2552 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c4851b22-eb50-4f84-aee3-38c7f46ed0c6-metrics-certs\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:53:49.830770 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:53:49.830568    2552 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c4851b22-eb50-4f84-aee3-38c7f46ed0c6-maas-api-tls\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:53:50.209339 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:53:50.209305    2552 generic.go:356] "Generic (PLEG): container finished" podID="c4851b22-eb50-4f84-aee3-38c7f46ed0c6" containerID="5ac2f719658b3e5ebb3b91a1f70ca6ec4214d7be8eba5708de3b6037244d0395" exitCode=0
Sep 17 03:53:50.209484 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:53:50.209353    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f6d581-55ccd4fbd4-phzll"
Sep 17 03:53:50.209484 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:53:50.209383    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f6d581-55ccd4fbd4-phzll" event={"ID":"c4851b22-eb50-4f84-aee3-38c7f46ed0c6","Type":"ContainerDied","Data":"5ac2f719658b3e5ebb3b91a1f70ca6ec4214d7be8eba5708de3b6037244d0395"}
Sep 17 03:53:50.209484 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:53:50.209430    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f6d581-55ccd4fbd4-phzll" event={"ID":"c4851b22-eb50-4f84-aee3-38c7f46ed0c6","Type":"ContainerDied","Data":"c0ad593a70c284ee6c16b960b267e2ebfcd903fc58e8433664a44b2f836dc709"}
Sep 17 03:53:50.209484 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:53:50.209452    2552 scope.go:117] "RemoveContainer" containerID="5ac2f719658b3e5ebb3b91a1f70ca6ec4214d7be8eba5708de3b6037244d0395"
Sep 17 03:53:50.218937 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:53:50.218921    2552 scope.go:117] "RemoveContainer" containerID="5ac2f719658b3e5ebb3b91a1f70ca6ec4214d7be8eba5708de3b6037244d0395"
Sep 17 03:53:50.219204 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:53:50.219186    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ac2f719658b3e5ebb3b91a1f70ca6ec4214d7be8eba5708de3b6037244d0395\": container with ID starting with 5ac2f719658b3e5ebb3b91a1f70ca6ec4214d7be8eba5708de3b6037244d0395 not found: ID does not exist" containerID="5ac2f719658b3e5ebb3b91a1f70ca6ec4214d7be8eba5708de3b6037244d0395"
Sep 17 03:53:50.219260 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:53:50.219212    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ac2f719658b3e5ebb3b91a1f70ca6ec4214d7be8eba5708de3b6037244d0395"} err="failed to get container status \"5ac2f719658b3e5ebb3b91a1f70ca6ec4214d7be8eba5708de3b6037244d0395\": rpc error: code = NotFound desc = could not find container \"5ac2f719658b3e5ebb3b91a1f70ca6ec4214d7be8eba5708de3b6037244d0395\": container with ID starting with 5ac2f719658b3e5ebb3b91a1f70ca6ec4214d7be8eba5708de3b6037244d0395 not found: ID does not exist"
Sep 17 03:53:50.231392 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:53:50.231367    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f6d581-55ccd4fbd4-phzll"]
Sep 17 03:53:50.235272 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:53:50.235253    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f6d581-55ccd4fbd4-phzll"]
Sep 17 03:53:50.985670 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:53:50.985636    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c2338e9b-a64c-47d6-8b38-d47b0d148690" path="/var/lib/kubelet/pods/c2338e9b-a64c-47d6-8b38-d47b0d148690/volumes"
Sep 17 03:53:50.986190 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:53:50.986170    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c4851b22-eb50-4f84-aee3-38c7f46ed0c6" path="/var/lib/kubelet/pods/c4851b22-eb50-4f84-aee3-38c7f46ed0c6/volumes"
Sep 17 03:53:56.137566 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:53:56.137535    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-f6d581-openshift-default-6955dfcbb6-szl9x"]
Sep 17 03:53:56.138080 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:53:56.137798    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-f6d581-openshift-default-6955dfcbb6-szl9x" podUID="c871cdcf-6205-4f39-809a-7a72f976e145" containerName="istio-proxy" containerID="cri-o://a58d3fd60bb301db0a8ebf535afceb75b18756fd3293213f286aa504ab7b73fc" gracePeriod=30
Sep 17 03:54:00.981220 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:54:00.981184    2552 patch_prober.go:28] interesting pod/e2e-shared-b-w3-f6d581-openshift-default-6955dfcbb6-szl9x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 17 03:54:00.981620 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:54:00.981252    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-f6d581-openshift-default-6955dfcbb6-szl9x" podUID="c871cdcf-6205-4f39-809a-7a72f976e145" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 03:54:01.273893 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:54:01.273855    2552 generic.go:356] "Generic (PLEG): container finished" podID="c871cdcf-6205-4f39-809a-7a72f976e145" containerID="a58d3fd60bb301db0a8ebf535afceb75b18756fd3293213f286aa504ab7b73fc" exitCode=0
Sep 17 03:54:01.274029 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:54:01.273920    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-f6d581-openshift-default-6955dfcbb6-szl9x" event={"ID":"c871cdcf-6205-4f39-809a-7a72f976e145","Type":"ContainerDied","Data":"a58d3fd60bb301db0a8ebf535afceb75b18756fd3293213f286aa504ab7b73fc"}
Sep 17 03:54:01.274029 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:54:01.273986    2552 scope.go:117] "RemoveContainer" containerID="1018875442a0413e9caf6819eb3790a3bc9d92401b79aa32e7d157f378fca484"
Sep 17 03:54:01.386261 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:54:01.386239    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-f6d581-openshift-default-6955dfcbb6-szl9x"
Sep 17 03:54:01.529361 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:54:01.529285    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c871cdcf-6205-4f39-809a-7a72f976e145-istio-data\") pod \"c871cdcf-6205-4f39-809a-7a72f976e145\" (UID: \"c871cdcf-6205-4f39-809a-7a72f976e145\") "
Sep 17 03:54:01.529361 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:54:01.529319    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c871cdcf-6205-4f39-809a-7a72f976e145-istio-envoy\") pod \"c871cdcf-6205-4f39-809a-7a72f976e145\" (UID: \"c871cdcf-6205-4f39-809a-7a72f976e145\") "
Sep 17 03:54:01.529361 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:54:01.529356    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c871cdcf-6205-4f39-809a-7a72f976e145-workload-certs\") pod \"c871cdcf-6205-4f39-809a-7a72f976e145\" (UID: \"c871cdcf-6205-4f39-809a-7a72f976e145\") "
Sep 17 03:54:01.529649 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:54:01.529385    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c871cdcf-6205-4f39-809a-7a72f976e145-istiod-ca-cert\") pod \"c871cdcf-6205-4f39-809a-7a72f976e145\" (UID: \"c871cdcf-6205-4f39-809a-7a72f976e145\") "
Sep 17 03:54:01.529649 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:54:01.529422    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c871cdcf-6205-4f39-809a-7a72f976e145-istio-podinfo\") pod \"c871cdcf-6205-4f39-809a-7a72f976e145\" (UID: \"c871cdcf-6205-4f39-809a-7a72f976e145\") "
Sep 17 03:54:01.529649 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:54:01.529453    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c871cdcf-6205-4f39-809a-7a72f976e145-credential-socket\") pod \"c871cdcf-6205-4f39-809a-7a72f976e145\" (UID: \"c871cdcf-6205-4f39-809a-7a72f976e145\") "
Sep 17 03:54:01.529649 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:54:01.529480    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c871cdcf-6205-4f39-809a-7a72f976e145-workload-socket\") pod \"c871cdcf-6205-4f39-809a-7a72f976e145\" (UID: \"c871cdcf-6205-4f39-809a-7a72f976e145\") "
Sep 17 03:54:01.529649 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:54:01.529516    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lmnxb\" (UniqueName: \"kubernetes.io/projected/c871cdcf-6205-4f39-809a-7a72f976e145-kube-api-access-lmnxb\") pod \"c871cdcf-6205-4f39-809a-7a72f976e145\" (UID: \"c871cdcf-6205-4f39-809a-7a72f976e145\") "
Sep 17 03:54:01.529649 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:54:01.529540    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c871cdcf-6205-4f39-809a-7a72f976e145-istio-token\") pod \"c871cdcf-6205-4f39-809a-7a72f976e145\" (UID: \"c871cdcf-6205-4f39-809a-7a72f976e145\") "
Sep 17 03:54:01.529649 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:54:01.529636    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c871cdcf-6205-4f39-809a-7a72f976e145-istio-data" (OuterVolumeSpecName: "istio-data") pod "c871cdcf-6205-4f39-809a-7a72f976e145" (UID: "c871cdcf-6205-4f39-809a-7a72f976e145"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:54:01.530018 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:54:01.529769    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c871cdcf-6205-4f39-809a-7a72f976e145-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c871cdcf-6205-4f39-809a-7a72f976e145" (UID: "c871cdcf-6205-4f39-809a-7a72f976e145"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:54:01.530018 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:54:01.529831    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c871cdcf-6205-4f39-809a-7a72f976e145-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c871cdcf-6205-4f39-809a-7a72f976e145" (UID: "c871cdcf-6205-4f39-809a-7a72f976e145"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:54:01.530018 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:54:01.529916    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c871cdcf-6205-4f39-809a-7a72f976e145-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c871cdcf-6205-4f39-809a-7a72f976e145" (UID: "c871cdcf-6205-4f39-809a-7a72f976e145"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 03:54:01.530018 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:54:01.529964    2552 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c871cdcf-6205-4f39-809a-7a72f976e145-workload-socket\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:54:01.530018 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:54:01.529984    2552 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c871cdcf-6205-4f39-809a-7a72f976e145-istio-data\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:54:01.530018 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:54:01.529997    2552 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c871cdcf-6205-4f39-809a-7a72f976e145-workload-certs\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:54:01.530330 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:54:01.530016    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c871cdcf-6205-4f39-809a-7a72f976e145-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c871cdcf-6205-4f39-809a-7a72f976e145" (UID: "c871cdcf-6205-4f39-809a-7a72f976e145"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:54:01.531633 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:54:01.531591    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c871cdcf-6205-4f39-809a-7a72f976e145-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c871cdcf-6205-4f39-809a-7a72f976e145" (UID: "c871cdcf-6205-4f39-809a-7a72f976e145"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:54:01.531793 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:54:01.531777    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c871cdcf-6205-4f39-809a-7a72f976e145-kube-api-access-lmnxb" (OuterVolumeSpecName: "kube-api-access-lmnxb") pod "c871cdcf-6205-4f39-809a-7a72f976e145" (UID: "c871cdcf-6205-4f39-809a-7a72f976e145"). InnerVolumeSpecName "kube-api-access-lmnxb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 03:54:01.531854 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:54:01.531806    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c871cdcf-6205-4f39-809a-7a72f976e145-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c871cdcf-6205-4f39-809a-7a72f976e145" (UID: "c871cdcf-6205-4f39-809a-7a72f976e145"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 17 03:54:01.531896 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:54:01.531859    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c871cdcf-6205-4f39-809a-7a72f976e145-istio-token" (OuterVolumeSpecName: "istio-token") pod "c871cdcf-6205-4f39-809a-7a72f976e145" (UID: "c871cdcf-6205-4f39-809a-7a72f976e145"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 03:54:01.631180 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:54:01.631155    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lmnxb\" (UniqueName: \"kubernetes.io/projected/c871cdcf-6205-4f39-809a-7a72f976e145-kube-api-access-lmnxb\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:54:01.631180 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:54:01.631177    2552 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c871cdcf-6205-4f39-809a-7a72f976e145-istio-token\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:54:01.631180 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:54:01.631185    2552 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c871cdcf-6205-4f39-809a-7a72f976e145-istio-envoy\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:54:01.631363 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:54:01.631193    2552 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c871cdcf-6205-4f39-809a-7a72f976e145-istiod-ca-cert\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:54:01.631363 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:54:01.631201    2552 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c871cdcf-6205-4f39-809a-7a72f976e145-istio-podinfo\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:54:01.631363 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:54:01.631208    2552 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c871cdcf-6205-4f39-809a-7a72f976e145-credential-socket\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:54:02.280719 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:54:02.280681    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-f6d581-openshift-default-6955dfcbb6-szl9x" event={"ID":"c871cdcf-6205-4f39-809a-7a72f976e145","Type":"ContainerDied","Data":"e642408157451ea3c8dd93aaa03d5dc051b807cb086e9f394f8e6b7fcf0a51c6"}
Sep 17 03:54:02.281163 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:54:02.280733    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-f6d581-openshift-default-6955dfcbb6-szl9x"
Sep 17 03:54:02.281163 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:54:02.280753    2552 scope.go:117] "RemoveContainer" containerID="a58d3fd60bb301db0a8ebf535afceb75b18756fd3293213f286aa504ab7b73fc"
Sep 17 03:54:02.307558 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:54:02.302579    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-f6d581-openshift-default-6955dfcbb6-szl9x"]
Sep 17 03:54:02.310850 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:54:02.310823    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-f6d581-openshift-default-6955dfcbb6-szl9x"]
Sep 17 03:54:02.985398 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:54:02.985366    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c871cdcf-6205-4f39-809a-7a72f976e145" path="/var/lib/kubelet/pods/c871cdcf-6205-4f39-809a-7a72f976e145/volumes"
Sep 17 03:54:25.596145 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:54:25.596115    2552 scope.go:117] "RemoveContainer" containerID="eea2344cff1f1fc900dc92ff258ccf5d57b77615c5f1713f1bdda12a2434398d"
Sep 17 03:57:25.642672 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:57:25.642641    2552 scope.go:117] "RemoveContainer" containerID="c891888b7fba48001e6bd5114ada486ce62ec217efce7d164db37615d3e0d6e3"
Sep 17 03:58:25.660465 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:58:25.660434    2552 scope.go:117] "RemoveContainer" containerID="a27aa5ff3eede4ca9ad46b51c97772b93fc9011b4d74af28247867b5df86737e"
Sep 17 03:58:42.677517 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:58:42.677485    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-836e3afb-kserve-dbd4cfc79-zh7mz"]
Sep 17 03:58:42.677934 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:58:42.677879    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4d4225c-5283-4143-a89e-9c2fbc0eb172" containerName="istio-proxy"
Sep 17 03:58:42.677934 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:58:42.677894    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4d4225c-5283-4143-a89e-9c2fbc0eb172" containerName="istio-proxy"
Sep 17 03:58:42.677934 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:58:42.677908    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ccecfb8-e85c-4656-b88d-0551baf776dd" containerName="istio-proxy"
Sep 17 03:58:42.677934 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:58:42.677916    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ccecfb8-e85c-4656-b88d-0551baf776dd" containerName="istio-proxy"
Sep 17 03:58:42.677934 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:58:42.677929    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28c32ecd-31c3-4069-8701-0997e5146b32" containerName="main"
Sep 17 03:58:42.677934 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:58:42.677935    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="28c32ecd-31c3-4069-8701-0997e5146b32" containerName="main"
Sep 17 03:58:42.678132 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:58:42.677948    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a78851c0-74a9-4e0c-9864-71b65331fcdb" containerName="istio-proxy"
Sep 17 03:58:42.678132 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:58:42.677953    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="a78851c0-74a9-4e0c-9864-71b65331fcdb" containerName="istio-proxy"
Sep 17 03:58:42.678132 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:58:42.677960    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c871cdcf-6205-4f39-809a-7a72f976e145" containerName="istio-proxy"
Sep 17 03:58:42.678132 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:58:42.677965    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="c871cdcf-6205-4f39-809a-7a72f976e145" containerName="istio-proxy"
Sep 17 03:58:42.678132 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:58:42.677971    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a78851c0-74a9-4e0c-9864-71b65331fcdb" containerName="istio-proxy"
Sep 17 03:58:42.678132 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:58:42.677976    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="a78851c0-74a9-4e0c-9864-71b65331fcdb" containerName="istio-proxy"
Sep 17 03:58:42.678132 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:58:42.677984    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c871cdcf-6205-4f39-809a-7a72f976e145" containerName="istio-proxy"
Sep 17 03:58:42.678132 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:58:42.677991    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="c871cdcf-6205-4f39-809a-7a72f976e145" containerName="istio-proxy"
Sep 17 03:58:42.678132 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:58:42.678005    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4d4225c-5283-4143-a89e-9c2fbc0eb172" containerName="istio-proxy"
Sep 17 03:58:42.678132 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:58:42.678009    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4d4225c-5283-4143-a89e-9c2fbc0eb172" containerName="istio-proxy"
Sep 17 03:58:42.678132 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:58:42.678014    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4851b22-eb50-4f84-aee3-38c7f46ed0c6" containerName="maas-api"
Sep 17 03:58:42.678132 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:58:42.678019    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4851b22-eb50-4f84-aee3-38c7f46ed0c6" containerName="maas-api"
Sep 17 03:58:42.678132 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:58:42.678027    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1afd61b-bf80-4b36-993a-433f7356c372" containerName="revoke-keys"
Sep 17 03:58:42.678132 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:58:42.678032    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1afd61b-bf80-4b36-993a-433f7356c372" containerName="revoke-keys"
Sep 17 03:58:42.678132 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:58:42.678105    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="c4d4225c-5283-4143-a89e-9c2fbc0eb172" containerName="istio-proxy"
Sep 17 03:58:42.678132 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:58:42.678113    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="c871cdcf-6205-4f39-809a-7a72f976e145" containerName="istio-proxy"
Sep 17 03:58:42.678132 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:58:42.678124    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="c4d4225c-5283-4143-a89e-9c2fbc0eb172" containerName="istio-proxy"
Sep 17 03:58:42.678132 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:58:42.678133    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="a78851c0-74a9-4e0c-9864-71b65331fcdb" containerName="istio-proxy"
Sep 17 03:58:42.678132 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:58:42.678139    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="a78851c0-74a9-4e0c-9864-71b65331fcdb" containerName="istio-proxy"
Sep 17 03:58:42.678870 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:58:42.678145    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="28c32ecd-31c3-4069-8701-0997e5146b32" containerName="main"
Sep 17 03:58:42.678870 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:58:42.678157    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="c4851b22-eb50-4f84-aee3-38c7f46ed0c6" containerName="maas-api"
Sep 17 03:58:42.678870 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:58:42.678165    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ccecfb8-e85c-4656-b88d-0551baf776dd" containerName="istio-proxy"
Sep 17 03:58:42.678870 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:58:42.678173    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ccecfb8-e85c-4656-b88d-0551baf776dd" containerName="istio-proxy"
Sep 17 03:58:42.678870 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:58:42.678182    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="d1afd61b-bf80-4b36-993a-433f7356c372" containerName="revoke-keys"
Sep 17 03:58:42.681284 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:58:42.681269    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-836e3afb-kserve-dbd4cfc79-zh7mz"
Sep 17 03:58:42.683718 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:58:42.683697    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-836e3afb-kserve-self-signed-certs\""
Sep 17 03:58:42.691357 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:58:42.691336    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-836e3afb-kserve-dbd4cfc79-zh7mz"]
Sep 17 03:58:42.824691 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:58:42.824663    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfad5f08-4211-4ea3-beac-21eea25db010-model-cache\") pod \"e2e-same-id-a-836e3afb-kserve-dbd4cfc79-zh7mz\" (UID: \"cfad5f08-4211-4ea3-beac-21eea25db010\") " pod="llm/e2e-same-id-a-836e3afb-kserve-dbd4cfc79-zh7mz"
Sep 17 03:58:42.824847 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:58:42.824701    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfad5f08-4211-4ea3-beac-21eea25db010-tmp-dir\") pod \"e2e-same-id-a-836e3afb-kserve-dbd4cfc79-zh7mz\" (UID: \"cfad5f08-4211-4ea3-beac-21eea25db010\") " pod="llm/e2e-same-id-a-836e3afb-kserve-dbd4cfc79-zh7mz"
Sep 17 03:58:42.824847 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:58:42.824761    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfad5f08-4211-4ea3-beac-21eea25db010-dshm\") pod \"e2e-same-id-a-836e3afb-kserve-dbd4cfc79-zh7mz\" (UID: \"cfad5f08-4211-4ea3-beac-21eea25db010\") " pod="llm/e2e-same-id-a-836e3afb-kserve-dbd4cfc79-zh7mz"
Sep 17 03:58:42.824847 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:58:42.824787    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwckg\" (UniqueName: \"kubernetes.io/projected/cfad5f08-4211-4ea3-beac-21eea25db010-kube-api-access-nwckg\") pod \"e2e-same-id-a-836e3afb-kserve-dbd4cfc79-zh7mz\" (UID: \"cfad5f08-4211-4ea3-beac-21eea25db010\") " pod="llm/e2e-same-id-a-836e3afb-kserve-dbd4cfc79-zh7mz"
Sep 17 03:58:42.824847 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:58:42.824813    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfad5f08-4211-4ea3-beac-21eea25db010-home\") pod \"e2e-same-id-a-836e3afb-kserve-dbd4cfc79-zh7mz\" (UID: \"cfad5f08-4211-4ea3-beac-21eea25db010\") " pod="llm/e2e-same-id-a-836e3afb-kserve-dbd4cfc79-zh7mz"
Sep 17 03:58:42.824847 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:58:42.824830    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfad5f08-4211-4ea3-beac-21eea25db010-tls-certs\") pod \"e2e-same-id-a-836e3afb-kserve-dbd4cfc79-zh7mz\" (UID: \"cfad5f08-4211-4ea3-beac-21eea25db010\") " pod="llm/e2e-same-id-a-836e3afb-kserve-dbd4cfc79-zh7mz"
Sep 17 03:58:42.926228 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:58:42.926196    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfad5f08-4211-4ea3-beac-21eea25db010-dshm\") pod \"e2e-same-id-a-836e3afb-kserve-dbd4cfc79-zh7mz\" (UID: \"cfad5f08-4211-4ea3-beac-21eea25db010\") " pod="llm/e2e-same-id-a-836e3afb-kserve-dbd4cfc79-zh7mz"
Sep 17 03:58:42.926393 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:58:42.926241    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwckg\" (UniqueName: \"kubernetes.io/projected/cfad5f08-4211-4ea3-beac-21eea25db010-kube-api-access-nwckg\") pod \"e2e-same-id-a-836e3afb-kserve-dbd4cfc79-zh7mz\" (UID: \"cfad5f08-4211-4ea3-beac-21eea25db010\") " pod="llm/e2e-same-id-a-836e3afb-kserve-dbd4cfc79-zh7mz"
Sep 17 03:58:42.926393 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:58:42.926279    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfad5f08-4211-4ea3-beac-21eea25db010-home\") pod \"e2e-same-id-a-836e3afb-kserve-dbd4cfc79-zh7mz\" (UID: \"cfad5f08-4211-4ea3-beac-21eea25db010\") " pod="llm/e2e-same-id-a-836e3afb-kserve-dbd4cfc79-zh7mz"
Sep 17 03:58:42.926393 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:58:42.926312    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfad5f08-4211-4ea3-beac-21eea25db010-tls-certs\") pod \"e2e-same-id-a-836e3afb-kserve-dbd4cfc79-zh7mz\" (UID: \"cfad5f08-4211-4ea3-beac-21eea25db010\") " pod="llm/e2e-same-id-a-836e3afb-kserve-dbd4cfc79-zh7mz"
Sep 17 03:58:42.926393 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:58:42.926373    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfad5f08-4211-4ea3-beac-21eea25db010-model-cache\") pod \"e2e-same-id-a-836e3afb-kserve-dbd4cfc79-zh7mz\" (UID: \"cfad5f08-4211-4ea3-beac-21eea25db010\") " pod="llm/e2e-same-id-a-836e3afb-kserve-dbd4cfc79-zh7mz"
Sep 17 03:58:42.926678 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:58:42.926399    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfad5f08-4211-4ea3-beac-21eea25db010-tmp-dir\") pod \"e2e-same-id-a-836e3afb-kserve-dbd4cfc79-zh7mz\" (UID: \"cfad5f08-4211-4ea3-beac-21eea25db010\") " pod="llm/e2e-same-id-a-836e3afb-kserve-dbd4cfc79-zh7mz"
Sep 17 03:58:42.926741 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:58:42.926700    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfad5f08-4211-4ea3-beac-21eea25db010-home\") pod \"e2e-same-id-a-836e3afb-kserve-dbd4cfc79-zh7mz\" (UID: \"cfad5f08-4211-4ea3-beac-21eea25db010\") " pod="llm/e2e-same-id-a-836e3afb-kserve-dbd4cfc79-zh7mz"
Sep 17 03:58:42.926799 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:58:42.926736    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfad5f08-4211-4ea3-beac-21eea25db010-model-cache\") pod \"e2e-same-id-a-836e3afb-kserve-dbd4cfc79-zh7mz\" (UID: \"cfad5f08-4211-4ea3-beac-21eea25db010\") " pod="llm/e2e-same-id-a-836e3afb-kserve-dbd4cfc79-zh7mz"
Sep 17 03:58:42.926852 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:58:42.926802    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfad5f08-4211-4ea3-beac-21eea25db010-tmp-dir\") pod \"e2e-same-id-a-836e3afb-kserve-dbd4cfc79-zh7mz\" (UID: \"cfad5f08-4211-4ea3-beac-21eea25db010\") " pod="llm/e2e-same-id-a-836e3afb-kserve-dbd4cfc79-zh7mz"
Sep 17 03:58:42.928503 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:58:42.928450    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfad5f08-4211-4ea3-beac-21eea25db010-dshm\") pod \"e2e-same-id-a-836e3afb-kserve-dbd4cfc79-zh7mz\" (UID: \"cfad5f08-4211-4ea3-beac-21eea25db010\") " pod="llm/e2e-same-id-a-836e3afb-kserve-dbd4cfc79-zh7mz"
Sep 17 03:58:42.928895 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:58:42.928873    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfad5f08-4211-4ea3-beac-21eea25db010-tls-certs\") pod \"e2e-same-id-a-836e3afb-kserve-dbd4cfc79-zh7mz\" (UID: \"cfad5f08-4211-4ea3-beac-21eea25db010\") " pod="llm/e2e-same-id-a-836e3afb-kserve-dbd4cfc79-zh7mz"
Sep 17 03:58:42.933775 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:58:42.933750    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwckg\" (UniqueName: \"kubernetes.io/projected/cfad5f08-4211-4ea3-beac-21eea25db010-kube-api-access-nwckg\") pod \"e2e-same-id-a-836e3afb-kserve-dbd4cfc79-zh7mz\" (UID: \"cfad5f08-4211-4ea3-beac-21eea25db010\") " pod="llm/e2e-same-id-a-836e3afb-kserve-dbd4cfc79-zh7mz"
Sep 17 03:58:42.991359 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:58:42.991331    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-836e3afb-kserve-dbd4cfc79-zh7mz"
Sep 17 03:58:43.262232 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:58:43.262207    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-836e3afb-kserve-dbd4cfc79-zh7mz"]
Sep 17 03:58:43.264038 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:58:43.264016    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcfad5f08_4211_4ea3_beac_21eea25db010.slice/crio-f01fbd93f42855a1cb2659b27e1a6a9c2397e55d978a0f3cd38648b9b00cb21e WatchSource:0}: Error finding container f01fbd93f42855a1cb2659b27e1a6a9c2397e55d978a0f3cd38648b9b00cb21e: Status 404 returned error can't find the container with id f01fbd93f42855a1cb2659b27e1a6a9c2397e55d978a0f3cd38648b9b00cb21e
Sep 17 03:58:43.637393 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:58:43.637305    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-836e3afb-kserve-dbd4cfc79-zh7mz" event={"ID":"cfad5f08-4211-4ea3-beac-21eea25db010","Type":"ContainerStarted","Data":"90e5af52d7b09ffc3897f75fb15e3cd2c382c7c2c271f230e0492ea49cc00db1"}
Sep 17 03:58:43.637393 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:58:43.637352    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-836e3afb-kserve-dbd4cfc79-zh7mz" event={"ID":"cfad5f08-4211-4ea3-beac-21eea25db010","Type":"ContainerStarted","Data":"f01fbd93f42855a1cb2659b27e1a6a9c2397e55d978a0f3cd38648b9b00cb21e"}
Sep 17 03:58:43.655931 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:58:43.655885    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-836e3afb-kserve-dbd4cfc79-zh7mz" podStartSLOduration=1.6558722000000001 podStartE2EDuration="1.6558722s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 03:58:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:58:43.654288185 +0000 UTC m=+2359.120695711" watchObservedRunningTime="2026-09-17 03:58:43.6558722 +0000 UTC m=+2359.122279765"
Sep 17 03:58:48.876146 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:58:48.876101    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-836e3afb-kserve-c59b4cddc-bp6k8"]
Sep 17 03:58:48.876732 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:58:48.876492    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ccecfb8-e85c-4656-b88d-0551baf776dd" containerName="istio-proxy"
Sep 17 03:58:48.876732 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:58:48.876505    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ccecfb8-e85c-4656-b88d-0551baf776dd" containerName="istio-proxy"
Sep 17 03:58:48.876732 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:58:48.876567    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="c871cdcf-6205-4f39-809a-7a72f976e145" containerName="istio-proxy"
Sep 17 03:58:48.879729 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:58:48.879710    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-836e3afb-kserve-c59b4cddc-bp6k8"
Sep 17 03:58:48.882101 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:58:48.882081    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-836e3afb-kserve-self-signed-certs\""
Sep 17 03:58:48.886714 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:58:48.886690    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-836e3afb-kserve-c59b4cddc-bp6k8"]
Sep 17 03:58:48.976869 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:58:48.976839    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2d031bc-025e-4f16-ad4e-0ea2bbbf2587-tls-certs\") pod \"e2e-same-id-b-836e3afb-kserve-c59b4cddc-bp6k8\" (UID: \"a2d031bc-025e-4f16-ad4e-0ea2bbbf2587\") " pod="llm/e2e-same-id-b-836e3afb-kserve-c59b4cddc-bp6k8"
Sep 17 03:58:48.977016 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:58:48.976887    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jsbsj\" (UniqueName: \"kubernetes.io/projected/a2d031bc-025e-4f16-ad4e-0ea2bbbf2587-kube-api-access-jsbsj\") pod \"e2e-same-id-b-836e3afb-kserve-c59b4cddc-bp6k8\" (UID: \"a2d031bc-025e-4f16-ad4e-0ea2bbbf2587\") " pod="llm/e2e-same-id-b-836e3afb-kserve-c59b4cddc-bp6k8"
Sep 17 03:58:48.977016 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:58:48.976962    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2d031bc-025e-4f16-ad4e-0ea2bbbf2587-tmp-dir\") pod \"e2e-same-id-b-836e3afb-kserve-c59b4cddc-bp6k8\" (UID: \"a2d031bc-025e-4f16-ad4e-0ea2bbbf2587\") " pod="llm/e2e-same-id-b-836e3afb-kserve-c59b4cddc-bp6k8"
Sep 17 03:58:48.977103 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:58:48.977024    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2d031bc-025e-4f16-ad4e-0ea2bbbf2587-dshm\") pod \"e2e-same-id-b-836e3afb-kserve-c59b4cddc-bp6k8\" (UID: \"a2d031bc-025e-4f16-ad4e-0ea2bbbf2587\") " pod="llm/e2e-same-id-b-836e3afb-kserve-c59b4cddc-bp6k8"
Sep 17 03:58:48.977103 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:58:48.977049    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2d031bc-025e-4f16-ad4e-0ea2bbbf2587-model-cache\") pod \"e2e-same-id-b-836e3afb-kserve-c59b4cddc-bp6k8\" (UID: \"a2d031bc-025e-4f16-ad4e-0ea2bbbf2587\") " pod="llm/e2e-same-id-b-836e3afb-kserve-c59b4cddc-bp6k8"
Sep 17 03:58:48.977103 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:58:48.977067    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2d031bc-025e-4f16-ad4e-0ea2bbbf2587-home\") pod \"e2e-same-id-b-836e3afb-kserve-c59b4cddc-bp6k8\" (UID: \"a2d031bc-025e-4f16-ad4e-0ea2bbbf2587\") " pod="llm/e2e-same-id-b-836e3afb-kserve-c59b4cddc-bp6k8"
Sep 17 03:58:49.078190 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:58:49.078159    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2d031bc-025e-4f16-ad4e-0ea2bbbf2587-tls-certs\") pod \"e2e-same-id-b-836e3afb-kserve-c59b4cddc-bp6k8\" (UID: \"a2d031bc-025e-4f16-ad4e-0ea2bbbf2587\") " pod="llm/e2e-same-id-b-836e3afb-kserve-c59b4cddc-bp6k8"
Sep 17 03:58:49.078339 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:58:49.078206    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jsbsj\" (UniqueName: \"kubernetes.io/projected/a2d031bc-025e-4f16-ad4e-0ea2bbbf2587-kube-api-access-jsbsj\") pod \"e2e-same-id-b-836e3afb-kserve-c59b4cddc-bp6k8\" (UID: \"a2d031bc-025e-4f16-ad4e-0ea2bbbf2587\") " pod="llm/e2e-same-id-b-836e3afb-kserve-c59b4cddc-bp6k8"
Sep 17 03:58:49.078339 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:58:49.078240    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2d031bc-025e-4f16-ad4e-0ea2bbbf2587-tmp-dir\") pod \"e2e-same-id-b-836e3afb-kserve-c59b4cddc-bp6k8\" (UID: \"a2d031bc-025e-4f16-ad4e-0ea2bbbf2587\") " pod="llm/e2e-same-id-b-836e3afb-kserve-c59b4cddc-bp6k8"
Sep 17 03:58:49.078339 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:58:49.078270    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2d031bc-025e-4f16-ad4e-0ea2bbbf2587-dshm\") pod \"e2e-same-id-b-836e3afb-kserve-c59b4cddc-bp6k8\" (UID: \"a2d031bc-025e-4f16-ad4e-0ea2bbbf2587\") " pod="llm/e2e-same-id-b-836e3afb-kserve-c59b4cddc-bp6k8"
Sep 17 03:58:49.078339 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:58:49.078294    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2d031bc-025e-4f16-ad4e-0ea2bbbf2587-model-cache\") pod \"e2e-same-id-b-836e3afb-kserve-c59b4cddc-bp6k8\" (UID: \"a2d031bc-025e-4f16-ad4e-0ea2bbbf2587\") " pod="llm/e2e-same-id-b-836e3afb-kserve-c59b4cddc-bp6k8"
Sep 17 03:58:49.078339 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:58:49.078319    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2d031bc-025e-4f16-ad4e-0ea2bbbf2587-home\") pod \"e2e-same-id-b-836e3afb-kserve-c59b4cddc-bp6k8\" (UID: \"a2d031bc-025e-4f16-ad4e-0ea2bbbf2587\") " pod="llm/e2e-same-id-b-836e3afb-kserve-c59b4cddc-bp6k8"
Sep 17 03:58:49.078764 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:58:49.078745    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2d031bc-025e-4f16-ad4e-0ea2bbbf2587-home\") pod \"e2e-same-id-b-836e3afb-kserve-c59b4cddc-bp6k8\" (UID: \"a2d031bc-025e-4f16-ad4e-0ea2bbbf2587\") " pod="llm/e2e-same-id-b-836e3afb-kserve-c59b4cddc-bp6k8"
Sep 17 03:58:49.078841 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:58:49.078812    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2d031bc-025e-4f16-ad4e-0ea2bbbf2587-tmp-dir\") pod \"e2e-same-id-b-836e3afb-kserve-c59b4cddc-bp6k8\" (UID: \"a2d031bc-025e-4f16-ad4e-0ea2bbbf2587\") " pod="llm/e2e-same-id-b-836e3afb-kserve-c59b4cddc-bp6k8"
Sep 17 03:58:49.078903 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:58:49.078834    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2d031bc-025e-4f16-ad4e-0ea2bbbf2587-model-cache\") pod \"e2e-same-id-b-836e3afb-kserve-c59b4cddc-bp6k8\" (UID: \"a2d031bc-025e-4f16-ad4e-0ea2bbbf2587\") " pod="llm/e2e-same-id-b-836e3afb-kserve-c59b4cddc-bp6k8"
Sep 17 03:58:49.080650 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:58:49.080627    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2d031bc-025e-4f16-ad4e-0ea2bbbf2587-dshm\") pod \"e2e-same-id-b-836e3afb-kserve-c59b4cddc-bp6k8\" (UID: \"a2d031bc-025e-4f16-ad4e-0ea2bbbf2587\") " pod="llm/e2e-same-id-b-836e3afb-kserve-c59b4cddc-bp6k8"
Sep 17 03:58:49.080809 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:58:49.080791    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2d031bc-025e-4f16-ad4e-0ea2bbbf2587-tls-certs\") pod \"e2e-same-id-b-836e3afb-kserve-c59b4cddc-bp6k8\" (UID: \"a2d031bc-025e-4f16-ad4e-0ea2bbbf2587\") " pod="llm/e2e-same-id-b-836e3afb-kserve-c59b4cddc-bp6k8"
Sep 17 03:58:49.086094 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:58:49.086073    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jsbsj\" (UniqueName: \"kubernetes.io/projected/a2d031bc-025e-4f16-ad4e-0ea2bbbf2587-kube-api-access-jsbsj\") pod \"e2e-same-id-b-836e3afb-kserve-c59b4cddc-bp6k8\" (UID: \"a2d031bc-025e-4f16-ad4e-0ea2bbbf2587\") " pod="llm/e2e-same-id-b-836e3afb-kserve-c59b4cddc-bp6k8"
Sep 17 03:58:49.190100 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:58:49.190037    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-836e3afb-kserve-c59b4cddc-bp6k8"
Sep 17 03:58:49.463806 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:58:49.463783    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-836e3afb-kserve-c59b4cddc-bp6k8"]
Sep 17 03:58:49.466139 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:58:49.466111    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda2d031bc_025e_4f16_ad4e_0ea2bbbf2587.slice/crio-68e2a9d2380c19a3fda2a280141c5d59af45e46502735442a32f2e8d78747bcd WatchSource:0}: Error finding container 68e2a9d2380c19a3fda2a280141c5d59af45e46502735442a32f2e8d78747bcd: Status 404 returned error can't find the container with id 68e2a9d2380c19a3fda2a280141c5d59af45e46502735442a32f2e8d78747bcd
Sep 17 03:58:49.666652 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:58:49.666590    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-836e3afb-kserve-c59b4cddc-bp6k8" event={"ID":"a2d031bc-025e-4f16-ad4e-0ea2bbbf2587","Type":"ContainerStarted","Data":"34a96ed9107942f2187dddb04cccd5a09052a9dd4897481165e8a9ba5c9f0f54"}
Sep 17 03:58:49.666819 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:58:49.666665    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-836e3afb-kserve-c59b4cddc-bp6k8" event={"ID":"a2d031bc-025e-4f16-ad4e-0ea2bbbf2587","Type":"ContainerStarted","Data":"68e2a9d2380c19a3fda2a280141c5d59af45e46502735442a32f2e8d78747bcd"}
Sep 17 03:58:49.684504 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:58:49.684457    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-836e3afb-kserve-c59b4cddc-bp6k8" podStartSLOduration=1.684443745 podStartE2EDuration="1.684443745s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 03:58:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:58:49.682339038 +0000 UTC m=+2365.148746564" watchObservedRunningTime="2026-09-17 03:58:49.684443745 +0000 UTC m=+2365.150851271"
Sep 17 03:58:52.991444 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:58:52.991413    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-836e3afb-kserve-dbd4cfc79-zh7mz"
Sep 17 03:58:52.991444 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:58:52.991448    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-836e3afb-kserve-dbd4cfc79-zh7mz"
Sep 17 03:58:53.004171 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:58:53.004142    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-836e3afb-kserve-dbd4cfc79-zh7mz"
Sep 17 03:58:53.699934 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:58:53.699906    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-836e3afb-kserve-dbd4cfc79-zh7mz"
Sep 17 03:58:59.190473 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:58:59.190434    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-836e3afb-kserve-c59b4cddc-bp6k8"
Sep 17 03:58:59.190983 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:58:59.190508    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-836e3afb-kserve-c59b4cddc-bp6k8"
Sep 17 03:58:59.203184 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:58:59.203154    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-836e3afb-kserve-c59b4cddc-bp6k8"
Sep 17 03:58:59.728872 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:58:59.728839    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-836e3afb-kserve-c59b4cddc-bp6k8"
Sep 17 03:59:19.457346 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:59:19.457313    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-836e3afb-kserve-dbd4cfc79-zh7mz"]
Sep 17 03:59:21.060282 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:59:21.060253    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-836e3afb-kserve-c59b4cddc-bp6k8"]
Sep 17 03:59:34.478915 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:59:34.478858    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-836e3afb-kserve-dbd4cfc79-zh7mz" podUID="cfad5f08-4211-4ea3-beac-21eea25db010" containerName="main" containerID="cri-o://90e5af52d7b09ffc3897f75fb15e3cd2c382c7c2c271f230e0492ea49cc00db1" gracePeriod=45
Sep 17 03:59:34.725556 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:59:34.725531    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-836e3afb-kserve-dbd4cfc79-zh7mz"
Sep 17 03:59:34.776762 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:59:34.776738    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfad5f08-4211-4ea3-beac-21eea25db010-tls-certs\") pod \"cfad5f08-4211-4ea3-beac-21eea25db010\" (UID: \"cfad5f08-4211-4ea3-beac-21eea25db010\") "
Sep 17 03:59:34.776891 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:59:34.776771    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfad5f08-4211-4ea3-beac-21eea25db010-home\") pod \"cfad5f08-4211-4ea3-beac-21eea25db010\" (UID: \"cfad5f08-4211-4ea3-beac-21eea25db010\") "
Sep 17 03:59:34.776891 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:59:34.776791    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nwckg\" (UniqueName: \"kubernetes.io/projected/cfad5f08-4211-4ea3-beac-21eea25db010-kube-api-access-nwckg\") pod \"cfad5f08-4211-4ea3-beac-21eea25db010\" (UID: \"cfad5f08-4211-4ea3-beac-21eea25db010\") "
Sep 17 03:59:34.776891 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:59:34.776823    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfad5f08-4211-4ea3-beac-21eea25db010-dshm\") pod \"cfad5f08-4211-4ea3-beac-21eea25db010\" (UID: \"cfad5f08-4211-4ea3-beac-21eea25db010\") "
Sep 17 03:59:34.776891 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:59:34.776875    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfad5f08-4211-4ea3-beac-21eea25db010-tmp-dir\") pod \"cfad5f08-4211-4ea3-beac-21eea25db010\" (UID: \"cfad5f08-4211-4ea3-beac-21eea25db010\") "
Sep 17 03:59:34.777108 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:59:34.776933    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfad5f08-4211-4ea3-beac-21eea25db010-model-cache\") pod \"cfad5f08-4211-4ea3-beac-21eea25db010\" (UID: \"cfad5f08-4211-4ea3-beac-21eea25db010\") "
Sep 17 03:59:34.777108 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:59:34.777048    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfad5f08-4211-4ea3-beac-21eea25db010-home" (OuterVolumeSpecName: "home") pod "cfad5f08-4211-4ea3-beac-21eea25db010" (UID: "cfad5f08-4211-4ea3-beac-21eea25db010"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:59:34.777226 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:59:34.777123    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfad5f08-4211-4ea3-beac-21eea25db010-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "cfad5f08-4211-4ea3-beac-21eea25db010" (UID: "cfad5f08-4211-4ea3-beac-21eea25db010"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:59:34.777271 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:59:34.777200    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfad5f08-4211-4ea3-beac-21eea25db010-model-cache" (OuterVolumeSpecName: "model-cache") pod "cfad5f08-4211-4ea3-beac-21eea25db010" (UID: "cfad5f08-4211-4ea3-beac-21eea25db010"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:59:34.777311 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:59:34.777285    2552 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfad5f08-4211-4ea3-beac-21eea25db010-tmp-dir\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:59:34.777311 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:59:34.777304    2552 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfad5f08-4211-4ea3-beac-21eea25db010-home\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:59:34.779028 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:59:34.779006    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cfad5f08-4211-4ea3-beac-21eea25db010-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cfad5f08-4211-4ea3-beac-21eea25db010" (UID: "cfad5f08-4211-4ea3-beac-21eea25db010"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 03:59:34.779132 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:59:34.779010    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfad5f08-4211-4ea3-beac-21eea25db010-dshm" (OuterVolumeSpecName: "dshm") pod "cfad5f08-4211-4ea3-beac-21eea25db010" (UID: "cfad5f08-4211-4ea3-beac-21eea25db010"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:59:34.779132 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:59:34.779010    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfad5f08-4211-4ea3-beac-21eea25db010-kube-api-access-nwckg" (OuterVolumeSpecName: "kube-api-access-nwckg") pod "cfad5f08-4211-4ea3-beac-21eea25db010" (UID: "cfad5f08-4211-4ea3-beac-21eea25db010"). InnerVolumeSpecName "kube-api-access-nwckg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 03:59:34.877678 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:59:34.877657    2552 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfad5f08-4211-4ea3-beac-21eea25db010-model-cache\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:59:34.877678 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:59:34.877676    2552 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfad5f08-4211-4ea3-beac-21eea25db010-tls-certs\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:59:34.877826 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:59:34.877685    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwckg\" (UniqueName: \"kubernetes.io/projected/cfad5f08-4211-4ea3-beac-21eea25db010-kube-api-access-nwckg\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:59:34.877826 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:59:34.877694    2552 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfad5f08-4211-4ea3-beac-21eea25db010-dshm\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:59:34.881550 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:59:34.881528    2552 generic.go:356] "Generic (PLEG): container finished" podID="cfad5f08-4211-4ea3-beac-21eea25db010" containerID="90e5af52d7b09ffc3897f75fb15e3cd2c382c7c2c271f230e0492ea49cc00db1" exitCode=0
Sep 17 03:59:34.881671 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:59:34.881573    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-836e3afb-kserve-dbd4cfc79-zh7mz"
Sep 17 03:59:34.881671 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:59:34.881634    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-836e3afb-kserve-dbd4cfc79-zh7mz" event={"ID":"cfad5f08-4211-4ea3-beac-21eea25db010","Type":"ContainerDied","Data":"90e5af52d7b09ffc3897f75fb15e3cd2c382c7c2c271f230e0492ea49cc00db1"}
Sep 17 03:59:34.881745 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:59:34.881678    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-836e3afb-kserve-dbd4cfc79-zh7mz" event={"ID":"cfad5f08-4211-4ea3-beac-21eea25db010","Type":"ContainerDied","Data":"f01fbd93f42855a1cb2659b27e1a6a9c2397e55d978a0f3cd38648b9b00cb21e"}
Sep 17 03:59:34.881745 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:59:34.881701    2552 scope.go:117] "RemoveContainer" containerID="90e5af52d7b09ffc3897f75fb15e3cd2c382c7c2c271f230e0492ea49cc00db1"
Sep 17 03:59:34.890679 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:59:34.890662    2552 scope.go:117] "RemoveContainer" containerID="90e5af52d7b09ffc3897f75fb15e3cd2c382c7c2c271f230e0492ea49cc00db1"
Sep 17 03:59:34.890939 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:59:34.890923    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90e5af52d7b09ffc3897f75fb15e3cd2c382c7c2c271f230e0492ea49cc00db1\": container with ID starting with 90e5af52d7b09ffc3897f75fb15e3cd2c382c7c2c271f230e0492ea49cc00db1 not found: ID does not exist" containerID="90e5af52d7b09ffc3897f75fb15e3cd2c382c7c2c271f230e0492ea49cc00db1"
Sep 17 03:59:34.890990 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:59:34.890946    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90e5af52d7b09ffc3897f75fb15e3cd2c382c7c2c271f230e0492ea49cc00db1"} err="failed to get container status \"90e5af52d7b09ffc3897f75fb15e3cd2c382c7c2c271f230e0492ea49cc00db1\": rpc error: code = NotFound desc = could not find container \"90e5af52d7b09ffc3897f75fb15e3cd2c382c7c2c271f230e0492ea49cc00db1\": container with ID starting with 90e5af52d7b09ffc3897f75fb15e3cd2c382c7c2c271f230e0492ea49cc00db1 not found: ID does not exist"
Sep 17 03:59:34.902419 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:59:34.902399    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-836e3afb-kserve-dbd4cfc79-zh7mz"]
Sep 17 03:59:34.906075 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:59:34.906047    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-836e3afb-kserve-dbd4cfc79-zh7mz"]
Sep 17 03:59:34.985239 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:59:34.985211    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cfad5f08-4211-4ea3-beac-21eea25db010" path="/var/lib/kubelet/pods/cfad5f08-4211-4ea3-beac-21eea25db010/volumes"
Sep 17 03:59:36.079863 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:59:36.079825    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-836e3afb-kserve-c59b4cddc-bp6k8" podUID="a2d031bc-025e-4f16-ad4e-0ea2bbbf2587" containerName="main" containerID="cri-o://34a96ed9107942f2187dddb04cccd5a09052a9dd4897481165e8a9ba5c9f0f54" gracePeriod=45
Sep 17 03:59:36.321057 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:59:36.321034    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-836e3afb-kserve-c59b4cddc-bp6k8"
Sep 17 03:59:36.391369 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:59:36.391281    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2d031bc-025e-4f16-ad4e-0ea2bbbf2587-model-cache\") pod \"a2d031bc-025e-4f16-ad4e-0ea2bbbf2587\" (UID: \"a2d031bc-025e-4f16-ad4e-0ea2bbbf2587\") "
Sep 17 03:59:36.391369 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:59:36.391326    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2d031bc-025e-4f16-ad4e-0ea2bbbf2587-dshm\") pod \"a2d031bc-025e-4f16-ad4e-0ea2bbbf2587\" (UID: \"a2d031bc-025e-4f16-ad4e-0ea2bbbf2587\") "
Sep 17 03:59:36.391369 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:59:36.391364    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jsbsj\" (UniqueName: \"kubernetes.io/projected/a2d031bc-025e-4f16-ad4e-0ea2bbbf2587-kube-api-access-jsbsj\") pod \"a2d031bc-025e-4f16-ad4e-0ea2bbbf2587\" (UID: \"a2d031bc-025e-4f16-ad4e-0ea2bbbf2587\") "
Sep 17 03:59:36.391641 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:59:36.391420    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2d031bc-025e-4f16-ad4e-0ea2bbbf2587-tls-certs\") pod \"a2d031bc-025e-4f16-ad4e-0ea2bbbf2587\" (UID: \"a2d031bc-025e-4f16-ad4e-0ea2bbbf2587\") "
Sep 17 03:59:36.391641 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:59:36.391449    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2d031bc-025e-4f16-ad4e-0ea2bbbf2587-tmp-dir\") pod \"a2d031bc-025e-4f16-ad4e-0ea2bbbf2587\" (UID: \"a2d031bc-025e-4f16-ad4e-0ea2bbbf2587\") "
Sep 17 03:59:36.391641 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:59:36.391498    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2d031bc-025e-4f16-ad4e-0ea2bbbf2587-home\") pod \"a2d031bc-025e-4f16-ad4e-0ea2bbbf2587\" (UID: \"a2d031bc-025e-4f16-ad4e-0ea2bbbf2587\") "
Sep 17 03:59:36.391641 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:59:36.391633    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2d031bc-025e-4f16-ad4e-0ea2bbbf2587-model-cache" (OuterVolumeSpecName: "model-cache") pod "a2d031bc-025e-4f16-ad4e-0ea2bbbf2587" (UID: "a2d031bc-025e-4f16-ad4e-0ea2bbbf2587"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:59:36.391863 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:59:36.391766    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2d031bc-025e-4f16-ad4e-0ea2bbbf2587-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a2d031bc-025e-4f16-ad4e-0ea2bbbf2587" (UID: "a2d031bc-025e-4f16-ad4e-0ea2bbbf2587"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:59:36.391863 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:59:36.391843    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2d031bc-025e-4f16-ad4e-0ea2bbbf2587-home" (OuterVolumeSpecName: "home") pod "a2d031bc-025e-4f16-ad4e-0ea2bbbf2587" (UID: "a2d031bc-025e-4f16-ad4e-0ea2bbbf2587"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:59:36.391950 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:59:36.391879    2552 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2d031bc-025e-4f16-ad4e-0ea2bbbf2587-model-cache\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:59:36.391950 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:59:36.391893    2552 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2d031bc-025e-4f16-ad4e-0ea2bbbf2587-tmp-dir\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:59:36.393621 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:59:36.393582    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2d031bc-025e-4f16-ad4e-0ea2bbbf2587-kube-api-access-jsbsj" (OuterVolumeSpecName: "kube-api-access-jsbsj") pod "a2d031bc-025e-4f16-ad4e-0ea2bbbf2587" (UID: "a2d031bc-025e-4f16-ad4e-0ea2bbbf2587"). InnerVolumeSpecName "kube-api-access-jsbsj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 03:59:36.393944 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:59:36.393923    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2d031bc-025e-4f16-ad4e-0ea2bbbf2587-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a2d031bc-025e-4f16-ad4e-0ea2bbbf2587" (UID: "a2d031bc-025e-4f16-ad4e-0ea2bbbf2587"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 03:59:36.393944 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:59:36.393932    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2d031bc-025e-4f16-ad4e-0ea2bbbf2587-dshm" (OuterVolumeSpecName: "dshm") pod "a2d031bc-025e-4f16-ad4e-0ea2bbbf2587" (UID: "a2d031bc-025e-4f16-ad4e-0ea2bbbf2587"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:59:36.493075 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:59:36.493044    2552 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2d031bc-025e-4f16-ad4e-0ea2bbbf2587-dshm\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:59:36.493075 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:59:36.493069    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jsbsj\" (UniqueName: \"kubernetes.io/projected/a2d031bc-025e-4f16-ad4e-0ea2bbbf2587-kube-api-access-jsbsj\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:59:36.493075 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:59:36.493081    2552 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2d031bc-025e-4f16-ad4e-0ea2bbbf2587-tls-certs\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:59:36.493317 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:59:36.493090    2552 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2d031bc-025e-4f16-ad4e-0ea2bbbf2587-home\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:59:36.892810 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:59:36.892778    2552 generic.go:356] "Generic (PLEG): container finished" podID="a2d031bc-025e-4f16-ad4e-0ea2bbbf2587" containerID="34a96ed9107942f2187dddb04cccd5a09052a9dd4897481165e8a9ba5c9f0f54" exitCode=0
Sep 17 03:59:36.892993 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:59:36.892818    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-836e3afb-kserve-c59b4cddc-bp6k8"
Sep 17 03:59:36.892993 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:59:36.892829    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-836e3afb-kserve-c59b4cddc-bp6k8" event={"ID":"a2d031bc-025e-4f16-ad4e-0ea2bbbf2587","Type":"ContainerDied","Data":"34a96ed9107942f2187dddb04cccd5a09052a9dd4897481165e8a9ba5c9f0f54"}
Sep 17 03:59:36.892993 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:59:36.892858    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-836e3afb-kserve-c59b4cddc-bp6k8" event={"ID":"a2d031bc-025e-4f16-ad4e-0ea2bbbf2587","Type":"ContainerDied","Data":"68e2a9d2380c19a3fda2a280141c5d59af45e46502735442a32f2e8d78747bcd"}
Sep 17 03:59:36.892993 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:59:36.892876    2552 scope.go:117] "RemoveContainer" containerID="34a96ed9107942f2187dddb04cccd5a09052a9dd4897481165e8a9ba5c9f0f54"
Sep 17 03:59:36.902568 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:59:36.902551    2552 scope.go:117] "RemoveContainer" containerID="34a96ed9107942f2187dddb04cccd5a09052a9dd4897481165e8a9ba5c9f0f54"
Sep 17 03:59:36.902847 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:59:36.902826    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34a96ed9107942f2187dddb04cccd5a09052a9dd4897481165e8a9ba5c9f0f54\": container with ID starting with 34a96ed9107942f2187dddb04cccd5a09052a9dd4897481165e8a9ba5c9f0f54 not found: ID does not exist" containerID="34a96ed9107942f2187dddb04cccd5a09052a9dd4897481165e8a9ba5c9f0f54"
Sep 17 03:59:36.902909 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:59:36.902856    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34a96ed9107942f2187dddb04cccd5a09052a9dd4897481165e8a9ba5c9f0f54"} err="failed to get container status \"34a96ed9107942f2187dddb04cccd5a09052a9dd4897481165e8a9ba5c9f0f54\": rpc error: code = NotFound desc = could not find container \"34a96ed9107942f2187dddb04cccd5a09052a9dd4897481165e8a9ba5c9f0f54\": container with ID starting with 34a96ed9107942f2187dddb04cccd5a09052a9dd4897481165e8a9ba5c9f0f54 not found: ID does not exist"
Sep 17 03:59:36.913671 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:59:36.913650    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-836e3afb-kserve-c59b4cddc-bp6k8"]
Sep 17 03:59:36.921744 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:59:36.921720    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-836e3afb-kserve-c59b4cddc-bp6k8"]
Sep 17 03:59:36.986346 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:59:36.986312    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a2d031bc-025e-4f16-ad4e-0ea2bbbf2587" path="/var/lib/kubelet/pods/a2d031bc-025e-4f16-ad4e-0ea2bbbf2587/volumes"
Sep 17 03:59:39.977112 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:59:39.977071    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5575f6cfb4-xwcjr"]
Sep 17 03:59:39.977638 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:59:39.977620    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cfad5f08-4211-4ea3-beac-21eea25db010" containerName="main"
Sep 17 03:59:39.977736 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:59:39.977640    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="cfad5f08-4211-4ea3-beac-21eea25db010" containerName="main"
Sep 17 03:59:39.977736 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:59:39.977653    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2d031bc-025e-4f16-ad4e-0ea2bbbf2587" containerName="main"
Sep 17 03:59:39.977736 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:59:39.977660    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2d031bc-025e-4f16-ad4e-0ea2bbbf2587" containerName="main"
Sep 17 03:59:39.977895 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:59:39.977763    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2d031bc-025e-4f16-ad4e-0ea2bbbf2587" containerName="main"
Sep 17 03:59:39.977895 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:59:39.977779    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="cfad5f08-4211-4ea3-beac-21eea25db010" containerName="main"
Sep 17 03:59:39.982577 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:59:39.982545    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5575f6cfb4-xwcjr"
Sep 17 03:59:39.986175 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:59:39.986153    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-trmz9\""
Sep 17 03:59:39.986295 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:59:39.986173    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 17 03:59:39.986295 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:59:39.986229    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 17 03:59:39.986295 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:59:39.986288    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 17 03:59:39.986427 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:59:39.986326    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 17 03:59:39.991939 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:59:39.991916    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5575f6cfb4-xwcjr"]
Sep 17 03:59:40.024512 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:59:40.024486    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkcwg\" (UniqueName: \"kubernetes.io/projected/8e5f5c5f-bcd4-469d-83fb-9a3c36adde9c-kube-api-access-gkcwg\") pod \"maas-api-5575f6cfb4-xwcjr\" (UID: \"8e5f5c5f-bcd4-469d-83fb-9a3c36adde9c\") " pod="odh-ai-gateway-infra/maas-api-5575f6cfb4-xwcjr"
Sep 17 03:59:40.024654 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:59:40.024556    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8e5f5c5f-bcd4-469d-83fb-9a3c36adde9c-maas-api-tls\") pod \"maas-api-5575f6cfb4-xwcjr\" (UID: \"8e5f5c5f-bcd4-469d-83fb-9a3c36adde9c\") " pod="odh-ai-gateway-infra/maas-api-5575f6cfb4-xwcjr"
Sep 17 03:59:40.024654 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:59:40.024587    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e5f5c5f-bcd4-469d-83fb-9a3c36adde9c-openshift-service-ca\") pod \"maas-api-5575f6cfb4-xwcjr\" (UID: \"8e5f5c5f-bcd4-469d-83fb-9a3c36adde9c\") " pod="odh-ai-gateway-infra/maas-api-5575f6cfb4-xwcjr"
Sep 17 03:59:40.024654 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:59:40.024647    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8e5f5c5f-bcd4-469d-83fb-9a3c36adde9c-metrics-certs\") pod \"maas-api-5575f6cfb4-xwcjr\" (UID: \"8e5f5c5f-bcd4-469d-83fb-9a3c36adde9c\") " pod="odh-ai-gateway-infra/maas-api-5575f6cfb4-xwcjr"
Sep 17 03:59:40.125982 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:59:40.125954    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e5f5c5f-bcd4-469d-83fb-9a3c36adde9c-openshift-service-ca\") pod \"maas-api-5575f6cfb4-xwcjr\" (UID: \"8e5f5c5f-bcd4-469d-83fb-9a3c36adde9c\") " pod="odh-ai-gateway-infra/maas-api-5575f6cfb4-xwcjr"
Sep 17 03:59:40.126129 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:59:40.125988    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8e5f5c5f-bcd4-469d-83fb-9a3c36adde9c-metrics-certs\") pod \"maas-api-5575f6cfb4-xwcjr\" (UID: \"8e5f5c5f-bcd4-469d-83fb-9a3c36adde9c\") " pod="odh-ai-gateway-infra/maas-api-5575f6cfb4-xwcjr"
Sep 17 03:59:40.126129 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:59:40.126035    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkcwg\" (UniqueName: \"kubernetes.io/projected/8e5f5c5f-bcd4-469d-83fb-9a3c36adde9c-kube-api-access-gkcwg\") pod \"maas-api-5575f6cfb4-xwcjr\" (UID: \"8e5f5c5f-bcd4-469d-83fb-9a3c36adde9c\") " pod="odh-ai-gateway-infra/maas-api-5575f6cfb4-xwcjr"
Sep 17 03:59:40.126129 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:59:40.126069    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8e5f5c5f-bcd4-469d-83fb-9a3c36adde9c-maas-api-tls\") pod \"maas-api-5575f6cfb4-xwcjr\" (UID: \"8e5f5c5f-bcd4-469d-83fb-9a3c36adde9c\") " pod="odh-ai-gateway-infra/maas-api-5575f6cfb4-xwcjr"
Sep 17 03:59:40.126678 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:59:40.126655    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e5f5c5f-bcd4-469d-83fb-9a3c36adde9c-openshift-service-ca\") pod \"maas-api-5575f6cfb4-xwcjr\" (UID: \"8e5f5c5f-bcd4-469d-83fb-9a3c36adde9c\") " pod="odh-ai-gateway-infra/maas-api-5575f6cfb4-xwcjr"
Sep 17 03:59:40.128530 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:59:40.128511    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8e5f5c5f-bcd4-469d-83fb-9a3c36adde9c-metrics-certs\") pod \"maas-api-5575f6cfb4-xwcjr\" (UID: \"8e5f5c5f-bcd4-469d-83fb-9a3c36adde9c\") " pod="odh-ai-gateway-infra/maas-api-5575f6cfb4-xwcjr"
Sep 17 03:59:40.128628 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:59:40.128561    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8e5f5c5f-bcd4-469d-83fb-9a3c36adde9c-maas-api-tls\") pod \"maas-api-5575f6cfb4-xwcjr\" (UID: \"8e5f5c5f-bcd4-469d-83fb-9a3c36adde9c\") " pod="odh-ai-gateway-infra/maas-api-5575f6cfb4-xwcjr"
Sep 17 03:59:40.133305 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:59:40.133279    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkcwg\" (UniqueName: \"kubernetes.io/projected/8e5f5c5f-bcd4-469d-83fb-9a3c36adde9c-kube-api-access-gkcwg\") pod \"maas-api-5575f6cfb4-xwcjr\" (UID: \"8e5f5c5f-bcd4-469d-83fb-9a3c36adde9c\") " pod="odh-ai-gateway-infra/maas-api-5575f6cfb4-xwcjr"
Sep 17 03:59:40.294254 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:59:40.294226    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5575f6cfb4-xwcjr"
Sep 17 03:59:40.565996 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:59:40.565911    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5575f6cfb4-xwcjr"]
Sep 17 03:59:40.569686 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:59:40.569658    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e5f5c5f_bcd4_469d_83fb_9a3c36adde9c.slice/crio-9004787789d8d268b4c0df71b0f81f97086dced4eb42c9063ff38d4d2ab4bc87 WatchSource:0}: Error finding container 9004787789d8d268b4c0df71b0f81f97086dced4eb42c9063ff38d4d2ab4bc87: Status 404 returned error can't find the container with id 9004787789d8d268b4c0df71b0f81f97086dced4eb42c9063ff38d4d2ab4bc87
Sep 17 03:59:40.571443 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:59:40.571421    2552 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 03:59:40.920487 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:59:40.920400    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5575f6cfb4-xwcjr" event={"ID":"8e5f5c5f-bcd4-469d-83fb-9a3c36adde9c","Type":"ContainerStarted","Data":"9004787789d8d268b4c0df71b0f81f97086dced4eb42c9063ff38d4d2ab4bc87"}
Sep 17 03:59:41.928510 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:59:41.928475    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5575f6cfb4-xwcjr" event={"ID":"8e5f5c5f-bcd4-469d-83fb-9a3c36adde9c","Type":"ContainerStarted","Data":"2aa88d22b8b2f79aca657ef1bd48d2b56491280041b8d1eebff0e9d4cd1cb334"}
Sep 17 03:59:41.928882 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:59:41.928626    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5575f6cfb4-xwcjr"
Sep 17 03:59:41.944828 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:59:41.944781    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5575f6cfb4-xwcjr" podStartSLOduration=2.543389634 podStartE2EDuration="2.944762303s" totalImagesPullingTime="401.372669ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 03:59:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:59:41.944123655 +0000 UTC m=+2417.410531205" watchObservedRunningTime="2026-09-17 03:59:41.944762303 +0000 UTC m=+2417.411169829"
Sep 17 03:59:47.549993 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:59:47.549910    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5575f6cfb4-xwcjr"]
Sep 17 03:59:47.550463 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:59:47.550095    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-5575f6cfb4-xwcjr" podUID="8e5f5c5f-bcd4-469d-83fb-9a3c36adde9c" containerName="maas-api" containerID="cri-o://2aa88d22b8b2f79aca657ef1bd48d2b56491280041b8d1eebff0e9d4cd1cb334" gracePeriod=30
Sep 17 03:59:47.555509 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:59:47.555485    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5575f6cfb4-xwcjr"
Sep 17 03:59:47.579621 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:59:47.579582    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-7c7d4dbbc6-w6z7w"]
Sep 17 03:59:47.582890 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:59:47.582871    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7c7d4dbbc6-w6z7w"
Sep 17 03:59:47.593260 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:59:47.593235    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7c7d4dbbc6-w6z7w"]
Sep 17 03:59:47.692554 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:59:47.692527    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjcg8\" (UniqueName: \"kubernetes.io/projected/a175f4e0-30b6-47da-b2bc-75436770c4d1-kube-api-access-tjcg8\") pod \"maas-api-7c7d4dbbc6-w6z7w\" (UID: \"a175f4e0-30b6-47da-b2bc-75436770c4d1\") " pod="odh-ai-gateway-infra/maas-api-7c7d4dbbc6-w6z7w"
Sep 17 03:59:47.692737 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:59:47.692569    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a175f4e0-30b6-47da-b2bc-75436770c4d1-maas-api-tls\") pod \"maas-api-7c7d4dbbc6-w6z7w\" (UID: \"a175f4e0-30b6-47da-b2bc-75436770c4d1\") " pod="odh-ai-gateway-infra/maas-api-7c7d4dbbc6-w6z7w"
Sep 17 03:59:47.692737 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:59:47.692610    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a175f4e0-30b6-47da-b2bc-75436770c4d1-metrics-certs\") pod \"maas-api-7c7d4dbbc6-w6z7w\" (UID: \"a175f4e0-30b6-47da-b2bc-75436770c4d1\") " pod="odh-ai-gateway-infra/maas-api-7c7d4dbbc6-w6z7w"
Sep 17 03:59:47.692737 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:59:47.692658    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a175f4e0-30b6-47da-b2bc-75436770c4d1-openshift-service-ca\") pod \"maas-api-7c7d4dbbc6-w6z7w\" (UID: \"a175f4e0-30b6-47da-b2bc-75436770c4d1\") " pod="odh-ai-gateway-infra/maas-api-7c7d4dbbc6-w6z7w"
Sep 17 03:59:47.791037 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:59:47.791016    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5575f6cfb4-xwcjr"
Sep 17 03:59:47.793220 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:59:47.793193    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjcg8\" (UniqueName: \"kubernetes.io/projected/a175f4e0-30b6-47da-b2bc-75436770c4d1-kube-api-access-tjcg8\") pod \"maas-api-7c7d4dbbc6-w6z7w\" (UID: \"a175f4e0-30b6-47da-b2bc-75436770c4d1\") " pod="odh-ai-gateway-infra/maas-api-7c7d4dbbc6-w6z7w"
Sep 17 03:59:47.793322 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:59:47.793257    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a175f4e0-30b6-47da-b2bc-75436770c4d1-maas-api-tls\") pod \"maas-api-7c7d4dbbc6-w6z7w\" (UID: \"a175f4e0-30b6-47da-b2bc-75436770c4d1\") " pod="odh-ai-gateway-infra/maas-api-7c7d4dbbc6-w6z7w"
Sep 17 03:59:47.793322 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:59:47.793292    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a175f4e0-30b6-47da-b2bc-75436770c4d1-metrics-certs\") pod \"maas-api-7c7d4dbbc6-w6z7w\" (UID: \"a175f4e0-30b6-47da-b2bc-75436770c4d1\") " pod="odh-ai-gateway-infra/maas-api-7c7d4dbbc6-w6z7w"
Sep 17 03:59:47.793322 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:59:47.793318    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a175f4e0-30b6-47da-b2bc-75436770c4d1-openshift-service-ca\") pod \"maas-api-7c7d4dbbc6-w6z7w\" (UID: \"a175f4e0-30b6-47da-b2bc-75436770c4d1\") " pod="odh-ai-gateway-infra/maas-api-7c7d4dbbc6-w6z7w"
Sep 17 03:59:47.794008 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:59:47.793987    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a175f4e0-30b6-47da-b2bc-75436770c4d1-openshift-service-ca\") pod \"maas-api-7c7d4dbbc6-w6z7w\" (UID: \"a175f4e0-30b6-47da-b2bc-75436770c4d1\") " pod="odh-ai-gateway-infra/maas-api-7c7d4dbbc6-w6z7w"
Sep 17 03:59:47.795639 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:59:47.795620    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a175f4e0-30b6-47da-b2bc-75436770c4d1-metrics-certs\") pod \"maas-api-7c7d4dbbc6-w6z7w\" (UID: \"a175f4e0-30b6-47da-b2bc-75436770c4d1\") " pod="odh-ai-gateway-infra/maas-api-7c7d4dbbc6-w6z7w"
Sep 17 03:59:47.795827 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:59:47.795811    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a175f4e0-30b6-47da-b2bc-75436770c4d1-maas-api-tls\") pod \"maas-api-7c7d4dbbc6-w6z7w\" (UID: \"a175f4e0-30b6-47da-b2bc-75436770c4d1\") " pod="odh-ai-gateway-infra/maas-api-7c7d4dbbc6-w6z7w"
Sep 17 03:59:47.800327 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:59:47.800287    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjcg8\" (UniqueName: \"kubernetes.io/projected/a175f4e0-30b6-47da-b2bc-75436770c4d1-kube-api-access-tjcg8\") pod \"maas-api-7c7d4dbbc6-w6z7w\" (UID: \"a175f4e0-30b6-47da-b2bc-75436770c4d1\") " pod="odh-ai-gateway-infra/maas-api-7c7d4dbbc6-w6z7w"
Sep 17 03:59:47.894534 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:59:47.894511    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8e5f5c5f-bcd4-469d-83fb-9a3c36adde9c-maas-api-tls\") pod \"8e5f5c5f-bcd4-469d-83fb-9a3c36adde9c\" (UID: \"8e5f5c5f-bcd4-469d-83fb-9a3c36adde9c\") "
Sep 17 03:59:47.894689 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:59:47.894546    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8e5f5c5f-bcd4-469d-83fb-9a3c36adde9c-metrics-certs\") pod \"8e5f5c5f-bcd4-469d-83fb-9a3c36adde9c\" (UID: \"8e5f5c5f-bcd4-469d-83fb-9a3c36adde9c\") "
Sep 17 03:59:47.894689 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:59:47.894638    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gkcwg\" (UniqueName: \"kubernetes.io/projected/8e5f5c5f-bcd4-469d-83fb-9a3c36adde9c-kube-api-access-gkcwg\") pod \"8e5f5c5f-bcd4-469d-83fb-9a3c36adde9c\" (UID: \"8e5f5c5f-bcd4-469d-83fb-9a3c36adde9c\") "
Sep 17 03:59:47.894817 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:59:47.894701    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e5f5c5f-bcd4-469d-83fb-9a3c36adde9c-openshift-service-ca\") pod \"8e5f5c5f-bcd4-469d-83fb-9a3c36adde9c\" (UID: \"8e5f5c5f-bcd4-469d-83fb-9a3c36adde9c\") "
Sep 17 03:59:47.895087 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:59:47.895048    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e5f5c5f-bcd4-469d-83fb-9a3c36adde9c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8e5f5c5f-bcd4-469d-83fb-9a3c36adde9c" (UID: "8e5f5c5f-bcd4-469d-83fb-9a3c36adde9c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 03:59:47.896554 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:59:47.896530    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e5f5c5f-bcd4-469d-83fb-9a3c36adde9c-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "8e5f5c5f-bcd4-469d-83fb-9a3c36adde9c" (UID: "8e5f5c5f-bcd4-469d-83fb-9a3c36adde9c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 03:59:47.896665 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:59:47.896563    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e5f5c5f-bcd4-469d-83fb-9a3c36adde9c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8e5f5c5f-bcd4-469d-83fb-9a3c36adde9c" (UID: "8e5f5c5f-bcd4-469d-83fb-9a3c36adde9c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 03:59:47.896665 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:59:47.896625    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e5f5c5f-bcd4-469d-83fb-9a3c36adde9c-kube-api-access-gkcwg" (OuterVolumeSpecName: "kube-api-access-gkcwg") pod "8e5f5c5f-bcd4-469d-83fb-9a3c36adde9c" (UID: "8e5f5c5f-bcd4-469d-83fb-9a3c36adde9c"). InnerVolumeSpecName "kube-api-access-gkcwg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 03:59:47.935412 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:59:47.935391    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7c7d4dbbc6-w6z7w"
Sep 17 03:59:47.958262 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:59:47.958239    2552 generic.go:356] "Generic (PLEG): container finished" podID="8e5f5c5f-bcd4-469d-83fb-9a3c36adde9c" containerID="2aa88d22b8b2f79aca657ef1bd48d2b56491280041b8d1eebff0e9d4cd1cb334" exitCode=0
Sep 17 03:59:47.958357 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:59:47.958272    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5575f6cfb4-xwcjr" event={"ID":"8e5f5c5f-bcd4-469d-83fb-9a3c36adde9c","Type":"ContainerDied","Data":"2aa88d22b8b2f79aca657ef1bd48d2b56491280041b8d1eebff0e9d4cd1cb334"}
Sep 17 03:59:47.958357 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:59:47.958282    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5575f6cfb4-xwcjr"
Sep 17 03:59:47.958357 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:59:47.958306    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5575f6cfb4-xwcjr" event={"ID":"8e5f5c5f-bcd4-469d-83fb-9a3c36adde9c","Type":"ContainerDied","Data":"9004787789d8d268b4c0df71b0f81f97086dced4eb42c9063ff38d4d2ab4bc87"}
Sep 17 03:59:47.958357 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:59:47.958328    2552 scope.go:117] "RemoveContainer" containerID="2aa88d22b8b2f79aca657ef1bd48d2b56491280041b8d1eebff0e9d4cd1cb334"
Sep 17 03:59:47.976350 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:59:47.976328    2552 scope.go:117] "RemoveContainer" containerID="2aa88d22b8b2f79aca657ef1bd48d2b56491280041b8d1eebff0e9d4cd1cb334"
Sep 17 03:59:47.976715 ip-10-0-139-53 kubenswrapper[2552]: E0917 03:59:47.976685    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2aa88d22b8b2f79aca657ef1bd48d2b56491280041b8d1eebff0e9d4cd1cb334\": container with ID starting with 2aa88d22b8b2f79aca657ef1bd48d2b56491280041b8d1eebff0e9d4cd1cb334 not found: ID does not exist" containerID="2aa88d22b8b2f79aca657ef1bd48d2b56491280041b8d1eebff0e9d4cd1cb334"
Sep 17 03:59:47.976800 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:59:47.976721    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2aa88d22b8b2f79aca657ef1bd48d2b56491280041b8d1eebff0e9d4cd1cb334"} err="failed to get container status \"2aa88d22b8b2f79aca657ef1bd48d2b56491280041b8d1eebff0e9d4cd1cb334\": rpc error: code = NotFound desc = could not find container \"2aa88d22b8b2f79aca657ef1bd48d2b56491280041b8d1eebff0e9d4cd1cb334\": container with ID starting with 2aa88d22b8b2f79aca657ef1bd48d2b56491280041b8d1eebff0e9d4cd1cb334 not found: ID does not exist"
Sep 17 03:59:47.983911 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:59:47.983884    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5575f6cfb4-xwcjr"]
Sep 17 03:59:47.987890 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:59:47.987868    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-5575f6cfb4-xwcjr"]
Sep 17 03:59:47.996019 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:59:47.995940    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gkcwg\" (UniqueName: \"kubernetes.io/projected/8e5f5c5f-bcd4-469d-83fb-9a3c36adde9c-kube-api-access-gkcwg\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:59:47.996019 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:59:47.995970    2552 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e5f5c5f-bcd4-469d-83fb-9a3c36adde9c-openshift-service-ca\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:59:47.996019 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:59:47.995986    2552 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8e5f5c5f-bcd4-469d-83fb-9a3c36adde9c-maas-api-tls\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:59:47.996019 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:59:47.996001    2552 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8e5f5c5f-bcd4-469d-83fb-9a3c36adde9c-metrics-certs\") on node \"ip-10-0-139-53.ec2.internal\" DevicePath \"\""
Sep 17 03:59:48.211182 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:59:48.211112    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7c7d4dbbc6-w6z7w"]
Sep 17 03:59:48.214629 ip-10-0-139-53 kubenswrapper[2552]: W0917 03:59:48.214582    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda175f4e0_30b6_47da_b2bc_75436770c4d1.slice/crio-8074e1cd895a2017b36026819a1cfb1a21217f057b10b2ddc08264e8aeff8ae3 WatchSource:0}: Error finding container 8074e1cd895a2017b36026819a1cfb1a21217f057b10b2ddc08264e8aeff8ae3: Status 404 returned error can't find the container with id 8074e1cd895a2017b36026819a1cfb1a21217f057b10b2ddc08264e8aeff8ae3
Sep 17 03:59:48.965357 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:59:48.965268    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7c7d4dbbc6-w6z7w" event={"ID":"a175f4e0-30b6-47da-b2bc-75436770c4d1","Type":"ContainerStarted","Data":"06d69b795dfbf2a1a44f096c47c80233f0d66f1d8a665e56f7ef74ec89b0a3ed"}
Sep 17 03:59:48.965357 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:59:48.965324    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7c7d4dbbc6-w6z7w" event={"ID":"a175f4e0-30b6-47da-b2bc-75436770c4d1","Type":"ContainerStarted","Data":"8074e1cd895a2017b36026819a1cfb1a21217f057b10b2ddc08264e8aeff8ae3"}
Sep 17 03:59:48.965830 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:59:48.965365    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-7c7d4dbbc6-w6z7w"
Sep 17 03:59:48.982651 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:59:48.982579    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-7c7d4dbbc6-w6z7w" podStartSLOduration=1.632102341 podStartE2EDuration="1.982565583s" totalImagesPullingTime="350.463242ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 03:59:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:59:48.981322795 +0000 UTC m=+2424.447730331" watchObservedRunningTime="2026-09-17 03:59:48.982565583 +0000 UTC m=+2424.448973108"
Sep 17 03:59:48.985529 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:59:48.985501    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8e5f5c5f-bcd4-469d-83fb-9a3c36adde9c" path="/var/lib/kubelet/pods/8e5f5c5f-bcd4-469d-83fb-9a3c36adde9c/volumes"
Sep 17 03:59:54.975969 ip-10-0-139-53 kubenswrapper[2552]: I0917 03:59:54.975943    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-7c7d4dbbc6-w6z7w"
Sep 17 04:00:10.742371 ip-10-0-139-53 kubenswrapper[2552]: I0917 04:00:10.742340    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5999746d68-vdmx6_b968d89b-da89-4601-9590-3a48d48e7c0c/authorino/0.log"
Sep 17 04:00:17.142752 ip-10-0-139-53 kubenswrapper[2552]: I0917 04:00:17.142716    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6977f7b7bc-css5r_f512d104-3bf3-4414-bed2-f7440c18ee3b/manager/0.log"
Sep 17 04:00:17.150567 ip-10-0-139-53 kubenswrapper[2552]: I0917 04:00:17.150544    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6977f7b7bc-css5r_f512d104-3bf3-4414-bed2-f7440c18ee3b/copy-manifests/0.log"
Sep 17 04:00:17.808977 ip-10-0-139-53 kubenswrapper[2552]: I0917 04:00:17.808952    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-7c7d4dbbc6-w6z7w_a175f4e0-30b6-47da-b2bc-75436770c4d1/maas-api/0.log"
Sep 17 04:00:17.916112 ip-10-0-139-53 kubenswrapper[2552]: I0917 04:00:17.916075    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29826930-lfrx5_3b326935-9539-47d9-903b-9f45e404c65a/cleanup/0.log"
Sep 17 04:00:18.019684 ip-10-0-139-53 kubenswrapper[2552]: I0917 04:00:18.019656    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29826945-s5d4q_ab9688c7-a95b-4430-8075-a0008c6595fc/cleanup/0.log"
Sep 17 04:00:19.038894 ip-10-0-139-53 kubenswrapper[2552]: I0917 04:00:19.038865    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5999746d68-vdmx6_b968d89b-da89-4601-9590-3a48d48e7c0c/authorino/0.log"
Sep 17 04:00:19.366380 ip-10-0-139-53 kubenswrapper[2552]: I0917 04:00:19.366304    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-bffwq_c8cc769e-e0d1-40bd-ba4d-3ee56d0c48b4/kuadrant-console-plugin/0.log"
Sep 17 04:00:19.756891 ip-10-0-139-53 kubenswrapper[2552]: I0917 04:00:19.756861    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-hlqc8_ffd803ff-57b5-4e73-ba88-a149336f8be5/manager/0.log"
Sep 17 04:00:20.443324 ip-10-0-139-53 kubenswrapper[2552]: I0917 04:00:20.443282    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5bc6dc7888-txj6c_60cb769a-2eb5-433e-b97d-5ae489598831/kube-auth-proxy/0.log"
Sep 17 04:00:21.469338 ip-10-0-139-53 kubenswrapper[2552]: I0917 04:00:21.469287    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-bcc9ff8d8-wwmhr_53da0826-1bd9-440c-9e83-e437508931ea/main/0.log"
Sep 17 04:00:21.949576 ip-10-0-139-53 kubenswrapper[2552]: I0917 04:00:21.949544    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-4hdgk_84edcbbb-cf1d-4a6e-b7ef-d48f9a776cf7/main/0.log"
Sep 17 04:00:26.832335 ip-10-0-139-53 kubenswrapper[2552]: I0917 04:00:26.832303    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-nv4hh/must-gather-ptcrm"]
Sep 17 04:00:26.832791 ip-10-0-139-53 kubenswrapper[2552]: I0917 04:00:26.832696    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e5f5c5f-bcd4-469d-83fb-9a3c36adde9c" containerName="maas-api"
Sep 17 04:00:26.832791 ip-10-0-139-53 kubenswrapper[2552]: I0917 04:00:26.832708    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e5f5c5f-bcd4-469d-83fb-9a3c36adde9c" containerName="maas-api"
Sep 17 04:00:26.832791 ip-10-0-139-53 kubenswrapper[2552]: I0917 04:00:26.832771    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e5f5c5f-bcd4-469d-83fb-9a3c36adde9c" containerName="maas-api"
Sep 17 04:00:26.837118 ip-10-0-139-53 kubenswrapper[2552]: I0917 04:00:26.837100    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-nv4hh/must-gather-ptcrm"
Sep 17 04:00:26.839726 ip-10-0-139-53 kubenswrapper[2552]: I0917 04:00:26.839702    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-nv4hh\"/\"kube-root-ca.crt\""
Sep 17 04:00:26.840804 ip-10-0-139-53 kubenswrapper[2552]: I0917 04:00:26.840784    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-nv4hh\"/\"openshift-service-ca.crt\""
Sep 17 04:00:26.840922 ip-10-0-139-53 kubenswrapper[2552]: I0917 04:00:26.840823    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-nv4hh\"/\"default-dockercfg-8vt88\""
Sep 17 04:00:26.853147 ip-10-0-139-53 kubenswrapper[2552]: I0917 04:00:26.853119    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-nv4hh/must-gather-ptcrm"]
Sep 17 04:00:26.922706 ip-10-0-139-53 kubenswrapper[2552]: I0917 04:00:26.922677    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8z27b\" (UniqueName: \"kubernetes.io/projected/70dcfdfd-d8e7-45fb-97cb-afc1c780c720-kube-api-access-8z27b\") pod \"must-gather-ptcrm\" (UID: \"70dcfdfd-d8e7-45fb-97cb-afc1c780c720\") " pod="openshift-must-gather-nv4hh/must-gather-ptcrm"
Sep 17 04:00:26.922828 ip-10-0-139-53 kubenswrapper[2552]: I0917 04:00:26.922717    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/70dcfdfd-d8e7-45fb-97cb-afc1c780c720-must-gather-output\") pod \"must-gather-ptcrm\" (UID: \"70dcfdfd-d8e7-45fb-97cb-afc1c780c720\") " pod="openshift-must-gather-nv4hh/must-gather-ptcrm"
Sep 17 04:00:27.024074 ip-10-0-139-53 kubenswrapper[2552]: I0917 04:00:27.024025    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8z27b\" (UniqueName: \"kubernetes.io/projected/70dcfdfd-d8e7-45fb-97cb-afc1c780c720-kube-api-access-8z27b\") pod \"must-gather-ptcrm\" (UID: \"70dcfdfd-d8e7-45fb-97cb-afc1c780c720\") " pod="openshift-must-gather-nv4hh/must-gather-ptcrm"
Sep 17 04:00:27.024247 ip-10-0-139-53 kubenswrapper[2552]: I0917 04:00:27.024094    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/70dcfdfd-d8e7-45fb-97cb-afc1c780c720-must-gather-output\") pod \"must-gather-ptcrm\" (UID: \"70dcfdfd-d8e7-45fb-97cb-afc1c780c720\") " pod="openshift-must-gather-nv4hh/must-gather-ptcrm"
Sep 17 04:00:27.024435 ip-10-0-139-53 kubenswrapper[2552]: I0917 04:00:27.024413    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/70dcfdfd-d8e7-45fb-97cb-afc1c780c720-must-gather-output\") pod \"must-gather-ptcrm\" (UID: \"70dcfdfd-d8e7-45fb-97cb-afc1c780c720\") " pod="openshift-must-gather-nv4hh/must-gather-ptcrm"
Sep 17 04:00:27.032527 ip-10-0-139-53 kubenswrapper[2552]: I0917 04:00:27.032498    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8z27b\" (UniqueName: \"kubernetes.io/projected/70dcfdfd-d8e7-45fb-97cb-afc1c780c720-kube-api-access-8z27b\") pod \"must-gather-ptcrm\" (UID: \"70dcfdfd-d8e7-45fb-97cb-afc1c780c720\") " pod="openshift-must-gather-nv4hh/must-gather-ptcrm"
Sep 17 04:00:27.148491 ip-10-0-139-53 kubenswrapper[2552]: I0917 04:00:27.148406    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-nv4hh/must-gather-ptcrm"
Sep 17 04:00:27.420114 ip-10-0-139-53 kubenswrapper[2552]: I0917 04:00:27.420091    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-nv4hh/must-gather-ptcrm"]
Sep 17 04:00:27.422317 ip-10-0-139-53 kubenswrapper[2552]: W0917 04:00:27.422287    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod70dcfdfd_d8e7_45fb_97cb_afc1c780c720.slice/crio-9086e63ebd66a264bdf2bb5819ff163f15c0eb16db1cdb6308223b7064760e6b WatchSource:0}: Error finding container 9086e63ebd66a264bdf2bb5819ff163f15c0eb16db1cdb6308223b7064760e6b: Status 404 returned error can't find the container with id 9086e63ebd66a264bdf2bb5819ff163f15c0eb16db1cdb6308223b7064760e6b
Sep 17 04:00:28.158518 ip-10-0-139-53 kubenswrapper[2552]: I0917 04:00:28.158474    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-nv4hh/must-gather-ptcrm" event={"ID":"70dcfdfd-d8e7-45fb-97cb-afc1c780c720","Type":"ContainerStarted","Data":"9086e63ebd66a264bdf2bb5819ff163f15c0eb16db1cdb6308223b7064760e6b"}
Sep 17 04:00:29.167661 ip-10-0-139-53 kubenswrapper[2552]: I0917 04:00:29.167616    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-nv4hh/must-gather-ptcrm" event={"ID":"70dcfdfd-d8e7-45fb-97cb-afc1c780c720","Type":"ContainerStarted","Data":"626d75ee617a4775be7f2a021cbe1023bfffde2f78005d64df5f5d8c41e6268b"}
Sep 17 04:00:29.167661 ip-10-0-139-53 kubenswrapper[2552]: I0917 04:00:29.167663    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-nv4hh/must-gather-ptcrm" event={"ID":"70dcfdfd-d8e7-45fb-97cb-afc1c780c720","Type":"ContainerStarted","Data":"e73c32041a333b013dfe146bd6cf94c8b1cf2c510c0ece00a5677fae039f7198"}
Sep 17 04:00:29.186508 ip-10-0-139-53 kubenswrapper[2552]: I0917 04:00:29.186440    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-nv4hh/must-gather-ptcrm" podStartSLOduration=2.367861987 podStartE2EDuration="3.186418231s" totalImagesPullingTime="818.556244ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 04:00:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 04:00:29.183950197 +0000 UTC m=+2464.650357723" watchObservedRunningTime="2026-09-17 04:00:29.186418231 +0000 UTC m=+2464.652825759"
Sep 17 04:00:29.742151 ip-10-0-139-53 kubenswrapper[2552]: I0917 04:00:29.742123    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-4944l_d1adf33b-0171-446e-aabe-f60fb74197e3/global-pull-secret-syncer/0.log"
Sep 17 04:00:29.885801 ip-10-0-139-53 kubenswrapper[2552]: I0917 04:00:29.885772    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-8mzx5_2a4ab360-f144-4999-97dc-6796bd4a44c2/konnectivity-agent/0.log"
Sep 17 04:00:29.976255 ip-10-0-139-53 kubenswrapper[2552]: I0917 04:00:29.976228    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-139-53.ec2.internal_ce8d1a2b33a64a4f4ec3c7eebb5340d9/haproxy/0.log"
Sep 17 04:00:34.762507 ip-10-0-139-53 kubenswrapper[2552]: I0917 04:00:34.762459    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5999746d68-vdmx6_b968d89b-da89-4601-9590-3a48d48e7c0c/authorino/0.log"
Sep 17 04:00:34.833409 ip-10-0-139-53 kubenswrapper[2552]: I0917 04:00:34.833379    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-bffwq_c8cc769e-e0d1-40bd-ba4d-3ee56d0c48b4/kuadrant-console-plugin/0.log"
Sep 17 04:00:34.963502 ip-10-0-139-53 kubenswrapper[2552]: I0917 04:00:34.963327    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-hlqc8_ffd803ff-57b5-4e73-ba88-a149336f8be5/manager/0.log"
Sep 17 04:00:36.384538 ip-10-0-139-53 kubenswrapper[2552]: I0917 04:00:36.384508    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_2e583985-a9da-4679-9bd7-ae059f1ce9e9/alertmanager/0.log"
Sep 17 04:00:36.402986 ip-10-0-139-53 kubenswrapper[2552]: I0917 04:00:36.402953    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_2e583985-a9da-4679-9bd7-ae059f1ce9e9/config-reloader/0.log"
Sep 17 04:00:36.438859 ip-10-0-139-53 kubenswrapper[2552]: I0917 04:00:36.438800    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_2e583985-a9da-4679-9bd7-ae059f1ce9e9/kube-rbac-proxy-web/0.log"
Sep 17 04:00:36.455709 ip-10-0-139-53 kubenswrapper[2552]: I0917 04:00:36.455587    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_2e583985-a9da-4679-9bd7-ae059f1ce9e9/kube-rbac-proxy/0.log"
Sep 17 04:00:36.473251 ip-10-0-139-53 kubenswrapper[2552]: I0917 04:00:36.473196    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_2e583985-a9da-4679-9bd7-ae059f1ce9e9/kube-rbac-proxy-metric/0.log"
Sep 17 04:00:36.492953 ip-10-0-139-53 kubenswrapper[2552]: I0917 04:00:36.492924    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_2e583985-a9da-4679-9bd7-ae059f1ce9e9/prom-label-proxy/0.log"
Sep 17 04:00:36.520708 ip-10-0-139-53 kubenswrapper[2552]: I0917 04:00:36.520665    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_2e583985-a9da-4679-9bd7-ae059f1ce9e9/init-config-reloader/0.log"
Sep 17 04:00:36.566923 ip-10-0-139-53 kubenswrapper[2552]: I0917 04:00:36.566888    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6cd954b897-vtbq6_4f8da691-30ee-4f3a-b816-d4d7b7a4b0a0/cluster-monitoring-operator/0.log"
Sep 17 04:00:36.906573 ip-10-0-139-53 kubenswrapper[2552]: I0917 04:00:36.906537    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vhrts_849238c4-27b8-445a-85bb-e2c786c9dbcd/node-exporter/0.log"
Sep 17 04:00:36.946109 ip-10-0-139-53 kubenswrapper[2552]: I0917 04:00:36.946041    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vhrts_849238c4-27b8-445a-85bb-e2c786c9dbcd/kube-rbac-proxy/0.log"
Sep 17 04:00:36.967790 ip-10-0-139-53 kubenswrapper[2552]: I0917 04:00:36.967767    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vhrts_849238c4-27b8-445a-85bb-e2c786c9dbcd/init-textfile/0.log"
Sep 17 04:00:37.369023 ip-10-0-139-53 kubenswrapper[2552]: I0917 04:00:37.368988    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7d84fddb95-j64fs_dc35de27-6f47-48d7-aaf1-0e40ecb04f5c/telemeter-client/0.log"
Sep 17 04:00:37.389837 ip-10-0-139-53 kubenswrapper[2552]: I0917 04:00:37.389807    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7d84fddb95-j64fs_dc35de27-6f47-48d7-aaf1-0e40ecb04f5c/reload/0.log"
Sep 17 04:00:37.413540 ip-10-0-139-53 kubenswrapper[2552]: I0917 04:00:37.413512    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7d84fddb95-j64fs_dc35de27-6f47-48d7-aaf1-0e40ecb04f5c/kube-rbac-proxy/0.log"
Sep 17 04:00:37.443890 ip-10-0-139-53 kubenswrapper[2552]: I0917 04:00:37.443857    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-574675c997-9jwg9_ab812fe0-d6fb-45c0-a99e-41a4e6192740/thanos-query/0.log"
Sep 17 04:00:37.462206 ip-10-0-139-53 kubenswrapper[2552]: I0917 04:00:37.462182    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-574675c997-9jwg9_ab812fe0-d6fb-45c0-a99e-41a4e6192740/kube-rbac-proxy-web/0.log"
Sep 17 04:00:37.480732 ip-10-0-139-53 kubenswrapper[2552]: I0917 04:00:37.480656    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-574675c997-9jwg9_ab812fe0-d6fb-45c0-a99e-41a4e6192740/kube-rbac-proxy/0.log"
Sep 17 04:00:37.511670 ip-10-0-139-53 kubenswrapper[2552]: I0917 04:00:37.511644    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-574675c997-9jwg9_ab812fe0-d6fb-45c0-a99e-41a4e6192740/prom-label-proxy/0.log"
Sep 17 04:00:37.569503 ip-10-0-139-53 kubenswrapper[2552]: I0917 04:00:37.569470    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-574675c997-9jwg9_ab812fe0-d6fb-45c0-a99e-41a4e6192740/kube-rbac-proxy-rules/0.log"
Sep 17 04:00:37.620611 ip-10-0-139-53 kubenswrapper[2552]: I0917 04:00:37.620504    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-574675c997-9jwg9_ab812fe0-d6fb-45c0-a99e-41a4e6192740/kube-rbac-proxy-metrics/0.log"
Sep 17 04:00:38.924633 ip-10-0-139-53 kubenswrapper[2552]: I0917 04:00:38.924583    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-5dcbc7df6b-qtch6_88221296-e256-4d59-aba5-9465442a5727/networking-console-plugin/0.log"
Sep 17 04:00:40.023883 ip-10-0-139-53 kubenswrapper[2552]: I0917 04:00:40.023851    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-76f99584f4-nqq7f_90f3715a-13ff-4388-87e8-84c472dd4917/download-server/0.log"
Sep 17 04:00:40.155004 ip-10-0-139-53 kubenswrapper[2552]: I0917 04:00:40.154956    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-nv4hh/perf-node-gather-daemonset-sbsvg"]
Sep 17 04:00:40.163265 ip-10-0-139-53 kubenswrapper[2552]: I0917 04:00:40.163242    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-nv4hh/perf-node-gather-daemonset-sbsvg"
Sep 17 04:00:40.168127 ip-10-0-139-53 kubenswrapper[2552]: I0917 04:00:40.168072    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-nv4hh/perf-node-gather-daemonset-sbsvg"]
Sep 17 04:00:40.277231 ip-10-0-139-53 kubenswrapper[2552]: I0917 04:00:40.277145    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b1304e8d-4ea3-468e-97a7-5b3d5bc3619a-lib-modules\") pod \"perf-node-gather-daemonset-sbsvg\" (UID: \"b1304e8d-4ea3-468e-97a7-5b3d5bc3619a\") " pod="openshift-must-gather-nv4hh/perf-node-gather-daemonset-sbsvg"
Sep 17 04:00:40.277231 ip-10-0-139-53 kubenswrapper[2552]: I0917 04:00:40.277194    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b1304e8d-4ea3-468e-97a7-5b3d5bc3619a-sys\") pod \"perf-node-gather-daemonset-sbsvg\" (UID: \"b1304e8d-4ea3-468e-97a7-5b3d5bc3619a\") " pod="openshift-must-gather-nv4hh/perf-node-gather-daemonset-sbsvg"
Sep 17 04:00:40.277455 ip-10-0-139-53 kubenswrapper[2552]: I0917 04:00:40.277272    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b1304e8d-4ea3-468e-97a7-5b3d5bc3619a-proc\") pod \"perf-node-gather-daemonset-sbsvg\" (UID: \"b1304e8d-4ea3-468e-97a7-5b3d5bc3619a\") " pod="openshift-must-gather-nv4hh/perf-node-gather-daemonset-sbsvg"
Sep 17 04:00:40.277455 ip-10-0-139-53 kubenswrapper[2552]: I0917 04:00:40.277370    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b1304e8d-4ea3-468e-97a7-5b3d5bc3619a-podres\") pod \"perf-node-gather-daemonset-sbsvg\" (UID: \"b1304e8d-4ea3-468e-97a7-5b3d5bc3619a\") " pod="openshift-must-gather-nv4hh/perf-node-gather-daemonset-sbsvg"
Sep 17 04:00:40.277455 ip-10-0-139-53 kubenswrapper[2552]: I0917 04:00:40.277402    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjv7p\" (UniqueName: \"kubernetes.io/projected/b1304e8d-4ea3-468e-97a7-5b3d5bc3619a-kube-api-access-vjv7p\") pod \"perf-node-gather-daemonset-sbsvg\" (UID: \"b1304e8d-4ea3-468e-97a7-5b3d5bc3619a\") " pod="openshift-must-gather-nv4hh/perf-node-gather-daemonset-sbsvg"
Sep 17 04:00:40.378122 ip-10-0-139-53 kubenswrapper[2552]: I0917 04:00:40.378078    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b1304e8d-4ea3-468e-97a7-5b3d5bc3619a-proc\") pod \"perf-node-gather-daemonset-sbsvg\" (UID: \"b1304e8d-4ea3-468e-97a7-5b3d5bc3619a\") " pod="openshift-must-gather-nv4hh/perf-node-gather-daemonset-sbsvg"
Sep 17 04:00:40.378557 ip-10-0-139-53 kubenswrapper[2552]: I0917 04:00:40.378419    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b1304e8d-4ea3-468e-97a7-5b3d5bc3619a-proc\") pod \"perf-node-gather-daemonset-sbsvg\" (UID: \"b1304e8d-4ea3-468e-97a7-5b3d5bc3619a\") " pod="openshift-must-gather-nv4hh/perf-node-gather-daemonset-sbsvg"
Sep 17 04:00:40.378809 ip-10-0-139-53 kubenswrapper[2552]: I0917 04:00:40.378757    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b1304e8d-4ea3-468e-97a7-5b3d5bc3619a-podres\") pod \"perf-node-gather-daemonset-sbsvg\" (UID: \"b1304e8d-4ea3-468e-97a7-5b3d5bc3619a\") " pod="openshift-must-gather-nv4hh/perf-node-gather-daemonset-sbsvg"
Sep 17 04:00:40.379153 ip-10-0-139-53 kubenswrapper[2552]: I0917 04:00:40.379126    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjv7p\" (UniqueName: \"kubernetes.io/projected/b1304e8d-4ea3-468e-97a7-5b3d5bc3619a-kube-api-access-vjv7p\") pod \"perf-node-gather-daemonset-sbsvg\" (UID: \"b1304e8d-4ea3-468e-97a7-5b3d5bc3619a\") " pod="openshift-must-gather-nv4hh/perf-node-gather-daemonset-sbsvg"
Sep 17 04:00:40.379727 ip-10-0-139-53 kubenswrapper[2552]: I0917 04:00:40.379045    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b1304e8d-4ea3-468e-97a7-5b3d5bc3619a-podres\") pod \"perf-node-gather-daemonset-sbsvg\" (UID: \"b1304e8d-4ea3-468e-97a7-5b3d5bc3619a\") " pod="openshift-must-gather-nv4hh/perf-node-gather-daemonset-sbsvg"
Sep 17 04:00:40.379999 ip-10-0-139-53 kubenswrapper[2552]: I0917 04:00:40.379959    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b1304e8d-4ea3-468e-97a7-5b3d5bc3619a-lib-modules\") pod \"perf-node-gather-daemonset-sbsvg\" (UID: \"b1304e8d-4ea3-468e-97a7-5b3d5bc3619a\") " pod="openshift-must-gather-nv4hh/perf-node-gather-daemonset-sbsvg"
Sep 17 04:00:40.380150 ip-10-0-139-53 kubenswrapper[2552]: I0917 04:00:40.379692    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b1304e8d-4ea3-468e-97a7-5b3d5bc3619a-lib-modules\") pod \"perf-node-gather-daemonset-sbsvg\" (UID: \"b1304e8d-4ea3-468e-97a7-5b3d5bc3619a\") " pod="openshift-must-gather-nv4hh/perf-node-gather-daemonset-sbsvg"
Sep 17 04:00:40.380257 ip-10-0-139-53 kubenswrapper[2552]: I0917 04:00:40.380196    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b1304e8d-4ea3-468e-97a7-5b3d5bc3619a-sys\") pod \"perf-node-gather-daemonset-sbsvg\" (UID: \"b1304e8d-4ea3-468e-97a7-5b3d5bc3619a\") " pod="openshift-must-gather-nv4hh/perf-node-gather-daemonset-sbsvg"
Sep 17 04:00:40.380302 ip-10-0-139-53 kubenswrapper[2552]: I0917 04:00:40.380292    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b1304e8d-4ea3-468e-97a7-5b3d5bc3619a-sys\") pod \"perf-node-gather-daemonset-sbsvg\" (UID: \"b1304e8d-4ea3-468e-97a7-5b3d5bc3619a\") " pod="openshift-must-gather-nv4hh/perf-node-gather-daemonset-sbsvg"
Sep 17 04:00:40.388463 ip-10-0-139-53 kubenswrapper[2552]: I0917 04:00:40.388435    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjv7p\" (UniqueName: \"kubernetes.io/projected/b1304e8d-4ea3-468e-97a7-5b3d5bc3619a-kube-api-access-vjv7p\") pod \"perf-node-gather-daemonset-sbsvg\" (UID: \"b1304e8d-4ea3-468e-97a7-5b3d5bc3619a\") " pod="openshift-must-gather-nv4hh/perf-node-gather-daemonset-sbsvg"
Sep 17 04:00:40.481974 ip-10-0-139-53 kubenswrapper[2552]: I0917 04:00:40.481935    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-nv4hh/perf-node-gather-daemonset-sbsvg"
Sep 17 04:00:40.698086 ip-10-0-139-53 kubenswrapper[2552]: I0917 04:00:40.698044    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-86695b7cc7-srhm6_bf40e783-559e-4067-b28b-5016ab6fb8c8/volume-data-source-validator/0.log"
Sep 17 04:00:40.779686 ip-10-0-139-53 kubenswrapper[2552]: I0917 04:00:40.779655    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-nv4hh/perf-node-gather-daemonset-sbsvg"]
Sep 17 04:00:40.793500 ip-10-0-139-53 kubenswrapper[2552]: W0917 04:00:40.793459    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podb1304e8d_4ea3_468e_97a7_5b3d5bc3619a.slice/crio-fb6f8c83abba673a65c9d783dfc9887d422b54e1f146e695e200f589239a66e3 WatchSource:0}: Error finding container fb6f8c83abba673a65c9d783dfc9887d422b54e1f146e695e200f589239a66e3: Status 404 returned error can't find the container with id fb6f8c83abba673a65c9d783dfc9887d422b54e1f146e695e200f589239a66e3
Sep 17 04:00:41.263509 ip-10-0-139-53 kubenswrapper[2552]: I0917 04:00:41.263466    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-nv4hh/perf-node-gather-daemonset-sbsvg" event={"ID":"b1304e8d-4ea3-468e-97a7-5b3d5bc3619a","Type":"ContainerStarted","Data":"0de67f18c13f64b35c431f113f896e62f943f152cf2bc418333a4b81ba4ef4da"}
Sep 17 04:00:41.264017 ip-10-0-139-53 kubenswrapper[2552]: I0917 04:00:41.263573    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-nv4hh/perf-node-gather-daemonset-sbsvg" event={"ID":"b1304e8d-4ea3-468e-97a7-5b3d5bc3619a","Type":"ContainerStarted","Data":"fb6f8c83abba673a65c9d783dfc9887d422b54e1f146e695e200f589239a66e3"}
Sep 17 04:00:41.264499 ip-10-0-139-53 kubenswrapper[2552]: I0917 04:00:41.264473    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-nv4hh/perf-node-gather-daemonset-sbsvg"
Sep 17 04:00:41.282258 ip-10-0-139-53 kubenswrapper[2552]: I0917 04:00:41.282194    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-nv4hh/perf-node-gather-daemonset-sbsvg" podStartSLOduration=1.282173479 podStartE2EDuration="1.282173479s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 04:00:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 04:00:41.279738609 +0000 UTC m=+2476.746146140" watchObservedRunningTime="2026-09-17 04:00:41.282173479 +0000 UTC m=+2476.748581006"
Sep 17 04:00:41.894098 ip-10-0-139-53 kubenswrapper[2552]: I0917 04:00:41.894075    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-vb2fm_a7d922cf-74cc-4afb-b900-b21d63d4a225/dns/0.log"
Sep 17 04:00:41.911630 ip-10-0-139-53 kubenswrapper[2552]: I0917 04:00:41.911583    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-vb2fm_a7d922cf-74cc-4afb-b900-b21d63d4a225/kube-rbac-proxy/0.log"
Sep 17 04:00:41.973490 ip-10-0-139-53 kubenswrapper[2552]: I0917 04:00:41.973464    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-gxmbv_b3bac7f7-fbb1-4cec-8a26-046b702f4d70/dns-node-resolver/0.log"
Sep 17 04:00:42.482395 ip-10-0-139-53 kubenswrapper[2552]: I0917 04:00:42.482367    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-7595f9b8-kknw8_67b60bb3-13c4-4e20-a1b6-e6d766d8bcbe/registry/0.log"
Sep 17 04:00:42.522400 ip-10-0-139-53 kubenswrapper[2552]: I0917 04:00:42.522368    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-rld2q_b7b79096-9bfc-4d82-9406-073a3f7961d0/node-ca/0.log"
Sep 17 04:00:43.876291 ip-10-0-139-53 kubenswrapper[2552]: I0917 04:00:43.876259    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5bc6dc7888-txj6c_60cb769a-2eb5-433e-b97d-5ae489598831/kube-auth-proxy/0.log"
Sep 17 04:00:44.877497 ip-10-0-139-53 kubenswrapper[2552]: I0917 04:00:44.877465    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-tbtsz_1a74da51-05c3-4067-b3a5-233734643f6e/serve-healthcheck-canary/0.log"
Sep 17 04:00:45.313536 ip-10-0-139-53 kubenswrapper[2552]: I0917 04:00:45.313502    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-68rbk_0f65c6b8-41ff-4572-9e32-c14e524fa713/insights-operator/0.log"
Sep 17 04:00:45.314498 ip-10-0-139-53 kubenswrapper[2552]: I0917 04:00:45.314460    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-68rbk_0f65c6b8-41ff-4572-9e32-c14e524fa713/insights-operator/1.log"
Sep 17 04:00:45.388810 ip-10-0-139-53 kubenswrapper[2552]: I0917 04:00:45.388780    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-4dd7t_8d497c2d-d126-4f70-a968-d7371e39793f/kube-rbac-proxy/0.log"
Sep 17 04:00:45.406473 ip-10-0-139-53 kubenswrapper[2552]: I0917 04:00:45.406446    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-4dd7t_8d497c2d-d126-4f70-a968-d7371e39793f/exporter/0.log"
Sep 17 04:00:45.424002 ip-10-0-139-53 kubenswrapper[2552]: I0917 04:00:45.423973    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-4dd7t_8d497c2d-d126-4f70-a968-d7371e39793f/extractor/0.log"
Sep 17 04:00:48.071862 ip-10-0-139-53 kubenswrapper[2552]: I0917 04:00:48.071823    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6977f7b7bc-css5r_f512d104-3bf3-4414-bed2-f7440c18ee3b/manager/0.log"
Sep 17 04:00:48.086644 ip-10-0-139-53 kubenswrapper[2552]: I0917 04:00:48.086620    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6977f7b7bc-css5r_f512d104-3bf3-4414-bed2-f7440c18ee3b/copy-manifests/0.log"
Sep 17 04:00:48.291138 ip-10-0-139-53 kubenswrapper[2552]: I0917 04:00:48.291111    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-nv4hh/perf-node-gather-daemonset-sbsvg"
Sep 17 04:00:49.387155 ip-10-0-139-53 kubenswrapper[2552]: I0917 04:00:49.387125    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-55d557bdbb-vbnn4_e13f17f0-340c-44a3-8974-330973d98f60/manager/0.log"
Sep 17 04:00:50.543862 ip-10-0-139-53 kubenswrapper[2552]: I0917 04:00:50.543833    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5bc6dc7888-txj6c_60cb769a-2eb5-433e-b97d-5ae489598831/kube-auth-proxy/0.log"
Sep 17 04:00:52.708528 ip-10-0-139-53 kubenswrapper[2552]: I0917 04:00:52.708502    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5bc6dc7888-txj6c_60cb769a-2eb5-433e-b97d-5ae489598831/kube-auth-proxy/0.log"
Sep 17 04:00:55.215073 ip-10-0-139-53 kubenswrapper[2552]: I0917 04:00:55.215041    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-khv5k_91c3490d-d515-4cc2-bafb-aa6c999ac3d0/kube-storage-version-migrator-operator/1.log"
Sep 17 04:00:55.215908 ip-10-0-139-53 kubenswrapper[2552]: I0917 04:00:55.215852    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-khv5k_91c3490d-d515-4cc2-bafb-aa6c999ac3d0/kube-storage-version-migrator-operator/0.log"
Sep 17 04:00:56.388610 ip-10-0-139-53 kubenswrapper[2552]: I0917 04:00:56.388566    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nkrxz_0cc4aedf-1bb8-47c6-88f2-266bbbf99058/kube-multus-additional-cni-plugins/0.log"
Sep 17 04:00:56.404806 ip-10-0-139-53 kubenswrapper[2552]: I0917 04:00:56.404778    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nkrxz_0cc4aedf-1bb8-47c6-88f2-266bbbf99058/egress-router-binary-copy/0.log"
Sep 17 04:00:56.420674 ip-10-0-139-53 kubenswrapper[2552]: I0917 04:00:56.420627    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nkrxz_0cc4aedf-1bb8-47c6-88f2-266bbbf99058/cni-plugins/0.log"
Sep 17 04:00:56.439729 ip-10-0-139-53 kubenswrapper[2552]: I0917 04:00:56.439707    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nkrxz_0cc4aedf-1bb8-47c6-88f2-266bbbf99058/bond-cni-plugin/0.log"
Sep 17 04:00:56.456562 ip-10-0-139-53 kubenswrapper[2552]: I0917 04:00:56.456537    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nkrxz_0cc4aedf-1bb8-47c6-88f2-266bbbf99058/routeoverride-cni/0.log"
Sep 17 04:00:56.471777 ip-10-0-139-53 kubenswrapper[2552]: I0917 04:00:56.471754    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nkrxz_0cc4aedf-1bb8-47c6-88f2-266bbbf99058/whereabouts-cni-bincopy/0.log"
Sep 17 04:00:56.488528 ip-10-0-139-53 kubenswrapper[2552]: I0917 04:00:56.488510    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nkrxz_0cc4aedf-1bb8-47c6-88f2-266bbbf99058/whereabouts-cni/0.log"
Sep 17 04:00:56.712945 ip-10-0-139-53 kubenswrapper[2552]: I0917 04:00:56.712872    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-wds8n_c6a03737-feb7-48d2-82ba-5a2188f80980/kube-multus/0.log"
Sep 17 04:00:56.838813 ip-10-0-139-53 kubenswrapper[2552]: I0917 04:00:56.838786    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-xl9wh_2758c0c8-e1ff-454a-9f0b-43cc8e1f8609/network-metrics-daemon/0.log"
Sep 17 04:00:56.852401 ip-10-0-139-53 kubenswrapper[2552]: I0917 04:00:56.852379    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-xl9wh_2758c0c8-e1ff-454a-9f0b-43cc8e1f8609/kube-rbac-proxy/0.log"
Sep 17 04:00:57.606112 ip-10-0-139-53 kubenswrapper[2552]: I0917 04:00:57.606085    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ldhl4_d3e92f88-fcb1-4d0f-b653-77b7ec5681fd/ovn-controller/0.log"
Sep 17 04:00:57.621295 ip-10-0-139-53 kubenswrapper[2552]: I0917 04:00:57.621271    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ldhl4_d3e92f88-fcb1-4d0f-b653-77b7ec5681fd/ovn-acl-logging/0.log"
Sep 17 04:00:57.640067 ip-10-0-139-53 kubenswrapper[2552]: I0917 04:00:57.640043    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ldhl4_d3e92f88-fcb1-4d0f-b653-77b7ec5681fd/kube-rbac-proxy-node/0.log"
Sep 17 04:00:57.657067 ip-10-0-139-53 kubenswrapper[2552]: I0917 04:00:57.657045    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ldhl4_d3e92f88-fcb1-4d0f-b653-77b7ec5681fd/kube-rbac-proxy-ovn-metrics/0.log"
Sep 17 04:00:57.674643 ip-10-0-139-53 kubenswrapper[2552]: I0917 04:00:57.674625    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ldhl4_d3e92f88-fcb1-4d0f-b653-77b7ec5681fd/northd/0.log"
Sep 17 04:00:57.692093 ip-10-0-139-53 kubenswrapper[2552]: I0917 04:00:57.692073    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ldhl4_d3e92f88-fcb1-4d0f-b653-77b7ec5681fd/nbdb/0.log"
Sep 17 04:00:57.712535 ip-10-0-139-53 kubenswrapper[2552]: I0917 04:00:57.712515    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ldhl4_d3e92f88-fcb1-4d0f-b653-77b7ec5681fd/sbdb/0.log"
Sep 17 04:00:57.777732 ip-10-0-139-53 kubenswrapper[2552]: I0917 04:00:57.777708    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ldhl4_d3e92f88-fcb1-4d0f-b653-77b7ec5681fd/ovnkube-controller/0.log"