Sep 08 15:30:38.909108 ip-10-0-143-61 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 08 15:30:38.909119 ip-10-0-143-61 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 08 15:30:38.909128 ip-10-0-143-61 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 08 15:30:38.909369 ip-10-0-143-61 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 08 15:30:49.143268 ip-10-0-143-61 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 08 15:30:49.143281 ip-10-0-143-61 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot ef9f6169c61549dcaa2bb8973a53894b --
Sep 08 15:32:41.797129 ip-10-0-143-61 systemd[1]: Starting Kubernetes Kubelet...
Sep 08 15:32:42.187267 ip-10-0-143-61 kubenswrapper[2549]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 08 15:32:42.187267 ip-10-0-143-61 kubenswrapper[2549]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 08 15:32:42.187267 ip-10-0-143-61 kubenswrapper[2549]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 08 15:32:42.187267 ip-10-0-143-61 kubenswrapper[2549]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 08 15:32:42.187267 ip-10-0-143-61 kubenswrapper[2549]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 08 15:32:42.189135 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.189060    2549 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 08 15:32:42.195346 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.195323    2549 feature_gate.go:328] unrecognized feature gate: Example
Sep 08 15:32:42.195346 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.195341    2549 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 08 15:32:42.195346 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.195345    2549 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 08 15:32:42.195346 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.195349    2549 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 08 15:32:42.195346 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.195352    2549 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 08 15:32:42.195539 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.195355    2549 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 08 15:32:42.195539 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.195358    2549 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 08 15:32:42.195539 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.195361    2549 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 08 15:32:42.195539 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.195364    2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 08 15:32:42.195539 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.195366    2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 08 15:32:42.195539 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.195369    2549 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 08 15:32:42.195539 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.195372    2549 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 08 15:32:42.195539 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.195375    2549 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 08 15:32:42.195539 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.195377    2549 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 08 15:32:42.195539 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.195380    2549 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 08 15:32:42.195539 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.195383    2549 feature_gate.go:328] unrecognized feature gate: Example2
Sep 08 15:32:42.195539 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.195385    2549 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 08 15:32:42.195539 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.195388    2549 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 08 15:32:42.195539 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.195392    2549 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 08 15:32:42.195539 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.195397    2549 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 08 15:32:42.195539 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.195402    2549 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 08 15:32:42.195539 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.195405    2549 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 08 15:32:42.195539 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.195408    2549 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 08 15:32:42.195539 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.195410    2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 08 15:32:42.195539 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.195413    2549 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 08 15:32:42.196040 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.195416    2549 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 08 15:32:42.196040 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.195419    2549 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 08 15:32:42.196040 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.195421    2549 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 08 15:32:42.196040 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.195427    2549 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 08 15:32:42.196040 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.195429    2549 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 08 15:32:42.196040 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.195432    2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 08 15:32:42.196040 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.195434    2549 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 08 15:32:42.196040 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.195437    2549 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 08 15:32:42.196040 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.195439    2549 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 08 15:32:42.196040 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.195442    2549 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 08 15:32:42.196040 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.195444    2549 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 08 15:32:42.196040 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.195447    2549 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 08 15:32:42.196040 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.195450    2549 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 08 15:32:42.196040 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.195453    2549 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 08 15:32:42.196040 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.195456    2549 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 08 15:32:42.196040 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.195458    2549 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 08 15:32:42.196040 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.195461    2549 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 08 15:32:42.196040 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.195463    2549 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 08 15:32:42.196040 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.195466    2549 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 08 15:32:42.196040 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.195468    2549 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 08 15:32:42.196609 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.195471    2549 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 08 15:32:42.196609 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.195474    2549 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 08 15:32:42.196609 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.195476    2549 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 08 15:32:42.196609 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.195479    2549 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 08 15:32:42.196609 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.195482    2549 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 08 15:32:42.196609 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.195485    2549 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 08 15:32:42.196609 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.195488    2549 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 08 15:32:42.196609 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.195490    2549 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 08 15:32:42.196609 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.195492    2549 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 08 15:32:42.196609 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.195495    2549 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 08 15:32:42.196609 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.195497    2549 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 08 15:32:42.196609 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.195500    2549 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 08 15:32:42.196609 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.195503    2549 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 08 15:32:42.196609 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.195505    2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 08 15:32:42.196609 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.195508    2549 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 08 15:32:42.196609 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.195511    2549 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 08 15:32:42.196609 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.195514    2549 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 08 15:32:42.196609 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.195516    2549 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 08 15:32:42.196609 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.195519    2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 08 15:32:42.196609 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.195521    2549 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 08 15:32:42.197155 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.195523    2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 08 15:32:42.197155 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.195526    2549 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 08 15:32:42.197155 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.195530    2549 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 08 15:32:42.197155 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.195533    2549 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 08 15:32:42.197155 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.195539    2549 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 08 15:32:42.197155 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.195542    2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 08 15:32:42.197155 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.195546    2549 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 08 15:32:42.197155 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.195549    2549 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 08 15:32:42.197155 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.195552    2549 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 08 15:32:42.197155 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.195555    2549 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 08 15:32:42.197155 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.195558    2549 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 08 15:32:42.197155 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.195561    2549 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 08 15:32:42.197155 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.195564    2549 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 08 15:32:42.197155 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.195566    2549 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 08 15:32:42.197155 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.195569    2549 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 08 15:32:42.197155 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.195572    2549 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 08 15:32:42.197155 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.195574    2549 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 08 15:32:42.197155 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.195577    2549 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 08 15:32:42.197155 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.195581    2549 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 08 15:32:42.197629 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.195583    2549 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 08 15:32:42.197629 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.195586    2549 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 08 15:32:42.197629 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.195589    2549 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 08 15:32:42.197629 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.195592    2549 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 08 15:32:42.197629 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.195594    2549 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 08 15:32:42.197629 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.195598    2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 08 15:32:42.197629 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.195601    2549 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 08 15:32:42.197629 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.195603    2549 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 08 15:32:42.197629 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.195606    2549 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 08 15:32:42.197629 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.195609    2549 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 08 15:32:42.197629 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.195612    2549 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 08 15:32:42.197629 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.195615    2549 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 08 15:32:42.197629 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.195618    2549 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 08 15:32:42.197629 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.195621    2549 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 08 15:32:42.197629 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.195624    2549 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 08 15:32:42.197629 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.197389    2549 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 08 15:32:42.197629 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.197398    2549 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 08 15:32:42.197629 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.197403    2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 08 15:32:42.197629 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.197406    2549 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 08 15:32:42.198100 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.197409    2549 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 08 15:32:42.198100 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.197412    2549 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 08 15:32:42.198100 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.197415    2549 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 08 15:32:42.198100 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.197418    2549 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 08 15:32:42.198100 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.197421    2549 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 08 15:32:42.198100 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.197424    2549 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 08 15:32:42.198100 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.197427    2549 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 08 15:32:42.198100 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.197430    2549 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 08 15:32:42.198100 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.197432    2549 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 08 15:32:42.198100 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.197435    2549 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 08 15:32:42.198100 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.197438    2549 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 08 15:32:42.198100 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.197442    2549 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 08 15:32:42.198100 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.197445    2549 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 08 15:32:42.198100 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.197449    2549 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 08 15:32:42.198100 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.197453    2549 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 08 15:32:42.198100 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.197456    2549 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 08 15:32:42.198100 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.197459    2549 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 08 15:32:42.198100 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.197462    2549 feature_gate.go:328] unrecognized feature gate: Example2
Sep 08 15:32:42.198100 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.197465    2549 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 08 15:32:42.198100 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.197468    2549 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 08 15:32:42.198630 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.197470    2549 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 08 15:32:42.198630 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.197473    2549 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 08 15:32:42.198630 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.197476    2549 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 08 15:32:42.198630 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.197478    2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 08 15:32:42.198630 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.197481    2549 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 08 15:32:42.198630 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.197484    2549 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 08 15:32:42.198630 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.197486    2549 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 08 15:32:42.198630 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.197489    2549 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 08 15:32:42.198630 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.197492    2549 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 08 15:32:42.198630 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.197494    2549 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 08 15:32:42.198630 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.197497    2549 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 08 15:32:42.198630 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.197500    2549 feature_gate.go:328] unrecognized feature gate: Example
Sep 08 15:32:42.198630 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.197502    2549 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 08 15:32:42.198630 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.197505    2549 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 08 15:32:42.198630 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.197508    2549 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 08 15:32:42.198630 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.197510    2549 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 08 15:32:42.198630 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.197513    2549 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 08 15:32:42.198630 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.197516    2549 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 08 15:32:42.198630 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.197518    2549 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 08 15:32:42.199112 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.197521    2549 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 08 15:32:42.199112 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.197523    2549 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 08 15:32:42.199112 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.197526    2549 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 08 15:32:42.199112 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.197530    2549 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 08 15:32:42.199112 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.197533    2549 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 08 15:32:42.199112 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.197536    2549 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 08 15:32:42.199112 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.197539    2549 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 08 15:32:42.199112 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.197543    2549 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 08 15:32:42.199112 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.197546    2549 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 08 15:32:42.199112 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.197549    2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 08 15:32:42.199112 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.197552    2549 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 08 15:32:42.199112 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.197555    2549 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 08 15:32:42.199112 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.197558    2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 08 15:32:42.199112 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.197560    2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 08 15:32:42.199112 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.197563    2549 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 08 15:32:42.199112 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.197566    2549 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 08 15:32:42.199112 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.197568    2549 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 08 15:32:42.199112 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.197571    2549 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 08 15:32:42.199112 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.197573    2549 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 08 15:32:42.199584 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.197577    2549 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 08 15:32:42.199584 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.197579    2549 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 08 15:32:42.199584 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.197592    2549 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 08 15:32:42.199584 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.197595    2549 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 08 15:32:42.199584 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.197598    2549 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 08 15:32:42.199584 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.197600    2549 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 08 15:32:42.199584 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.197603    2549 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 08 15:32:42.199584 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.197605    2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 08 15:32:42.199584 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.197608    2549 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 08 15:32:42.199584 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.197611    2549 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 08 15:32:42.199584 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.197614    2549 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 08 15:32:42.199584 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.197616    2549 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 08 15:32:42.199584 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.197619    2549 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 08 15:32:42.199584 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.197622    2549 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 08 15:32:42.199584 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.197624    2549 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 08 15:32:42.199584 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.197627    2549 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 08 15:32:42.199584 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.197629    2549 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 08 15:32:42.199584 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.197632    2549 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 08 15:32:42.199584 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.197635    2549 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 08 15:32:42.199584 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.197638    2549 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 08 15:32:42.200094 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.197641    2549 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 08 15:32:42.200094 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.197645    2549 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 08 15:32:42.200094 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.197649    2549 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 08 15:32:42.200094 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.197651    2549 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 08 15:32:42.200094 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.197655    2549 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 08 15:32:42.200094 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.197658    2549 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 08 15:32:42.200094 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.197661    2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 08 15:32:42.200094 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.197663    2549 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 08 15:32:42.200094 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.197666    2549 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 08 15:32:42.200094 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.197668    2549 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 08 15:32:42.200094 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.197671    2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 08 15:32:42.200094 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.197673    2549 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 08 15:32:42.200094 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.197676    2549 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 08 15:32:42.200094 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.197678    2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 08 15:32:42.200094 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.197681    2549 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 08 15:32:42.200094 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.197683    2549 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 08 15:32:42.200094 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.197686    2549 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 08 15:32:42.200094 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.198066    2549 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 08 15:32:42.200094 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.198072    2549 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 08 15:32:42.200094 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.198075    2549 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 08 15:32:42.200590 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.198078    2549 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 08 15:32:42.200590 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.198080    2549 feature_gate.go:328] unrecognized feature gate: Example2
Sep 08 15:32:42.200590 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.198084    2549 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 08 15:32:42.200590 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.198086    2549 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 08 15:32:42.200590 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.198089    2549 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 08 15:32:42.200590 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.198092    2549 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 08 15:32:42.200590 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.198096    2549 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 08 15:32:42.200590 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.198100    2549 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 08 15:32:42.200590 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.198103    2549 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 08 15:32:42.200590 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.198106    2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 08 15:32:42.200590 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.198109    2549 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 08 15:32:42.200590 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.198112    2549 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 08 15:32:42.200590 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.198115    2549 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 08 15:32:42.200590 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.198118    2549 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 08 15:32:42.200590 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.198122    2549 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 08 15:32:42.200590 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.198124    2549 feature_gate.go:328] unrecognized feature gate: Example
Sep 08 15:32:42.200590 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.198127    2549 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 08 15:32:42.200590 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.198130    2549 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 08 15:32:42.200590 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.198132    2549 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 08 15:32:42.201069 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.198136    2549 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 08 15:32:42.201069 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.198138    2549 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 08 15:32:42.201069 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.198140    2549 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 08 15:32:42.201069 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.198143    2549 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 08 15:32:42.201069 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.198146    2549 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 08 15:32:42.201069 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.198148    2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 08 15:32:42.201069 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.198151    2549 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 08 15:32:42.201069 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.198154    2549 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 08 15:32:42.201069 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.198156    2549 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 08 15:32:42.201069 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.198159    2549 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 08 15:32:42.201069 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.198161    2549 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 08 15:32:42.201069 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.198164    2549 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 08 15:32:42.201069 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.198166    2549 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 08 15:32:42.201069 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.198169    2549 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 08 15:32:42.201069 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.198171    2549 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 08 15:32:42.201069 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.198174    2549 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 08 15:32:42.201069 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.198177    2549 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 08 15:32:42.201069 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.198180    2549 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 08 15:32:42.201069 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.198182    2549 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 08 15:32:42.201069 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.198185    2549 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 08 15:32:42.201575 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.198189    2549 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 08 15:32:42.201575 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.198192    2549 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 08 15:32:42.201575 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.198196    2549 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 08 15:32:42.201575 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.198200    2549 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 08 15:32:42.201575 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.198202    2549 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 08 15:32:42.201575 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.198205    2549 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 08 15:32:42.201575 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.198208    2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 08 15:32:42.201575 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.198211    2549 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 08 15:32:42.201575 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.198213    2549 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 08 15:32:42.201575 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.198216    2549 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 08 15:32:42.201575 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.198219    2549 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 08 15:32:42.201575 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.198221    2549 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 08 15:32:42.201575 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.198224    2549 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 08 15:32:42.201575 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.198226    2549 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 08 15:32:42.201575 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.198229    2549 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 08 15:32:42.201575 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.198231    2549 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 08 15:32:42.201575 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.198234    2549 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 08 15:32:42.201575 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.198236    2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 08 15:32:42.201575 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.198240    2549 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 08 15:32:42.202094 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.198243    2549 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 08 15:32:42.202094 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.198245    2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 08 15:32:42.202094 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.198248    2549 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 08 15:32:42.202094 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.198250    2549 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 08 15:32:42.202094 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.198253    2549 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 08 15:32:42.202094 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.198256    2549 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 08 15:32:42.202094 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.198258    2549 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 08 15:32:42.202094 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.198260    2549 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 08 15:32:42.202094 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.198263    2549 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 08 15:32:42.202094 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.198265    2549 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 08 15:32:42.202094 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.198268    2549 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 08 15:32:42.202094 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.198271    2549 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 08 15:32:42.202094 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.198273    2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 08 15:32:42.202094 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.198276    2549 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 08 15:32:42.202094 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.198279    2549 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 08 15:32:42.202094 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.198281    2549 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 08 15:32:42.202094 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.198284    2549 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 08 15:32:42.202094 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.198286    2549 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 08 15:32:42.202094 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.198289    2549 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 08 15:32:42.202094 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.198292    2549 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 08 15:32:42.202581 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.198294    2549 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 08 15:32:42.202581 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.198297    2549 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 08 15:32:42.202581 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.198300    2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 08 15:32:42.202581 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.198302    2549 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 08 15:32:42.202581 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.198305    2549 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 08 15:32:42.202581 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.198307    2549 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 08 15:32:42.202581 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.198309    2549 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 08 15:32:42.202581 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.198312    2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 08 15:32:42.202581 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.198314    2549 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 08 15:32:42.202581 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.198317    2549 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 08 15:32:42.202581 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.198319    2549 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 08 15:32:42.202581 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.198322    2549 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 08 15:32:42.202581 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.198324    2549 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 08 15:32:42.202581 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.198328    2549 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 08 15:32:42.202581 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.198331    2549 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 08 15:32:42.202581 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.198333    2549 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 08 15:32:42.202581 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.198336    2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 08 15:32:42.202581 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.198338    2549 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 08 15:32:42.202581 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.198410    2549 flags.go:64] FLAG: --address="0.0.0.0"
Sep 08 15:32:42.202581 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.198444    2549 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 08 15:32:42.203086 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.198450    2549 flags.go:64] FLAG: --anonymous-auth="true"
Sep 08 15:32:42.203086 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.198455    2549 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 08 15:32:42.203086 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.198464    2549 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 08 15:32:42.203086 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.198467    2549 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 08 15:32:42.203086 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.198471    2549 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 08 15:32:42.203086 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.198479    2549 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 08 15:32:42.203086 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.198482    2549 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 08 15:32:42.203086 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.198485    2549 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 08 15:32:42.203086 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.198488    2549 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 08 15:32:42.203086 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.198492    2549 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 08 15:32:42.203086 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.198497    2549 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 08 15:32:42.203086 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.198500    2549 flags.go:64] FLAG: --cgroup-root=""
Sep 08 15:32:42.203086 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.198503    2549 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 08 15:32:42.203086 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.198505    2549 flags.go:64] FLAG: --client-ca-file=""
Sep 08 15:32:42.203086 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.198509    2549 flags.go:64] FLAG: --cloud-provider="external"
Sep 08 15:32:42.203086 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.198512    2549 flags.go:64] FLAG: --cluster-dns="[]"
Sep 08 15:32:42.203086 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.198515    2549 flags.go:64] FLAG: --cluster-domain=""
Sep 08 15:32:42.203086 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.198518    2549 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 08 15:32:42.203086 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.198521    2549 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 08 15:32:42.203086 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.198524    2549 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 08 15:32:42.203086 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.198527    2549 flags.go:64] FLAG: --container-log-max-files="5"
Sep 08 15:32:42.203086 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.198530    2549 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 08 15:32:42.203086 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.198533    2549 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 08 15:32:42.203666 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.198536    2549 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 08 15:32:42.203666 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.198541    2549 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 08 15:32:42.203666 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.198544    2549 flags.go:64] FLAG: --contention-profiling="false"
Sep 08 15:32:42.203666 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.198547    2549 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 08 15:32:42.203666 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.198550    2549 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 08 15:32:42.203666 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.198553    2549 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 08 15:32:42.203666 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.198555    2549 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 08 15:32:42.203666 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.198559    2549 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 08 15:32:42.203666 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.198562    2549 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 08 15:32:42.203666 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.198565    2549 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 08 15:32:42.203666 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.198568    2549 flags.go:64] FLAG: --enable-load-reader="false"
Sep 08 15:32:42.203666 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.198570    2549 flags.go:64] FLAG: --enable-server="true"
Sep 08 15:32:42.203666 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.198573    2549 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 08 15:32:42.203666 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.198577    2549 flags.go:64] FLAG: --event-burst="100"
Sep 08 15:32:42.203666 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.198580    2549 flags.go:64] FLAG: --event-qps="50"
Sep 08 15:32:42.203666 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.198583    2549 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 08 15:32:42.203666 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.198586    2549 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 08 15:32:42.203666 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.198589    2549 flags.go:64] FLAG: --eviction-hard=""
Sep 08 15:32:42.203666 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.198592    2549 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 08 15:32:42.203666 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.198596    2549 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 08 15:32:42.203666 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.198599    2549 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 08 15:32:42.203666 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.198602    2549 flags.go:64] FLAG: --eviction-soft=""
Sep 08 15:32:42.203666 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.198605    2549 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 08 15:32:42.203666 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.198608    2549 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 08 15:32:42.203666 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.198612    2549 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 08 15:32:42.204316 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.198615    2549 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 08 15:32:42.204316 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.198617    2549 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 08 15:32:42.204316 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.198620    2549 flags.go:64] FLAG: --fail-swap-on="true"
Sep 08 15:32:42.204316 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.198623    2549 flags.go:64] FLAG: --feature-gates=""
Sep 08 15:32:42.204316 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.198627    2549 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 08 15:32:42.204316 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.198630    2549 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 08 15:32:42.204316 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.198633    2549 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 08 15:32:42.204316 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.198636    2549 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 08 15:32:42.204316 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.198642    2549 flags.go:64] FLAG: --healthz-port="10248"
Sep 08 15:32:42.204316 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.198645    2549 flags.go:64] FLAG: --help="false"
Sep 08 15:32:42.204316 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.198648    2549 flags.go:64] FLAG: --hostname-override="ip-10-0-143-61.ec2.internal"
Sep 08 15:32:42.204316 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.198651    2549 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 08 15:32:42.204316 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.198654    2549 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 08 15:32:42.204316 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.198656    2549 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 08 15:32:42.204316 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.198660    2549 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 08 15:32:42.204316 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.198663    2549 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 08 15:32:42.204316 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.198666    2549 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 08 15:32:42.204316 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.198669    2549 flags.go:64] FLAG: --image-service-endpoint=""
Sep 08 15:32:42.204316 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.198671    2549 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 08 15:32:42.204316 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.198674    2549 flags.go:64] FLAG: --kube-api-burst="100"
Sep 08 15:32:42.204316 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.198677    2549 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 08 15:32:42.204316 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.198680    2549 flags.go:64] FLAG: --kube-api-qps="50"
Sep 08 15:32:42.204316 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.198683    2549 flags.go:64] FLAG: --kube-reserved=""
Sep 08 15:32:42.204316 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.198686    2549 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 08 15:32:42.204956 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.198688    2549 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 08 15:32:42.204956 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.198691    2549 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 08 15:32:42.204956 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.198695    2549 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 08 15:32:42.204956 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.198698    2549 flags.go:64] FLAG: --lock-file=""
Sep 08 15:32:42.204956 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.198701    2549 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 08 15:32:42.204956 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.198704    2549 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 08 15:32:42.204956 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.198706    2549 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 08 15:32:42.204956 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.198712    2549 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 08 15:32:42.204956 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.198715    2549 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 08 15:32:42.204956 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.198718    2549 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 08 15:32:42.204956 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.198721    2549 flags.go:64] FLAG: --logging-format="text"
Sep 08 15:32:42.204956 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.198724    2549 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 08 15:32:42.204956 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.198727    2549 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 08 15:32:42.204956 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.198730    2549 flags.go:64] FLAG: --manifest-url=""
Sep 08 15:32:42.204956 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.198733    2549 flags.go:64] FLAG: --manifest-url-header=""
Sep 08 15:32:42.204956 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.198737    2549 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 08 15:32:42.204956 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.198742    2549 flags.go:64] FLAG: --max-open-files="1000000"
Sep 08 15:32:42.204956 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.198745    2549 flags.go:64] FLAG: --max-pods="110"
Sep 08 15:32:42.204956 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.198748    2549 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 08 15:32:42.204956 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.198751    2549 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 08 15:32:42.204956 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.198754    2549 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 08 15:32:42.204956 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.198757    2549 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 08 15:32:42.204956 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.198760    2549 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 08 15:32:42.204956 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.198762    2549 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 08 15:32:42.204956 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.198765    2549 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 08 15:32:42.205628 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.198772    2549 flags.go:64] FLAG: --node-status-max-images="50"
Sep 08 15:32:42.205628 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.198775    2549 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 08 15:32:42.205628 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.198777    2549 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 08 15:32:42.205628 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.198781    2549 flags.go:64] FLAG: --pod-cidr=""
Sep 08 15:32:42.205628 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.198784    2549 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1"
Sep 08 15:32:42.205628 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.198789    2549 flags.go:64] FLAG: --pod-manifest-path=""
Sep 08 15:32:42.205628 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.198791    2549 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 08 15:32:42.205628 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.198794    2549 flags.go:64] FLAG: --pods-per-core="0"
Sep 08 15:32:42.205628 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.198797    2549 flags.go:64] FLAG: --port="10250"
Sep 08 15:32:42.205628 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.198801    2549 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 08 15:32:42.205628 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.198804    2549 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0703b34b13642c9d0"
Sep 08 15:32:42.205628 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.198807    2549 flags.go:64] FLAG: --qos-reserved=""
Sep 08 15:32:42.205628 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.198810    2549 flags.go:64] FLAG: --read-only-port="10255"
Sep 08 15:32:42.205628 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.198813    2549 flags.go:64] FLAG: --register-node="true"
Sep 08 15:32:42.205628 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.198816    2549 flags.go:64] FLAG: --register-with-taints=""
Sep 08 15:32:42.205628 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.198820    2549 flags.go:64] FLAG: --registry-burst="10"
Sep 08 15:32:42.205628 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.198823    2549 flags.go:64] FLAG: --registry-qps="5"
Sep 08 15:32:42.205628 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.198826    2549 flags.go:64] FLAG: --reserved-cpus=""
Sep 08 15:32:42.205628 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.198829    2549 flags.go:64] FLAG: --reserved-memory=""
Sep 08 15:32:42.205628 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.198833    2549 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 08 15:32:42.205628 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.198835    2549 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 08 15:32:42.205628 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.198838    2549 flags.go:64] FLAG: --rotate-certificates="false"
Sep 08 15:32:42.205628 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.198841    2549 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 08 15:32:42.205628 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.198844    2549 flags.go:64] FLAG: --runonce="false"
Sep 08 15:32:42.205628 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.198849    2549 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 08 15:32:42.206266 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.198866    2549 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 08 15:32:42.206266 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.198870    2549 flags.go:64] FLAG: --seccomp-default="false"
Sep 08 15:32:42.206266 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.198873    2549 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 08 15:32:42.206266 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.198876    2549 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 08 15:32:42.206266 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.198879    2549 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 08 15:32:42.206266 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.198882    2549 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 08 15:32:42.206266 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.198885    2549 flags.go:64] FLAG: --storage-driver-password="root"
Sep 08 15:32:42.206266 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.198887    2549 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 08 15:32:42.206266 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.198890    2549 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 08 15:32:42.206266 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.198893    2549 flags.go:64] FLAG: --storage-driver-user="root"
Sep 08 15:32:42.206266 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.198896    2549 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 08 15:32:42.206266 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.198899    2549 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 08 15:32:42.206266 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.198901    2549 flags.go:64] FLAG: --system-cgroups=""
Sep 08 15:32:42.206266 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.198904    2549 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 08 15:32:42.206266 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.198910    2549 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 08 15:32:42.206266 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.198913    2549 flags.go:64] FLAG: --tls-cert-file=""
Sep 08 15:32:42.206266 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.198916    2549 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 08 15:32:42.206266 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.198920    2549 flags.go:64] FLAG: --tls-min-version=""
Sep 08 15:32:42.206266 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.198923    2549 flags.go:64] FLAG: --tls-private-key-file=""
Sep 08 15:32:42.206266 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.198926    2549 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 08 15:32:42.206266 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.198928    2549 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 08 15:32:42.206266 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.198931    2549 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 08 15:32:42.206266 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.198934    2549 flags.go:64] FLAG: --v="2"
Sep 08 15:32:42.206266 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.198937    2549 flags.go:64] FLAG: --version="false"
Sep 08 15:32:42.206266 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.198942    2549 flags.go:64] FLAG: --vmodule=""
Sep 08 15:32:42.206895 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.198946    2549 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 08 15:32:42.206895 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.198949    2549 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 08 15:32:42.206895 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.199060    2549 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 08 15:32:42.206895 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.199063    2549 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 08 15:32:42.206895 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.199067    2549 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 08 15:32:42.206895 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.199069    2549 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 08 15:32:42.206895 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.199074    2549 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 08 15:32:42.206895 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.199076    2549 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 08 15:32:42.206895 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.199079    2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 08 15:32:42.206895 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.199082    2549 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 08 15:32:42.206895 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.199085    2549 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 08 15:32:42.206895 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.199088    2549 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 08 15:32:42.206895 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.199091    2549 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 08 15:32:42.206895 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.199093    2549 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 08 15:32:42.206895 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.199096    2549 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 08 15:32:42.206895 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.199098    2549 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 08 15:32:42.206895 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.199101    2549 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 08 15:32:42.206895 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.199104    2549 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 08 15:32:42.206895 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.199106    2549 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 08 15:32:42.206895 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.199109    2549 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 08 15:32:42.207384 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.199111    2549 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 08 15:32:42.207384 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.199114    2549 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 08 15:32:42.207384 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.199117    2549 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 08 15:32:42.207384 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.199121    2549 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 08 15:32:42.207384 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.199124    2549 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 08 15:32:42.207384 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.199126    2549 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 08 15:32:42.207384 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.199129    2549 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 08 15:32:42.207384 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.199132    2549 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 08 15:32:42.207384 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.199134    2549 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 08 15:32:42.207384 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.199137    2549 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 08 15:32:42.207384 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.199139    2549 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 08 15:32:42.207384 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.199142    2549 feature_gate.go:328] unrecognized feature gate: Example2
Sep 08 15:32:42.207384 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.199145    2549 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 08 15:32:42.207384 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.199147    2549 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 08 15:32:42.207384 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.199150    2549 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 08 15:32:42.207384 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.199152    2549 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 08 15:32:42.207384 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.199155    2549 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 08 15:32:42.207384 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.199158    2549 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 08 15:32:42.207384 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.199161    2549 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 08 15:32:42.207841 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.199164    2549 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 08 15:32:42.207841 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.199167    2549 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 08 15:32:42.207841 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.199169    2549 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 08 15:32:42.207841 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.199172    2549 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 08 15:32:42.207841 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.199175    2549 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 08 15:32:42.207841 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.199177    2549 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 08 15:32:42.207841 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.199180    2549 feature_gate.go:328] unrecognized feature gate: Example
Sep 08 15:32:42.207841 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.199182    2549 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 08 15:32:42.207841 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.199185    2549 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 08 15:32:42.207841 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.199187    2549 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 08 15:32:42.207841 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.199191    2549 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 08 15:32:42.207841 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.199195    2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 08 15:32:42.207841 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.199206    2549 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 08 15:32:42.207841 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.199209    2549 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 08 15:32:42.207841 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.199213    2549 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 08 15:32:42.207841 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.199216    2549 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 08 15:32:42.207841 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.199220    2549 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 08 15:32:42.207841 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.199223    2549 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 08 15:32:42.207841 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.199226    2549 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 08 15:32:42.208330 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.199229    2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 08 15:32:42.208330 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.199231    2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 08 15:32:42.208330 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.199234    2549 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 08 15:32:42.208330 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.199237    2549 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 08 15:32:42.208330 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.199239    2549 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 08 15:32:42.208330 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.199242    2549 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 08 15:32:42.208330 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.199245    2549 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 08 15:32:42.208330 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.199248    2549 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 08 15:32:42.208330 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.199251    2549 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 08 15:32:42.208330 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.199253    2549 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 08 15:32:42.208330 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.199256    2549 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 08 15:32:42.208330 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.199258    2549 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 08 15:32:42.208330 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.199262    2549 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 08 15:32:42.208330 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.199264    2549 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 08 15:32:42.208330 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.199267    2549 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 08 15:32:42.208330 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.199270    2549 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 08 15:32:42.208330 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.199273    2549 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 08 15:32:42.208330 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.199275    2549 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 08 15:32:42.208330 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.199278    2549 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 08 15:32:42.208780 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.199280    2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 08 15:32:42.208780 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.199283    2549 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 08 15:32:42.208780 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.199285    2549 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 08 15:32:42.208780 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.199288    2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 08 15:32:42.208780 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.199292    2549 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 08 15:32:42.208780 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.199295    2549 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 08 15:32:42.208780 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.199298    2549 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 08 15:32:42.208780 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.199301    2549 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 08 15:32:42.208780 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.199304    2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 08 15:32:42.208780 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.199307    2549 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 08 15:32:42.208780 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.199311    2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 08 15:32:42.208780 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.199313    2549 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 08 15:32:42.208780 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.199316    2549 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 08 15:32:42.208780 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.199319    2549 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 08 15:32:42.208780 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.199321    2549 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 08 15:32:42.208780 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.199324    2549 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 08 15:32:42.208780 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.199326    2549 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 08 15:32:42.208780 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.199330    2549 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 08 15:32:42.208780 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.199332    2549 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 08 15:32:42.208780 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.199335    2549 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 08 15:32:42.209341 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.199337    2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 08 15:32:42.209341 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.199340    2549 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 08 15:32:42.209341 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.199343    2549 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 08 15:32:42.209341 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.199345    2549 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 08 15:32:42.209341 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.199895    2549 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 08 15:32:42.209341 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.206570    2549 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 08 15:32:42.209341 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.206584    2549 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 08 15:32:42.209341 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.208192    2549 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 08 15:32:42.209341 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.208201    2549 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 08 15:32:42.209341 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208287    2549 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 08 15:32:42.209341 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208293    2549 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 08 15:32:42.209341 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208297    2549 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 08 15:32:42.209341 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208300    2549 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 08 15:32:42.209341 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208303    2549 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 08 15:32:42.209341 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208306    2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 08 15:32:42.209717 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208310    2549 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 08 15:32:42.209717 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208312    2549 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 08 15:32:42.209717 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208316    2549 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 08 15:32:42.209717 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208318    2549 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 08 15:32:42.209717 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208321    2549 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 08 15:32:42.209717 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208324    2549 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 08 15:32:42.209717 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208326    2549 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 08 15:32:42.209717 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208329    2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 08 15:32:42.209717 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208332    2549 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 08 15:32:42.209717 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208335    2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 08 15:32:42.209717 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208338    2549 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 08 15:32:42.209717 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208340    2549 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 08 15:32:42.209717 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208343    2549 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 08 15:32:42.209717 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208347    2549 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 08 15:32:42.209717 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208351    2549 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 08 15:32:42.209717 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208355    2549 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 08 15:32:42.209717 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208358    2549 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 08 15:32:42.209717 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208360    2549 feature_gate.go:328] unrecognized feature gate: Example2
Sep 08 15:32:42.209717 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208369    2549 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 08 15:32:42.210199 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208372    2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 08 15:32:42.210199 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208375    2549 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 08 15:32:42.210199 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208378    2549 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 08 15:32:42.210199 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208380    2549 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 08 15:32:42.210199 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208384    2549 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 08 15:32:42.210199 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208387    2549 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 08 15:32:42.210199 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208390    2549 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 08 15:32:42.210199 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208393    2549 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 08 15:32:42.210199 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208395    2549 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 08 15:32:42.210199 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208398    2549 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 08 15:32:42.210199 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208401    2549 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 08 15:32:42.210199 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208403    2549 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 08 15:32:42.210199 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208406    2549 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 08 15:32:42.210199 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208410    2549 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 08 15:32:42.210199 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208413    2549 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 08 15:32:42.210199 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208417    2549 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 08 15:32:42.210199 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208419    2549 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 08 15:32:42.210199 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208422    2549 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 08 15:32:42.210199 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208425    2549 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 08 15:32:42.210673 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208428    2549 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 08 15:32:42.210673 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208430    2549 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 08 15:32:42.210673 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208433    2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 08 15:32:42.210673 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208436    2549 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 08 15:32:42.210673 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208438    2549 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 08 15:32:42.210673 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208441    2549 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 08 15:32:42.210673 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208443    2549 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 08 15:32:42.210673 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208446    2549 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 08 15:32:42.210673 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208449    2549 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 08 15:32:42.210673 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208451    2549 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 08 15:32:42.210673 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208454    2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 08 15:32:42.210673 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208456    2549 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 08 15:32:42.210673 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208459    2549 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 08 15:32:42.210673 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208467    2549 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 08 15:32:42.210673 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208469    2549 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 08 15:32:42.210673 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208472    2549 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 08 15:32:42.210673 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208475    2549 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 08 15:32:42.210673 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208478    2549 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 08 15:32:42.210673 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208482    2549 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 08 15:32:42.210673 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208485    2549 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 08 15:32:42.211237 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208488    2549 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 08 15:32:42.211237 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208491    2549 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 08 15:32:42.211237 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208493    2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 08 15:32:42.211237 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208496    2549 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 08 15:32:42.211237 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208498    2549 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 08 15:32:42.211237 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208501    2549 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 08 15:32:42.211237 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208503    2549 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 08 15:32:42.211237 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208506    2549 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 08 15:32:42.211237 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208509    2549 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 08 15:32:42.211237 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208511    2549 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 08 15:32:42.211237 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208514    2549 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 08 15:32:42.211237 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208516    2549 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 08 15:32:42.211237 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208519    2549 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 08 15:32:42.211237 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208522    2549 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 08 15:32:42.211237 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208524    2549 feature_gate.go:328] unrecognized feature gate: Example
Sep 08 15:32:42.211237 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208527    2549 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 08 15:32:42.211237 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208530    2549 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 08 15:32:42.211237 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208533    2549 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 08 15:32:42.211237 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208535    2549 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 08 15:32:42.211714 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208538    2549 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 08 15:32:42.211714 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208541    2549 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 08 15:32:42.211714 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208543    2549 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 08 15:32:42.211714 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208545    2549 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 08 15:32:42.211714 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208548    2549 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 08 15:32:42.211714 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208551    2549 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 08 15:32:42.211714 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208553    2549 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 08 15:32:42.211714 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208561    2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 08 15:32:42.211714 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208564    2549 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 08 15:32:42.211714 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208567    2549 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 08 15:32:42.211714 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208570    2549 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 08 15:32:42.211714 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208573    2549 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 08 15:32:42.211714 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208575    2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 08 15:32:42.211714 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208578    2549 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 08 15:32:42.211714 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208580    2549 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 08 15:32:42.211714 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208583    2549 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 08 15:32:42.212214 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.208588    2549 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 08 15:32:42.212214 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208726    2549 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 08 15:32:42.212214 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208734    2549 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 08 15:32:42.212214 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208737    2549 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 08 15:32:42.212214 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208740    2549 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 08 15:32:42.212214 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208743    2549 feature_gate.go:328] unrecognized feature gate: Example2
Sep 08 15:32:42.212214 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208746    2549 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 08 15:32:42.212214 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208749    2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 08 15:32:42.212214 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208752    2549 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 08 15:32:42.212214 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208755    2549 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 08 15:32:42.212214 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208758    2549 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 08 15:32:42.212214 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208761    2549 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 08 15:32:42.212214 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208764    2549 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 08 15:32:42.212214 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208766    2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 08 15:32:42.212214 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208769    2549 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 08 15:32:42.212601 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208771    2549 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 08 15:32:42.212601 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208774    2549 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 08 15:32:42.212601 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208776    2549 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 08 15:32:42.212601 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208779    2549 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 08 15:32:42.212601 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208783    2549 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 08 15:32:42.212601 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208785    2549 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 08 15:32:42.212601 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208788    2549 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 08 15:32:42.212601 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208791    2549 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 08 15:32:42.212601 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208793    2549 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 08 15:32:42.212601 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208801    2549 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 08 15:32:42.212601 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208804    2549 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 08 15:32:42.212601 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208808    2549 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 08 15:32:42.212601 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208811    2549 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 08 15:32:42.212601 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208814    2549 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 08 15:32:42.212601 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208816    2549 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 08 15:32:42.212601 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208819    2549 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 08 15:32:42.212601 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208821    2549 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 08 15:32:42.212601 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208824    2549 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 08 15:32:42.212601 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208828    2549 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 08 15:32:42.213150 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208831    2549 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 08 15:32:42.213150 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208834    2549 feature_gate.go:328] unrecognized feature gate: Example
Sep 08 15:32:42.213150 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208837    2549 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 08 15:32:42.213150 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208839    2549 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 08 15:32:42.213150 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208842    2549 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 08 15:32:42.213150 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208845    2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 08 15:32:42.213150 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208847    2549 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 08 15:32:42.213150 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208850    2549 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 08 15:32:42.213150 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208866    2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 08 15:32:42.213150 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208870    2549 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 08 15:32:42.213150 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208872    2549 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 08 15:32:42.213150 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208875    2549 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 08 15:32:42.213150 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208878    2549 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 08 15:32:42.213150 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208880    2549 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 08 15:32:42.213150 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208883    2549 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 08 15:32:42.213150 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208885    2549 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 08 15:32:42.213150 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208888    2549 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 08 15:32:42.213150 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208890    2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 08 15:32:42.213150 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208893    2549 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 08 15:32:42.213150 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208895    2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 08 15:32:42.213975 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208898    2549 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 08 15:32:42.213975 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208900    2549 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 08 15:32:42.213975 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208902    2549 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 08 15:32:42.213975 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208913    2549 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 08 15:32:42.213975 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208916    2549 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 08 15:32:42.213975 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208918    2549 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 08 15:32:42.213975 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208921    2549 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 08 15:32:42.213975 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208924    2549 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 08 15:32:42.213975 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208926    2549 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 08 15:32:42.213975 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208929    2549 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 08 15:32:42.213975 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208931    2549 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 08 15:32:42.213975 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208934    2549 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 08 15:32:42.213975 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208937    2549 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 08 15:32:42.213975 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208939    2549 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 08 15:32:42.213975 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208942    2549 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 08 15:32:42.213975 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208944    2549 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 08 15:32:42.213975 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208947    2549 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 08 15:32:42.213975 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208949    2549 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 08 15:32:42.213975 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208953    2549 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 08 15:32:42.214482 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208957    2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 08 15:32:42.214482 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208959    2549 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 08 15:32:42.214482 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208962    2549 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 08 15:32:42.214482 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208965    2549 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 08 15:32:42.214482 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208968    2549 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 08 15:32:42.214482 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208970    2549 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 08 15:32:42.214482 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208973    2549 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 08 15:32:42.214482 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208975    2549 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 08 15:32:42.214482 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208978    2549 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 08 15:32:42.214482 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208981    2549 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 08 15:32:42.214482 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208983    2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 08 15:32:42.214482 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208986    2549 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 08 15:32:42.214482 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208989    2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 08 15:32:42.214482 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208991    2549 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 08 15:32:42.214482 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.208999    2549 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 08 15:32:42.214482 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.209002    2549 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 08 15:32:42.214482 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.209005    2549 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 08 15:32:42.214482 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.209014    2549 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 08 15:32:42.214482 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.209017    2549 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 08 15:32:42.214482 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.209019    2549 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 08 15:32:42.215111 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.209022    2549 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 08 15:32:42.215111 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.209024    2549 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 08 15:32:42.215111 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.209027    2549 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 08 15:32:42.215111 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.209029    2549 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 08 15:32:42.215111 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.209032    2549 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 08 15:32:42.215111 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.209034    2549 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 08 15:32:42.215111 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:42.209037    2549 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 08 15:32:42.215111 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.209042    2549 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 08 15:32:42.215111 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.209139    2549 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 08 15:32:42.215111 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.210985    2549 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 08 15:32:42.215111 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.211829    2549 server.go:1019] "Starting client certificate rotation"
Sep 08 15:32:42.215111 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.211896    2549 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 08 15:32:42.215111 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.211936    2549 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 08 15:32:42.215521 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.214972    2549 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 08 15:32:42.223929 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.223911    2549 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 08 15:32:42.236729 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.236713    2549 log.go:25] "Validated CRI v1 runtime API"
Sep 08 15:32:42.240301 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.240281    2549 log.go:25] "Validated CRI v1 image API"
Sep 08 15:32:42.241198 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.241178    2549 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 08 15:32:42.243620 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.243599    2549 fs.go:135] Filesystem UUIDs: map[00c11be2-d511-4726-8802-40ca45a7251f:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 963c3bbd-f2dd-412b-8c06-0816a44ea1ba:/dev/nvme0n1p4]
Sep 08 15:32:42.243691 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.243619    2549 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 08 15:32:42.249100 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.249084    2549 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 08 15:32:42.249569 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.249466    2549 manager.go:216] Machine: {Timestamp:2026-09-08 15:32:42.248103138 +0000 UTC m=+0.337554217 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3097931 MemoryCapacity:33164476416 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec29f7add86b4bf7cd2e5955892fee83 SystemUUID:ec29f7ad-d86b-4bf7-cd2e-5955892fee83 BootID:ef9f6169-c615-49dc-aa2b-b8973a53894b Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:82:eb:2b:8f:59 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:82:eb:2b:8f:59 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:c2:ab:bc:a1:e4:a2 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164476416 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 08 15:32:42.249631 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.249571    2549 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 08 15:32:42.249660 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.249653    2549 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 08 15:32:42.252368 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.252348    2549 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 08 15:32:42.252504 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.252370    2549 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-143-61.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 08 15:32:42.252955 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.252945    2549 topology_manager.go:138] "Creating topology manager with none policy"
Sep 08 15:32:42.252992 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.252957    2549 container_manager_linux.go:309] "Creating device plugin manager"
Sep 08 15:32:42.252992 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.252965    2549 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 08 15:32:42.253650 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.253639    2549 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 08 15:32:42.254805 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.254796    2549 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 08 15:32:42.258489 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.258478    2549 state_mem.go:36] "Initialized new in-memory state store"
Sep 08 15:32:42.258589 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.258581    2549 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 08 15:32:42.258624 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.258616    2549 kubelet.go:486] "Attempting to sync node with API server"
Sep 08 15:32:42.258652 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.258630    2549 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 08 15:32:42.258652 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.258643    2549 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 08 15:32:42.258705 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.258659    2549 kubelet.go:398] "Adding apiserver pod source"
Sep 08 15:32:42.258705 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.258668    2549 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 08 15:32:42.259004 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.258992    2549 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 08 15:32:42.259051 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.259028    2549 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 08 15:32:42.262123 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.262091    2549 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 08 15:32:42.264525 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.264508    2549 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 08 15:32:42.264604 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.264539    2549 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 08 15:32:42.264712 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.264702    2549 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 08 15:32:42.264750 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.264717    2549 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 08 15:32:42.264750 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.264723    2549 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 08 15:32:42.264750 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.264728    2549 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 08 15:32:42.264750 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.264736    2549 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 08 15:32:42.264750 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.264741    2549 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 08 15:32:42.265308 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.265296    2549 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 08 15:32:42.265355 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.265309    2549 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 08 15:32:42.265355 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.265320    2549 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 08 15:32:42.265355 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.265329    2549 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 08 15:32:42.265355 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.265341    2549 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 08 15:32:42.265355 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.265353    2549 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 08 15:32:42.266197 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.266182    2549 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 08 15:32:42.266197 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.266197    2549 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 08 15:32:42.268045 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:32:42.268023    2549 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-143-61.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 08 15:32:42.268045 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:32:42.268021    2549 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 08 15:32:42.269841 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.269825    2549 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-k6nb8"
Sep 08 15:32:42.269913 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.269890    2549 server.go:1268] "Started kubelet"
Sep 08 15:32:42.269986 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.269957    2549 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 08 15:32:42.270091 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.270006    2549 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 08 15:32:42.270091 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.270077    2549 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 08 15:32:42.270569 ip-10-0-143-61 systemd[1]: Started Kubernetes Kubelet.
Sep 08 15:32:42.271562 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.271039    2549 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 08 15:32:42.272000 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.271986    2549 server.go:310] "Adding debug handlers to kubelet server"
Sep 08 15:32:42.275733 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:32:42.275714    2549 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 08 15:32:42.276340 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.276321    2549 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 08 15:32:42.276340 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.276327    2549 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 08 15:32:42.276553 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:32:42.276533    2549 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-61.ec2.internal\" not found"
Sep 08 15:32:42.276772 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.276744    2549 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-61.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 08 15:32:42.276936 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.276917    2549 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 08 15:32:42.277078 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.277060    2549 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 08 15:32:42.277165 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.277075    2549 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-k6nb8"
Sep 08 15:32:42.277165 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.277081    2549 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 08 15:32:42.277165 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.277152    2549 reconstruct.go:97] "Volume reconstruction finished"
Sep 08 15:32:42.277165 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.277159    2549 reconciler.go:29] "Reconciler: start to sync state"
Sep 08 15:32:42.277664 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:32:42.276788    2549 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-61.ec2.internal.18d3624d90575dd1  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-61.ec2.internal,UID:ip-10-0-143-61.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-143-61.ec2.internal,},FirstTimestamp:2026-09-08 15:32:42.269867473 +0000 UTC m=+0.359318553,LastTimestamp:2026-09-08 15:32:42.269867473 +0000 UTC m=+0.359318553,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-61.ec2.internal,}"
Sep 08 15:32:42.277771 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.277737    2549 factory.go:153] Registering CRI-O factory
Sep 08 15:32:42.277771 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.277751    2549 factory.go:223] Registration of the crio container factory successfully
Sep 08 15:32:42.277892 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.277799    2549 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 08 15:32:42.277892 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.277809    2549 factory.go:55] Registering systemd factory
Sep 08 15:32:42.277892 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.277818    2549 factory.go:223] Registration of the systemd container factory successfully
Sep 08 15:32:42.277892 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.277839    2549 factory.go:103] Registering Raw factory
Sep 08 15:32:42.277892 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.277869    2549 manager.go:1196] Started watching for new ooms in manager
Sep 08 15:32:42.278253 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.278238    2549 manager.go:349] Starting recovery of all containers
Sep 08 15:32:42.280382 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:32:42.280354    2549 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-61.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 08 15:32:42.280471 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:32:42.280451    2549 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 08 15:32:42.289942 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.289925    2549 manager.go:354] Recovery completed
Sep 08 15:32:42.294298 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.294285    2549 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 15:32:42.296007 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.295990    2549 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-61.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 15:32:42.296078 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.296021    2549 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-61.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 15:32:42.296078 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.296031    2549 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-61.ec2.internal" event="NodeHasSufficientPID"
Sep 08 15:32:42.297546 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.297530    2549 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 08 15:32:42.297546 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.297545    2549 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 08 15:32:42.297691 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.297564    2549 state_mem.go:36] "Initialized new in-memory state store"
Sep 08 15:32:42.297691 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.297545    2549 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 15:32:42.298318 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.298302    2549 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-61.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 15:32:42.298402 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.298332    2549 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-61.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 15:32:42.298402 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.298346    2549 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-61.ec2.internal" event="NodeHasSufficientPID"
Sep 08 15:32:42.299637 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.299624    2549 policy_none.go:49] "None policy: Start"
Sep 08 15:32:42.299692 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.299640    2549 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 08 15:32:42.299692 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.299649    2549 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 08 15:32:42.301649 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.301636    2549 policy_none.go:47] "Start"
Sep 08 15:32:42.325450 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.325436    2549 manager.go:337] "Starting Device Plugin manager"
Sep 08 15:32:42.352999 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:32:42.325476    2549 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 08 15:32:42.352999 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.325488    2549 server.go:87] "Starting device plugin registration server"
Sep 08 15:32:42.352999 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.325699    2549 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 08 15:32:42.352999 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.325719    2549 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 08 15:32:42.352999 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.325794    2549 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 08 15:32:42.352999 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.325884    2549 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 08 15:32:42.352999 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.325893    2549 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 08 15:32:42.352999 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:32:42.326347    2549 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 08 15:32:42.352999 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:32:42.326389    2549 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-61.ec2.internal\" not found"
Sep 08 15:32:42.392851 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.392821    2549 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 08 15:32:42.393966 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.393950    2549 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 08 15:32:42.394063 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.393991    2549 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 08 15:32:42.394063 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.394010    2549 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 08 15:32:42.394063 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:32:42.394042    2549 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 08 15:32:42.395963 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.395945    2549 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 08 15:32:42.426655 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.426638    2549 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 15:32:42.428071 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.428055    2549 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-61.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 15:32:42.428155 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.428089    2549 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-61.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 15:32:42.428155 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.428104    2549 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-61.ec2.internal" event="NodeHasSufficientPID"
Sep 08 15:32:42.428155 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.428130    2549 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-61.ec2.internal"
Sep 08 15:32:42.436851 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.436835    2549 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-143-61.ec2.internal"
Sep 08 15:32:42.436926 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:32:42.436871    2549 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-143-61.ec2.internal\": node \"ip-10-0-143-61.ec2.internal\" not found"
Sep 08 15:32:42.451895 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:32:42.451838    2549 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-61.ec2.internal\" not found"
Sep 08 15:32:42.494865 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.494829    2549 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-61.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-143-61.ec2.internal"]
Sep 08 15:32:42.494954 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.494909    2549 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 15:32:42.496326 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.496300    2549 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-61.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 15:32:42.496417 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.496335    2549 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-61.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 15:32:42.496417 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.496348    2549 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-61.ec2.internal" event="NodeHasSufficientPID"
Sep 08 15:32:42.498405 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.498392    2549 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 15:32:42.498527 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.498514    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-61.ec2.internal"
Sep 08 15:32:42.498564 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.498541    2549 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 15:32:42.499144 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.499127    2549 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-61.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 15:32:42.499212 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.499130    2549 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-61.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 15:32:42.499212 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.499160    2549 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-61.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 15:32:42.499212 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.499174    2549 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-61.ec2.internal" event="NodeHasSufficientPID"
Sep 08 15:32:42.499212 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.499179    2549 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-61.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 15:32:42.499212 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.499194    2549 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-61.ec2.internal" event="NodeHasSufficientPID"
Sep 08 15:32:42.501356 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.501337    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-61.ec2.internal"
Sep 08 15:32:42.501432 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.501364    2549 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 15:32:42.502229 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.502213    2549 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-61.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 15:32:42.502309 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.502241    2549 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-61.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 15:32:42.502309 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.502253    2549 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-61.ec2.internal" event="NodeHasSufficientPID"
Sep 08 15:32:42.525414 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:32:42.525394    2549 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-61.ec2.internal\" not found" node="ip-10-0-143-61.ec2.internal"
Sep 08 15:32:42.529691 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:32:42.529676    2549 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-61.ec2.internal\" not found" node="ip-10-0-143-61.ec2.internal"
Sep 08 15:32:42.552314 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:32:42.552297    2549 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-61.ec2.internal\" not found"
Sep 08 15:32:42.578630 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.578613    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e8cf966721b2ec8fb0cd4ca3ebd9318e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-61.ec2.internal\" (UID: \"e8cf966721b2ec8fb0cd4ca3ebd9318e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-61.ec2.internal"
Sep 08 15:32:42.578682 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.578641    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4a20335aaf3733c5ea5fa6f08be94600-config\") pod \"kube-apiserver-proxy-ip-10-0-143-61.ec2.internal\" (UID: \"4a20335aaf3733c5ea5fa6f08be94600\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-61.ec2.internal"
Sep 08 15:32:42.578682 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.578656    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e8cf966721b2ec8fb0cd4ca3ebd9318e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-61.ec2.internal\" (UID: \"e8cf966721b2ec8fb0cd4ca3ebd9318e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-61.ec2.internal"
Sep 08 15:32:42.653401 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:32:42.653379    2549 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-61.ec2.internal\" not found"
Sep 08 15:32:42.679731 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.679713    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e8cf966721b2ec8fb0cd4ca3ebd9318e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-61.ec2.internal\" (UID: \"e8cf966721b2ec8fb0cd4ca3ebd9318e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-61.ec2.internal"
Sep 08 15:32:42.679800 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.679751    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e8cf966721b2ec8fb0cd4ca3ebd9318e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-61.ec2.internal\" (UID: \"e8cf966721b2ec8fb0cd4ca3ebd9318e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-61.ec2.internal"
Sep 08 15:32:42.679800 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.679781    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e8cf966721b2ec8fb0cd4ca3ebd9318e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-61.ec2.internal\" (UID: \"e8cf966721b2ec8fb0cd4ca3ebd9318e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-61.ec2.internal"
Sep 08 15:32:42.679918 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.679802    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4a20335aaf3733c5ea5fa6f08be94600-config\") pod \"kube-apiserver-proxy-ip-10-0-143-61.ec2.internal\" (UID: \"4a20335aaf3733c5ea5fa6f08be94600\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-61.ec2.internal"
Sep 08 15:32:42.679918 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.679812    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e8cf966721b2ec8fb0cd4ca3ebd9318e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-61.ec2.internal\" (UID: \"e8cf966721b2ec8fb0cd4ca3ebd9318e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-61.ec2.internal"
Sep 08 15:32:42.679918 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.679820    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4a20335aaf3733c5ea5fa6f08be94600-config\") pod \"kube-apiserver-proxy-ip-10-0-143-61.ec2.internal\" (UID: \"4a20335aaf3733c5ea5fa6f08be94600\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-61.ec2.internal"
Sep 08 15:32:42.754026 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:32:42.753977    2549 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-61.ec2.internal\" not found"
Sep 08 15:32:42.827596 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.827570    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-61.ec2.internal"
Sep 08 15:32:42.832344 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:42.832324    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-61.ec2.internal"
Sep 08 15:32:42.854749 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:32:42.854721    2549 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-61.ec2.internal\" not found"
Sep 08 15:32:42.955279 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:32:42.955254    2549 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-61.ec2.internal\" not found"
Sep 08 15:32:43.056103 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:32:43.056072    2549 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-61.ec2.internal\" not found"
Sep 08 15:32:43.103803 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.103781    2549 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 08 15:32:43.156764 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:32:43.156738    2549 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-61.ec2.internal\" not found"
Sep 08 15:32:43.157803 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.157788    2549 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 08 15:32:43.177051 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.177021    2549 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-61.ec2.internal"
Sep 08 15:32:43.187933 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.187918    2549 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 08 15:32:43.189076 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.189055    2549 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-61.ec2.internal"
Sep 08 15:32:43.199374 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.199350    2549 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 08 15:32:43.211364 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.211342    2549 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 08 15:32:43.211464 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.211446    2549 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 08 15:32:43.211515 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.211487    2549 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 08 15:32:43.211515 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.211490    2549 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 08 15:32:43.258775 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.258758    2549 apiserver.go:52] "Watching apiserver"
Sep 08 15:32:43.277055 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.277004    2549 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 08 15:32:43.278121 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.278101    2549 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 08 15:32:43.278458 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.278436    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-st9hm","openshift-dns/node-resolver-vp9nk","openshift-multus/multus-95s5s","openshift-network-diagnostics/network-check-target-5vt6c","openshift-network-operator/iptables-alerter-p4rw9","openshift-ovn-kubernetes/ovnkube-node-f4vvt","kube-system/global-pull-secret-syncer-fw5jh","kube-system/kube-apiserver-proxy-ip-10-0-143-61.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kfw4l","openshift-cluster-node-tuning-operator/tuned-klnj5","openshift-image-registry/node-ca-6q7cq","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-61.ec2.internal","openshift-multus/multus-additional-cni-plugins-rpthc","openshift-multus/network-metrics-daemon-h4cml"]
Sep 08 15:32:43.280265 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.280229    2549 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-07 15:27:42 +0000 UTC" deadline="2028-02-10 03:44:10.802607845 +0000 UTC"
Sep 08 15:32:43.280328 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.280266    2549 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12468h11m27.522346206s"
Sep 08 15:32:43.281169 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.281155    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-st9hm"
Sep 08 15:32:43.284011 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.283993    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-vp9nk"
Sep 08 15:32:43.284286 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.284264    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 08 15:32:43.284383 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.284298    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-sb9v8\""
Sep 08 15:32:43.284383 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.284303    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 08 15:32:43.286472 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.286455    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-89p88\""
Sep 08 15:32:43.286551 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.286487    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 08 15:32:43.286551 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.286464    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 08 15:32:43.286779 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.286763    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-95s5s"
Sep 08 15:32:43.288925 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.288909    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-vzwmq\""
Sep 08 15:32:43.289118 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.289104    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 08 15:32:43.289263 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.289250    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 08 15:32:43.289331 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.289281    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 08 15:32:43.289497 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.289482    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 08 15:32:43.289577 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.289563    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5vt6c"
Sep 08 15:32:43.289674 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:32:43.289654    2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5vt6c" podUID="3d62444d-76ba-4184-869c-496ba4da8d04"
Sep 08 15:32:43.290231 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.290216    2549 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 08 15:32:43.292642 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.292625    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-p4rw9"
Sep 08 15:32:43.296544 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.295419    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-g45lg\""
Sep 08 15:32:43.296544 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.295622    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 08 15:32:43.296544 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.295792    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 08 15:32:43.296544 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.296057    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-f4vvt"
Sep 08 15:32:43.297162 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.297147    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 08 15:32:43.298767 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.298745    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 08 15:32:43.298925 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.298907    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fw5jh"
Sep 08 15:32:43.299020 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:32:43.298998    2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fw5jh" podUID="84663ea3-1feb-4c87-8609-c580fa60416e"
Sep 08 15:32:43.299095 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.299073    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 08 15:32:43.299292 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.299274    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-2fdcv\""
Sep 08 15:32:43.299443 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.299426    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 08 15:32:43.300698 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.300665    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 08 15:32:43.300803 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.300706    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 08 15:32:43.300803 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.300707    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 08 15:32:43.301621 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.301603    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kfw4l"
Sep 08 15:32:43.303941 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.303912    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 08 15:32:43.304065 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.303976    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-fjjbn\""
Sep 08 15:32:43.304122 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.304111    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 08 15:32:43.304187 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.304161    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 08 15:32:43.305254 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.305237    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-klnj5"
Sep 08 15:32:43.307698 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.307676    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-6q7cq"
Sep 08 15:32:43.307854 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.307781    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-rfmfd\""
Sep 08 15:32:43.307949 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.307866    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 08 15:32:43.308749 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.308725    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 08 15:32:43.310244 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.310225    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 08 15:32:43.311483 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.310638    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-rpthc"
Sep 08 15:32:43.311483 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.310722    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 08 15:32:43.311483 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.311126    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-7476w\""
Sep 08 15:32:43.311483 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.311196    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 08 15:32:43.313356 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.313332    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h4cml"
Sep 08 15:32:43.313466 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:32:43.313404    2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h4cml" podUID="7d4d759e-9d3c-497a-b546-462ae2b48785"
Sep 08 15:32:43.313466 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.313438    2549 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-wb4tl"
Sep 08 15:32:43.313964 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.313934    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 08 15:32:43.314081 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.314000    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 08 15:32:43.314495 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.314477    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-smvfq\""
Sep 08 15:32:43.321052 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.321017    2549 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-wb4tl"
Sep 08 15:32:43.378044 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.378026    2549 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 08 15:32:43.383898 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.383878    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fnhz\" (UniqueName: \"kubernetes.io/projected/5f9598ad-fb27-4dda-ab17-98d54cbd30aa-kube-api-access-2fnhz\") pod \"node-ca-6q7cq\" (UID: \"5f9598ad-fb27-4dda-ab17-98d54cbd30aa\") " pod="openshift-image-registry/node-ca-6q7cq"
Sep 08 15:32:43.383992 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.383906    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/37ec7079-542c-4b91-8b2b-83e8e89fcbb7-etc-systemd\") pod \"tuned-klnj5\" (UID: \"37ec7079-542c-4b91-8b2b-83e8e89fcbb7\") " pod="openshift-cluster-node-tuning-operator/tuned-klnj5"
Sep 08 15:32:43.383992 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.383941    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/37ec7079-542c-4b91-8b2b-83e8e89fcbb7-sys\") pod \"tuned-klnj5\" (UID: \"37ec7079-542c-4b91-8b2b-83e8e89fcbb7\") " pod="openshift-cluster-node-tuning-operator/tuned-klnj5"
Sep 08 15:32:43.384075 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.383995    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/95c9958b-a005-439c-b856-0cbb99f08700-host-run-k8s-cni-cncf-io\") pod \"multus-95s5s\" (UID: \"95c9958b-a005-439c-b856-0cbb99f08700\") " pod="openshift-multus/multus-95s5s"
Sep 08 15:32:43.384075 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.384020    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/95c9958b-a005-439c-b856-0cbb99f08700-host-run-netns\") pod \"multus-95s5s\" (UID: \"95c9958b-a005-439c-b856-0cbb99f08700\") " pod="openshift-multus/multus-95s5s"
Sep 08 15:32:43.384075 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.384039    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/95c9958b-a005-439c-b856-0cbb99f08700-multus-daemon-config\") pod \"multus-95s5s\" (UID: \"95c9958b-a005-439c-b856-0cbb99f08700\") " pod="openshift-multus/multus-95s5s"
Sep 08 15:32:43.384075 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.384059    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/65a8a5a6-5e05-4ee5-99a6-c026327dfa43-host-slash\") pod \"iptables-alerter-p4rw9\" (UID: \"65a8a5a6-5e05-4ee5-99a6-c026327dfa43\") " pod="openshift-network-operator/iptables-alerter-p4rw9"
Sep 08 15:32:43.384216 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.384077    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/195f1a74-19bc-49c9-aada-fa1c13488480-log-socket\") pod \"ovnkube-node-f4vvt\" (UID: \"195f1a74-19bc-49c9-aada-fa1c13488480\") " pod="openshift-ovn-kubernetes/ovnkube-node-f4vvt"
Sep 08 15:32:43.384216 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.384095    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c9873505-399e-48b7-b74c-685ae3d9f7b5-etc-selinux\") pod \"aws-ebs-csi-driver-node-kfw4l\" (UID: \"c9873505-399e-48b7-b74c-685ae3d9f7b5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kfw4l"
Sep 08 15:32:43.384216 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.384110    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c9873505-399e-48b7-b74c-685ae3d9f7b5-sys-fs\") pod \"aws-ebs-csi-driver-node-kfw4l\" (UID: \"c9873505-399e-48b7-b74c-685ae3d9f7b5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kfw4l"
Sep 08 15:32:43.384216 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.384125    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/37ec7079-542c-4b91-8b2b-83e8e89fcbb7-etc-sysconfig\") pod \"tuned-klnj5\" (UID: \"37ec7079-542c-4b91-8b2b-83e8e89fcbb7\") " pod="openshift-cluster-node-tuning-operator/tuned-klnj5"
Sep 08 15:32:43.384216 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.384155    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a955f56c-5ac8-422a-8ddc-ce17a8361eb6-tmp-dir\") pod \"node-resolver-vp9nk\" (UID: \"a955f56c-5ac8-422a-8ddc-ce17a8361eb6\") " pod="openshift-dns/node-resolver-vp9nk"
Sep 08 15:32:43.384216 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.384186    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/95c9958b-a005-439c-b856-0cbb99f08700-system-cni-dir\") pod \"multus-95s5s\" (UID: \"95c9958b-a005-439c-b856-0cbb99f08700\") " pod="openshift-multus/multus-95s5s"
Sep 08 15:32:43.384426 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.384214    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/195f1a74-19bc-49c9-aada-fa1c13488480-host-kubelet\") pod \"ovnkube-node-f4vvt\" (UID: \"195f1a74-19bc-49c9-aada-fa1c13488480\") " pod="openshift-ovn-kubernetes/ovnkube-node-f4vvt"
Sep 08 15:32:43.384426 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.384243    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wc458\" (UniqueName: \"kubernetes.io/projected/c9873505-399e-48b7-b74c-685ae3d9f7b5-kube-api-access-wc458\") pod \"aws-ebs-csi-driver-node-kfw4l\" (UID: \"c9873505-399e-48b7-b74c-685ae3d9f7b5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kfw4l"
Sep 08 15:32:43.384426 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.384261    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3236f6c9-60de-47f9-84a9-5980933c58cd-system-cni-dir\") pod \"multus-additional-cni-plugins-rpthc\" (UID: \"3236f6c9-60de-47f9-84a9-5980933c58cd\") " pod="openshift-multus/multus-additional-cni-plugins-rpthc"
Sep 08 15:32:43.384426 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.384292    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3236f6c9-60de-47f9-84a9-5980933c58cd-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rpthc\" (UID: \"3236f6c9-60de-47f9-84a9-5980933c58cd\") " pod="openshift-multus/multus-additional-cni-plugins-rpthc"
Sep 08 15:32:43.384426 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.384316    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d4d759e-9d3c-497a-b546-462ae2b48785-metrics-certs\") pod \"network-metrics-daemon-h4cml\" (UID: \"7d4d759e-9d3c-497a-b546-462ae2b48785\") " pod="openshift-multus/network-metrics-daemon-h4cml"
Sep 08 15:32:43.384426 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.384332    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/95c9958b-a005-439c-b856-0cbb99f08700-multus-conf-dir\") pod \"multus-95s5s\" (UID: \"95c9958b-a005-439c-b856-0cbb99f08700\") " pod="openshift-multus/multus-95s5s"
Sep 08 15:32:43.384426 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.384353    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/195f1a74-19bc-49c9-aada-fa1c13488480-node-log\") pod \"ovnkube-node-f4vvt\" (UID: \"195f1a74-19bc-49c9-aada-fa1c13488480\") " pod="openshift-ovn-kubernetes/ovnkube-node-f4vvt"
Sep 08 15:32:43.384426 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.384374    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c9873505-399e-48b7-b74c-685ae3d9f7b5-socket-dir\") pod \"aws-ebs-csi-driver-node-kfw4l\" (UID: \"c9873505-399e-48b7-b74c-685ae3d9f7b5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kfw4l"
Sep 08 15:32:43.384426 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.384400    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c9873505-399e-48b7-b74c-685ae3d9f7b5-device-dir\") pod \"aws-ebs-csi-driver-node-kfw4l\" (UID: \"c9873505-399e-48b7-b74c-685ae3d9f7b5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kfw4l"
Sep 08 15:32:43.384426 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.384415    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3236f6c9-60de-47f9-84a9-5980933c58cd-cnibin\") pod \"multus-additional-cni-plugins-rpthc\" (UID: \"3236f6c9-60de-47f9-84a9-5980933c58cd\") " pod="openshift-multus/multus-additional-cni-plugins-rpthc"
Sep 08 15:32:43.384753 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.384452    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/95c9958b-a005-439c-b856-0cbb99f08700-host-var-lib-cni-bin\") pod \"multus-95s5s\" (UID: \"95c9958b-a005-439c-b856-0cbb99f08700\") " pod="openshift-multus/multus-95s5s"
Sep 08 15:32:43.384753 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.384480    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/37ec7079-542c-4b91-8b2b-83e8e89fcbb7-etc-tuned\") pod \"tuned-klnj5\" (UID: \"37ec7079-542c-4b91-8b2b-83e8e89fcbb7\") " pod="openshift-cluster-node-tuning-operator/tuned-klnj5"
Sep 08 15:32:43.384753 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.384503    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9498\" (UniqueName: \"kubernetes.io/projected/37ec7079-542c-4b91-8b2b-83e8e89fcbb7-kube-api-access-z9498\") pod \"tuned-klnj5\" (UID: \"37ec7079-542c-4b91-8b2b-83e8e89fcbb7\") " pod="openshift-cluster-node-tuning-operator/tuned-klnj5"
Sep 08 15:32:43.384753 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.384529    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79p6n\" (UniqueName: \"kubernetes.io/projected/3d62444d-76ba-4184-869c-496ba4da8d04-kube-api-access-79p6n\") pod \"network-check-target-5vt6c\" (UID: \"3d62444d-76ba-4184-869c-496ba4da8d04\") " pod="openshift-network-diagnostics/network-check-target-5vt6c"
Sep 08 15:32:43.384753 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.384545    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/95c9958b-a005-439c-b856-0cbb99f08700-host-var-lib-cni-multus\") pod \"multus-95s5s\" (UID: \"95c9958b-a005-439c-b856-0cbb99f08700\") " pod="openshift-multus/multus-95s5s"
Sep 08 15:32:43.384753 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.384568    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/195f1a74-19bc-49c9-aada-fa1c13488480-host-run-netns\") pod \"ovnkube-node-f4vvt\" (UID: \"195f1a74-19bc-49c9-aada-fa1c13488480\") " pod="openshift-ovn-kubernetes/ovnkube-node-f4vvt"
Sep 08 15:32:43.384753 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.384584    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/195f1a74-19bc-49c9-aada-fa1c13488480-host-run-ovn-kubernetes\") pod \"ovnkube-node-f4vvt\" (UID: \"195f1a74-19bc-49c9-aada-fa1c13488480\") " pod="openshift-ovn-kubernetes/ovnkube-node-f4vvt"
Sep 08 15:32:43.384753 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.384604    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/37ec7079-542c-4b91-8b2b-83e8e89fcbb7-run\") pod \"tuned-klnj5\" (UID: \"37ec7079-542c-4b91-8b2b-83e8e89fcbb7\") " pod="openshift-cluster-node-tuning-operator/tuned-klnj5"
Sep 08 15:32:43.384753 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.384621    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/95c9958b-a005-439c-b856-0cbb99f08700-host-run-multus-certs\") pod \"multus-95s5s\" (UID: \"95c9958b-a005-439c-b856-0cbb99f08700\") " pod="openshift-multus/multus-95s5s"
Sep 08 15:32:43.384753 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.384644    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/195f1a74-19bc-49c9-aada-fa1c13488480-ovn-node-metrics-cert\") pod \"ovnkube-node-f4vvt\" (UID: \"195f1a74-19bc-49c9-aada-fa1c13488480\") " pod="openshift-ovn-kubernetes/ovnkube-node-f4vvt"
Sep 08 15:32:43.384753 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.384665    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/84663ea3-1feb-4c87-8609-c580fa60416e-kubelet-config\") pod \"global-pull-secret-syncer-fw5jh\" (UID: \"84663ea3-1feb-4c87-8609-c580fa60416e\") " pod="kube-system/global-pull-secret-syncer-fw5jh"
Sep 08 15:32:43.384753 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.384680    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/65a8a5a6-5e05-4ee5-99a6-c026327dfa43-iptables-alerter-script\") pod \"iptables-alerter-p4rw9\" (UID: \"65a8a5a6-5e05-4ee5-99a6-c026327dfa43\") " pod="openshift-network-operator/iptables-alerter-p4rw9"
Sep 08 15:32:43.384753 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.384696    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/28cd0557-2fec-4948-af98-6c67fb1e568d-agent-certs\") pod \"konnectivity-agent-st9hm\" (UID: \"28cd0557-2fec-4948-af98-6c67fb1e568d\") " pod="kube-system/konnectivity-agent-st9hm"
Sep 08 15:32:43.384753 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.384716    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/195f1a74-19bc-49c9-aada-fa1c13488480-systemd-units\") pod \"ovnkube-node-f4vvt\" (UID: \"195f1a74-19bc-49c9-aada-fa1c13488480\") " pod="openshift-ovn-kubernetes/ovnkube-node-f4vvt"
Sep 08 15:32:43.384753 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.384736    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/195f1a74-19bc-49c9-aada-fa1c13488480-run-openvswitch\") pod \"ovnkube-node-f4vvt\" (UID: \"195f1a74-19bc-49c9-aada-fa1c13488480\") " pod="openshift-ovn-kubernetes/ovnkube-node-f4vvt"
Sep 08 15:32:43.384753 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.384759    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/84663ea3-1feb-4c87-8609-c580fa60416e-dbus\") pod \"global-pull-secret-syncer-fw5jh\" (UID: \"84663ea3-1feb-4c87-8609-c580fa60416e\") " pod="kube-system/global-pull-secret-syncer-fw5jh"
Sep 08 15:32:43.385423 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.384778    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c9873505-399e-48b7-b74c-685ae3d9f7b5-registration-dir\") pod \"aws-ebs-csi-driver-node-kfw4l\" (UID: \"c9873505-399e-48b7-b74c-685ae3d9f7b5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kfw4l"
Sep 08 15:32:43.385423 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.384796    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bblp\" (UniqueName: \"kubernetes.io/projected/65a8a5a6-5e05-4ee5-99a6-c026327dfa43-kube-api-access-7bblp\") pod \"iptables-alerter-p4rw9\" (UID: \"65a8a5a6-5e05-4ee5-99a6-c026327dfa43\") " pod="openshift-network-operator/iptables-alerter-p4rw9"
Sep 08 15:32:43.385423 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.384833    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/95c9958b-a005-439c-b856-0cbb99f08700-multus-socket-dir-parent\") pod \"multus-95s5s\" (UID: \"95c9958b-a005-439c-b856-0cbb99f08700\") " pod="openshift-multus/multus-95s5s"
Sep 08 15:32:43.385423 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.384849    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/95c9958b-a005-439c-b856-0cbb99f08700-hostroot\") pod \"multus-95s5s\" (UID: \"95c9958b-a005-439c-b856-0cbb99f08700\") " pod="openshift-multus/multus-95s5s"
Sep 08 15:32:43.385423 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.384865    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/195f1a74-19bc-49c9-aada-fa1c13488480-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-f4vvt\" (UID: \"195f1a74-19bc-49c9-aada-fa1c13488480\") " pod="openshift-ovn-kubernetes/ovnkube-node-f4vvt"
Sep 08 15:32:43.385423 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.384903    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/37ec7079-542c-4b91-8b2b-83e8e89fcbb7-var-lib-kubelet\") pod \"tuned-klnj5\" (UID: \"37ec7079-542c-4b91-8b2b-83e8e89fcbb7\") " pod="openshift-cluster-node-tuning-operator/tuned-klnj5"
Sep 08 15:32:43.385423 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.384918    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/37ec7079-542c-4b91-8b2b-83e8e89fcbb7-host\") pod \"tuned-klnj5\" (UID: \"37ec7079-542c-4b91-8b2b-83e8e89fcbb7\") " pod="openshift-cluster-node-tuning-operator/tuned-klnj5"
Sep 08 15:32:43.385423 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.384933    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/195f1a74-19bc-49c9-aada-fa1c13488480-var-lib-openvswitch\") pod \"ovnkube-node-f4vvt\" (UID: \"195f1a74-19bc-49c9-aada-fa1c13488480\") " pod="openshift-ovn-kubernetes/ovnkube-node-f4vvt"
Sep 08 15:32:43.385423 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.384947    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/195f1a74-19bc-49c9-aada-fa1c13488480-ovnkube-script-lib\") pod \"ovnkube-node-f4vvt\" (UID: \"195f1a74-19bc-49c9-aada-fa1c13488480\") " pod="openshift-ovn-kubernetes/ovnkube-node-f4vvt"
Sep 08 15:32:43.385423 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.384974    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/37ec7079-542c-4b91-8b2b-83e8e89fcbb7-etc-kubernetes\") pod \"tuned-klnj5\" (UID: \"37ec7079-542c-4b91-8b2b-83e8e89fcbb7\") " pod="openshift-cluster-node-tuning-operator/tuned-klnj5"
Sep 08 15:32:43.385423 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.384988    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/37ec7079-542c-4b91-8b2b-83e8e89fcbb7-etc-sysctl-d\") pod \"tuned-klnj5\" (UID: \"37ec7079-542c-4b91-8b2b-83e8e89fcbb7\") " pod="openshift-cluster-node-tuning-operator/tuned-klnj5"
Sep 08 15:32:43.385423 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.385003    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/37ec7079-542c-4b91-8b2b-83e8e89fcbb7-etc-sysctl-conf\") pod \"tuned-klnj5\" (UID: \"37ec7079-542c-4b91-8b2b-83e8e89fcbb7\") " pod="openshift-cluster-node-tuning-operator/tuned-klnj5"
Sep 08 15:32:43.385423 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.385026    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tx6tv\" (UniqueName: \"kubernetes.io/projected/3236f6c9-60de-47f9-84a9-5980933c58cd-kube-api-access-tx6tv\") pod \"multus-additional-cni-plugins-rpthc\" (UID: \"3236f6c9-60de-47f9-84a9-5980933c58cd\") " pod="openshift-multus/multus-additional-cni-plugins-rpthc"
Sep 08 15:32:43.385423 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.385053    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/95c9958b-a005-439c-b856-0cbb99f08700-multus-cni-dir\") pod \"multus-95s5s\" (UID: \"95c9958b-a005-439c-b856-0cbb99f08700\") " pod="openshift-multus/multus-95s5s"
Sep 08 15:32:43.385423 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.385074    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/195f1a74-19bc-49c9-aada-fa1c13488480-etc-openvswitch\") pod \"ovnkube-node-f4vvt\" (UID: \"195f1a74-19bc-49c9-aada-fa1c13488480\") " pod="openshift-ovn-kubernetes/ovnkube-node-f4vvt"
Sep 08 15:32:43.385423 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.385091    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/195f1a74-19bc-49c9-aada-fa1c13488480-host-cni-bin\") pod \"ovnkube-node-f4vvt\" (UID: \"195f1a74-19bc-49c9-aada-fa1c13488480\") " pod="openshift-ovn-kubernetes/ovnkube-node-f4vvt"
Sep 08 15:32:43.386024 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.385106    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqqvf\" (UniqueName: \"kubernetes.io/projected/195f1a74-19bc-49c9-aada-fa1c13488480-kube-api-access-lqqvf\") pod \"ovnkube-node-f4vvt\" (UID: \"195f1a74-19bc-49c9-aada-fa1c13488480\") " pod="openshift-ovn-kubernetes/ovnkube-node-f4vvt"
Sep 08 15:32:43.386024 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.385138    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c9873505-399e-48b7-b74c-685ae3d9f7b5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kfw4l\" (UID: \"c9873505-399e-48b7-b74c-685ae3d9f7b5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kfw4l"
Sep 08 15:32:43.386024 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.385166    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/84663ea3-1feb-4c87-8609-c580fa60416e-original-pull-secret\") pod \"global-pull-secret-syncer-fw5jh\" (UID: \"84663ea3-1feb-4c87-8609-c580fa60416e\") " pod="kube-system/global-pull-secret-syncer-fw5jh"
Sep 08 15:32:43.386024 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.385184    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/95c9958b-a005-439c-b856-0cbb99f08700-host-var-lib-kubelet\") pod \"multus-95s5s\" (UID: \"95c9958b-a005-439c-b856-0cbb99f08700\") " pod="openshift-multus/multus-95s5s"
Sep 08 15:32:43.386024 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.385206    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dggm\" (UniqueName: \"kubernetes.io/projected/95c9958b-a005-439c-b856-0cbb99f08700-kube-api-access-4dggm\") pod \"multus-95s5s\" (UID: \"95c9958b-a005-439c-b856-0cbb99f08700\") " pod="openshift-multus/multus-95s5s"
Sep 08 15:32:43.386024 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.385225    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/195f1a74-19bc-49c9-aada-fa1c13488480-run-ovn\") pod \"ovnkube-node-f4vvt\" (UID: \"195f1a74-19bc-49c9-aada-fa1c13488480\") " pod="openshift-ovn-kubernetes/ovnkube-node-f4vvt"
Sep 08 15:32:43.386024 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.385239    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/195f1a74-19bc-49c9-aada-fa1c13488480-host-cni-netd\") pod \"ovnkube-node-f4vvt\" (UID: \"195f1a74-19bc-49c9-aada-fa1c13488480\") " pod="openshift-ovn-kubernetes/ovnkube-node-f4vvt"
Sep 08 15:32:43.386024 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.385254    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/37ec7079-542c-4b91-8b2b-83e8e89fcbb7-etc-modprobe-d\") pod \"tuned-klnj5\" (UID: \"37ec7079-542c-4b91-8b2b-83e8e89fcbb7\") " pod="openshift-cluster-node-tuning-operator/tuned-klnj5"
Sep 08 15:32:43.386024 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.385268    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5f9598ad-fb27-4dda-ab17-98d54cbd30aa-host\") pod \"node-ca-6q7cq\" (UID: \"5f9598ad-fb27-4dda-ab17-98d54cbd30aa\") " pod="openshift-image-registry/node-ca-6q7cq"
Sep 08 15:32:43.386024 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.385300    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5f9598ad-fb27-4dda-ab17-98d54cbd30aa-serviceca\") pod \"node-ca-6q7cq\" (UID: \"5f9598ad-fb27-4dda-ab17-98d54cbd30aa\") " pod="openshift-image-registry/node-ca-6q7cq"
Sep 08 15:32:43.386024 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.385325    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/28cd0557-2fec-4948-af98-6c67fb1e568d-konnectivity-ca\") pod \"konnectivity-agent-st9hm\" (UID: \"28cd0557-2fec-4948-af98-6c67fb1e568d\") " pod="kube-system/konnectivity-agent-st9hm"
Sep 08 15:32:43.386024 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.385352    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/95c9958b-a005-439c-b856-0cbb99f08700-cnibin\") pod \"multus-95s5s\" (UID: \"95c9958b-a005-439c-b856-0cbb99f08700\") " pod="openshift-multus/multus-95s5s"
Sep 08 15:32:43.386024 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.385375    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/95c9958b-a005-439c-b856-0cbb99f08700-cni-binary-copy\") pod \"multus-95s5s\" (UID: \"95c9958b-a005-439c-b856-0cbb99f08700\") " pod="openshift-multus/multus-95s5s"
Sep 08 15:32:43.386024 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.385405    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/95c9958b-a005-439c-b856-0cbb99f08700-etc-kubernetes\") pod \"multus-95s5s\" (UID: \"95c9958b-a005-439c-b856-0cbb99f08700\") " pod="openshift-multus/multus-95s5s"
Sep 08 15:32:43.386024 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.385439    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/195f1a74-19bc-49c9-aada-fa1c13488480-ovnkube-config\") pod \"ovnkube-node-f4vvt\" (UID: \"195f1a74-19bc-49c9-aada-fa1c13488480\") " pod="openshift-ovn-kubernetes/ovnkube-node-f4vvt"
Sep 08 15:32:43.386024 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.385466    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/195f1a74-19bc-49c9-aada-fa1c13488480-env-overrides\") pod \"ovnkube-node-f4vvt\" (UID: \"195f1a74-19bc-49c9-aada-fa1c13488480\") " pod="openshift-ovn-kubernetes/ovnkube-node-f4vvt"
Sep 08 15:32:43.386024 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.385488    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/195f1a74-19bc-49c9-aada-fa1c13488480-host-slash\") pod \"ovnkube-node-f4vvt\" (UID: \"195f1a74-19bc-49c9-aada-fa1c13488480\") " pod="openshift-ovn-kubernetes/ovnkube-node-f4vvt"
Sep 08 15:32:43.386577 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.385521    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/195f1a74-19bc-49c9-aada-fa1c13488480-run-systemd\") pod \"ovnkube-node-f4vvt\" (UID: \"195f1a74-19bc-49c9-aada-fa1c13488480\") " pod="openshift-ovn-kubernetes/ovnkube-node-f4vvt"
Sep 08 15:32:43.386577 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.385549    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/37ec7079-542c-4b91-8b2b-83e8e89fcbb7-lib-modules\") pod \"tuned-klnj5\" (UID: \"37ec7079-542c-4b91-8b2b-83e8e89fcbb7\") " pod="openshift-cluster-node-tuning-operator/tuned-klnj5"
Sep 08 15:32:43.386577 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.385571    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/37ec7079-542c-4b91-8b2b-83e8e89fcbb7-tmp\") pod \"tuned-klnj5\" (UID: \"37ec7079-542c-4b91-8b2b-83e8e89fcbb7\") " pod="openshift-cluster-node-tuning-operator/tuned-klnj5"
Sep 08 15:32:43.386577 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.385605    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3236f6c9-60de-47f9-84a9-5980933c58cd-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rpthc\" (UID: \"3236f6c9-60de-47f9-84a9-5980933c58cd\") " pod="openshift-multus/multus-additional-cni-plugins-rpthc"
Sep 08 15:32:43.386577 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.385637    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3236f6c9-60de-47f9-84a9-5980933c58cd-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rpthc\" (UID: \"3236f6c9-60de-47f9-84a9-5980933c58cd\") " pod="openshift-multus/multus-additional-cni-plugins-rpthc"
Sep 08 15:32:43.386577 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.385682    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a955f56c-5ac8-422a-8ddc-ce17a8361eb6-hosts-file\") pod \"node-resolver-vp9nk\" (UID: \"a955f56c-5ac8-422a-8ddc-ce17a8361eb6\") " pod="openshift-dns/node-resolver-vp9nk"
Sep 08 15:32:43.386577 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.385707    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjcrr\" (UniqueName: \"kubernetes.io/projected/7d4d759e-9d3c-497a-b546-462ae2b48785-kube-api-access-vjcrr\") pod \"network-metrics-daemon-h4cml\" (UID: \"7d4d759e-9d3c-497a-b546-462ae2b48785\") " pod="openshift-multus/network-metrics-daemon-h4cml"
Sep 08 15:32:43.386577 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.385732    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3236f6c9-60de-47f9-84a9-5980933c58cd-cni-binary-copy\") pod \"multus-additional-cni-plugins-rpthc\" (UID: \"3236f6c9-60de-47f9-84a9-5980933c58cd\") " pod="openshift-multus/multus-additional-cni-plugins-rpthc"
Sep 08 15:32:43.386577 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.385768    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nkb9\" (UniqueName: \"kubernetes.io/projected/a955f56c-5ac8-422a-8ddc-ce17a8361eb6-kube-api-access-7nkb9\") pod \"node-resolver-vp9nk\" (UID: \"a955f56c-5ac8-422a-8ddc-ce17a8361eb6\") " pod="openshift-dns/node-resolver-vp9nk"
Sep 08 15:32:43.452453 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:43.452406    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8cf966721b2ec8fb0cd4ca3ebd9318e.slice/crio-c01f9bd6425649322405931ef99d72694b3dec0503ba6aeef60913cf327fc4f5 WatchSource:0}: Error finding container c01f9bd6425649322405931ef99d72694b3dec0503ba6aeef60913cf327fc4f5: Status 404 returned error can't find the container with id c01f9bd6425649322405931ef99d72694b3dec0503ba6aeef60913cf327fc4f5
Sep 08 15:32:43.452744 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:43.452726    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a20335aaf3733c5ea5fa6f08be94600.slice/crio-5ed6e3aef0d28d77ab2548d29e4e31157dbf57ce2f168859aad44b977c8bfecb WatchSource:0}: Error finding container 5ed6e3aef0d28d77ab2548d29e4e31157dbf57ce2f168859aad44b977c8bfecb: Status 404 returned error can't find the container with id 5ed6e3aef0d28d77ab2548d29e4e31157dbf57ce2f168859aad44b977c8bfecb
Sep 08 15:32:43.459886 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.459859    2549 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 08 15:32:43.486498 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.486472    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nkb9\" (UniqueName: \"kubernetes.io/projected/a955f56c-5ac8-422a-8ddc-ce17a8361eb6-kube-api-access-7nkb9\") pod \"node-resolver-vp9nk\" (UID: \"a955f56c-5ac8-422a-8ddc-ce17a8361eb6\") " pod="openshift-dns/node-resolver-vp9nk"
Sep 08 15:32:43.486616 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.486511    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fnhz\" (UniqueName: \"kubernetes.io/projected/5f9598ad-fb27-4dda-ab17-98d54cbd30aa-kube-api-access-2fnhz\") pod \"node-ca-6q7cq\" (UID: \"5f9598ad-fb27-4dda-ab17-98d54cbd30aa\") " pod="openshift-image-registry/node-ca-6q7cq"
Sep 08 15:32:43.486616 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.486534    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/37ec7079-542c-4b91-8b2b-83e8e89fcbb7-etc-systemd\") pod \"tuned-klnj5\" (UID: \"37ec7079-542c-4b91-8b2b-83e8e89fcbb7\") " pod="openshift-cluster-node-tuning-operator/tuned-klnj5"
Sep 08 15:32:43.486616 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.486557    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/37ec7079-542c-4b91-8b2b-83e8e89fcbb7-sys\") pod \"tuned-klnj5\" (UID: \"37ec7079-542c-4b91-8b2b-83e8e89fcbb7\") " pod="openshift-cluster-node-tuning-operator/tuned-klnj5"
Sep 08 15:32:43.486616 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.486580    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/95c9958b-a005-439c-b856-0cbb99f08700-host-run-k8s-cni-cncf-io\") pod \"multus-95s5s\" (UID: \"95c9958b-a005-439c-b856-0cbb99f08700\") " pod="openshift-multus/multus-95s5s"
Sep 08 15:32:43.486830 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.486634    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/95c9958b-a005-439c-b856-0cbb99f08700-host-run-k8s-cni-cncf-io\") pod \"multus-95s5s\" (UID: \"95c9958b-a005-439c-b856-0cbb99f08700\") " pod="openshift-multus/multus-95s5s"
Sep 08 15:32:43.486830 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.486639    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/37ec7079-542c-4b91-8b2b-83e8e89fcbb7-etc-systemd\") pod \"tuned-klnj5\" (UID: \"37ec7079-542c-4b91-8b2b-83e8e89fcbb7\") " pod="openshift-cluster-node-tuning-operator/tuned-klnj5"
Sep 08 15:32:43.486830 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.486649    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/37ec7079-542c-4b91-8b2b-83e8e89fcbb7-sys\") pod \"tuned-klnj5\" (UID: \"37ec7079-542c-4b91-8b2b-83e8e89fcbb7\") " pod="openshift-cluster-node-tuning-operator/tuned-klnj5"
Sep 08 15:32:43.486830 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.486685    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/95c9958b-a005-439c-b856-0cbb99f08700-host-run-netns\") pod \"multus-95s5s\" (UID: \"95c9958b-a005-439c-b856-0cbb99f08700\") " pod="openshift-multus/multus-95s5s"
Sep 08 15:32:43.486830 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.486713    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/95c9958b-a005-439c-b856-0cbb99f08700-multus-daemon-config\") pod \"multus-95s5s\" (UID: \"95c9958b-a005-439c-b856-0cbb99f08700\") " pod="openshift-multus/multus-95s5s"
Sep 08 15:32:43.486830 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.486741    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/65a8a5a6-5e05-4ee5-99a6-c026327dfa43-host-slash\") pod \"iptables-alerter-p4rw9\" (UID: \"65a8a5a6-5e05-4ee5-99a6-c026327dfa43\") " pod="openshift-network-operator/iptables-alerter-p4rw9"
Sep 08 15:32:43.486830 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.486761    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/95c9958b-a005-439c-b856-0cbb99f08700-host-run-netns\") pod \"multus-95s5s\" (UID: \"95c9958b-a005-439c-b856-0cbb99f08700\") " pod="openshift-multus/multus-95s5s"
Sep 08 15:32:43.486830 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.486763    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/195f1a74-19bc-49c9-aada-fa1c13488480-log-socket\") pod \"ovnkube-node-f4vvt\" (UID: \"195f1a74-19bc-49c9-aada-fa1c13488480\") " pod="openshift-ovn-kubernetes/ovnkube-node-f4vvt"
Sep 08 15:32:43.486830 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.486794    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/195f1a74-19bc-49c9-aada-fa1c13488480-log-socket\") pod \"ovnkube-node-f4vvt\" (UID: \"195f1a74-19bc-49c9-aada-fa1c13488480\") " pod="openshift-ovn-kubernetes/ovnkube-node-f4vvt"
Sep 08 15:32:43.486830 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.486799    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/65a8a5a6-5e05-4ee5-99a6-c026327dfa43-host-slash\") pod \"iptables-alerter-p4rw9\" (UID: \"65a8a5a6-5e05-4ee5-99a6-c026327dfa43\") " pod="openshift-network-operator/iptables-alerter-p4rw9"
Sep 08 15:32:43.486830 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.486807    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c9873505-399e-48b7-b74c-685ae3d9f7b5-etc-selinux\") pod \"aws-ebs-csi-driver-node-kfw4l\" (UID: \"c9873505-399e-48b7-b74c-685ae3d9f7b5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kfw4l"
Sep 08 15:32:43.487431 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.486852    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c9873505-399e-48b7-b74c-685ae3d9f7b5-sys-fs\") pod \"aws-ebs-csi-driver-node-kfw4l\" (UID: \"c9873505-399e-48b7-b74c-685ae3d9f7b5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kfw4l"
Sep 08 15:32:43.487431 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.486878    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/37ec7079-542c-4b91-8b2b-83e8e89fcbb7-etc-sysconfig\") pod \"tuned-klnj5\" (UID: \"37ec7079-542c-4b91-8b2b-83e8e89fcbb7\") " pod="openshift-cluster-node-tuning-operator/tuned-klnj5"
Sep 08 15:32:43.487431 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.486909    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c9873505-399e-48b7-b74c-685ae3d9f7b5-sys-fs\") pod \"aws-ebs-csi-driver-node-kfw4l\" (UID: \"c9873505-399e-48b7-b74c-685ae3d9f7b5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kfw4l"
Sep 08 15:32:43.487431 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.486913    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c9873505-399e-48b7-b74c-685ae3d9f7b5-etc-selinux\") pod \"aws-ebs-csi-driver-node-kfw4l\" (UID: \"c9873505-399e-48b7-b74c-685ae3d9f7b5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kfw4l"
Sep 08 15:32:43.487431 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.486919    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a955f56c-5ac8-422a-8ddc-ce17a8361eb6-tmp-dir\") pod \"node-resolver-vp9nk\" (UID: \"a955f56c-5ac8-422a-8ddc-ce17a8361eb6\") " pod="openshift-dns/node-resolver-vp9nk"
Sep 08 15:32:43.487431 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.486948    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/95c9958b-a005-439c-b856-0cbb99f08700-system-cni-dir\") pod \"multus-95s5s\" (UID: \"95c9958b-a005-439c-b856-0cbb99f08700\") " pod="openshift-multus/multus-95s5s"
Sep 08 15:32:43.487431 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.486959    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/37ec7079-542c-4b91-8b2b-83e8e89fcbb7-etc-sysconfig\") pod \"tuned-klnj5\" (UID: \"37ec7079-542c-4b91-8b2b-83e8e89fcbb7\") " pod="openshift-cluster-node-tuning-operator/tuned-klnj5"
Sep 08 15:32:43.487431 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.486972    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/195f1a74-19bc-49c9-aada-fa1c13488480-host-kubelet\") pod \"ovnkube-node-f4vvt\" (UID: \"195f1a74-19bc-49c9-aada-fa1c13488480\") " pod="openshift-ovn-kubernetes/ovnkube-node-f4vvt"
Sep 08 15:32:43.487431 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.487000    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wc458\" (UniqueName: \"kubernetes.io/projected/c9873505-399e-48b7-b74c-685ae3d9f7b5-kube-api-access-wc458\") pod \"aws-ebs-csi-driver-node-kfw4l\" (UID: \"c9873505-399e-48b7-b74c-685ae3d9f7b5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kfw4l"
Sep 08 15:32:43.487431 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.487028    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3236f6c9-60de-47f9-84a9-5980933c58cd-system-cni-dir\") pod \"multus-additional-cni-plugins-rpthc\" (UID: \"3236f6c9-60de-47f9-84a9-5980933c58cd\") " pod="openshift-multus/multus-additional-cni-plugins-rpthc"
Sep 08 15:32:43.487431 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.487052    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/195f1a74-19bc-49c9-aada-fa1c13488480-host-kubelet\") pod \"ovnkube-node-f4vvt\" (UID: \"195f1a74-19bc-49c9-aada-fa1c13488480\") " pod="openshift-ovn-kubernetes/ovnkube-node-f4vvt"
Sep 08 15:32:43.487431 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.487062    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/95c9958b-a005-439c-b856-0cbb99f08700-system-cni-dir\") pod \"multus-95s5s\" (UID: \"95c9958b-a005-439c-b856-0cbb99f08700\") " pod="openshift-multus/multus-95s5s"
Sep 08 15:32:43.487431 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.487055    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3236f6c9-60de-47f9-84a9-5980933c58cd-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rpthc\" (UID: \"3236f6c9-60de-47f9-84a9-5980933c58cd\") " pod="openshift-multus/multus-additional-cni-plugins-rpthc"
Sep 08 15:32:43.487431 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.487103    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3236f6c9-60de-47f9-84a9-5980933c58cd-system-cni-dir\") pod \"multus-additional-cni-plugins-rpthc\" (UID: \"3236f6c9-60de-47f9-84a9-5980933c58cd\") " pod="openshift-multus/multus-additional-cni-plugins-rpthc"
Sep 08 15:32:43.487431 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.487107    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a955f56c-5ac8-422a-8ddc-ce17a8361eb6-tmp-dir\") pod \"node-resolver-vp9nk\" (UID: \"a955f56c-5ac8-422a-8ddc-ce17a8361eb6\") " pod="openshift-dns/node-resolver-vp9nk"
Sep 08 15:32:43.487431 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.487122    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d4d759e-9d3c-497a-b546-462ae2b48785-metrics-certs\") pod \"network-metrics-daemon-h4cml\" (UID: \"7d4d759e-9d3c-497a-b546-462ae2b48785\") " pod="openshift-multus/network-metrics-daemon-h4cml"
Sep 08 15:32:43.487431 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.487149    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/95c9958b-a005-439c-b856-0cbb99f08700-multus-conf-dir\") pod \"multus-95s5s\" (UID: \"95c9958b-a005-439c-b856-0cbb99f08700\") " pod="openshift-multus/multus-95s5s"
Sep 08 15:32:43.488361 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.487164    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/195f1a74-19bc-49c9-aada-fa1c13488480-node-log\") pod \"ovnkube-node-f4vvt\" (UID: \"195f1a74-19bc-49c9-aada-fa1c13488480\") " pod="openshift-ovn-kubernetes/ovnkube-node-f4vvt"
Sep 08 15:32:43.488361 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.487186    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c9873505-399e-48b7-b74c-685ae3d9f7b5-socket-dir\") pod \"aws-ebs-csi-driver-node-kfw4l\" (UID: \"c9873505-399e-48b7-b74c-685ae3d9f7b5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kfw4l"
Sep 08 15:32:43.488361 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:32:43.487213    2549 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 15:32:43.488361 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.487221    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/95c9958b-a005-439c-b856-0cbb99f08700-multus-conf-dir\") pod \"multus-95s5s\" (UID: \"95c9958b-a005-439c-b856-0cbb99f08700\") " pod="openshift-multus/multus-95s5s"
Sep 08 15:32:43.488361 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.487230    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/195f1a74-19bc-49c9-aada-fa1c13488480-node-log\") pod \"ovnkube-node-f4vvt\" (UID: \"195f1a74-19bc-49c9-aada-fa1c13488480\") " pod="openshift-ovn-kubernetes/ovnkube-node-f4vvt"
Sep 08 15:32:43.488361 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.487212    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c9873505-399e-48b7-b74c-685ae3d9f7b5-device-dir\") pod \"aws-ebs-csi-driver-node-kfw4l\" (UID: \"c9873505-399e-48b7-b74c-685ae3d9f7b5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kfw4l"
Sep 08 15:32:43.488361 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.487247    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3236f6c9-60de-47f9-84a9-5980933c58cd-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rpthc\" (UID: \"3236f6c9-60de-47f9-84a9-5980933c58cd\") " pod="openshift-multus/multus-additional-cni-plugins-rpthc"
Sep 08 15:32:43.488361 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:32:43.487276    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d4d759e-9d3c-497a-b546-462ae2b48785-metrics-certs podName:7d4d759e-9d3c-497a-b546-462ae2b48785 nodeName:}" failed. No retries permitted until 2026-09-08 15:32:43.987252604 +0000 UTC m=+2.076703682 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7d4d759e-9d3c-497a-b546-462ae2b48785-metrics-certs") pod "network-metrics-daemon-h4cml" (UID: "7d4d759e-9d3c-497a-b546-462ae2b48785") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 15:32:43.488361 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.487259    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c9873505-399e-48b7-b74c-685ae3d9f7b5-device-dir\") pod \"aws-ebs-csi-driver-node-kfw4l\" (UID: \"c9873505-399e-48b7-b74c-685ae3d9f7b5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kfw4l"
Sep 08 15:32:43.488361 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.487310    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3236f6c9-60de-47f9-84a9-5980933c58cd-cnibin\") pod \"multus-additional-cni-plugins-rpthc\" (UID: \"3236f6c9-60de-47f9-84a9-5980933c58cd\") " pod="openshift-multus/multus-additional-cni-plugins-rpthc"
Sep 08 15:32:43.488361 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.487317    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c9873505-399e-48b7-b74c-685ae3d9f7b5-socket-dir\") pod \"aws-ebs-csi-driver-node-kfw4l\" (UID: \"c9873505-399e-48b7-b74c-685ae3d9f7b5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kfw4l"
Sep 08 15:32:43.488361 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.487329    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/95c9958b-a005-439c-b856-0cbb99f08700-host-var-lib-cni-bin\") pod \"multus-95s5s\" (UID: \"95c9958b-a005-439c-b856-0cbb99f08700\") " pod="openshift-multus/multus-95s5s"
Sep 08 15:32:43.488361 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.487344    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/37ec7079-542c-4b91-8b2b-83e8e89fcbb7-etc-tuned\") pod \"tuned-klnj5\" (UID: \"37ec7079-542c-4b91-8b2b-83e8e89fcbb7\") " pod="openshift-cluster-node-tuning-operator/tuned-klnj5"
Sep 08 15:32:43.488361 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.487363    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9498\" (UniqueName: \"kubernetes.io/projected/37ec7079-542c-4b91-8b2b-83e8e89fcbb7-kube-api-access-z9498\") pod \"tuned-klnj5\" (UID: \"37ec7079-542c-4b91-8b2b-83e8e89fcbb7\") " pod="openshift-cluster-node-tuning-operator/tuned-klnj5"
Sep 08 15:32:43.488361 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.487383    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/95c9958b-a005-439c-b856-0cbb99f08700-host-var-lib-cni-bin\") pod \"multus-95s5s\" (UID: \"95c9958b-a005-439c-b856-0cbb99f08700\") " pod="openshift-multus/multus-95s5s"
Sep 08 15:32:43.488361 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.487384    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3236f6c9-60de-47f9-84a9-5980933c58cd-cnibin\") pod \"multus-additional-cni-plugins-rpthc\" (UID: \"3236f6c9-60de-47f9-84a9-5980933c58cd\") " pod="openshift-multus/multus-additional-cni-plugins-rpthc"
Sep 08 15:32:43.488361 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.487387    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79p6n\" (UniqueName: \"kubernetes.io/projected/3d62444d-76ba-4184-869c-496ba4da8d04-kube-api-access-79p6n\") pod \"network-check-target-5vt6c\" (UID: \"3d62444d-76ba-4184-869c-496ba4da8d04\") " pod="openshift-network-diagnostics/network-check-target-5vt6c"
Sep 08 15:32:43.489109 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.487430    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/95c9958b-a005-439c-b856-0cbb99f08700-host-var-lib-cni-multus\") pod \"multus-95s5s\" (UID: \"95c9958b-a005-439c-b856-0cbb99f08700\") " pod="openshift-multus/multus-95s5s"
Sep 08 15:32:43.489109 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.487458    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/195f1a74-19bc-49c9-aada-fa1c13488480-host-run-netns\") pod \"ovnkube-node-f4vvt\" (UID: \"195f1a74-19bc-49c9-aada-fa1c13488480\") " pod="openshift-ovn-kubernetes/ovnkube-node-f4vvt"
Sep 08 15:32:43.489109 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.487486    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/195f1a74-19bc-49c9-aada-fa1c13488480-host-run-ovn-kubernetes\") pod \"ovnkube-node-f4vvt\" (UID: \"195f1a74-19bc-49c9-aada-fa1c13488480\") " pod="openshift-ovn-kubernetes/ovnkube-node-f4vvt"
Sep 08 15:32:43.489109 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.487534    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/195f1a74-19bc-49c9-aada-fa1c13488480-host-run-ovn-kubernetes\") pod \"ovnkube-node-f4vvt\" (UID: \"195f1a74-19bc-49c9-aada-fa1c13488480\") " pod="openshift-ovn-kubernetes/ovnkube-node-f4vvt"
Sep 08 15:32:43.489109 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.487535    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/195f1a74-19bc-49c9-aada-fa1c13488480-host-run-netns\") pod \"ovnkube-node-f4vvt\" (UID: \"195f1a74-19bc-49c9-aada-fa1c13488480\") " pod="openshift-ovn-kubernetes/ovnkube-node-f4vvt"
Sep 08 15:32:43.489109 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.487558    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/37ec7079-542c-4b91-8b2b-83e8e89fcbb7-run\") pod \"tuned-klnj5\" (UID: \"37ec7079-542c-4b91-8b2b-83e8e89fcbb7\") " pod="openshift-cluster-node-tuning-operator/tuned-klnj5"
Sep 08 15:32:43.489109 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.487590    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/95c9958b-a005-439c-b856-0cbb99f08700-host-run-multus-certs\") pod \"multus-95s5s\" (UID: \"95c9958b-a005-439c-b856-0cbb99f08700\") " pod="openshift-multus/multus-95s5s"
Sep 08 15:32:43.489109 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.487534    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/95c9958b-a005-439c-b856-0cbb99f08700-host-var-lib-cni-multus\") pod \"multus-95s5s\" (UID: \"95c9958b-a005-439c-b856-0cbb99f08700\") " pod="openshift-multus/multus-95s5s"
Sep 08 15:32:43.489109 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.487617    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/195f1a74-19bc-49c9-aada-fa1c13488480-ovn-node-metrics-cert\") pod \"ovnkube-node-f4vvt\" (UID: \"195f1a74-19bc-49c9-aada-fa1c13488480\") " pod="openshift-ovn-kubernetes/ovnkube-node-f4vvt"
Sep 08 15:32:43.489109 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.487663    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/37ec7079-542c-4b91-8b2b-83e8e89fcbb7-run\") pod \"tuned-klnj5\" (UID: \"37ec7079-542c-4b91-8b2b-83e8e89fcbb7\") " pod="openshift-cluster-node-tuning-operator/tuned-klnj5"
Sep 08 15:32:43.489109 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.487669    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/84663ea3-1feb-4c87-8609-c580fa60416e-kubelet-config\") pod \"global-pull-secret-syncer-fw5jh\" (UID: \"84663ea3-1feb-4c87-8609-c580fa60416e\") " pod="kube-system/global-pull-secret-syncer-fw5jh"
Sep 08 15:32:43.489109 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.487684    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/95c9958b-a005-439c-b856-0cbb99f08700-host-run-multus-certs\") pod \"multus-95s5s\" (UID: \"95c9958b-a005-439c-b856-0cbb99f08700\") " pod="openshift-multus/multus-95s5s"
Sep 08 15:32:43.489109 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.487643    2549 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 08 15:32:43.489109 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.487710    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/65a8a5a6-5e05-4ee5-99a6-c026327dfa43-iptables-alerter-script\") pod \"iptables-alerter-p4rw9\" (UID: \"65a8a5a6-5e05-4ee5-99a6-c026327dfa43\") " pod="openshift-network-operator/iptables-alerter-p4rw9"
Sep 08 15:32:43.489109 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.487719    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/84663ea3-1feb-4c87-8609-c580fa60416e-kubelet-config\") pod \"global-pull-secret-syncer-fw5jh\" (UID: \"84663ea3-1feb-4c87-8609-c580fa60416e\") " pod="kube-system/global-pull-secret-syncer-fw5jh"
Sep 08 15:32:43.489109 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.487737    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/28cd0557-2fec-4948-af98-6c67fb1e568d-agent-certs\") pod \"konnectivity-agent-st9hm\" (UID: \"28cd0557-2fec-4948-af98-6c67fb1e568d\") " pod="kube-system/konnectivity-agent-st9hm"
Sep 08 15:32:43.489109 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.487764    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/195f1a74-19bc-49c9-aada-fa1c13488480-systemd-units\") pod \"ovnkube-node-f4vvt\" (UID: \"195f1a74-19bc-49c9-aada-fa1c13488480\") " pod="openshift-ovn-kubernetes/ovnkube-node-f4vvt"
Sep 08 15:32:43.489109 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.487789    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/195f1a74-19bc-49c9-aada-fa1c13488480-run-openvswitch\") pod \"ovnkube-node-f4vvt\" (UID: \"195f1a74-19bc-49c9-aada-fa1c13488480\") " pod="openshift-ovn-kubernetes/ovnkube-node-f4vvt"
Sep 08 15:32:43.490048 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.487804    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/195f1a74-19bc-49c9-aada-fa1c13488480-systemd-units\") pod \"ovnkube-node-f4vvt\" (UID: \"195f1a74-19bc-49c9-aada-fa1c13488480\") " pod="openshift-ovn-kubernetes/ovnkube-node-f4vvt"
Sep 08 15:32:43.490048 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.487813    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/84663ea3-1feb-4c87-8609-c580fa60416e-dbus\") pod \"global-pull-secret-syncer-fw5jh\" (UID: \"84663ea3-1feb-4c87-8609-c580fa60416e\") " pod="kube-system/global-pull-secret-syncer-fw5jh"
Sep 08 15:32:43.490048 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.487839    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c9873505-399e-48b7-b74c-685ae3d9f7b5-registration-dir\") pod \"aws-ebs-csi-driver-node-kfw4l\" (UID: \"c9873505-399e-48b7-b74c-685ae3d9f7b5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kfw4l"
Sep 08 15:32:43.490048 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.487850    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/195f1a74-19bc-49c9-aada-fa1c13488480-run-openvswitch\") pod \"ovnkube-node-f4vvt\" (UID: \"195f1a74-19bc-49c9-aada-fa1c13488480\") " pod="openshift-ovn-kubernetes/ovnkube-node-f4vvt"
Sep 08 15:32:43.490048 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.487862    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bblp\" (UniqueName: \"kubernetes.io/projected/65a8a5a6-5e05-4ee5-99a6-c026327dfa43-kube-api-access-7bblp\") pod \"iptables-alerter-p4rw9\" (UID: \"65a8a5a6-5e05-4ee5-99a6-c026327dfa43\") " pod="openshift-network-operator/iptables-alerter-p4rw9"
Sep 08 15:32:43.490048 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.487908    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/95c9958b-a005-439c-b856-0cbb99f08700-multus-socket-dir-parent\") pod \"multus-95s5s\" (UID: \"95c9958b-a005-439c-b856-0cbb99f08700\") " pod="openshift-multus/multus-95s5s"
Sep 08 15:32:43.490048 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.487936    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/95c9958b-a005-439c-b856-0cbb99f08700-hostroot\") pod \"multus-95s5s\" (UID: \"95c9958b-a005-439c-b856-0cbb99f08700\") " pod="openshift-multus/multus-95s5s"
Sep 08 15:32:43.490048 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.487946    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/95c9958b-a005-439c-b856-0cbb99f08700-multus-daemon-config\") pod \"multus-95s5s\" (UID: \"95c9958b-a005-439c-b856-0cbb99f08700\") " pod="openshift-multus/multus-95s5s"
Sep 08 15:32:43.490048 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.487961    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/195f1a74-19bc-49c9-aada-fa1c13488480-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-f4vvt\" (UID: \"195f1a74-19bc-49c9-aada-fa1c13488480\") " pod="openshift-ovn-kubernetes/ovnkube-node-f4vvt"
Sep 08 15:32:43.490048 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.487989    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/37ec7079-542c-4b91-8b2b-83e8e89fcbb7-var-lib-kubelet\") pod \"tuned-klnj5\" (UID: \"37ec7079-542c-4b91-8b2b-83e8e89fcbb7\") " pod="openshift-cluster-node-tuning-operator/tuned-klnj5"
Sep 08 15:32:43.490048 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.488020    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/37ec7079-542c-4b91-8b2b-83e8e89fcbb7-host\") pod \"tuned-klnj5\" (UID: \"37ec7079-542c-4b91-8b2b-83e8e89fcbb7\") " pod="openshift-cluster-node-tuning-operator/tuned-klnj5"
Sep 08 15:32:43.490048 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.488048    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/195f1a74-19bc-49c9-aada-fa1c13488480-var-lib-openvswitch\") pod \"ovnkube-node-f4vvt\" (UID: \"195f1a74-19bc-49c9-aada-fa1c13488480\") " pod="openshift-ovn-kubernetes/ovnkube-node-f4vvt"
Sep 08 15:32:43.490048 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.488072    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/195f1a74-19bc-49c9-aada-fa1c13488480-ovnkube-script-lib\") pod \"ovnkube-node-f4vvt\" (UID: \"195f1a74-19bc-49c9-aada-fa1c13488480\") " pod="openshift-ovn-kubernetes/ovnkube-node-f4vvt"
Sep 08 15:32:43.490048 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.488095    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/37ec7079-542c-4b91-8b2b-83e8e89fcbb7-etc-kubernetes\") pod \"tuned-klnj5\" (UID: \"37ec7079-542c-4b91-8b2b-83e8e89fcbb7\") " pod="openshift-cluster-node-tuning-operator/tuned-klnj5"
Sep 08 15:32:43.490048 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.488105    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/84663ea3-1feb-4c87-8609-c580fa60416e-dbus\") pod \"global-pull-secret-syncer-fw5jh\" (UID: \"84663ea3-1feb-4c87-8609-c580fa60416e\") " pod="kube-system/global-pull-secret-syncer-fw5jh"
Sep 08 15:32:43.490048 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.488119    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/37ec7079-542c-4b91-8b2b-83e8e89fcbb7-etc-sysctl-d\") pod \"tuned-klnj5\" (UID: \"37ec7079-542c-4b91-8b2b-83e8e89fcbb7\") " pod="openshift-cluster-node-tuning-operator/tuned-klnj5"
Sep 08 15:32:43.490048 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.488143    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c9873505-399e-48b7-b74c-685ae3d9f7b5-registration-dir\") pod \"aws-ebs-csi-driver-node-kfw4l\" (UID: \"c9873505-399e-48b7-b74c-685ae3d9f7b5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kfw4l"
Sep 08 15:32:43.490883 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.488163    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/195f1a74-19bc-49c9-aada-fa1c13488480-var-lib-openvswitch\") pod \"ovnkube-node-f4vvt\" (UID: \"195f1a74-19bc-49c9-aada-fa1c13488480\") " pod="openshift-ovn-kubernetes/ovnkube-node-f4vvt"
Sep 08 15:32:43.490883 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.488167    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/37ec7079-542c-4b91-8b2b-83e8e89fcbb7-var-lib-kubelet\") pod \"tuned-klnj5\" (UID: \"37ec7079-542c-4b91-8b2b-83e8e89fcbb7\") " pod="openshift-cluster-node-tuning-operator/tuned-klnj5"
Sep 08 15:32:43.490883 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.488197    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/195f1a74-19bc-49c9-aada-fa1c13488480-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-f4vvt\" (UID: \"195f1a74-19bc-49c9-aada-fa1c13488480\") " pod="openshift-ovn-kubernetes/ovnkube-node-f4vvt"
Sep 08 15:32:43.490883 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.488217    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/37ec7079-542c-4b91-8b2b-83e8e89fcbb7-host\") pod \"tuned-klnj5\" (UID: \"37ec7079-542c-4b91-8b2b-83e8e89fcbb7\") " pod="openshift-cluster-node-tuning-operator/tuned-klnj5"
Sep 08 15:32:43.490883 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.488218    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/37ec7079-542c-4b91-8b2b-83e8e89fcbb7-etc-kubernetes\") pod \"tuned-klnj5\" (UID: \"37ec7079-542c-4b91-8b2b-83e8e89fcbb7\") " pod="openshift-cluster-node-tuning-operator/tuned-klnj5"
Sep 08 15:32:43.490883 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.488234    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/37ec7079-542c-4b91-8b2b-83e8e89fcbb7-etc-sysctl-conf\") pod \"tuned-klnj5\" (UID: \"37ec7079-542c-4b91-8b2b-83e8e89fcbb7\") " pod="openshift-cluster-node-tuning-operator/tuned-klnj5"
Sep 08 15:32:43.490883 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.488233    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/65a8a5a6-5e05-4ee5-99a6-c026327dfa43-iptables-alerter-script\") pod \"iptables-alerter-p4rw9\" (UID: \"65a8a5a6-5e05-4ee5-99a6-c026327dfa43\") " pod="openshift-network-operator/iptables-alerter-p4rw9"
Sep 08 15:32:43.490883 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.488245    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/95c9958b-a005-439c-b856-0cbb99f08700-multus-socket-dir-parent\") pod \"multus-95s5s\" (UID: \"95c9958b-a005-439c-b856-0cbb99f08700\") " pod="openshift-multus/multus-95s5s"
Sep 08 15:32:43.490883 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.488258    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tx6tv\" (UniqueName: \"kubernetes.io/projected/3236f6c9-60de-47f9-84a9-5980933c58cd-kube-api-access-tx6tv\") pod \"multus-additional-cni-plugins-rpthc\" (UID: \"3236f6c9-60de-47f9-84a9-5980933c58cd\") " pod="openshift-multus/multus-additional-cni-plugins-rpthc"
Sep 08 15:32:43.490883 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.488280    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/95c9958b-a005-439c-b856-0cbb99f08700-multus-cni-dir\") pod \"multus-95s5s\" (UID: \"95c9958b-a005-439c-b856-0cbb99f08700\") " pod="openshift-multus/multus-95s5s"
Sep 08 15:32:43.490883 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.488301    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/195f1a74-19bc-49c9-aada-fa1c13488480-etc-openvswitch\") pod \"ovnkube-node-f4vvt\" (UID: \"195f1a74-19bc-49c9-aada-fa1c13488480\") " pod="openshift-ovn-kubernetes/ovnkube-node-f4vvt"
Sep 08 15:32:43.490883 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.488313    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/37ec7079-542c-4b91-8b2b-83e8e89fcbb7-etc-sysctl-d\") pod \"tuned-klnj5\" (UID: \"37ec7079-542c-4b91-8b2b-83e8e89fcbb7\") " pod="openshift-cluster-node-tuning-operator/tuned-klnj5"
Sep 08 15:32:43.490883 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.488323    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/195f1a74-19bc-49c9-aada-fa1c13488480-host-cni-bin\") pod \"ovnkube-node-f4vvt\" (UID: \"195f1a74-19bc-49c9-aada-fa1c13488480\") " pod="openshift-ovn-kubernetes/ovnkube-node-f4vvt"
Sep 08 15:32:43.490883 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.488344    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/37ec7079-542c-4b91-8b2b-83e8e89fcbb7-etc-sysctl-conf\") pod \"tuned-klnj5\" (UID: \"37ec7079-542c-4b91-8b2b-83e8e89fcbb7\") " pod="openshift-cluster-node-tuning-operator/tuned-klnj5"
Sep 08 15:32:43.490883 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.488349    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqqvf\" (UniqueName: \"kubernetes.io/projected/195f1a74-19bc-49c9-aada-fa1c13488480-kube-api-access-lqqvf\") pod \"ovnkube-node-f4vvt\" (UID: \"195f1a74-19bc-49c9-aada-fa1c13488480\") " pod="openshift-ovn-kubernetes/ovnkube-node-f4vvt"
Sep 08 15:32:43.490883 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.488369    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/95c9958b-a005-439c-b856-0cbb99f08700-multus-cni-dir\") pod \"multus-95s5s\" (UID: \"95c9958b-a005-439c-b856-0cbb99f08700\") " pod="openshift-multus/multus-95s5s"
Sep 08 15:32:43.490883 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.488387    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c9873505-399e-48b7-b74c-685ae3d9f7b5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kfw4l\" (UID: \"c9873505-399e-48b7-b74c-685ae3d9f7b5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kfw4l"
Sep 08 15:32:43.491720 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.488403    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/195f1a74-19bc-49c9-aada-fa1c13488480-host-cni-bin\") pod \"ovnkube-node-f4vvt\" (UID: \"195f1a74-19bc-49c9-aada-fa1c13488480\") " pod="openshift-ovn-kubernetes/ovnkube-node-f4vvt"
Sep 08 15:32:43.491720 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.488410    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/195f1a74-19bc-49c9-aada-fa1c13488480-etc-openvswitch\") pod \"ovnkube-node-f4vvt\" (UID: \"195f1a74-19bc-49c9-aada-fa1c13488480\") " pod="openshift-ovn-kubernetes/ovnkube-node-f4vvt"
Sep 08 15:32:43.491720 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.488417    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/84663ea3-1feb-4c87-8609-c580fa60416e-original-pull-secret\") pod \"global-pull-secret-syncer-fw5jh\" (UID: \"84663ea3-1feb-4c87-8609-c580fa60416e\") " pod="kube-system/global-pull-secret-syncer-fw5jh"
Sep 08 15:32:43.491720 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.488445    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/95c9958b-a005-439c-b856-0cbb99f08700-host-var-lib-kubelet\") pod \"multus-95s5s\" (UID: \"95c9958b-a005-439c-b856-0cbb99f08700\") " pod="openshift-multus/multus-95s5s"
Sep 08 15:32:43.491720 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.488471    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dggm\" (UniqueName: \"kubernetes.io/projected/95c9958b-a005-439c-b856-0cbb99f08700-kube-api-access-4dggm\") pod \"multus-95s5s\" (UID: \"95c9958b-a005-439c-b856-0cbb99f08700\") " pod="openshift-multus/multus-95s5s"
Sep 08 15:32:43.491720 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.488494    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/195f1a74-19bc-49c9-aada-fa1c13488480-run-ovn\") pod \"ovnkube-node-f4vvt\" (UID: \"195f1a74-19bc-49c9-aada-fa1c13488480\") " pod="openshift-ovn-kubernetes/ovnkube-node-f4vvt"
Sep 08 15:32:43.491720 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.488518    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/195f1a74-19bc-49c9-aada-fa1c13488480-host-cni-netd\") pod \"ovnkube-node-f4vvt\" (UID: \"195f1a74-19bc-49c9-aada-fa1c13488480\") " pod="openshift-ovn-kubernetes/ovnkube-node-f4vvt"
Sep 08 15:32:43.491720 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.488281    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/95c9958b-a005-439c-b856-0cbb99f08700-hostroot\") pod \"multus-95s5s\" (UID: \"95c9958b-a005-439c-b856-0cbb99f08700\") " pod="openshift-multus/multus-95s5s"
Sep 08 15:32:43.491720 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.488543    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/37ec7079-542c-4b91-8b2b-83e8e89fcbb7-etc-modprobe-d\") pod \"tuned-klnj5\" (UID: \"37ec7079-542c-4b91-8b2b-83e8e89fcbb7\") " pod="openshift-cluster-node-tuning-operator/tuned-klnj5"
Sep 08 15:32:43.491720 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.488580    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/95c9958b-a005-439c-b856-0cbb99f08700-host-var-lib-kubelet\") pod \"multus-95s5s\" (UID: \"95c9958b-a005-439c-b856-0cbb99f08700\") " pod="openshift-multus/multus-95s5s"
Sep 08 15:32:43.491720 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.488606    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5f9598ad-fb27-4dda-ab17-98d54cbd30aa-host\") pod \"node-ca-6q7cq\" (UID: \"5f9598ad-fb27-4dda-ab17-98d54cbd30aa\") " pod="openshift-image-registry/node-ca-6q7cq"
Sep 08 15:32:43.491720 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.488624    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/37ec7079-542c-4b91-8b2b-83e8e89fcbb7-etc-modprobe-d\") pod \"tuned-klnj5\" (UID: \"37ec7079-542c-4b91-8b2b-83e8e89fcbb7\") " pod="openshift-cluster-node-tuning-operator/tuned-klnj5"
Sep 08 15:32:43.491720 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.488631    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5f9598ad-fb27-4dda-ab17-98d54cbd30aa-serviceca\") pod \"node-ca-6q7cq\" (UID: \"5f9598ad-fb27-4dda-ab17-98d54cbd30aa\") " pod="openshift-image-registry/node-ca-6q7cq"
Sep 08 15:32:43.491720 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.488643    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c9873505-399e-48b7-b74c-685ae3d9f7b5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kfw4l\" (UID: \"c9873505-399e-48b7-b74c-685ae3d9f7b5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kfw4l"
Sep 08 15:32:43.491720 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.488656    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/28cd0557-2fec-4948-af98-6c67fb1e568d-konnectivity-ca\") pod \"konnectivity-agent-st9hm\" (UID: \"28cd0557-2fec-4948-af98-6c67fb1e568d\") " pod="kube-system/konnectivity-agent-st9hm"
Sep 08 15:32:43.491720 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.488672    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/195f1a74-19bc-49c9-aada-fa1c13488480-run-ovn\") pod \"ovnkube-node-f4vvt\" (UID: \"195f1a74-19bc-49c9-aada-fa1c13488480\") " pod="openshift-ovn-kubernetes/ovnkube-node-f4vvt"
Sep 08 15:32:43.491720 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.488683    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/95c9958b-a005-439c-b856-0cbb99f08700-cnibin\") pod \"multus-95s5s\" (UID: \"95c9958b-a005-439c-b856-0cbb99f08700\") " pod="openshift-multus/multus-95s5s"
Sep 08 15:32:43.491720 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.488709    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/95c9958b-a005-439c-b856-0cbb99f08700-cni-binary-copy\") pod \"multus-95s5s\" (UID: \"95c9958b-a005-439c-b856-0cbb99f08700\") " pod="openshift-multus/multus-95s5s"
Sep 08 15:32:43.492312 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.488711    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/195f1a74-19bc-49c9-aada-fa1c13488480-host-cni-netd\") pod \"ovnkube-node-f4vvt\" (UID: \"195f1a74-19bc-49c9-aada-fa1c13488480\") " pod="openshift-ovn-kubernetes/ovnkube-node-f4vvt"
Sep 08 15:32:43.492312 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.488733    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/95c9958b-a005-439c-b856-0cbb99f08700-etc-kubernetes\") pod \"multus-95s5s\" (UID: \"95c9958b-a005-439c-b856-0cbb99f08700\") " pod="openshift-multus/multus-95s5s"
Sep 08 15:32:43.492312 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:32:43.488739    2549 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 15:32:43.492312 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.488757    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/195f1a74-19bc-49c9-aada-fa1c13488480-ovnkube-config\") pod \"ovnkube-node-f4vvt\" (UID: \"195f1a74-19bc-49c9-aada-fa1c13488480\") " pod="openshift-ovn-kubernetes/ovnkube-node-f4vvt"
Sep 08 15:32:43.492312 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.488762    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/195f1a74-19bc-49c9-aada-fa1c13488480-ovnkube-script-lib\") pod \"ovnkube-node-f4vvt\" (UID: \"195f1a74-19bc-49c9-aada-fa1c13488480\") " pod="openshift-ovn-kubernetes/ovnkube-node-f4vvt"
Sep 08 15:32:43.492312 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:32:43.488779    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84663ea3-1feb-4c87-8609-c580fa60416e-original-pull-secret podName:84663ea3-1feb-4c87-8609-c580fa60416e nodeName:}" failed. No retries permitted until 2026-09-08 15:32:43.988765769 +0000 UTC m=+2.078216838 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/84663ea3-1feb-4c87-8609-c580fa60416e-original-pull-secret") pod "global-pull-secret-syncer-fw5jh" (UID: "84663ea3-1feb-4c87-8609-c580fa60416e") : object "kube-system"/"original-pull-secret" not registered
Sep 08 15:32:43.492312 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.488804    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/195f1a74-19bc-49c9-aada-fa1c13488480-env-overrides\") pod \"ovnkube-node-f4vvt\" (UID: \"195f1a74-19bc-49c9-aada-fa1c13488480\") " pod="openshift-ovn-kubernetes/ovnkube-node-f4vvt"
Sep 08 15:32:43.492312 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.488862    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/195f1a74-19bc-49c9-aada-fa1c13488480-host-slash\") pod \"ovnkube-node-f4vvt\" (UID: \"195f1a74-19bc-49c9-aada-fa1c13488480\") " pod="openshift-ovn-kubernetes/ovnkube-node-f4vvt"
Sep 08 15:32:43.492312 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.488887    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/195f1a74-19bc-49c9-aada-fa1c13488480-run-systemd\") pod \"ovnkube-node-f4vvt\" (UID: \"195f1a74-19bc-49c9-aada-fa1c13488480\") " pod="openshift-ovn-kubernetes/ovnkube-node-f4vvt"
Sep 08 15:32:43.492312 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.488908    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/37ec7079-542c-4b91-8b2b-83e8e89fcbb7-lib-modules\") pod \"tuned-klnj5\" (UID: \"37ec7079-542c-4b91-8b2b-83e8e89fcbb7\") " pod="openshift-cluster-node-tuning-operator/tuned-klnj5"
Sep 08 15:32:43.492312 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.488949    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/37ec7079-542c-4b91-8b2b-83e8e89fcbb7-tmp\") pod \"tuned-klnj5\" (UID: \"37ec7079-542c-4b91-8b2b-83e8e89fcbb7\") " pod="openshift-cluster-node-tuning-operator/tuned-klnj5"
Sep 08 15:32:43.492312 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.489145    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5f9598ad-fb27-4dda-ab17-98d54cbd30aa-serviceca\") pod \"node-ca-6q7cq\" (UID: \"5f9598ad-fb27-4dda-ab17-98d54cbd30aa\") " pod="openshift-image-registry/node-ca-6q7cq"
Sep 08 15:32:43.492312 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.489166    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3236f6c9-60de-47f9-84a9-5980933c58cd-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rpthc\" (UID: \"3236f6c9-60de-47f9-84a9-5980933c58cd\") " pod="openshift-multus/multus-additional-cni-plugins-rpthc"
Sep 08 15:32:43.492312 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.489205    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3236f6c9-60de-47f9-84a9-5980933c58cd-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rpthc\" (UID: \"3236f6c9-60de-47f9-84a9-5980933c58cd\") " pod="openshift-multus/multus-additional-cni-plugins-rpthc"
Sep 08 15:32:43.492312 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.489215    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/95c9958b-a005-439c-b856-0cbb99f08700-cnibin\") pod \"multus-95s5s\" (UID: \"95c9958b-a005-439c-b856-0cbb99f08700\") " pod="openshift-multus/multus-95s5s"
Sep 08 15:32:43.492312 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.489235    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a955f56c-5ac8-422a-8ddc-ce17a8361eb6-hosts-file\") pod \"node-resolver-vp9nk\" (UID: \"a955f56c-5ac8-422a-8ddc-ce17a8361eb6\") " pod="openshift-dns/node-resolver-vp9nk"
Sep 08 15:32:43.492312 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.489264    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjcrr\" (UniqueName: \"kubernetes.io/projected/7d4d759e-9d3c-497a-b546-462ae2b48785-kube-api-access-vjcrr\") pod \"network-metrics-daemon-h4cml\" (UID: \"7d4d759e-9d3c-497a-b546-462ae2b48785\") " pod="openshift-multus/network-metrics-daemon-h4cml"
Sep 08 15:32:43.492813 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.489292    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3236f6c9-60de-47f9-84a9-5980933c58cd-cni-binary-copy\") pod \"multus-additional-cni-plugins-rpthc\" (UID: \"3236f6c9-60de-47f9-84a9-5980933c58cd\") " pod="openshift-multus/multus-additional-cni-plugins-rpthc"
Sep 08 15:32:43.492813 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.489302    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/95c9958b-a005-439c-b856-0cbb99f08700-etc-kubernetes\") pod \"multus-95s5s\" (UID: \"95c9958b-a005-439c-b856-0cbb99f08700\") " pod="openshift-multus/multus-95s5s"
Sep 08 15:32:43.492813 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.489369    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a955f56c-5ac8-422a-8ddc-ce17a8361eb6-hosts-file\") pod \"node-resolver-vp9nk\" (UID: \"a955f56c-5ac8-422a-8ddc-ce17a8361eb6\") " pod="openshift-dns/node-resolver-vp9nk"
Sep 08 15:32:43.492813 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.489446    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/195f1a74-19bc-49c9-aada-fa1c13488480-host-slash\") pod \"ovnkube-node-f4vvt\" (UID: \"195f1a74-19bc-49c9-aada-fa1c13488480\") " pod="openshift-ovn-kubernetes/ovnkube-node-f4vvt"
Sep 08 15:32:43.492813 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.489265    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/195f1a74-19bc-49c9-aada-fa1c13488480-ovnkube-config\") pod \"ovnkube-node-f4vvt\" (UID: \"195f1a74-19bc-49c9-aada-fa1c13488480\") " pod="openshift-ovn-kubernetes/ovnkube-node-f4vvt"
Sep 08 15:32:43.492813 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.489690    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/95c9958b-a005-439c-b856-0cbb99f08700-cni-binary-copy\") pod \"multus-95s5s\" (UID: \"95c9958b-a005-439c-b856-0cbb99f08700\") " pod="openshift-multus/multus-95s5s"
Sep 08 15:32:43.492813 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.489763    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/195f1a74-19bc-49c9-aada-fa1c13488480-run-systemd\") pod \"ovnkube-node-f4vvt\" (UID: \"195f1a74-19bc-49c9-aada-fa1c13488480\") " pod="openshift-ovn-kubernetes/ovnkube-node-f4vvt"
Sep 08 15:32:43.492813 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.489865    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/37ec7079-542c-4b91-8b2b-83e8e89fcbb7-lib-modules\") pod \"tuned-klnj5\" (UID: \"37ec7079-542c-4b91-8b2b-83e8e89fcbb7\") " pod="openshift-cluster-node-tuning-operator/tuned-klnj5"
Sep 08 15:32:43.492813 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.488686    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5f9598ad-fb27-4dda-ab17-98d54cbd30aa-host\") pod \"node-ca-6q7cq\" (UID: \"5f9598ad-fb27-4dda-ab17-98d54cbd30aa\") " pod="openshift-image-registry/node-ca-6q7cq"
Sep 08 15:32:43.492813 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.489915    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3236f6c9-60de-47f9-84a9-5980933c58cd-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rpthc\" (UID: \"3236f6c9-60de-47f9-84a9-5980933c58cd\") " pod="openshift-multus/multus-additional-cni-plugins-rpthc"
Sep 08 15:32:43.492813 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.489962    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/195f1a74-19bc-49c9-aada-fa1c13488480-env-overrides\") pod \"ovnkube-node-f4vvt\" (UID: \"195f1a74-19bc-49c9-aada-fa1c13488480\") " pod="openshift-ovn-kubernetes/ovnkube-node-f4vvt"
Sep 08 15:32:43.492813 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.490055    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3236f6c9-60de-47f9-84a9-5980933c58cd-cni-binary-copy\") pod \"multus-additional-cni-plugins-rpthc\" (UID: \"3236f6c9-60de-47f9-84a9-5980933c58cd\") " pod="openshift-multus/multus-additional-cni-plugins-rpthc"
Sep 08 15:32:43.492813 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.490292    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/28cd0557-2fec-4948-af98-6c67fb1e568d-konnectivity-ca\") pod \"konnectivity-agent-st9hm\" (UID: \"28cd0557-2fec-4948-af98-6c67fb1e568d\") " pod="kube-system/konnectivity-agent-st9hm"
Sep 08 15:32:43.492813 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.490414    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3236f6c9-60de-47f9-84a9-5980933c58cd-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rpthc\" (UID: \"3236f6c9-60de-47f9-84a9-5980933c58cd\") " pod="openshift-multus/multus-additional-cni-plugins-rpthc"
Sep 08 15:32:43.492813 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.491751    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/195f1a74-19bc-49c9-aada-fa1c13488480-ovn-node-metrics-cert\") pod \"ovnkube-node-f4vvt\" (UID: \"195f1a74-19bc-49c9-aada-fa1c13488480\") " pod="openshift-ovn-kubernetes/ovnkube-node-f4vvt"
Sep 08 15:32:43.492813 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.491799    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/37ec7079-542c-4b91-8b2b-83e8e89fcbb7-tmp\") pod \"tuned-klnj5\" (UID: \"37ec7079-542c-4b91-8b2b-83e8e89fcbb7\") " pod="openshift-cluster-node-tuning-operator/tuned-klnj5"
Sep 08 15:32:43.492813 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.491839    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/37ec7079-542c-4b91-8b2b-83e8e89fcbb7-etc-tuned\") pod \"tuned-klnj5\" (UID: \"37ec7079-542c-4b91-8b2b-83e8e89fcbb7\") " pod="openshift-cluster-node-tuning-operator/tuned-klnj5"
Sep 08 15:32:43.492813 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.492023    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/28cd0557-2fec-4948-af98-6c67fb1e568d-agent-certs\") pod \"konnectivity-agent-st9hm\" (UID: \"28cd0557-2fec-4948-af98-6c67fb1e568d\") " pod="kube-system/konnectivity-agent-st9hm"
Sep 08 15:32:43.497218 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:32:43.497202    2549 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 15:32:43.497267 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:32:43.497222    2549 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 15:32:43.497267 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:32:43.497232    2549 projected.go:196] Error preparing data for projected volume kube-api-access-79p6n for pod openshift-network-diagnostics/network-check-target-5vt6c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 15:32:43.497380 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:32:43.497279    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3d62444d-76ba-4184-869c-496ba4da8d04-kube-api-access-79p6n podName:3d62444d-76ba-4184-869c-496ba4da8d04 nodeName:}" failed. No retries permitted until 2026-09-08 15:32:43.997261095 +0000 UTC m=+2.086712169 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-79p6n" (UniqueName: "kubernetes.io/projected/3d62444d-76ba-4184-869c-496ba4da8d04-kube-api-access-79p6n") pod "network-check-target-5vt6c" (UID: "3d62444d-76ba-4184-869c-496ba4da8d04") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 15:32:43.499763 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.499650    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fnhz\" (UniqueName: \"kubernetes.io/projected/5f9598ad-fb27-4dda-ab17-98d54cbd30aa-kube-api-access-2fnhz\") pod \"node-ca-6q7cq\" (UID: \"5f9598ad-fb27-4dda-ab17-98d54cbd30aa\") " pod="openshift-image-registry/node-ca-6q7cq"
Sep 08 15:32:43.499763 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.499702    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9498\" (UniqueName: \"kubernetes.io/projected/37ec7079-542c-4b91-8b2b-83e8e89fcbb7-kube-api-access-z9498\") pod \"tuned-klnj5\" (UID: \"37ec7079-542c-4b91-8b2b-83e8e89fcbb7\") " pod="openshift-cluster-node-tuning-operator/tuned-klnj5"
Sep 08 15:32:43.499981 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.499951    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nkb9\" (UniqueName: \"kubernetes.io/projected/a955f56c-5ac8-422a-8ddc-ce17a8361eb6-kube-api-access-7nkb9\") pod \"node-resolver-vp9nk\" (UID: \"a955f56c-5ac8-422a-8ddc-ce17a8361eb6\") " pod="openshift-dns/node-resolver-vp9nk"
Sep 08 15:32:43.500452 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.500423    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wc458\" (UniqueName: \"kubernetes.io/projected/c9873505-399e-48b7-b74c-685ae3d9f7b5-kube-api-access-wc458\") pod \"aws-ebs-csi-driver-node-kfw4l\" (UID: \"c9873505-399e-48b7-b74c-685ae3d9f7b5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kfw4l"
Sep 08 15:32:43.501179 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.501151    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bblp\" (UniqueName: \"kubernetes.io/projected/65a8a5a6-5e05-4ee5-99a6-c026327dfa43-kube-api-access-7bblp\") pod \"iptables-alerter-p4rw9\" (UID: \"65a8a5a6-5e05-4ee5-99a6-c026327dfa43\") " pod="openshift-network-operator/iptables-alerter-p4rw9"
Sep 08 15:32:43.501671 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.501651    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqqvf\" (UniqueName: \"kubernetes.io/projected/195f1a74-19bc-49c9-aada-fa1c13488480-kube-api-access-lqqvf\") pod \"ovnkube-node-f4vvt\" (UID: \"195f1a74-19bc-49c9-aada-fa1c13488480\") " pod="openshift-ovn-kubernetes/ovnkube-node-f4vvt"
Sep 08 15:32:43.501746 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.501681    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjcrr\" (UniqueName: \"kubernetes.io/projected/7d4d759e-9d3c-497a-b546-462ae2b48785-kube-api-access-vjcrr\") pod \"network-metrics-daemon-h4cml\" (UID: \"7d4d759e-9d3c-497a-b546-462ae2b48785\") " pod="openshift-multus/network-metrics-daemon-h4cml"
Sep 08 15:32:43.501855 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.501837    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dggm\" (UniqueName: \"kubernetes.io/projected/95c9958b-a005-439c-b856-0cbb99f08700-kube-api-access-4dggm\") pod \"multus-95s5s\" (UID: \"95c9958b-a005-439c-b856-0cbb99f08700\") " pod="openshift-multus/multus-95s5s"
Sep 08 15:32:43.502083 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.502065    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tx6tv\" (UniqueName: \"kubernetes.io/projected/3236f6c9-60de-47f9-84a9-5980933c58cd-kube-api-access-tx6tv\") pod \"multus-additional-cni-plugins-rpthc\" (UID: \"3236f6c9-60de-47f9-84a9-5980933c58cd\") " pod="openshift-multus/multus-additional-cni-plugins-rpthc"
Sep 08 15:32:43.599011 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.598956    2549 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 08 15:32:43.611758 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.611729    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-st9hm"
Sep 08 15:32:43.622244 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:43.622223    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28cd0557_2fec_4948_af98_6c67fb1e568d.slice/crio-bec06a783c1565c94151d38c052eda971c12cdce728981b9acf2dc497dcbd2a6 WatchSource:0}: Error finding container bec06a783c1565c94151d38c052eda971c12cdce728981b9acf2dc497dcbd2a6: Status 404 returned error can't find the container with id bec06a783c1565c94151d38c052eda971c12cdce728981b9acf2dc497dcbd2a6
Sep 08 15:32:43.628399 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.628382    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-vp9nk"
Sep 08 15:32:43.636546 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:43.636523    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda955f56c_5ac8_422a_8ddc_ce17a8361eb6.slice/crio-ada114877c4c89db371b5034cc63393f318c4a3a41a976bdb348fd4c8b872d41 WatchSource:0}: Error finding container ada114877c4c89db371b5034cc63393f318c4a3a41a976bdb348fd4c8b872d41: Status 404 returned error can't find the container with id ada114877c4c89db371b5034cc63393f318c4a3a41a976bdb348fd4c8b872d41
Sep 08 15:32:43.648199 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.648180    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-95s5s"
Sep 08 15:32:43.655180 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.655163    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-p4rw9"
Sep 08 15:32:43.658025 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:43.658001    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95c9958b_a005_439c_b856_0cbb99f08700.slice/crio-bc4c19f9f59c85d1b879d5e3f39d584afa3254f40ddd1a9d3b05d24fe59f0b41 WatchSource:0}: Error finding container bc4c19f9f59c85d1b879d5e3f39d584afa3254f40ddd1a9d3b05d24fe59f0b41: Status 404 returned error can't find the container with id bc4c19f9f59c85d1b879d5e3f39d584afa3254f40ddd1a9d3b05d24fe59f0b41
Sep 08 15:32:43.664551 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:43.664529    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65a8a5a6_5e05_4ee5_99a6_c026327dfa43.slice/crio-f5beb9b809a3c2d0abcb2a3f66a7881f7cb2f0107abb4b5eeb407fdddc2ff35f WatchSource:0}: Error finding container f5beb9b809a3c2d0abcb2a3f66a7881f7cb2f0107abb4b5eeb407fdddc2ff35f: Status 404 returned error can't find the container with id f5beb9b809a3c2d0abcb2a3f66a7881f7cb2f0107abb4b5eeb407fdddc2ff35f
Sep 08 15:32:43.665946 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.665920    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-f4vvt"
Sep 08 15:32:43.674170 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:43.674147    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod195f1a74_19bc_49c9_aada_fa1c13488480.slice/crio-48eeca26f719b79306bc2a9a473c2ff4da73255aefaed16e3ba672dd86005a5f WatchSource:0}: Error finding container 48eeca26f719b79306bc2a9a473c2ff4da73255aefaed16e3ba672dd86005a5f: Status 404 returned error can't find the container with id 48eeca26f719b79306bc2a9a473c2ff4da73255aefaed16e3ba672dd86005a5f
Sep 08 15:32:43.679645 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.679627    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kfw4l"
Sep 08 15:32:43.687337 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:43.687315    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9873505_399e_48b7_b74c_685ae3d9f7b5.slice/crio-87995fb72b7a24fade0abcdff47ca2f47d26e4652e756328f4feeaf78367a7a8 WatchSource:0}: Error finding container 87995fb72b7a24fade0abcdff47ca2f47d26e4652e756328f4feeaf78367a7a8: Status 404 returned error can't find the container with id 87995fb72b7a24fade0abcdff47ca2f47d26e4652e756328f4feeaf78367a7a8
Sep 08 15:32:43.692685 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.692667    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-klnj5"
Sep 08 15:32:43.700833 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:43.700816    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37ec7079_542c_4b91_8b2b_83e8e89fcbb7.slice/crio-149500553ce727a03dc36c0e4bffe12a1693e121f76cd46ff2b631ae62640d29 WatchSource:0}: Error finding container 149500553ce727a03dc36c0e4bffe12a1693e121f76cd46ff2b631ae62640d29: Status 404 returned error can't find the container with id 149500553ce727a03dc36c0e4bffe12a1693e121f76cd46ff2b631ae62640d29
Sep 08 15:32:43.710747 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.710722    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-6q7cq"
Sep 08 15:32:43.715705 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.715686    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-rpthc"
Sep 08 15:32:43.718391 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:43.718368    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f9598ad_fb27_4dda_ab17_98d54cbd30aa.slice/crio-ff12150cd374a9d620ca73855e242217a46b9581720bfae4f7b07f703e4b2efa WatchSource:0}: Error finding container ff12150cd374a9d620ca73855e242217a46b9581720bfae4f7b07f703e4b2efa: Status 404 returned error can't find the container with id ff12150cd374a9d620ca73855e242217a46b9581720bfae4f7b07f703e4b2efa
Sep 08 15:32:43.724055 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:32:43.724035    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3236f6c9_60de_47f9_84a9_5980933c58cd.slice/crio-90d44cc426ce722e2621df7b126ebf02470a4981b8d0871568fa3cff64592ed7 WatchSource:0}: Error finding container 90d44cc426ce722e2621df7b126ebf02470a4981b8d0871568fa3cff64592ed7: Status 404 returned error can't find the container with id 90d44cc426ce722e2621df7b126ebf02470a4981b8d0871568fa3cff64592ed7
Sep 08 15:32:43.993499 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.993413    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/84663ea3-1feb-4c87-8609-c580fa60416e-original-pull-secret\") pod \"global-pull-secret-syncer-fw5jh\" (UID: \"84663ea3-1feb-4c87-8609-c580fa60416e\") " pod="kube-system/global-pull-secret-syncer-fw5jh"
Sep 08 15:32:43.993499 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:43.993464    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d4d759e-9d3c-497a-b546-462ae2b48785-metrics-certs\") pod \"network-metrics-daemon-h4cml\" (UID: \"7d4d759e-9d3c-497a-b546-462ae2b48785\") " pod="openshift-multus/network-metrics-daemon-h4cml"
Sep 08 15:32:43.993664 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:32:43.993540    2549 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 15:32:43.993664 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:32:43.993599    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84663ea3-1feb-4c87-8609-c580fa60416e-original-pull-secret podName:84663ea3-1feb-4c87-8609-c580fa60416e nodeName:}" failed. No retries permitted until 2026-09-08 15:32:44.993584508 +0000 UTC m=+3.083035582 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/84663ea3-1feb-4c87-8609-c580fa60416e-original-pull-secret") pod "global-pull-secret-syncer-fw5jh" (UID: "84663ea3-1feb-4c87-8609-c580fa60416e") : object "kube-system"/"original-pull-secret" not registered
Sep 08 15:32:43.993664 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:32:43.993545    2549 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 15:32:43.993775 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:32:43.993678    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d4d759e-9d3c-497a-b546-462ae2b48785-metrics-certs podName:7d4d759e-9d3c-497a-b546-462ae2b48785 nodeName:}" failed. No retries permitted until 2026-09-08 15:32:44.993666129 +0000 UTC m=+3.083117203 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7d4d759e-9d3c-497a-b546-462ae2b48785-metrics-certs") pod "network-metrics-daemon-h4cml" (UID: "7d4d759e-9d3c-497a-b546-462ae2b48785") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 15:32:44.094326 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:44.094219    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79p6n\" (UniqueName: \"kubernetes.io/projected/3d62444d-76ba-4184-869c-496ba4da8d04-kube-api-access-79p6n\") pod \"network-check-target-5vt6c\" (UID: \"3d62444d-76ba-4184-869c-496ba4da8d04\") " pod="openshift-network-diagnostics/network-check-target-5vt6c"
Sep 08 15:32:44.094507 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:32:44.094376    2549 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 15:32:44.094507 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:32:44.094396    2549 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 15:32:44.094507 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:32:44.094408    2549 projected.go:196] Error preparing data for projected volume kube-api-access-79p6n for pod openshift-network-diagnostics/network-check-target-5vt6c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 15:32:44.094507 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:32:44.094471    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3d62444d-76ba-4184-869c-496ba4da8d04-kube-api-access-79p6n podName:3d62444d-76ba-4184-869c-496ba4da8d04 nodeName:}" failed. No retries permitted until 2026-09-08 15:32:45.094452221 +0000 UTC m=+3.183903308 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-79p6n" (UniqueName: "kubernetes.io/projected/3d62444d-76ba-4184-869c-496ba4da8d04-kube-api-access-79p6n") pod "network-check-target-5vt6c" (UID: "3d62444d-76ba-4184-869c-496ba4da8d04") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 15:32:44.297014 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:44.296936    2549 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 08 15:32:44.324560 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:44.324519    2549 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-07 15:27:44 +0000 UTC" deadline="2028-05-31 04:37:26.907954758 +0000 UTC"
Sep 08 15:32:44.324560 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:44.324562    2549 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15133h4m42.583400842s"
Sep 08 15:32:44.402921 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:44.402847    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-st9hm" event={"ID":"28cd0557-2fec-4948-af98-6c67fb1e568d","Type":"ContainerStarted","Data":"bec06a783c1565c94151d38c052eda971c12cdce728981b9acf2dc497dcbd2a6"}
Sep 08 15:32:44.405575 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:44.405542    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-61.ec2.internal" event={"ID":"e8cf966721b2ec8fb0cd4ca3ebd9318e","Type":"ContainerStarted","Data":"c01f9bd6425649322405931ef99d72694b3dec0503ba6aeef60913cf327fc4f5"}
Sep 08 15:32:44.407436 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:44.407412    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-61.ec2.internal" event={"ID":"4a20335aaf3733c5ea5fa6f08be94600","Type":"ContainerStarted","Data":"5ed6e3aef0d28d77ab2548d29e4e31157dbf57ce2f168859aad44b977c8bfecb"}
Sep 08 15:32:44.410510 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:44.410484    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-6q7cq" event={"ID":"5f9598ad-fb27-4dda-ab17-98d54cbd30aa","Type":"ContainerStarted","Data":"ff12150cd374a9d620ca73855e242217a46b9581720bfae4f7b07f703e4b2efa"}
Sep 08 15:32:44.415380 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:44.413537    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-p4rw9" event={"ID":"65a8a5a6-5e05-4ee5-99a6-c026327dfa43","Type":"ContainerStarted","Data":"f5beb9b809a3c2d0abcb2a3f66a7881f7cb2f0107abb4b5eeb407fdddc2ff35f"}
Sep 08 15:32:44.415943 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:44.415674    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-95s5s" event={"ID":"95c9958b-a005-439c-b856-0cbb99f08700","Type":"ContainerStarted","Data":"bc4c19f9f59c85d1b879d5e3f39d584afa3254f40ddd1a9d3b05d24fe59f0b41"}
Sep 08 15:32:44.417798 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:44.417768    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rpthc" event={"ID":"3236f6c9-60de-47f9-84a9-5980933c58cd","Type":"ContainerStarted","Data":"90d44cc426ce722e2621df7b126ebf02470a4981b8d0871568fa3cff64592ed7"}
Sep 08 15:32:44.419822 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:44.419799    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-klnj5" event={"ID":"37ec7079-542c-4b91-8b2b-83e8e89fcbb7","Type":"ContainerStarted","Data":"149500553ce727a03dc36c0e4bffe12a1693e121f76cd46ff2b631ae62640d29"}
Sep 08 15:32:44.422445 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:44.422419    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kfw4l" event={"ID":"c9873505-399e-48b7-b74c-685ae3d9f7b5","Type":"ContainerStarted","Data":"87995fb72b7a24fade0abcdff47ca2f47d26e4652e756328f4feeaf78367a7a8"}
Sep 08 15:32:44.424704 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:44.424681    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-f4vvt" event={"ID":"195f1a74-19bc-49c9-aada-fa1c13488480","Type":"ContainerStarted","Data":"48eeca26f719b79306bc2a9a473c2ff4da73255aefaed16e3ba672dd86005a5f"}
Sep 08 15:32:44.430078 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:44.430049    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-vp9nk" event={"ID":"a955f56c-5ac8-422a-8ddc-ce17a8361eb6","Type":"ContainerStarted","Data":"ada114877c4c89db371b5034cc63393f318c4a3a41a976bdb348fd4c8b872d41"}
Sep 08 15:32:45.004086 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:45.004039    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d4d759e-9d3c-497a-b546-462ae2b48785-metrics-certs\") pod \"network-metrics-daemon-h4cml\" (UID: \"7d4d759e-9d3c-497a-b546-462ae2b48785\") " pod="openshift-multus/network-metrics-daemon-h4cml"
Sep 08 15:32:45.004267 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:45.004147    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/84663ea3-1feb-4c87-8609-c580fa60416e-original-pull-secret\") pod \"global-pull-secret-syncer-fw5jh\" (UID: \"84663ea3-1feb-4c87-8609-c580fa60416e\") " pod="kube-system/global-pull-secret-syncer-fw5jh"
Sep 08 15:32:45.004327 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:32:45.004311    2549 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 15:32:45.004481 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:32:45.004425    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84663ea3-1feb-4c87-8609-c580fa60416e-original-pull-secret podName:84663ea3-1feb-4c87-8609-c580fa60416e nodeName:}" failed. No retries permitted until 2026-09-08 15:32:47.00440171 +0000 UTC m=+5.093852795 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/84663ea3-1feb-4c87-8609-c580fa60416e-original-pull-secret") pod "global-pull-secret-syncer-fw5jh" (UID: "84663ea3-1feb-4c87-8609-c580fa60416e") : object "kube-system"/"original-pull-secret" not registered
Sep 08 15:32:45.004996 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:32:45.004974    2549 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 15:32:45.005099 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:32:45.005037    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d4d759e-9d3c-497a-b546-462ae2b48785-metrics-certs podName:7d4d759e-9d3c-497a-b546-462ae2b48785 nodeName:}" failed. No retries permitted until 2026-09-08 15:32:47.005018663 +0000 UTC m=+5.094469742 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7d4d759e-9d3c-497a-b546-462ae2b48785-metrics-certs") pod "network-metrics-daemon-h4cml" (UID: "7d4d759e-9d3c-497a-b546-462ae2b48785") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 15:32:45.104662 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:45.104619    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79p6n\" (UniqueName: \"kubernetes.io/projected/3d62444d-76ba-4184-869c-496ba4da8d04-kube-api-access-79p6n\") pod \"network-check-target-5vt6c\" (UID: \"3d62444d-76ba-4184-869c-496ba4da8d04\") " pod="openshift-network-diagnostics/network-check-target-5vt6c"
Sep 08 15:32:45.104825 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:32:45.104800    2549 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 15:32:45.104825 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:32:45.104819    2549 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 15:32:45.104984 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:32:45.104833    2549 projected.go:196] Error preparing data for projected volume kube-api-access-79p6n for pod openshift-network-diagnostics/network-check-target-5vt6c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 15:32:45.106561 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:32:45.106536    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3d62444d-76ba-4184-869c-496ba4da8d04-kube-api-access-79p6n podName:3d62444d-76ba-4184-869c-496ba4da8d04 nodeName:}" failed. No retries permitted until 2026-09-08 15:32:47.104877231 +0000 UTC m=+5.194328315 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-79p6n" (UniqueName: "kubernetes.io/projected/3d62444d-76ba-4184-869c-496ba4da8d04-kube-api-access-79p6n") pod "network-check-target-5vt6c" (UID: "3d62444d-76ba-4184-869c-496ba4da8d04") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 15:32:45.325398 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:45.325279    2549 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-07 15:27:44 +0000 UTC" deadline="2028-04-04 06:35:09.471730599 +0000 UTC"
Sep 08 15:32:45.325398 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:45.325346    2549 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13767h2m24.146388432s"
Sep 08 15:32:45.365349 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:45.365138    2549 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 08 15:32:45.395342 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:45.394452    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fw5jh"
Sep 08 15:32:45.395342 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:32:45.394582    2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fw5jh" podUID="84663ea3-1feb-4c87-8609-c580fa60416e"
Sep 08 15:32:45.395342 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:45.394962    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5vt6c"
Sep 08 15:32:45.395342 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:32:45.395075    2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5vt6c" podUID="3d62444d-76ba-4184-869c-496ba4da8d04"
Sep 08 15:32:45.395342 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:45.395152    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h4cml"
Sep 08 15:32:45.395342 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:32:45.395299    2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h4cml" podUID="7d4d759e-9d3c-497a-b546-462ae2b48785"
Sep 08 15:32:45.934863 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:45.934826    2549 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 08 15:32:47.020405 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:47.019982    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d4d759e-9d3c-497a-b546-462ae2b48785-metrics-certs\") pod \"network-metrics-daemon-h4cml\" (UID: \"7d4d759e-9d3c-497a-b546-462ae2b48785\") " pod="openshift-multus/network-metrics-daemon-h4cml"
Sep 08 15:32:47.020405 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:47.020063    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/84663ea3-1feb-4c87-8609-c580fa60416e-original-pull-secret\") pod \"global-pull-secret-syncer-fw5jh\" (UID: \"84663ea3-1feb-4c87-8609-c580fa60416e\") " pod="kube-system/global-pull-secret-syncer-fw5jh"
Sep 08 15:32:47.020405 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:32:47.020191    2549 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 15:32:47.020405 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:32:47.020253    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84663ea3-1feb-4c87-8609-c580fa60416e-original-pull-secret podName:84663ea3-1feb-4c87-8609-c580fa60416e nodeName:}" failed. No retries permitted until 2026-09-08 15:32:51.020233969 +0000 UTC m=+9.109685060 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/84663ea3-1feb-4c87-8609-c580fa60416e-original-pull-secret") pod "global-pull-secret-syncer-fw5jh" (UID: "84663ea3-1feb-4c87-8609-c580fa60416e") : object "kube-system"/"original-pull-secret" not registered
Sep 08 15:32:47.020405 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:32:47.020316    2549 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 15:32:47.020405 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:32:47.020358    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d4d759e-9d3c-497a-b546-462ae2b48785-metrics-certs podName:7d4d759e-9d3c-497a-b546-462ae2b48785 nodeName:}" failed. No retries permitted until 2026-09-08 15:32:51.02034402 +0000 UTC m=+9.109795104 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7d4d759e-9d3c-497a-b546-462ae2b48785-metrics-certs") pod "network-metrics-daemon-h4cml" (UID: "7d4d759e-9d3c-497a-b546-462ae2b48785") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 15:32:47.121506 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:47.121466    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79p6n\" (UniqueName: \"kubernetes.io/projected/3d62444d-76ba-4184-869c-496ba4da8d04-kube-api-access-79p6n\") pod \"network-check-target-5vt6c\" (UID: \"3d62444d-76ba-4184-869c-496ba4da8d04\") " pod="openshift-network-diagnostics/network-check-target-5vt6c"
Sep 08 15:32:47.121692 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:32:47.121653    2549 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 15:32:47.121692 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:32:47.121679    2549 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 15:32:47.121692 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:32:47.121692    2549 projected.go:196] Error preparing data for projected volume kube-api-access-79p6n for pod openshift-network-diagnostics/network-check-target-5vt6c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 15:32:47.121855 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:32:47.121760    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3d62444d-76ba-4184-869c-496ba4da8d04-kube-api-access-79p6n podName:3d62444d-76ba-4184-869c-496ba4da8d04 nodeName:}" failed. No retries permitted until 2026-09-08 15:32:51.12174064 +0000 UTC m=+9.211191728 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-79p6n" (UniqueName: "kubernetes.io/projected/3d62444d-76ba-4184-869c-496ba4da8d04-kube-api-access-79p6n") pod "network-check-target-5vt6c" (UID: "3d62444d-76ba-4184-869c-496ba4da8d04") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 15:32:47.395030 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:47.394942    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h4cml"
Sep 08 15:32:47.395030 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:47.394974    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5vt6c"
Sep 08 15:32:47.395266 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:47.394942    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fw5jh"
Sep 08 15:32:47.395266 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:32:47.395098    2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h4cml" podUID="7d4d759e-9d3c-497a-b546-462ae2b48785"
Sep 08 15:32:47.395266 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:32:47.395194    2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fw5jh" podUID="84663ea3-1feb-4c87-8609-c580fa60416e"
Sep 08 15:32:47.395438 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:32:47.395277    2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5vt6c" podUID="3d62444d-76ba-4184-869c-496ba4da8d04"
Sep 08 15:32:49.394688 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:49.394644    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h4cml"
Sep 08 15:32:49.394688 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:49.394687    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fw5jh"
Sep 08 15:32:49.395208 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:32:49.394809    2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h4cml" podUID="7d4d759e-9d3c-497a-b546-462ae2b48785"
Sep 08 15:32:49.395208 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:49.394833    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5vt6c"
Sep 08 15:32:49.395208 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:32:49.394960    2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5vt6c" podUID="3d62444d-76ba-4184-869c-496ba4da8d04"
Sep 08 15:32:49.395208 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:32:49.395071    2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fw5jh" podUID="84663ea3-1feb-4c87-8609-c580fa60416e"
Sep 08 15:32:51.054753 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:51.054662    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/84663ea3-1feb-4c87-8609-c580fa60416e-original-pull-secret\") pod \"global-pull-secret-syncer-fw5jh\" (UID: \"84663ea3-1feb-4c87-8609-c580fa60416e\") " pod="kube-system/global-pull-secret-syncer-fw5jh"
Sep 08 15:32:51.054753 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:51.054750    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d4d759e-9d3c-497a-b546-462ae2b48785-metrics-certs\") pod \"network-metrics-daemon-h4cml\" (UID: \"7d4d759e-9d3c-497a-b546-462ae2b48785\") " pod="openshift-multus/network-metrics-daemon-h4cml"
Sep 08 15:32:51.055341 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:32:51.054815    2549 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 15:32:51.055341 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:32:51.054871    2549 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 15:32:51.055341 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:32:51.054924    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84663ea3-1feb-4c87-8609-c580fa60416e-original-pull-secret podName:84663ea3-1feb-4c87-8609-c580fa60416e nodeName:}" failed. No retries permitted until 2026-09-08 15:32:59.054904266 +0000 UTC m=+17.144355351 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/84663ea3-1feb-4c87-8609-c580fa60416e-original-pull-secret") pod "global-pull-secret-syncer-fw5jh" (UID: "84663ea3-1feb-4c87-8609-c580fa60416e") : object "kube-system"/"original-pull-secret" not registered
Sep 08 15:32:51.055341 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:32:51.054973    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d4d759e-9d3c-497a-b546-462ae2b48785-metrics-certs podName:7d4d759e-9d3c-497a-b546-462ae2b48785 nodeName:}" failed. No retries permitted until 2026-09-08 15:32:59.054960763 +0000 UTC m=+17.144411833 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7d4d759e-9d3c-497a-b546-462ae2b48785-metrics-certs") pod "network-metrics-daemon-h4cml" (UID: "7d4d759e-9d3c-497a-b546-462ae2b48785") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 15:32:51.156337 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:51.155692    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79p6n\" (UniqueName: \"kubernetes.io/projected/3d62444d-76ba-4184-869c-496ba4da8d04-kube-api-access-79p6n\") pod \"network-check-target-5vt6c\" (UID: \"3d62444d-76ba-4184-869c-496ba4da8d04\") " pod="openshift-network-diagnostics/network-check-target-5vt6c"
Sep 08 15:32:51.156337 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:32:51.155868    2549 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 15:32:51.156337 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:32:51.155900    2549 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 15:32:51.156337 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:32:51.155915    2549 projected.go:196] Error preparing data for projected volume kube-api-access-79p6n for pod openshift-network-diagnostics/network-check-target-5vt6c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 15:32:51.156337 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:32:51.155974    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3d62444d-76ba-4184-869c-496ba4da8d04-kube-api-access-79p6n podName:3d62444d-76ba-4184-869c-496ba4da8d04 nodeName:}" failed. No retries permitted until 2026-09-08 15:32:59.155955324 +0000 UTC m=+17.245406410 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-79p6n" (UniqueName: "kubernetes.io/projected/3d62444d-76ba-4184-869c-496ba4da8d04-kube-api-access-79p6n") pod "network-check-target-5vt6c" (UID: "3d62444d-76ba-4184-869c-496ba4da8d04") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 15:32:51.395424 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:51.395019    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fw5jh"
Sep 08 15:32:51.395424 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:32:51.395145    2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fw5jh" podUID="84663ea3-1feb-4c87-8609-c580fa60416e"
Sep 08 15:32:51.395424 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:51.395190    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5vt6c"
Sep 08 15:32:51.395424 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:32:51.395305    2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5vt6c" podUID="3d62444d-76ba-4184-869c-496ba4da8d04"
Sep 08 15:32:51.395424 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:51.395383    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h4cml"
Sep 08 15:32:51.395789 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:32:51.395469    2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h4cml" podUID="7d4d759e-9d3c-497a-b546-462ae2b48785"
Sep 08 15:32:53.394505 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:53.394467    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fw5jh"
Sep 08 15:32:53.394966 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:53.394514    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5vt6c"
Sep 08 15:32:53.394966 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:53.394514    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h4cml"
Sep 08 15:32:53.394966 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:32:53.394604    2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fw5jh" podUID="84663ea3-1feb-4c87-8609-c580fa60416e"
Sep 08 15:32:53.395136 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:32:53.395016    2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5vt6c" podUID="3d62444d-76ba-4184-869c-496ba4da8d04"
Sep 08 15:32:53.395136 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:32:53.395109    2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h4cml" podUID="7d4d759e-9d3c-497a-b546-462ae2b48785"
Sep 08 15:32:55.394492 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:55.394411    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fw5jh"
Sep 08 15:32:55.394932 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:32:55.394534    2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fw5jh" podUID="84663ea3-1feb-4c87-8609-c580fa60416e"
Sep 08 15:32:55.394932 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:55.394412    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5vt6c"
Sep 08 15:32:55.394932 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:55.394411    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h4cml"
Sep 08 15:32:55.394932 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:32:55.394635    2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5vt6c" podUID="3d62444d-76ba-4184-869c-496ba4da8d04"
Sep 08 15:32:55.394932 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:32:55.394714    2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h4cml" podUID="7d4d759e-9d3c-497a-b546-462ae2b48785"
Sep 08 15:32:57.394689 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:57.394657    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fw5jh"
Sep 08 15:32:57.395084 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:57.394657    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5vt6c"
Sep 08 15:32:57.395084 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:32:57.394763    2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fw5jh" podUID="84663ea3-1feb-4c87-8609-c580fa60416e"
Sep 08 15:32:57.395084 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:57.394656    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h4cml"
Sep 08 15:32:57.395084 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:32:57.394869    2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5vt6c" podUID="3d62444d-76ba-4184-869c-496ba4da8d04"
Sep 08 15:32:57.395084 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:32:57.394965    2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h4cml" podUID="7d4d759e-9d3c-497a-b546-462ae2b48785"
Sep 08 15:32:59.115511 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:59.115465    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d4d759e-9d3c-497a-b546-462ae2b48785-metrics-certs\") pod \"network-metrics-daemon-h4cml\" (UID: \"7d4d759e-9d3c-497a-b546-462ae2b48785\") " pod="openshift-multus/network-metrics-daemon-h4cml"
Sep 08 15:32:59.115997 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:59.115543    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/84663ea3-1feb-4c87-8609-c580fa60416e-original-pull-secret\") pod \"global-pull-secret-syncer-fw5jh\" (UID: \"84663ea3-1feb-4c87-8609-c580fa60416e\") " pod="kube-system/global-pull-secret-syncer-fw5jh"
Sep 08 15:32:59.115997 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:32:59.115620    2549 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 15:32:59.115997 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:32:59.115648    2549 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 15:32:59.115997 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:32:59.115702    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d4d759e-9d3c-497a-b546-462ae2b48785-metrics-certs podName:7d4d759e-9d3c-497a-b546-462ae2b48785 nodeName:}" failed. No retries permitted until 2026-09-08 15:33:15.115680368 +0000 UTC m=+33.205131452 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7d4d759e-9d3c-497a-b546-462ae2b48785-metrics-certs") pod "network-metrics-daemon-h4cml" (UID: "7d4d759e-9d3c-497a-b546-462ae2b48785") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 15:32:59.115997 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:32:59.115722    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84663ea3-1feb-4c87-8609-c580fa60416e-original-pull-secret podName:84663ea3-1feb-4c87-8609-c580fa60416e nodeName:}" failed. No retries permitted until 2026-09-08 15:33:15.115712737 +0000 UTC m=+33.205163807 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/84663ea3-1feb-4c87-8609-c580fa60416e-original-pull-secret") pod "global-pull-secret-syncer-fw5jh" (UID: "84663ea3-1feb-4c87-8609-c580fa60416e") : object "kube-system"/"original-pull-secret" not registered
Sep 08 15:32:59.216299 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:59.216270    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79p6n\" (UniqueName: \"kubernetes.io/projected/3d62444d-76ba-4184-869c-496ba4da8d04-kube-api-access-79p6n\") pod \"network-check-target-5vt6c\" (UID: \"3d62444d-76ba-4184-869c-496ba4da8d04\") " pod="openshift-network-diagnostics/network-check-target-5vt6c"
Sep 08 15:32:59.216393 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:32:59.216380    2549 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 15:32:59.216449 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:32:59.216396    2549 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 15:32:59.216449 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:32:59.216406    2549 projected.go:196] Error preparing data for projected volume kube-api-access-79p6n for pod openshift-network-diagnostics/network-check-target-5vt6c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 15:32:59.216521 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:32:59.216453    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3d62444d-76ba-4184-869c-496ba4da8d04-kube-api-access-79p6n podName:3d62444d-76ba-4184-869c-496ba4da8d04 nodeName:}" failed. No retries permitted until 2026-09-08 15:33:15.216436714 +0000 UTC m=+33.305887789 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-79p6n" (UniqueName: "kubernetes.io/projected/3d62444d-76ba-4184-869c-496ba4da8d04-kube-api-access-79p6n") pod "network-check-target-5vt6c" (UID: "3d62444d-76ba-4184-869c-496ba4da8d04") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 15:32:59.394587 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:59.394506    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fw5jh"
Sep 08 15:32:59.394723 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:59.394618    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h4cml"
Sep 08 15:32:59.394723 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:32:59.394626    2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fw5jh" podUID="84663ea3-1feb-4c87-8609-c580fa60416e"
Sep 08 15:32:59.394723 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:32:59.394655    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5vt6c"
Sep 08 15:32:59.394898 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:32:59.394761    2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h4cml" podUID="7d4d759e-9d3c-497a-b546-462ae2b48785"
Sep 08 15:32:59.394898 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:32:59.394841    2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5vt6c" podUID="3d62444d-76ba-4184-869c-496ba4da8d04"
Sep 08 15:33:01.395117 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:01.395087    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h4cml"
Sep 08 15:33:01.395549 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:01.395086    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5vt6c"
Sep 08 15:33:01.395549 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:33:01.395226    2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h4cml" podUID="7d4d759e-9d3c-497a-b546-462ae2b48785"
Sep 08 15:33:01.395549 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:01.395086    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fw5jh"
Sep 08 15:33:01.395549 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:33:01.395287    2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5vt6c" podUID="3d62444d-76ba-4184-869c-496ba4da8d04"
Sep 08 15:33:01.395549 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:33:01.395363    2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fw5jh" podUID="84663ea3-1feb-4c87-8609-c580fa60416e"
Sep 08 15:33:03.394775 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:03.394426    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h4cml"
Sep 08 15:33:03.395130 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:03.394446    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fw5jh"
Sep 08 15:33:03.395130 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:33:03.394885    2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h4cml" podUID="7d4d759e-9d3c-497a-b546-462ae2b48785"
Sep 08 15:33:03.395130 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:33:03.394957    2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fw5jh" podUID="84663ea3-1feb-4c87-8609-c580fa60416e"
Sep 08 15:33:03.395130 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:03.394451    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5vt6c"
Sep 08 15:33:03.395130 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:33:03.395033    2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5vt6c" podUID="3d62444d-76ba-4184-869c-496ba4da8d04"
Sep 08 15:33:03.475842 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:03.475725    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-95s5s" event={"ID":"95c9958b-a005-439c-b856-0cbb99f08700","Type":"ContainerStarted","Data":"8a64d9f3a7878283c949285b17c74cd72990a390a7aafd74861ae7aee4211573"}
Sep 08 15:33:03.478597 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:03.478572    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-klnj5" event={"ID":"37ec7079-542c-4b91-8b2b-83e8e89fcbb7","Type":"ContainerStarted","Data":"f716264b2f3e2386d272991ab0ffb880caababc902a1f82fd43a0938735e52ec"}
Sep 08 15:33:03.484470 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:03.484446    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-f4vvt_195f1a74-19bc-49c9-aada-fa1c13488480/ovn-acl-logging/0.log"
Sep 08 15:33:03.484903 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:03.484829    2549 generic.go:356] "Generic (PLEG): container finished" podID="195f1a74-19bc-49c9-aada-fa1c13488480" containerID="31170c2071ff98138477b049f6d0722307f926088182974a1ce3a76568573cbc" exitCode=1
Sep 08 15:33:03.485012 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:03.484926    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-f4vvt" event={"ID":"195f1a74-19bc-49c9-aada-fa1c13488480","Type":"ContainerStarted","Data":"37406d9aa41b0cdc8c26372e03453c469e5bc015d3cee03070ee9decd863013a"}
Sep 08 15:33:03.485012 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:03.484996    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-f4vvt" event={"ID":"195f1a74-19bc-49c9-aada-fa1c13488480","Type":"ContainerStarted","Data":"43258e37a6b0b7f33e08b633b3c0f43b00a4c1d8f6e944a5188d05073dc14cf3"}
Sep 08 15:33:03.485114 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:03.485028    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-f4vvt" event={"ID":"195f1a74-19bc-49c9-aada-fa1c13488480","Type":"ContainerDied","Data":"31170c2071ff98138477b049f6d0722307f926088182974a1ce3a76568573cbc"}
Sep 08 15:33:03.485114 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:03.485092    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-f4vvt" event={"ID":"195f1a74-19bc-49c9-aada-fa1c13488480","Type":"ContainerStarted","Data":"62433dc0050ed5c600c715ba5ed468ebe55990a68c2a9f122678462a6a6ec6c1"}
Sep 08 15:33:03.486964 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:03.486942    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-61.ec2.internal" event={"ID":"4a20335aaf3733c5ea5fa6f08be94600","Type":"ContainerStarted","Data":"d21f77081cca2315f252d529fe265492034de6ab849cb8db83b89d2724f1d4f6"}
Sep 08 15:33:03.496562 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:03.496474    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-95s5s" podStartSLOduration=1.071348409 podStartE2EDuration="20.496383888s" totalImagesPullingTime="19.425035479s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 15:32:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:33:03.495826279 +0000 UTC m=+21.585277367" watchObservedRunningTime="2026-09-08 15:33:03.496383888 +0000 UTC m=+21.585834979"
Sep 08 15:33:03.511237 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:03.511129    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-61.ec2.internal" podStartSLOduration=20.511112011 podStartE2EDuration="20.511112011s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 15:32:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:33:03.510294276 +0000 UTC m=+21.599745367" watchObservedRunningTime="2026-09-08 15:33:03.511112011 +0000 UTC m=+21.600563102"
Sep 08 15:33:03.528058 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:03.528010    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-klnj5" podStartSLOduration=1.219357428 podStartE2EDuration="20.527992861s" totalImagesPullingTime="19.308635433s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 15:32:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:33:03.527792231 +0000 UTC m=+21.617243324" watchObservedRunningTime="2026-09-08 15:33:03.527992861 +0000 UTC m=+21.617443951"
Sep 08 15:33:04.491701 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:04.491612    2549 generic.go:356] "Generic (PLEG): container finished" podID="3236f6c9-60de-47f9-84a9-5980933c58cd" containerID="0f4a9e9cdd28117cf54473af2f7557fabad00a12f0efe8e23cf2b175a8b73ee7" exitCode=0
Sep 08 15:33:04.492463 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:04.491701    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rpthc" event={"ID":"3236f6c9-60de-47f9-84a9-5980933c58cd","Type":"ContainerDied","Data":"0f4a9e9cdd28117cf54473af2f7557fabad00a12f0efe8e23cf2b175a8b73ee7"}
Sep 08 15:33:04.493605 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:04.493575    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kfw4l" event={"ID":"c9873505-399e-48b7-b74c-685ae3d9f7b5","Type":"ContainerStarted","Data":"de9b321c41268c285f5624741fb4cda251e87b3252c017cb437f4a477c6dcad4"}
Sep 08 15:33:04.498266 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:04.498246    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-f4vvt_195f1a74-19bc-49c9-aada-fa1c13488480/ovn-acl-logging/0.log"
Sep 08 15:33:04.498699 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:04.498673    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-f4vvt" event={"ID":"195f1a74-19bc-49c9-aada-fa1c13488480","Type":"ContainerStarted","Data":"fb450c85c3cd2cf828fcc328d6306be4d4290b8f745a88ca6337c08060b24854"}
Sep 08 15:33:04.498779 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:04.498749    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-f4vvt" event={"ID":"195f1a74-19bc-49c9-aada-fa1c13488480","Type":"ContainerStarted","Data":"7c35ff40db187574c52003d38a211566ce6a972ffb5dafb5e0859f40a727d6ce"}
Sep 08 15:33:04.500335 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:04.500312    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-vp9nk" event={"ID":"a955f56c-5ac8-422a-8ddc-ce17a8361eb6","Type":"ContainerStarted","Data":"c84f39e10d6e93de858a7cf9c1180ec8207fa685d3264c5f3b202bed7d4ab41e"}
Sep 08 15:33:04.501865 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:04.501831    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-st9hm" event={"ID":"28cd0557-2fec-4948-af98-6c67fb1e568d","Type":"ContainerStarted","Data":"d3a12e3ab3b8bd072a0a6d64cb6df6a82c60b6b0522aa09870910370c56dc770"}
Sep 08 15:33:04.503509 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:04.503486    2549 generic.go:356] "Generic (PLEG): container finished" podID="e8cf966721b2ec8fb0cd4ca3ebd9318e" containerID="984f05c6fb7101e5dc6adf3a974e95b05fcc3a8028b97dc3a73dda87487b8554" exitCode=0
Sep 08 15:33:04.503618 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:04.503587    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-61.ec2.internal" event={"ID":"e8cf966721b2ec8fb0cd4ca3ebd9318e","Type":"ContainerDied","Data":"984f05c6fb7101e5dc6adf3a974e95b05fcc3a8028b97dc3a73dda87487b8554"}
Sep 08 15:33:04.505312 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:04.505281    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-6q7cq" event={"ID":"5f9598ad-fb27-4dda-ab17-98d54cbd30aa","Type":"ContainerStarted","Data":"78d307cb229a0bcfb4ea5573060db1fdc94b8915431cb37bc2f30bb92d665d8a"}
Sep 08 15:33:04.507055 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:04.507025    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-p4rw9" event={"ID":"65a8a5a6-5e05-4ee5-99a6-c026327dfa43","Type":"ContainerStarted","Data":"f4e72d9cf22149fdc52f6e0b6634714eb8fbcf7ce0d02ebbdd0d9440a29cfa7f"}
Sep 08 15:33:04.547046 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:04.546998    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-p4rw9" podStartSLOduration=2.222603289 podStartE2EDuration="21.54698104s" totalImagesPullingTime="19.324377751s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 15:32:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:33:04.54681531 +0000 UTC m=+22.636266399" watchObservedRunningTime="2026-09-08 15:33:04.54698104 +0000 UTC m=+22.636432129"
Sep 08 15:33:04.582323 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:04.582196    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-6q7cq" podStartSLOduration=2.291162874 podStartE2EDuration="21.582182842s" totalImagesPullingTime="19.291019968s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 15:32:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:33:04.565262804 +0000 UTC m=+22.654713895" watchObservedRunningTime="2026-09-08 15:33:04.582182842 +0000 UTC m=+22.671633931"
Sep 08 15:33:04.582470 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:04.582451    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-st9hm" podStartSLOduration=10.249662706 podStartE2EDuration="21.582445944s" totalImagesPullingTime="11.332783238s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:32:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:33:04.582039059 +0000 UTC m=+22.671490147" watchObservedRunningTime="2026-09-08 15:33:04.582445944 +0000 UTC m=+22.671897032"
Sep 08 15:33:05.220726 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:05.220704    2549 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 08 15:33:05.340317 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:05.340186    2549 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-08T15:33:05.22072244Z","UUID":"73348159-9514-4bd6-89d1-d8cc96e7d4ad","Handler":null,"Name":"","Endpoint":""}
Sep 08 15:33:05.342002 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:05.341975    2549 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 08 15:33:05.342002 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:05.342008    2549 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 08 15:33:05.395190 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:05.395160    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fw5jh"
Sep 08 15:33:05.395318 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:05.395160    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5vt6c"
Sep 08 15:33:05.395318 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:33:05.395264    2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fw5jh" podUID="84663ea3-1feb-4c87-8609-c580fa60416e"
Sep 08 15:33:05.395434 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:33:05.395388    2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5vt6c" podUID="3d62444d-76ba-4184-869c-496ba4da8d04"
Sep 08 15:33:05.395434 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:05.395163    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h4cml"
Sep 08 15:33:05.395535 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:33:05.395514    2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h4cml" podUID="7d4d759e-9d3c-497a-b546-462ae2b48785"
Sep 08 15:33:05.511331 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:05.511304    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kfw4l" event={"ID":"c9873505-399e-48b7-b74c-685ae3d9f7b5","Type":"ContainerStarted","Data":"72e8070c61f2d3aba052a37090ce463a6377f0135010c970acfb84b1bace13f4"}
Sep 08 15:33:05.513451 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:05.513397    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-61.ec2.internal" event={"ID":"e8cf966721b2ec8fb0cd4ca3ebd9318e","Type":"ContainerStarted","Data":"a2b1da03575673d2f7185d85d65b392f27f5fbe7dd7b479bc275d83c81fadb6d"}
Sep 08 15:33:05.528916 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:05.528867    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-vp9nk" podStartSLOduration=3.177359626 podStartE2EDuration="22.528838122s" totalImagesPullingTime="19.351478496s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 15:32:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:33:04.600490126 +0000 UTC m=+22.689941216" watchObservedRunningTime="2026-09-08 15:33:05.528838122 +0000 UTC m=+23.618289210"
Sep 08 15:33:05.529549 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:05.529504    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-61.ec2.internal" podStartSLOduration=22.52949448 podStartE2EDuration="22.52949448s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 15:32:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:33:05.528748557 +0000 UTC m=+23.618199645" watchObservedRunningTime="2026-09-08 15:33:05.52949448 +0000 UTC m=+23.618945568"
Sep 08 15:33:06.520778 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:06.520747    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-f4vvt_195f1a74-19bc-49c9-aada-fa1c13488480/ovn-acl-logging/0.log"
Sep 08 15:33:06.522150 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:06.522114    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-f4vvt" event={"ID":"195f1a74-19bc-49c9-aada-fa1c13488480","Type":"ContainerStarted","Data":"b1dd351e8cf05aeb5ed24cbd7fc3778ae1d8279267184f8df5bf9fbc35b9838c"}
Sep 08 15:33:07.014738 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:07.014706    2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-st9hm"
Sep 08 15:33:07.015400 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:07.015380    2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-st9hm"
Sep 08 15:33:07.394706 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:07.394678    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h4cml"
Sep 08 15:33:07.394890 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:07.394686    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fw5jh"
Sep 08 15:33:07.394890 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:33:07.394799    2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h4cml" podUID="7d4d759e-9d3c-497a-b546-462ae2b48785"
Sep 08 15:33:07.394890 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:07.394687    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5vt6c"
Sep 08 15:33:07.394890 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:33:07.394876    2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fw5jh" podUID="84663ea3-1feb-4c87-8609-c580fa60416e"
Sep 08 15:33:07.395096 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:33:07.394958    2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5vt6c" podUID="3d62444d-76ba-4184-869c-496ba4da8d04"
Sep 08 15:33:07.527050 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:07.527011    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kfw4l" event={"ID":"c9873505-399e-48b7-b74c-685ae3d9f7b5","Type":"ContainerStarted","Data":"590a22c0a8510161e9bab8dd4a0b14e5f3d9e3832852d7414d16016dfdf0760f"}
Sep 08 15:33:07.527524 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:07.527231    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-st9hm"
Sep 08 15:33:07.527984 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:07.527964    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-st9hm"
Sep 08 15:33:07.562991 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:07.562944    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kfw4l" podStartSLOduration=10.0827007 podStartE2EDuration="24.562928548s" totalImagesPullingTime="14.480227848s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 15:32:43 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:33:07.546201396 +0000 UTC m=+25.635652484" watchObservedRunningTime="2026-09-08 15:33:07.562928548 +0000 UTC m=+25.652379637"
Sep 08 15:33:09.395256 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:09.395093    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fw5jh"
Sep 08 15:33:09.395615 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:09.395140    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5vt6c"
Sep 08 15:33:09.395615 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:33:09.395334    2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fw5jh" podUID="84663ea3-1feb-4c87-8609-c580fa60416e"
Sep 08 15:33:09.395615 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:33:09.395427    2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5vt6c" podUID="3d62444d-76ba-4184-869c-496ba4da8d04"
Sep 08 15:33:09.395615 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:09.395159    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h4cml"
Sep 08 15:33:09.395615 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:33:09.395587    2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h4cml" podUID="7d4d759e-9d3c-497a-b546-462ae2b48785"
Sep 08 15:33:09.532972 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:09.532944    2549 generic.go:356] "Generic (PLEG): container finished" podID="3236f6c9-60de-47f9-84a9-5980933c58cd" containerID="cf03028efb674e717bc1396704346878225d172c98eb6dcd36530d1eae2bd990" exitCode=0
Sep 08 15:33:09.533129 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:09.533024    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rpthc" event={"ID":"3236f6c9-60de-47f9-84a9-5980933c58cd","Type":"ContainerDied","Data":"cf03028efb674e717bc1396704346878225d172c98eb6dcd36530d1eae2bd990"}
Sep 08 15:33:09.537008 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:09.536989    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-f4vvt_195f1a74-19bc-49c9-aada-fa1c13488480/ovn-acl-logging/0.log"
Sep 08 15:33:09.537384 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:09.537355    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-f4vvt" event={"ID":"195f1a74-19bc-49c9-aada-fa1c13488480","Type":"ContainerStarted","Data":"e704bd750d6ab7e38373658559f347e01ce7f9baa23c48e244f7512774e2db56"}
Sep 08 15:33:09.537559 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:09.537543    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-f4vvt"
Sep 08 15:33:09.537634 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:09.537608    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-f4vvt"
Sep 08 15:33:09.537752 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:09.537736    2549 scope.go:117] "RemoveContainer" containerID="31170c2071ff98138477b049f6d0722307f926088182974a1ce3a76568573cbc"
Sep 08 15:33:09.554105 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:09.554088    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-f4vvt"
Sep 08 15:33:10.543115 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:10.543093    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-f4vvt_195f1a74-19bc-49c9-aada-fa1c13488480/ovn-acl-logging/0.log"
Sep 08 15:33:10.543595 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:10.543453    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-f4vvt" event={"ID":"195f1a74-19bc-49c9-aada-fa1c13488480","Type":"ContainerStarted","Data":"ed557fd21c0ca0d7a8fbb3c311796d8d993e61ee9805bbbe04c2283820c9b89f"}
Sep 08 15:33:10.543737 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:10.543716    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-f4vvt"
Sep 08 15:33:10.557450 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:10.557428    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-f4vvt"
Sep 08 15:33:10.574817 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:10.574772    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-f4vvt" podStartSLOduration=8.208312753 podStartE2EDuration="27.574755106s" totalImagesPullingTime="19.366442353s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 15:32:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:33:10.574532021 +0000 UTC m=+28.663983109" watchObservedRunningTime="2026-09-08 15:33:10.574755106 +0000 UTC m=+28.664206195"
Sep 08 15:33:11.045513 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:11.045376    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-fw5jh"]
Sep 08 15:33:11.045639 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:11.045624    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fw5jh"
Sep 08 15:33:11.045754 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:33:11.045738    2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fw5jh" podUID="84663ea3-1feb-4c87-8609-c580fa60416e"
Sep 08 15:33:11.048395 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:11.048357    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-5vt6c"]
Sep 08 15:33:11.048487 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:11.048471    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5vt6c"
Sep 08 15:33:11.048566 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:33:11.048549    2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5vt6c" podUID="3d62444d-76ba-4184-869c-496ba4da8d04"
Sep 08 15:33:11.049404 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:11.049293    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-h4cml"]
Sep 08 15:33:11.049404 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:11.049400    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h4cml"
Sep 08 15:33:11.049564 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:33:11.049498    2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h4cml" podUID="7d4d759e-9d3c-497a-b546-462ae2b48785"
Sep 08 15:33:11.547286 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:11.547248    2549 generic.go:356] "Generic (PLEG): container finished" podID="3236f6c9-60de-47f9-84a9-5980933c58cd" containerID="b14960dec63bae16834c922a4c6eb3564d8b82a30c2a0240ca9a38ece08c7743" exitCode=0
Sep 08 15:33:11.547816 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:11.547335    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rpthc" event={"ID":"3236f6c9-60de-47f9-84a9-5980933c58cd","Type":"ContainerDied","Data":"b14960dec63bae16834c922a4c6eb3564d8b82a30c2a0240ca9a38ece08c7743"}
Sep 08 15:33:12.395920 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:12.395897    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fw5jh"
Sep 08 15:33:12.396040 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:33:12.395979    2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fw5jh" podUID="84663ea3-1feb-4c87-8609-c580fa60416e"
Sep 08 15:33:12.396040 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:12.396005    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5vt6c"
Sep 08 15:33:12.396118 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:33:12.396049    2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5vt6c" podUID="3d62444d-76ba-4184-869c-496ba4da8d04"
Sep 08 15:33:13.394997 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:13.394965    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h4cml"
Sep 08 15:33:13.395349 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:33:13.395081    2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h4cml" podUID="7d4d759e-9d3c-497a-b546-462ae2b48785"
Sep 08 15:33:13.556737 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:13.556663    2549 generic.go:356] "Generic (PLEG): container finished" podID="3236f6c9-60de-47f9-84a9-5980933c58cd" containerID="4e8b4352627507b8c0abfa4608be1e0a321cd3c5f43c1aa3df7b2daaf462bab7" exitCode=0
Sep 08 15:33:13.556737 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:13.556728    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rpthc" event={"ID":"3236f6c9-60de-47f9-84a9-5980933c58cd","Type":"ContainerDied","Data":"4e8b4352627507b8c0abfa4608be1e0a321cd3c5f43c1aa3df7b2daaf462bab7"}
Sep 08 15:33:14.394625 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:14.394595    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5vt6c"
Sep 08 15:33:14.394804 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:33:14.394711    2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5vt6c" podUID="3d62444d-76ba-4184-869c-496ba4da8d04"
Sep 08 15:33:14.394804 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:14.394768    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fw5jh"
Sep 08 15:33:14.394945 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:33:14.394881    2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fw5jh" podUID="84663ea3-1feb-4c87-8609-c580fa60416e"
Sep 08 15:33:15.139363 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:15.139327    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/84663ea3-1feb-4c87-8609-c580fa60416e-original-pull-secret\") pod \"global-pull-secret-syncer-fw5jh\" (UID: \"84663ea3-1feb-4c87-8609-c580fa60416e\") " pod="kube-system/global-pull-secret-syncer-fw5jh"
Sep 08 15:33:15.139804 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:15.139395    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d4d759e-9d3c-497a-b546-462ae2b48785-metrics-certs\") pod \"network-metrics-daemon-h4cml\" (UID: \"7d4d759e-9d3c-497a-b546-462ae2b48785\") " pod="openshift-multus/network-metrics-daemon-h4cml"
Sep 08 15:33:15.139804 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:33:15.139480    2549 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 15:33:15.139804 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:33:15.139507    2549 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 15:33:15.139804 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:33:15.139552    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84663ea3-1feb-4c87-8609-c580fa60416e-original-pull-secret podName:84663ea3-1feb-4c87-8609-c580fa60416e nodeName:}" failed. No retries permitted until 2026-09-08 15:33:47.139529813 +0000 UTC m=+65.228980894 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/84663ea3-1feb-4c87-8609-c580fa60416e-original-pull-secret") pod "global-pull-secret-syncer-fw5jh" (UID: "84663ea3-1feb-4c87-8609-c580fa60416e") : object "kube-system"/"original-pull-secret" not registered
Sep 08 15:33:15.139804 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:33:15.139571    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d4d759e-9d3c-497a-b546-462ae2b48785-metrics-certs podName:7d4d759e-9d3c-497a-b546-462ae2b48785 nodeName:}" failed. No retries permitted until 2026-09-08 15:33:47.139561477 +0000 UTC m=+65.229012548 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7d4d759e-9d3c-497a-b546-462ae2b48785-metrics-certs") pod "network-metrics-daemon-h4cml" (UID: "7d4d759e-9d3c-497a-b546-462ae2b48785") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 15:33:15.240093 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:15.240057    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79p6n\" (UniqueName: \"kubernetes.io/projected/3d62444d-76ba-4184-869c-496ba4da8d04-kube-api-access-79p6n\") pod \"network-check-target-5vt6c\" (UID: \"3d62444d-76ba-4184-869c-496ba4da8d04\") " pod="openshift-network-diagnostics/network-check-target-5vt6c"
Sep 08 15:33:15.240272 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:33:15.240208    2549 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 15:33:15.240272 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:33:15.240231    2549 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 15:33:15.240272 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:33:15.240243    2549 projected.go:196] Error preparing data for projected volume kube-api-access-79p6n for pod openshift-network-diagnostics/network-check-target-5vt6c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 15:33:15.240422 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:33:15.240306    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3d62444d-76ba-4184-869c-496ba4da8d04-kube-api-access-79p6n podName:3d62444d-76ba-4184-869c-496ba4da8d04 nodeName:}" failed. No retries permitted until 2026-09-08 15:33:47.240287874 +0000 UTC m=+65.329738961 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-79p6n" (UniqueName: "kubernetes.io/projected/3d62444d-76ba-4184-869c-496ba4da8d04-kube-api-access-79p6n") pod "network-check-target-5vt6c" (UID: "3d62444d-76ba-4184-869c-496ba4da8d04") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 15:33:15.394827 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:15.394803    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h4cml"
Sep 08 15:33:15.394992 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:33:15.394947    2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h4cml" podUID="7d4d759e-9d3c-497a-b546-462ae2b48785"
Sep 08 15:33:16.164051 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:16.164027    2549 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-61.ec2.internal" event="NodeReady"
Sep 08 15:33:16.164433 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:16.164143    2549 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 08 15:33:16.216519 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:16.216491    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-tt5hr"]
Sep 08 15:33:16.234072 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:16.234042    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-vhsng"]
Sep 08 15:33:16.234335 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:16.234292    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-tt5hr"
Sep 08 15:33:16.238449 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:16.238425    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 08 15:33:16.238636 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:16.238614    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 08 15:33:16.238716 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:16.238657    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-nqnwb\""
Sep 08 15:33:16.238870 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:16.238843    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 08 15:33:16.259922 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:16.259896    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-tt5hr"]
Sep 08 15:33:16.260040 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:16.259939    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-vhsng"]
Sep 08 15:33:16.260040 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:16.259952    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-vhsng"
Sep 08 15:33:16.263400 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:16.263373    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 08 15:33:16.263514 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:16.263448    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 08 15:33:16.263514 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:16.263487    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-5prt2\""
Sep 08 15:33:16.347261 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:16.347175    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/57cef312-9005-43bc-9e2f-b59db9e864c3-metrics-tls\") pod \"dns-default-vhsng\" (UID: \"57cef312-9005-43bc-9e2f-b59db9e864c3\") " pod="openshift-dns/dns-default-vhsng"
Sep 08 15:33:16.347261 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:16.347225    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgdz2\" (UniqueName: \"kubernetes.io/projected/0e985291-302f-4109-b4ba-8569a56235e4-kube-api-access-hgdz2\") pod \"ingress-canary-tt5hr\" (UID: \"0e985291-302f-4109-b4ba-8569a56235e4\") " pod="openshift-ingress-canary/ingress-canary-tt5hr"
Sep 08 15:33:16.347261 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:16.347248    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84xgf\" (UniqueName: \"kubernetes.io/projected/57cef312-9005-43bc-9e2f-b59db9e864c3-kube-api-access-84xgf\") pod \"dns-default-vhsng\" (UID: \"57cef312-9005-43bc-9e2f-b59db9e864c3\") " pod="openshift-dns/dns-default-vhsng"
Sep 08 15:33:16.347514 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:16.347369    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/57cef312-9005-43bc-9e2f-b59db9e864c3-config-volume\") pod \"dns-default-vhsng\" (UID: \"57cef312-9005-43bc-9e2f-b59db9e864c3\") " pod="openshift-dns/dns-default-vhsng"
Sep 08 15:33:16.347514 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:16.347467    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0e985291-302f-4109-b4ba-8569a56235e4-cert\") pod \"ingress-canary-tt5hr\" (UID: \"0e985291-302f-4109-b4ba-8569a56235e4\") " pod="openshift-ingress-canary/ingress-canary-tt5hr"
Sep 08 15:33:16.347514 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:16.347502    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57cef312-9005-43bc-9e2f-b59db9e864c3-tmp-dir\") pod \"dns-default-vhsng\" (UID: \"57cef312-9005-43bc-9e2f-b59db9e864c3\") " pod="openshift-dns/dns-default-vhsng"
Sep 08 15:33:16.398625 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:16.398598    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fw5jh"
Sep 08 15:33:16.398782 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:16.398768    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5vt6c"
Sep 08 15:33:16.401840 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:16.401821    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 08 15:33:16.401840 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:16.401829    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-xt4hq\""
Sep 08 15:33:16.402034 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:16.401821    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 08 15:33:16.402034 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:16.401831    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 08 15:33:16.448552 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:16.448526    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0e985291-302f-4109-b4ba-8569a56235e4-cert\") pod \"ingress-canary-tt5hr\" (UID: \"0e985291-302f-4109-b4ba-8569a56235e4\") " pod="openshift-ingress-canary/ingress-canary-tt5hr"
Sep 08 15:33:16.448652 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:16.448561    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57cef312-9005-43bc-9e2f-b59db9e864c3-tmp-dir\") pod \"dns-default-vhsng\" (UID: \"57cef312-9005-43bc-9e2f-b59db9e864c3\") " pod="openshift-dns/dns-default-vhsng"
Sep 08 15:33:16.448652 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:16.448594    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/57cef312-9005-43bc-9e2f-b59db9e864c3-metrics-tls\") pod \"dns-default-vhsng\" (UID: \"57cef312-9005-43bc-9e2f-b59db9e864c3\") " pod="openshift-dns/dns-default-vhsng"
Sep 08 15:33:16.448652 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:16.448623    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgdz2\" (UniqueName: \"kubernetes.io/projected/0e985291-302f-4109-b4ba-8569a56235e4-kube-api-access-hgdz2\") pod \"ingress-canary-tt5hr\" (UID: \"0e985291-302f-4109-b4ba-8569a56235e4\") " pod="openshift-ingress-canary/ingress-canary-tt5hr"
Sep 08 15:33:16.448652 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:16.448646    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84xgf\" (UniqueName: \"kubernetes.io/projected/57cef312-9005-43bc-9e2f-b59db9e864c3-kube-api-access-84xgf\") pod \"dns-default-vhsng\" (UID: \"57cef312-9005-43bc-9e2f-b59db9e864c3\") " pod="openshift-dns/dns-default-vhsng"
Sep 08 15:33:16.448845 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:16.448694    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/57cef312-9005-43bc-9e2f-b59db9e864c3-config-volume\") pod \"dns-default-vhsng\" (UID: \"57cef312-9005-43bc-9e2f-b59db9e864c3\") " pod="openshift-dns/dns-default-vhsng"
Sep 08 15:33:16.448845 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:33:16.448777    2549 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 15:33:16.448845 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:33:16.448842    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57cef312-9005-43bc-9e2f-b59db9e864c3-metrics-tls podName:57cef312-9005-43bc-9e2f-b59db9e864c3 nodeName:}" failed. No retries permitted until 2026-09-08 15:33:16.948820347 +0000 UTC m=+35.038271428 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/57cef312-9005-43bc-9e2f-b59db9e864c3-metrics-tls") pod "dns-default-vhsng" (UID: "57cef312-9005-43bc-9e2f-b59db9e864c3") : secret "dns-default-metrics-tls" not found
Sep 08 15:33:16.449022 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:16.448912    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57cef312-9005-43bc-9e2f-b59db9e864c3-tmp-dir\") pod \"dns-default-vhsng\" (UID: \"57cef312-9005-43bc-9e2f-b59db9e864c3\") " pod="openshift-dns/dns-default-vhsng"
Sep 08 15:33:16.449203 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:16.449185    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/57cef312-9005-43bc-9e2f-b59db9e864c3-config-volume\") pod \"dns-default-vhsng\" (UID: \"57cef312-9005-43bc-9e2f-b59db9e864c3\") " pod="openshift-dns/dns-default-vhsng"
Sep 08 15:33:16.452919 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:16.452897    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0e985291-302f-4109-b4ba-8569a56235e4-cert\") pod \"ingress-canary-tt5hr\" (UID: \"0e985291-302f-4109-b4ba-8569a56235e4\") " pod="openshift-ingress-canary/ingress-canary-tt5hr"
Sep 08 15:33:16.458326 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:16.458305    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-84xgf\" (UniqueName: \"kubernetes.io/projected/57cef312-9005-43bc-9e2f-b59db9e864c3-kube-api-access-84xgf\") pod \"dns-default-vhsng\" (UID: \"57cef312-9005-43bc-9e2f-b59db9e864c3\") " pod="openshift-dns/dns-default-vhsng"
Sep 08 15:33:16.461138 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:16.461120    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgdz2\" (UniqueName: \"kubernetes.io/projected/0e985291-302f-4109-b4ba-8569a56235e4-kube-api-access-hgdz2\") pod \"ingress-canary-tt5hr\" (UID: \"0e985291-302f-4109-b4ba-8569a56235e4\") " pod="openshift-ingress-canary/ingress-canary-tt5hr"
Sep 08 15:33:16.546281 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:16.546248    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-tt5hr"
Sep 08 15:33:16.951616 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:16.951577    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/57cef312-9005-43bc-9e2f-b59db9e864c3-metrics-tls\") pod \"dns-default-vhsng\" (UID: \"57cef312-9005-43bc-9e2f-b59db9e864c3\") " pod="openshift-dns/dns-default-vhsng"
Sep 08 15:33:16.951811 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:33:16.951717    2549 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 15:33:16.951811 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:33:16.951790    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57cef312-9005-43bc-9e2f-b59db9e864c3-metrics-tls podName:57cef312-9005-43bc-9e2f-b59db9e864c3 nodeName:}" failed. No retries permitted until 2026-09-08 15:33:17.951774403 +0000 UTC m=+36.041225475 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/57cef312-9005-43bc-9e2f-b59db9e864c3-metrics-tls") pod "dns-default-vhsng" (UID: "57cef312-9005-43bc-9e2f-b59db9e864c3") : secret "dns-default-metrics-tls" not found
Sep 08 15:33:17.394404 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:17.394311    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h4cml"
Sep 08 15:33:17.397492 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:17.397464    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 08 15:33:17.397628 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:17.397539    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-xsqld\""
Sep 08 15:33:17.960599 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:17.960399    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/57cef312-9005-43bc-9e2f-b59db9e864c3-metrics-tls\") pod \"dns-default-vhsng\" (UID: \"57cef312-9005-43bc-9e2f-b59db9e864c3\") " pod="openshift-dns/dns-default-vhsng"
Sep 08 15:33:17.960599 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:33:17.960553    2549 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 15:33:17.960833 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:33:17.960627    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57cef312-9005-43bc-9e2f-b59db9e864c3-metrics-tls podName:57cef312-9005-43bc-9e2f-b59db9e864c3 nodeName:}" failed. No retries permitted until 2026-09-08 15:33:19.960610664 +0000 UTC m=+38.050061741 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/57cef312-9005-43bc-9e2f-b59db9e864c3-metrics-tls") pod "dns-default-vhsng" (UID: "57cef312-9005-43bc-9e2f-b59db9e864c3") : secret "dns-default-metrics-tls" not found
Sep 08 15:33:17.972915 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:17.972890    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-tt5hr"]
Sep 08 15:33:17.976651 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:33:17.976632    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e985291_302f_4109_b4ba_8569a56235e4.slice/crio-089a6278964358031e5dcf10498c03911cbf72c018862ed8625d118305c0b19b WatchSource:0}: Error finding container 089a6278964358031e5dcf10498c03911cbf72c018862ed8625d118305c0b19b: Status 404 returned error can't find the container with id 089a6278964358031e5dcf10498c03911cbf72c018862ed8625d118305c0b19b
Sep 08 15:33:18.569745 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:18.569710    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-tt5hr" event={"ID":"0e985291-302f-4109-b4ba-8569a56235e4","Type":"ContainerStarted","Data":"089a6278964358031e5dcf10498c03911cbf72c018862ed8625d118305c0b19b"}
Sep 08 15:33:18.575104 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:18.575075    2549 generic.go:356] "Generic (PLEG): container finished" podID="3236f6c9-60de-47f9-84a9-5980933c58cd" containerID="7751fe8476bcfd49a6e7c207c8f9390956d43c8efda1deca2feafb224a71e4f6" exitCode=0
Sep 08 15:33:18.575263 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:18.575154    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rpthc" event={"ID":"3236f6c9-60de-47f9-84a9-5980933c58cd","Type":"ContainerDied","Data":"7751fe8476bcfd49a6e7c207c8f9390956d43c8efda1deca2feafb224a71e4f6"}
Sep 08 15:33:19.580951 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:19.580913    2549 generic.go:356] "Generic (PLEG): container finished" podID="3236f6c9-60de-47f9-84a9-5980933c58cd" containerID="62738e3bdf521878f23c045c120ef3f6a30d5852b047ece3a93bf61abbbf75c2" exitCode=0
Sep 08 15:33:19.581616 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:19.580968    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rpthc" event={"ID":"3236f6c9-60de-47f9-84a9-5980933c58cd","Type":"ContainerDied","Data":"62738e3bdf521878f23c045c120ef3f6a30d5852b047ece3a93bf61abbbf75c2"}
Sep 08 15:33:19.976225 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:19.976189    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/57cef312-9005-43bc-9e2f-b59db9e864c3-metrics-tls\") pod \"dns-default-vhsng\" (UID: \"57cef312-9005-43bc-9e2f-b59db9e864c3\") " pod="openshift-dns/dns-default-vhsng"
Sep 08 15:33:19.976404 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:33:19.976364    2549 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 15:33:19.976458 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:33:19.976442    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57cef312-9005-43bc-9e2f-b59db9e864c3-metrics-tls podName:57cef312-9005-43bc-9e2f-b59db9e864c3 nodeName:}" failed. No retries permitted until 2026-09-08 15:33:23.976425587 +0000 UTC m=+42.065876658 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/57cef312-9005-43bc-9e2f-b59db9e864c3-metrics-tls") pod "dns-default-vhsng" (UID: "57cef312-9005-43bc-9e2f-b59db9e864c3") : secret "dns-default-metrics-tls" not found
Sep 08 15:33:20.587472 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:20.587448    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rpthc" event={"ID":"3236f6c9-60de-47f9-84a9-5980933c58cd","Type":"ContainerStarted","Data":"424b6b577e43da997b259fe058dc3d9c575f3c40d18db1cee58edb10c322acc5"}
Sep 08 15:33:20.614016 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:20.613971    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-rpthc" podStartSLOduration=6.734595023 podStartE2EDuration="37.613956274s" totalImagesPullingTime="30.879361251s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 15:32:43 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:33:20.612084782 +0000 UTC m=+38.701535870" watchObservedRunningTime="2026-09-08 15:33:20.613956274 +0000 UTC m=+38.703407363"
Sep 08 15:33:21.591794 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:21.591757    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-tt5hr" event={"ID":"0e985291-302f-4109-b4ba-8569a56235e4","Type":"ContainerStarted","Data":"873eb1bddcf7a3dc2dcb3258597f13f9ca55779bfb1edeb77e1b88aae57ff471"}
Sep 08 15:33:21.609484 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:21.609444    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-tt5hr" podStartSLOduration=3.088641939 podStartE2EDuration="5.609429266s" totalImagesPullingTime="2.520787327s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:33:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:33:21.608046479 +0000 UTC m=+39.697497568" watchObservedRunningTime="2026-09-08 15:33:21.609429266 +0000 UTC m=+39.698880354"
Sep 08 15:33:24.005189 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:24.005145    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/57cef312-9005-43bc-9e2f-b59db9e864c3-metrics-tls\") pod \"dns-default-vhsng\" (UID: \"57cef312-9005-43bc-9e2f-b59db9e864c3\") " pod="openshift-dns/dns-default-vhsng"
Sep 08 15:33:24.005574 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:33:24.005284    2549 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 15:33:24.005574 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:33:24.005346    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57cef312-9005-43bc-9e2f-b59db9e864c3-metrics-tls podName:57cef312-9005-43bc-9e2f-b59db9e864c3 nodeName:}" failed. No retries permitted until 2026-09-08 15:33:32.005330352 +0000 UTC m=+50.094781422 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/57cef312-9005-43bc-9e2f-b59db9e864c3-metrics-tls") pod "dns-default-vhsng" (UID: "57cef312-9005-43bc-9e2f-b59db9e864c3") : secret "dns-default-metrics-tls" not found
Sep 08 15:33:32.054571 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:32.054523    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/57cef312-9005-43bc-9e2f-b59db9e864c3-metrics-tls\") pod \"dns-default-vhsng\" (UID: \"57cef312-9005-43bc-9e2f-b59db9e864c3\") " pod="openshift-dns/dns-default-vhsng"
Sep 08 15:33:32.055085 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:33:32.054669    2549 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 15:33:32.055085 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:33:32.054734    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57cef312-9005-43bc-9e2f-b59db9e864c3-metrics-tls podName:57cef312-9005-43bc-9e2f-b59db9e864c3 nodeName:}" failed. No retries permitted until 2026-09-08 15:33:48.05471947 +0000 UTC m=+66.144170536 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/57cef312-9005-43bc-9e2f-b59db9e864c3-metrics-tls") pod "dns-default-vhsng" (UID: "57cef312-9005-43bc-9e2f-b59db9e864c3") : secret "dns-default-metrics-tls" not found
Sep 08 15:33:42.565645 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:42.565619    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-f4vvt"
Sep 08 15:33:44.467911 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:44.467879    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-7mfs7"]
Sep 08 15:33:44.470383 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:44.470363    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-7mfs7"
Sep 08 15:33:44.473506 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:44.473483    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Sep 08 15:33:44.473624 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:44.473483    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Sep 08 15:33:44.473624 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:44.473543    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Sep 08 15:33:44.474679 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:44.474663    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-gzgz5\""
Sep 08 15:33:44.474756 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:44.474736    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Sep 08 15:33:44.482421 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:44.482399    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-7mfs7"]
Sep 08 15:33:44.639327 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:44.639300    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d1d7e194-2f46-486c-aa08-f87ef377f54f-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-7mfs7\" (UID: \"d1d7e194-2f46-486c-aa08-f87ef377f54f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-7mfs7"
Sep 08 15:33:44.639476 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:44.639365    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d1d7e194-2f46-486c-aa08-f87ef377f54f-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-7mfs7\" (UID: \"d1d7e194-2f46-486c-aa08-f87ef377f54f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-7mfs7"
Sep 08 15:33:44.639476 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:44.639463    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2p5hr\" (UniqueName: \"kubernetes.io/projected/d1d7e194-2f46-486c-aa08-f87ef377f54f-kube-api-access-2p5hr\") pod \"kube-storage-version-migrator-operator-8d7dd4675-7mfs7\" (UID: \"d1d7e194-2f46-486c-aa08-f87ef377f54f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-7mfs7"
Sep 08 15:33:44.670825 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:44.670799    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-vhng9"]
Sep 08 15:33:44.672460 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:44.672445    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-vhng9"
Sep 08 15:33:44.675179 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:44.675142    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Sep 08 15:33:44.675323 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:44.675225    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Sep 08 15:33:44.675372 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:44.675356    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Sep 08 15:33:44.675420 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:44.675404    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-r7q8k\""
Sep 08 15:33:44.679749 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:44.679728    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-85b454874b-wpwdq"]
Sep 08 15:33:44.681225 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:44.681210    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-95fbf4c66-9k8sq"]
Sep 08 15:33:44.681319 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:44.681306    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-85b454874b-wpwdq"
Sep 08 15:33:44.683034 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:44.683017    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-95fbf4c66-9k8sq"
Sep 08 15:33:44.686026 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:44.685870    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Sep 08 15:33:44.686134 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:44.685882    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-89c8v\""
Sep 08 15:33:44.686200 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:44.685889    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 08 15:33:44.686200 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:44.686166    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Sep 08 15:33:44.686300 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:44.685918    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Sep 08 15:33:44.686502 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:44.686485    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-vjkq6\""
Sep 08 15:33:44.687162 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:44.687132    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 08 15:33:44.687296 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:44.687170    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 08 15:33:44.687544 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:44.687458    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Sep 08 15:33:44.687915 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:44.687889    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-vhng9"]
Sep 08 15:33:44.696417 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:44.696397    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 08 15:33:44.698219 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:44.698200    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-85b454874b-wpwdq"]
Sep 08 15:33:44.698716 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:44.698698    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Sep 08 15:33:44.699142 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:44.699125    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-95fbf4c66-9k8sq"]
Sep 08 15:33:44.740150 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:44.740100    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d1d7e194-2f46-486c-aa08-f87ef377f54f-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-7mfs7\" (UID: \"d1d7e194-2f46-486c-aa08-f87ef377f54f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-7mfs7"
Sep 08 15:33:44.740150 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:44.740142    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2p5hr\" (UniqueName: \"kubernetes.io/projected/d1d7e194-2f46-486c-aa08-f87ef377f54f-kube-api-access-2p5hr\") pod \"kube-storage-version-migrator-operator-8d7dd4675-7mfs7\" (UID: \"d1d7e194-2f46-486c-aa08-f87ef377f54f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-7mfs7"
Sep 08 15:33:44.740296 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:44.740162    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d1d7e194-2f46-486c-aa08-f87ef377f54f-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-7mfs7\" (UID: \"d1d7e194-2f46-486c-aa08-f87ef377f54f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-7mfs7"
Sep 08 15:33:44.740563 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:44.740546    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d1d7e194-2f46-486c-aa08-f87ef377f54f-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-7mfs7\" (UID: \"d1d7e194-2f46-486c-aa08-f87ef377f54f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-7mfs7"
Sep 08 15:33:44.743295 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:44.743278    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d1d7e194-2f46-486c-aa08-f87ef377f54f-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-7mfs7\" (UID: \"d1d7e194-2f46-486c-aa08-f87ef377f54f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-7mfs7"
Sep 08 15:33:44.747774 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:44.747755    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2p5hr\" (UniqueName: \"kubernetes.io/projected/d1d7e194-2f46-486c-aa08-f87ef377f54f-kube-api-access-2p5hr\") pod \"kube-storage-version-migrator-operator-8d7dd4675-7mfs7\" (UID: \"d1d7e194-2f46-486c-aa08-f87ef377f54f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-7mfs7"
Sep 08 15:33:44.780636 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:44.780615    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-7mfs7"
Sep 08 15:33:44.841381 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:44.841354    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0ecfb56c-9113-49a2-99c8-226c564c32d5-installation-pull-secrets\") pod \"image-registry-95fbf4c66-9k8sq\" (UID: \"0ecfb56c-9113-49a2-99c8-226c564c32d5\") " pod="openshift-image-registry/image-registry-95fbf4c66-9k8sq"
Sep 08 15:33:44.841504 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:44.841402    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0ecfb56c-9113-49a2-99c8-226c564c32d5-registry-certificates\") pod \"image-registry-95fbf4c66-9k8sq\" (UID: \"0ecfb56c-9113-49a2-99c8-226c564c32d5\") " pod="openshift-image-registry/image-registry-95fbf4c66-9k8sq"
Sep 08 15:33:44.841504 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:44.841429    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5f7b0f68-a8e3-4058-b032-15f419682b2a-trusted-ca\") pod \"console-operator-85b454874b-wpwdq\" (UID: \"5f7b0f68-a8e3-4058-b032-15f419682b2a\") " pod="openshift-console-operator/console-operator-85b454874b-wpwdq"
Sep 08 15:33:44.841504 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:44.841487    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8k4v\" (UniqueName: \"kubernetes.io/projected/5f7b0f68-a8e3-4058-b032-15f419682b2a-kube-api-access-l8k4v\") pod \"console-operator-85b454874b-wpwdq\" (UID: \"5f7b0f68-a8e3-4058-b032-15f419682b2a\") " pod="openshift-console-operator/console-operator-85b454874b-wpwdq"
Sep 08 15:33:44.841662 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:44.841513    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0ecfb56c-9113-49a2-99c8-226c564c32d5-registry-tls\") pod \"image-registry-95fbf4c66-9k8sq\" (UID: \"0ecfb56c-9113-49a2-99c8-226c564c32d5\") " pod="openshift-image-registry/image-registry-95fbf4c66-9k8sq"
Sep 08 15:33:44.841662 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:44.841536    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c4ad30bb-757a-48cc-9d83-e788cf5cebb7-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-vhng9\" (UID: \"c4ad30bb-757a-48cc-9d83-e788cf5cebb7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-vhng9"
Sep 08 15:33:44.841662 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:44.841560    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5f7b0f68-a8e3-4058-b032-15f419682b2a-serving-cert\") pod \"console-operator-85b454874b-wpwdq\" (UID: \"5f7b0f68-a8e3-4058-b032-15f419682b2a\") " pod="openshift-console-operator/console-operator-85b454874b-wpwdq"
Sep 08 15:33:44.841662 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:44.841590    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0ecfb56c-9113-49a2-99c8-226c564c32d5-ca-trust-extracted\") pod \"image-registry-95fbf4c66-9k8sq\" (UID: \"0ecfb56c-9113-49a2-99c8-226c564c32d5\") " pod="openshift-image-registry/image-registry-95fbf4c66-9k8sq"
Sep 08 15:33:44.841662 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:44.841617    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0ecfb56c-9113-49a2-99c8-226c564c32d5-trusted-ca\") pod \"image-registry-95fbf4c66-9k8sq\" (UID: \"0ecfb56c-9113-49a2-99c8-226c564c32d5\") " pod="openshift-image-registry/image-registry-95fbf4c66-9k8sq"
Sep 08 15:33:44.841662 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:44.841655    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0ecfb56c-9113-49a2-99c8-226c564c32d5-image-registry-private-configuration\") pod \"image-registry-95fbf4c66-9k8sq\" (UID: \"0ecfb56c-9113-49a2-99c8-226c564c32d5\") " pod="openshift-image-registry/image-registry-95fbf4c66-9k8sq"
Sep 08 15:33:44.841971 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:44.841702    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8xxz\" (UniqueName: \"kubernetes.io/projected/c4ad30bb-757a-48cc-9d83-e788cf5cebb7-kube-api-access-z8xxz\") pod \"cluster-samples-operator-7cdcfcf76c-vhng9\" (UID: \"c4ad30bb-757a-48cc-9d83-e788cf5cebb7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-vhng9"
Sep 08 15:33:44.841971 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:44.841745    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wm2zh\" (UniqueName: \"kubernetes.io/projected/0ecfb56c-9113-49a2-99c8-226c564c32d5-kube-api-access-wm2zh\") pod \"image-registry-95fbf4c66-9k8sq\" (UID: \"0ecfb56c-9113-49a2-99c8-226c564c32d5\") " pod="openshift-image-registry/image-registry-95fbf4c66-9k8sq"
Sep 08 15:33:44.841971 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:44.841768    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5f7b0f68-a8e3-4058-b032-15f419682b2a-config\") pod \"console-operator-85b454874b-wpwdq\" (UID: \"5f7b0f68-a8e3-4058-b032-15f419682b2a\") " pod="openshift-console-operator/console-operator-85b454874b-wpwdq"
Sep 08 15:33:44.841971 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:44.841797    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0ecfb56c-9113-49a2-99c8-226c564c32d5-bound-sa-token\") pod \"image-registry-95fbf4c66-9k8sq\" (UID: \"0ecfb56c-9113-49a2-99c8-226c564c32d5\") " pod="openshift-image-registry/image-registry-95fbf4c66-9k8sq"
Sep 08 15:33:44.943055 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:44.943028    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0ecfb56c-9113-49a2-99c8-226c564c32d5-installation-pull-secrets\") pod \"image-registry-95fbf4c66-9k8sq\" (UID: \"0ecfb56c-9113-49a2-99c8-226c564c32d5\") " pod="openshift-image-registry/image-registry-95fbf4c66-9k8sq"
Sep 08 15:33:44.943210 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:44.943063    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0ecfb56c-9113-49a2-99c8-226c564c32d5-registry-certificates\") pod \"image-registry-95fbf4c66-9k8sq\" (UID: \"0ecfb56c-9113-49a2-99c8-226c564c32d5\") " pod="openshift-image-registry/image-registry-95fbf4c66-9k8sq"
Sep 08 15:33:44.943210 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:44.943082    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5f7b0f68-a8e3-4058-b032-15f419682b2a-trusted-ca\") pod \"console-operator-85b454874b-wpwdq\" (UID: \"5f7b0f68-a8e3-4058-b032-15f419682b2a\") " pod="openshift-console-operator/console-operator-85b454874b-wpwdq"
Sep 08 15:33:44.943210 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:44.943114    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8k4v\" (UniqueName: \"kubernetes.io/projected/5f7b0f68-a8e3-4058-b032-15f419682b2a-kube-api-access-l8k4v\") pod \"console-operator-85b454874b-wpwdq\" (UID: \"5f7b0f68-a8e3-4058-b032-15f419682b2a\") " pod="openshift-console-operator/console-operator-85b454874b-wpwdq"
Sep 08 15:33:44.943210 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:44.943131    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0ecfb56c-9113-49a2-99c8-226c564c32d5-registry-tls\") pod \"image-registry-95fbf4c66-9k8sq\" (UID: \"0ecfb56c-9113-49a2-99c8-226c564c32d5\") " pod="openshift-image-registry/image-registry-95fbf4c66-9k8sq"
Sep 08 15:33:44.943210 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:44.943145    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c4ad30bb-757a-48cc-9d83-e788cf5cebb7-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-vhng9\" (UID: \"c4ad30bb-757a-48cc-9d83-e788cf5cebb7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-vhng9"
Sep 08 15:33:44.943210 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:44.943168    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5f7b0f68-a8e3-4058-b032-15f419682b2a-serving-cert\") pod \"console-operator-85b454874b-wpwdq\" (UID: \"5f7b0f68-a8e3-4058-b032-15f419682b2a\") " pod="openshift-console-operator/console-operator-85b454874b-wpwdq"
Sep 08 15:33:44.943210 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:44.943185    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0ecfb56c-9113-49a2-99c8-226c564c32d5-ca-trust-extracted\") pod \"image-registry-95fbf4c66-9k8sq\" (UID: \"0ecfb56c-9113-49a2-99c8-226c564c32d5\") " pod="openshift-image-registry/image-registry-95fbf4c66-9k8sq"
Sep 08 15:33:44.943210 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:44.943210    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0ecfb56c-9113-49a2-99c8-226c564c32d5-trusted-ca\") pod \"image-registry-95fbf4c66-9k8sq\" (UID: \"0ecfb56c-9113-49a2-99c8-226c564c32d5\") " pod="openshift-image-registry/image-registry-95fbf4c66-9k8sq"
Sep 08 15:33:44.943614 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:44.943242    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0ecfb56c-9113-49a2-99c8-226c564c32d5-image-registry-private-configuration\") pod \"image-registry-95fbf4c66-9k8sq\" (UID: \"0ecfb56c-9113-49a2-99c8-226c564c32d5\") " pod="openshift-image-registry/image-registry-95fbf4c66-9k8sq"
Sep 08 15:33:44.943614 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:44.943310    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8xxz\" (UniqueName: \"kubernetes.io/projected/c4ad30bb-757a-48cc-9d83-e788cf5cebb7-kube-api-access-z8xxz\") pod \"cluster-samples-operator-7cdcfcf76c-vhng9\" (UID: \"c4ad30bb-757a-48cc-9d83-e788cf5cebb7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-vhng9"
Sep 08 15:33:44.943614 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:44.943350    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wm2zh\" (UniqueName: \"kubernetes.io/projected/0ecfb56c-9113-49a2-99c8-226c564c32d5-kube-api-access-wm2zh\") pod \"image-registry-95fbf4c66-9k8sq\" (UID: \"0ecfb56c-9113-49a2-99c8-226c564c32d5\") " pod="openshift-image-registry/image-registry-95fbf4c66-9k8sq"
Sep 08 15:33:44.943614 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:44.943373    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5f7b0f68-a8e3-4058-b032-15f419682b2a-config\") pod \"console-operator-85b454874b-wpwdq\" (UID: \"5f7b0f68-a8e3-4058-b032-15f419682b2a\") " pod="openshift-console-operator/console-operator-85b454874b-wpwdq"
Sep 08 15:33:44.943614 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:44.943402    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0ecfb56c-9113-49a2-99c8-226c564c32d5-bound-sa-token\") pod \"image-registry-95fbf4c66-9k8sq\" (UID: \"0ecfb56c-9113-49a2-99c8-226c564c32d5\") " pod="openshift-image-registry/image-registry-95fbf4c66-9k8sq"
Sep 08 15:33:44.943614 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:44.943573    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0ecfb56c-9113-49a2-99c8-226c564c32d5-ca-trust-extracted\") pod \"image-registry-95fbf4c66-9k8sq\" (UID: \"0ecfb56c-9113-49a2-99c8-226c564c32d5\") " pod="openshift-image-registry/image-registry-95fbf4c66-9k8sq"
Sep 08 15:33:44.943614 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:33:44.943603    2549 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 08 15:33:44.943968 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:33:44.943664    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c4ad30bb-757a-48cc-9d83-e788cf5cebb7-samples-operator-tls podName:c4ad30bb-757a-48cc-9d83-e788cf5cebb7 nodeName:}" failed. No retries permitted until 2026-09-08 15:33:45.443645577 +0000 UTC m=+63.533096665 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c4ad30bb-757a-48cc-9d83-e788cf5cebb7-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-vhng9" (UID: "c4ad30bb-757a-48cc-9d83-e788cf5cebb7") : secret "samples-operator-tls" not found
Sep 08 15:33:44.943968 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:44.943739    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0ecfb56c-9113-49a2-99c8-226c564c32d5-registry-certificates\") pod \"image-registry-95fbf4c66-9k8sq\" (UID: \"0ecfb56c-9113-49a2-99c8-226c564c32d5\") " pod="openshift-image-registry/image-registry-95fbf4c66-9k8sq"
Sep 08 15:33:44.944078 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:33:44.943964    2549 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 08 15:33:44.944078 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:33:44.943982    2549 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-95fbf4c66-9k8sq: secret "image-registry-tls" not found
Sep 08 15:33:44.944078 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:33:44.944046    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0ecfb56c-9113-49a2-99c8-226c564c32d5-registry-tls podName:0ecfb56c-9113-49a2-99c8-226c564c32d5 nodeName:}" failed. No retries permitted until 2026-09-08 15:33:45.444028909 +0000 UTC m=+63.533479976 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0ecfb56c-9113-49a2-99c8-226c564c32d5-registry-tls") pod "image-registry-95fbf4c66-9k8sq" (UID: "0ecfb56c-9113-49a2-99c8-226c564c32d5") : secret "image-registry-tls" not found
Sep 08 15:33:44.944219 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:44.944143    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0ecfb56c-9113-49a2-99c8-226c564c32d5-trusted-ca\") pod \"image-registry-95fbf4c66-9k8sq\" (UID: \"0ecfb56c-9113-49a2-99c8-226c564c32d5\") " pod="openshift-image-registry/image-registry-95fbf4c66-9k8sq"
Sep 08 15:33:44.944323 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:44.944293    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5f7b0f68-a8e3-4058-b032-15f419682b2a-config\") pod \"console-operator-85b454874b-wpwdq\" (UID: \"5f7b0f68-a8e3-4058-b032-15f419682b2a\") " pod="openshift-console-operator/console-operator-85b454874b-wpwdq"
Sep 08 15:33:44.944534 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:44.944515    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5f7b0f68-a8e3-4058-b032-15f419682b2a-trusted-ca\") pod \"console-operator-85b454874b-wpwdq\" (UID: \"5f7b0f68-a8e3-4058-b032-15f419682b2a\") " pod="openshift-console-operator/console-operator-85b454874b-wpwdq"
Sep 08 15:33:44.945896 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:44.945876    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0ecfb56c-9113-49a2-99c8-226c564c32d5-image-registry-private-configuration\") pod \"image-registry-95fbf4c66-9k8sq\" (UID: \"0ecfb56c-9113-49a2-99c8-226c564c32d5\") " pod="openshift-image-registry/image-registry-95fbf4c66-9k8sq"
Sep 08 15:33:44.945978 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:44.945908    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0ecfb56c-9113-49a2-99c8-226c564c32d5-installation-pull-secrets\") pod \"image-registry-95fbf4c66-9k8sq\" (UID: \"0ecfb56c-9113-49a2-99c8-226c564c32d5\") " pod="openshift-image-registry/image-registry-95fbf4c66-9k8sq"
Sep 08 15:33:44.945978 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:44.945954    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5f7b0f68-a8e3-4058-b032-15f419682b2a-serving-cert\") pod \"console-operator-85b454874b-wpwdq\" (UID: \"5f7b0f68-a8e3-4058-b032-15f419682b2a\") " pod="openshift-console-operator/console-operator-85b454874b-wpwdq"
Sep 08 15:33:44.954123 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:44.954094    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0ecfb56c-9113-49a2-99c8-226c564c32d5-bound-sa-token\") pod \"image-registry-95fbf4c66-9k8sq\" (UID: \"0ecfb56c-9113-49a2-99c8-226c564c32d5\") " pod="openshift-image-registry/image-registry-95fbf4c66-9k8sq"
Sep 08 15:33:44.954357 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:44.954326    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8k4v\" (UniqueName: \"kubernetes.io/projected/5f7b0f68-a8e3-4058-b032-15f419682b2a-kube-api-access-l8k4v\") pod \"console-operator-85b454874b-wpwdq\" (UID: \"5f7b0f68-a8e3-4058-b032-15f419682b2a\") " pod="openshift-console-operator/console-operator-85b454874b-wpwdq"
Sep 08 15:33:44.954448 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:44.954420    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8xxz\" (UniqueName: \"kubernetes.io/projected/c4ad30bb-757a-48cc-9d83-e788cf5cebb7-kube-api-access-z8xxz\") pod \"cluster-samples-operator-7cdcfcf76c-vhng9\" (UID: \"c4ad30bb-757a-48cc-9d83-e788cf5cebb7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-vhng9"
Sep 08 15:33:44.954696 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:44.954679    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wm2zh\" (UniqueName: \"kubernetes.io/projected/0ecfb56c-9113-49a2-99c8-226c564c32d5-kube-api-access-wm2zh\") pod \"image-registry-95fbf4c66-9k8sq\" (UID: \"0ecfb56c-9113-49a2-99c8-226c564c32d5\") " pod="openshift-image-registry/image-registry-95fbf4c66-9k8sq"
Sep 08 15:33:44.991780 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:44.991732    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-85b454874b-wpwdq"
Sep 08 15:33:45.050272 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:45.050245    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-7mfs7"]
Sep 08 15:33:45.260080 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:45.260019    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-85b454874b-wpwdq"]
Sep 08 15:33:45.262570 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:33:45.262550    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f7b0f68_a8e3_4058_b032_15f419682b2a.slice/crio-ee8e7b3b9b95ae1dea88f9eda9abf7b1503e0267c21d8cd4b3b5ced6c5af5bb0 WatchSource:0}: Error finding container ee8e7b3b9b95ae1dea88f9eda9abf7b1503e0267c21d8cd4b3b5ced6c5af5bb0: Status 404 returned error can't find the container with id ee8e7b3b9b95ae1dea88f9eda9abf7b1503e0267c21d8cd4b3b5ced6c5af5bb0
Sep 08 15:33:45.446250 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:45.446225    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0ecfb56c-9113-49a2-99c8-226c564c32d5-registry-tls\") pod \"image-registry-95fbf4c66-9k8sq\" (UID: \"0ecfb56c-9113-49a2-99c8-226c564c32d5\") " pod="openshift-image-registry/image-registry-95fbf4c66-9k8sq"
Sep 08 15:33:45.446373 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:45.446255    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c4ad30bb-757a-48cc-9d83-e788cf5cebb7-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-vhng9\" (UID: \"c4ad30bb-757a-48cc-9d83-e788cf5cebb7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-vhng9"
Sep 08 15:33:45.446373 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:33:45.446363    2549 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 08 15:33:45.446446 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:33:45.446380    2549 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 08 15:33:45.446446 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:33:45.446424    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c4ad30bb-757a-48cc-9d83-e788cf5cebb7-samples-operator-tls podName:c4ad30bb-757a-48cc-9d83-e788cf5cebb7 nodeName:}" failed. No retries permitted until 2026-09-08 15:33:46.44641078 +0000 UTC m=+64.535861847 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c4ad30bb-757a-48cc-9d83-e788cf5cebb7-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-vhng9" (UID: "c4ad30bb-757a-48cc-9d83-e788cf5cebb7") : secret "samples-operator-tls" not found
Sep 08 15:33:45.446529 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:33:45.446382    2549 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-95fbf4c66-9k8sq: secret "image-registry-tls" not found
Sep 08 15:33:45.446529 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:33:45.446500    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0ecfb56c-9113-49a2-99c8-226c564c32d5-registry-tls podName:0ecfb56c-9113-49a2-99c8-226c564c32d5 nodeName:}" failed. No retries permitted until 2026-09-08 15:33:46.446487151 +0000 UTC m=+64.535938234 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0ecfb56c-9113-49a2-99c8-226c564c32d5-registry-tls") pod "image-registry-95fbf4c66-9k8sq" (UID: "0ecfb56c-9113-49a2-99c8-226c564c32d5") : secret "image-registry-tls" not found
Sep 08 15:33:45.650046 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:45.650014    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-wpwdq" event={"ID":"5f7b0f68-a8e3-4058-b032-15f419682b2a","Type":"ContainerStarted","Data":"ee8e7b3b9b95ae1dea88f9eda9abf7b1503e0267c21d8cd4b3b5ced6c5af5bb0"}
Sep 08 15:33:45.651338 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:45.651306    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-7mfs7" event={"ID":"d1d7e194-2f46-486c-aa08-f87ef377f54f","Type":"ContainerStarted","Data":"69863b0e8bb0b7bd2b5f93d64917c0c446dd430531b1cf82621211a697a0bfe0"}
Sep 08 15:33:46.452716 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:46.452682    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0ecfb56c-9113-49a2-99c8-226c564c32d5-registry-tls\") pod \"image-registry-95fbf4c66-9k8sq\" (UID: \"0ecfb56c-9113-49a2-99c8-226c564c32d5\") " pod="openshift-image-registry/image-registry-95fbf4c66-9k8sq"
Sep 08 15:33:46.452918 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:46.452730    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c4ad30bb-757a-48cc-9d83-e788cf5cebb7-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-vhng9\" (UID: \"c4ad30bb-757a-48cc-9d83-e788cf5cebb7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-vhng9"
Sep 08 15:33:46.452918 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:33:46.452839    2549 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 08 15:33:46.452918 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:33:46.452875    2549 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-95fbf4c66-9k8sq: secret "image-registry-tls" not found
Sep 08 15:33:46.452918 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:33:46.452882    2549 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 08 15:33:46.453089 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:33:46.452948    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0ecfb56c-9113-49a2-99c8-226c564c32d5-registry-tls podName:0ecfb56c-9113-49a2-99c8-226c564c32d5 nodeName:}" failed. No retries permitted until 2026-09-08 15:33:48.45292891 +0000 UTC m=+66.542379989 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0ecfb56c-9113-49a2-99c8-226c564c32d5-registry-tls") pod "image-registry-95fbf4c66-9k8sq" (UID: "0ecfb56c-9113-49a2-99c8-226c564c32d5") : secret "image-registry-tls" not found
Sep 08 15:33:46.453089 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:33:46.452968    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c4ad30bb-757a-48cc-9d83-e788cf5cebb7-samples-operator-tls podName:c4ad30bb-757a-48cc-9d83-e788cf5cebb7 nodeName:}" failed. No retries permitted until 2026-09-08 15:33:48.452958788 +0000 UTC m=+66.542409865 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c4ad30bb-757a-48cc-9d83-e788cf5cebb7-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-vhng9" (UID: "c4ad30bb-757a-48cc-9d83-e788cf5cebb7") : secret "samples-operator-tls" not found
Sep 08 15:33:47.159082 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:47.159043    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/84663ea3-1feb-4c87-8609-c580fa60416e-original-pull-secret\") pod \"global-pull-secret-syncer-fw5jh\" (UID: \"84663ea3-1feb-4c87-8609-c580fa60416e\") " pod="kube-system/global-pull-secret-syncer-fw5jh"
Sep 08 15:33:47.159551 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:47.159223    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d4d759e-9d3c-497a-b546-462ae2b48785-metrics-certs\") pod \"network-metrics-daemon-h4cml\" (UID: \"7d4d759e-9d3c-497a-b546-462ae2b48785\") " pod="openshift-multus/network-metrics-daemon-h4cml"
Sep 08 15:33:47.163660 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:47.163631    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 08 15:33:47.163790 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:47.163678    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 08 15:33:47.170158 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:33:47.170125    2549 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 08 15:33:47.170272 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:33:47.170206    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d4d759e-9d3c-497a-b546-462ae2b48785-metrics-certs podName:7d4d759e-9d3c-497a-b546-462ae2b48785 nodeName:}" failed. No retries permitted until 2026-09-08 15:34:51.170184656 +0000 UTC m=+129.259635724 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7d4d759e-9d3c-497a-b546-462ae2b48785-metrics-certs") pod "network-metrics-daemon-h4cml" (UID: "7d4d759e-9d3c-497a-b546-462ae2b48785") : secret "metrics-daemon-secret" not found
Sep 08 15:33:47.172581 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:47.172560    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/84663ea3-1feb-4c87-8609-c580fa60416e-original-pull-secret\") pod \"global-pull-secret-syncer-fw5jh\" (UID: \"84663ea3-1feb-4c87-8609-c580fa60416e\") " pod="kube-system/global-pull-secret-syncer-fw5jh"
Sep 08 15:33:47.259783 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:47.259751    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79p6n\" (UniqueName: \"kubernetes.io/projected/3d62444d-76ba-4184-869c-496ba4da8d04-kube-api-access-79p6n\") pod \"network-check-target-5vt6c\" (UID: \"3d62444d-76ba-4184-869c-496ba4da8d04\") " pod="openshift-network-diagnostics/network-check-target-5vt6c"
Sep 08 15:33:47.262987 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:47.262955    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 08 15:33:47.273200 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:47.273173    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 08 15:33:47.283102 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:47.283082    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-79p6n\" (UniqueName: \"kubernetes.io/projected/3d62444d-76ba-4184-869c-496ba4da8d04-kube-api-access-79p6n\") pod \"network-check-target-5vt6c\" (UID: \"3d62444d-76ba-4184-869c-496ba4da8d04\") " pod="openshift-network-diagnostics/network-check-target-5vt6c"
Sep 08 15:33:47.309801 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:47.309780    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fw5jh"
Sep 08 15:33:47.319598 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:47.319574    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-xt4hq\""
Sep 08 15:33:47.326094 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:47.326075    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5vt6c"
Sep 08 15:33:47.658554 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:47.658534    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-wpwdq_5f7b0f68-a8e3-4058-b032-15f419682b2a/console-operator/0.log"
Sep 08 15:33:47.658729 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:47.658573    2549 generic.go:356] "Generic (PLEG): container finished" podID="5f7b0f68-a8e3-4058-b032-15f419682b2a" containerID="22f852118e0a690600bac6cb75b6039e2ba96c8711a87c8f3d00622b064ab70f" exitCode=255
Sep 08 15:33:47.658729 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:47.658604    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-wpwdq" event={"ID":"5f7b0f68-a8e3-4058-b032-15f419682b2a","Type":"ContainerDied","Data":"22f852118e0a690600bac6cb75b6039e2ba96c8711a87c8f3d00622b064ab70f"}
Sep 08 15:33:47.658893 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:47.658877    2549 scope.go:117] "RemoveContainer" containerID="22f852118e0a690600bac6cb75b6039e2ba96c8711a87c8f3d00622b064ab70f"
Sep 08 15:33:47.660190 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:47.660163    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-7mfs7" event={"ID":"d1d7e194-2f46-486c-aa08-f87ef377f54f","Type":"ContainerStarted","Data":"1d9b3fda97b5f50803637ff4e564b84647d5f2f39ece21b97c6fb0895619b65c"}
Sep 08 15:33:47.693954 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:47.693912    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-7mfs7" podStartSLOduration=1.275994007 podStartE2EDuration="3.693899169s" totalImagesPullingTime="2.417905162s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:33:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:33:47.692795556 +0000 UTC m=+65.782246645" watchObservedRunningTime="2026-09-08 15:33:47.693899169 +0000 UTC m=+65.783350255"
Sep 08 15:33:47.748468 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:47.748398    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-fw5jh"]
Sep 08 15:33:47.749602 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:47.749541    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-5vt6c"]
Sep 08 15:33:47.751138 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:33:47.751112    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d62444d_76ba_4184_869c_496ba4da8d04.slice/crio-f1da652e2453e213deb4634f3273f1e4b3144caa4fa1cb2a490b9b22b90267a3 WatchSource:0}: Error finding container f1da652e2453e213deb4634f3273f1e4b3144caa4fa1cb2a490b9b22b90267a3: Status 404 returned error can't find the container with id f1da652e2453e213deb4634f3273f1e4b3144caa4fa1cb2a490b9b22b90267a3
Sep 08 15:33:47.752026 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:33:47.751960    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84663ea3_1feb_4c87_8609_c580fa60416e.slice/crio-5b0c3667d24cf0e2653a2899f881754851c1e8b5b0ccd18a8d657e37bf5430af WatchSource:0}: Error finding container 5b0c3667d24cf0e2653a2899f881754851c1e8b5b0ccd18a8d657e37bf5430af: Status 404 returned error can't find the container with id 5b0c3667d24cf0e2653a2899f881754851c1e8b5b0ccd18a8d657e37bf5430af
Sep 08 15:33:48.066165 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:48.066089    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/57cef312-9005-43bc-9e2f-b59db9e864c3-metrics-tls\") pod \"dns-default-vhsng\" (UID: \"57cef312-9005-43bc-9e2f-b59db9e864c3\") " pod="openshift-dns/dns-default-vhsng"
Sep 08 15:33:48.066290 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:33:48.066200    2549 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 15:33:48.066290 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:33:48.066251    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57cef312-9005-43bc-9e2f-b59db9e864c3-metrics-tls podName:57cef312-9005-43bc-9e2f-b59db9e864c3 nodeName:}" failed. No retries permitted until 2026-09-08 15:34:20.066237828 +0000 UTC m=+98.155688909 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/57cef312-9005-43bc-9e2f-b59db9e864c3-metrics-tls") pod "dns-default-vhsng" (UID: "57cef312-9005-43bc-9e2f-b59db9e864c3") : secret "dns-default-metrics-tls" not found
Sep 08 15:33:48.080846 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:48.080820    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-6tn6s"]
Sep 08 15:33:48.083845 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:48.083828    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-6tn6s"
Sep 08 15:33:48.088710 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:48.088685    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 08 15:33:48.088817 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:48.088721    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 08 15:33:48.088817 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:48.088764    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 08 15:33:48.089084 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:48.089067    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 08 15:33:48.089166 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:48.089071    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-l67b9\""
Sep 08 15:33:48.099542 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:48.099500    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-6tn6s"]
Sep 08 15:33:48.166465 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:48.166436    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3b1fd2bf-0e75-4f18-b224-db2a474aba0e-data-volume\") pod \"insights-runtime-extractor-6tn6s\" (UID: \"3b1fd2bf-0e75-4f18-b224-db2a474aba0e\") " pod="openshift-insights/insights-runtime-extractor-6tn6s"
Sep 08 15:33:48.166825 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:48.166467    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3b1fd2bf-0e75-4f18-b224-db2a474aba0e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6tn6s\" (UID: \"3b1fd2bf-0e75-4f18-b224-db2a474aba0e\") " pod="openshift-insights/insights-runtime-extractor-6tn6s"
Sep 08 15:33:48.166825 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:48.166514    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3b1fd2bf-0e75-4f18-b224-db2a474aba0e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6tn6s\" (UID: \"3b1fd2bf-0e75-4f18-b224-db2a474aba0e\") " pod="openshift-insights/insights-runtime-extractor-6tn6s"
Sep 08 15:33:48.166825 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:48.166625    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3b1fd2bf-0e75-4f18-b224-db2a474aba0e-crio-socket\") pod \"insights-runtime-extractor-6tn6s\" (UID: \"3b1fd2bf-0e75-4f18-b224-db2a474aba0e\") " pod="openshift-insights/insights-runtime-extractor-6tn6s"
Sep 08 15:33:48.166825 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:48.166661    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6p6pd\" (UniqueName: \"kubernetes.io/projected/3b1fd2bf-0e75-4f18-b224-db2a474aba0e-kube-api-access-6p6pd\") pod \"insights-runtime-extractor-6tn6s\" (UID: \"3b1fd2bf-0e75-4f18-b224-db2a474aba0e\") " pod="openshift-insights/insights-runtime-extractor-6tn6s"
Sep 08 15:33:48.267109 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:48.267079    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3b1fd2bf-0e75-4f18-b224-db2a474aba0e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6tn6s\" (UID: \"3b1fd2bf-0e75-4f18-b224-db2a474aba0e\") " pod="openshift-insights/insights-runtime-extractor-6tn6s"
Sep 08 15:33:48.267321 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:48.267120    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3b1fd2bf-0e75-4f18-b224-db2a474aba0e-crio-socket\") pod \"insights-runtime-extractor-6tn6s\" (UID: \"3b1fd2bf-0e75-4f18-b224-db2a474aba0e\") " pod="openshift-insights/insights-runtime-extractor-6tn6s"
Sep 08 15:33:48.267321 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:48.267150    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6p6pd\" (UniqueName: \"kubernetes.io/projected/3b1fd2bf-0e75-4f18-b224-db2a474aba0e-kube-api-access-6p6pd\") pod \"insights-runtime-extractor-6tn6s\" (UID: \"3b1fd2bf-0e75-4f18-b224-db2a474aba0e\") " pod="openshift-insights/insights-runtime-extractor-6tn6s"
Sep 08 15:33:48.267321 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:33:48.267253    2549 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 08 15:33:48.267321 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:48.267269    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3b1fd2bf-0e75-4f18-b224-db2a474aba0e-data-volume\") pod \"insights-runtime-extractor-6tn6s\" (UID: \"3b1fd2bf-0e75-4f18-b224-db2a474aba0e\") " pod="openshift-insights/insights-runtime-extractor-6tn6s"
Sep 08 15:33:48.267321 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:48.267294    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3b1fd2bf-0e75-4f18-b224-db2a474aba0e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6tn6s\" (UID: \"3b1fd2bf-0e75-4f18-b224-db2a474aba0e\") " pod="openshift-insights/insights-runtime-extractor-6tn6s"
Sep 08 15:33:48.267599 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:33:48.267342    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b1fd2bf-0e75-4f18-b224-db2a474aba0e-insights-runtime-extractor-tls podName:3b1fd2bf-0e75-4f18-b224-db2a474aba0e nodeName:}" failed. No retries permitted until 2026-09-08 15:33:48.767321851 +0000 UTC m=+66.856772926 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3b1fd2bf-0e75-4f18-b224-db2a474aba0e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6tn6s" (UID: "3b1fd2bf-0e75-4f18-b224-db2a474aba0e") : secret "insights-runtime-extractor-tls" not found
Sep 08 15:33:48.267599 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:33:48.267372    2549 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 08 15:33:48.267599 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:33:48.267433    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3b1fd2bf-0e75-4f18-b224-db2a474aba0e-kube-rbac-proxy-cm podName:3b1fd2bf-0e75-4f18-b224-db2a474aba0e nodeName:}" failed. No retries permitted until 2026-09-08 15:33:48.767418176 +0000 UTC m=+66.856869256 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3b1fd2bf-0e75-4f18-b224-db2a474aba0e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6tn6s" (UID: "3b1fd2bf-0e75-4f18-b224-db2a474aba0e") : configmap "kube-rbac-proxy" not found
Sep 08 15:33:48.267599 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:48.267256    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3b1fd2bf-0e75-4f18-b224-db2a474aba0e-crio-socket\") pod \"insights-runtime-extractor-6tn6s\" (UID: \"3b1fd2bf-0e75-4f18-b224-db2a474aba0e\") " pod="openshift-insights/insights-runtime-extractor-6tn6s"
Sep 08 15:33:48.267814 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:48.267689    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3b1fd2bf-0e75-4f18-b224-db2a474aba0e-data-volume\") pod \"insights-runtime-extractor-6tn6s\" (UID: \"3b1fd2bf-0e75-4f18-b224-db2a474aba0e\") " pod="openshift-insights/insights-runtime-extractor-6tn6s"
Sep 08 15:33:48.276966 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:48.276943    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6p6pd\" (UniqueName: \"kubernetes.io/projected/3b1fd2bf-0e75-4f18-b224-db2a474aba0e-kube-api-access-6p6pd\") pod \"insights-runtime-extractor-6tn6s\" (UID: \"3b1fd2bf-0e75-4f18-b224-db2a474aba0e\") " pod="openshift-insights/insights-runtime-extractor-6tn6s"
Sep 08 15:33:48.468694 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:48.468659    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0ecfb56c-9113-49a2-99c8-226c564c32d5-registry-tls\") pod \"image-registry-95fbf4c66-9k8sq\" (UID: \"0ecfb56c-9113-49a2-99c8-226c564c32d5\") " pod="openshift-image-registry/image-registry-95fbf4c66-9k8sq"
Sep 08 15:33:48.468883 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:48.468706    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c4ad30bb-757a-48cc-9d83-e788cf5cebb7-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-vhng9\" (UID: \"c4ad30bb-757a-48cc-9d83-e788cf5cebb7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-vhng9"
Sep 08 15:33:48.468960 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:33:48.468877    2549 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 08 15:33:48.468960 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:33:48.468900    2549 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-95fbf4c66-9k8sq: secret "image-registry-tls" not found
Sep 08 15:33:48.468960 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:33:48.468938    2549 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 08 15:33:48.469124 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:33:48.468986    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0ecfb56c-9113-49a2-99c8-226c564c32d5-registry-tls podName:0ecfb56c-9113-49a2-99c8-226c564c32d5 nodeName:}" failed. No retries permitted until 2026-09-08 15:33:52.468963273 +0000 UTC m=+70.558414351 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0ecfb56c-9113-49a2-99c8-226c564c32d5-registry-tls") pod "image-registry-95fbf4c66-9k8sq" (UID: "0ecfb56c-9113-49a2-99c8-226c564c32d5") : secret "image-registry-tls" not found
Sep 08 15:33:48.469124 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:33:48.469004    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c4ad30bb-757a-48cc-9d83-e788cf5cebb7-samples-operator-tls podName:c4ad30bb-757a-48cc-9d83-e788cf5cebb7 nodeName:}" failed. No retries permitted until 2026-09-08 15:33:52.46899401 +0000 UTC m=+70.558445094 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c4ad30bb-757a-48cc-9d83-e788cf5cebb7-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-vhng9" (UID: "c4ad30bb-757a-48cc-9d83-e788cf5cebb7") : secret "samples-operator-tls" not found
Sep 08 15:33:48.665329 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:48.665269    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-5vt6c" event={"ID":"3d62444d-76ba-4184-869c-496ba4da8d04","Type":"ContainerStarted","Data":"f1da652e2453e213deb4634f3273f1e4b3144caa4fa1cb2a490b9b22b90267a3"}
Sep 08 15:33:48.667007 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:48.666977    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-fw5jh" event={"ID":"84663ea3-1feb-4c87-8609-c580fa60416e","Type":"ContainerStarted","Data":"5b0c3667d24cf0e2653a2899f881754851c1e8b5b0ccd18a8d657e37bf5430af"}
Sep 08 15:33:48.669116 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:48.669089    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-wpwdq_5f7b0f68-a8e3-4058-b032-15f419682b2a/console-operator/1.log"
Sep 08 15:33:48.669664 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:48.669646    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-wpwdq_5f7b0f68-a8e3-4058-b032-15f419682b2a/console-operator/0.log"
Sep 08 15:33:48.669744 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:48.669687    2549 generic.go:356] "Generic (PLEG): container finished" podID="5f7b0f68-a8e3-4058-b032-15f419682b2a" containerID="fe0f67b144eef7d9a35b32180e0cd385f9acfefa04bb2af361dc8aa4561c3abb" exitCode=255
Sep 08 15:33:48.670905 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:48.670264    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-wpwdq" event={"ID":"5f7b0f68-a8e3-4058-b032-15f419682b2a","Type":"ContainerDied","Data":"fe0f67b144eef7d9a35b32180e0cd385f9acfefa04bb2af361dc8aa4561c3abb"}
Sep 08 15:33:48.670905 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:48.670443    2549 scope.go:117] "RemoveContainer" containerID="fe0f67b144eef7d9a35b32180e0cd385f9acfefa04bb2af361dc8aa4561c3abb"
Sep 08 15:33:48.670905 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:33:48.670588    2549 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-wpwdq_openshift-console-operator(5f7b0f68-a8e3-4058-b032-15f419682b2a)\"" pod="openshift-console-operator/console-operator-85b454874b-wpwdq" podUID="5f7b0f68-a8e3-4058-b032-15f419682b2a"
Sep 08 15:33:48.670905 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:48.670650    2549 scope.go:117] "RemoveContainer" containerID="22f852118e0a690600bac6cb75b6039e2ba96c8711a87c8f3d00622b064ab70f"
Sep 08 15:33:48.771314 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:48.771239    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3b1fd2bf-0e75-4f18-b224-db2a474aba0e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6tn6s\" (UID: \"3b1fd2bf-0e75-4f18-b224-db2a474aba0e\") " pod="openshift-insights/insights-runtime-extractor-6tn6s"
Sep 08 15:33:48.771458 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:33:48.771317    2549 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 08 15:33:48.771458 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:48.771347    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3b1fd2bf-0e75-4f18-b224-db2a474aba0e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6tn6s\" (UID: \"3b1fd2bf-0e75-4f18-b224-db2a474aba0e\") " pod="openshift-insights/insights-runtime-extractor-6tn6s"
Sep 08 15:33:48.771458 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:33:48.771386    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3b1fd2bf-0e75-4f18-b224-db2a474aba0e-kube-rbac-proxy-cm podName:3b1fd2bf-0e75-4f18-b224-db2a474aba0e nodeName:}" failed. No retries permitted until 2026-09-08 15:33:49.771367484 +0000 UTC m=+67.860818551 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3b1fd2bf-0e75-4f18-b224-db2a474aba0e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6tn6s" (UID: "3b1fd2bf-0e75-4f18-b224-db2a474aba0e") : configmap "kube-rbac-proxy" not found
Sep 08 15:33:48.771458 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:33:48.771426    2549 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 08 15:33:48.771673 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:33:48.771473    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b1fd2bf-0e75-4f18-b224-db2a474aba0e-insights-runtime-extractor-tls podName:3b1fd2bf-0e75-4f18-b224-db2a474aba0e nodeName:}" failed. No retries permitted until 2026-09-08 15:33:49.771458268 +0000 UTC m=+67.860909347 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3b1fd2bf-0e75-4f18-b224-db2a474aba0e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6tn6s" (UID: "3b1fd2bf-0e75-4f18-b224-db2a474aba0e") : secret "insights-runtime-extractor-tls" not found
Sep 08 15:33:49.674616 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:49.674582    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-wpwdq_5f7b0f68-a8e3-4058-b032-15f419682b2a/console-operator/1.log"
Sep 08 15:33:49.675033 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:49.675015    2549 scope.go:117] "RemoveContainer" containerID="fe0f67b144eef7d9a35b32180e0cd385f9acfefa04bb2af361dc8aa4561c3abb"
Sep 08 15:33:49.675241 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:33:49.675219    2549 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-wpwdq_openshift-console-operator(5f7b0f68-a8e3-4058-b032-15f419682b2a)\"" pod="openshift-console-operator/console-operator-85b454874b-wpwdq" podUID="5f7b0f68-a8e3-4058-b032-15f419682b2a"
Sep 08 15:33:49.779129 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:49.779092    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3b1fd2bf-0e75-4f18-b224-db2a474aba0e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6tn6s\" (UID: \"3b1fd2bf-0e75-4f18-b224-db2a474aba0e\") " pod="openshift-insights/insights-runtime-extractor-6tn6s"
Sep 08 15:33:49.779285 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:49.779177    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3b1fd2bf-0e75-4f18-b224-db2a474aba0e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6tn6s\" (UID: \"3b1fd2bf-0e75-4f18-b224-db2a474aba0e\") " pod="openshift-insights/insights-runtime-extractor-6tn6s"
Sep 08 15:33:49.779339 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:33:49.779320    2549 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 08 15:33:49.779399 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:33:49.779392    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b1fd2bf-0e75-4f18-b224-db2a474aba0e-insights-runtime-extractor-tls podName:3b1fd2bf-0e75-4f18-b224-db2a474aba0e nodeName:}" failed. No retries permitted until 2026-09-08 15:33:51.779371602 +0000 UTC m=+69.868822691 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3b1fd2bf-0e75-4f18-b224-db2a474aba0e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6tn6s" (UID: "3b1fd2bf-0e75-4f18-b224-db2a474aba0e") : secret "insights-runtime-extractor-tls" not found
Sep 08 15:33:49.779547 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:33:49.779515    2549 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 08 15:33:49.779620 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:33:49.779602    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3b1fd2bf-0e75-4f18-b224-db2a474aba0e-kube-rbac-proxy-cm podName:3b1fd2bf-0e75-4f18-b224-db2a474aba0e nodeName:}" failed. No retries permitted until 2026-09-08 15:33:51.779585386 +0000 UTC m=+69.869036486 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3b1fd2bf-0e75-4f18-b224-db2a474aba0e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6tn6s" (UID: "3b1fd2bf-0e75-4f18-b224-db2a474aba0e") : configmap "kube-rbac-proxy" not found
Sep 08 15:33:50.230794 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:50.230766    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-vp9nk_a955f56c-5ac8-422a-8ddc-ce17a8361eb6/dns-node-resolver/0.log"
Sep 08 15:33:50.830935 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:50.830899    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-6q7cq_5f9598ad-fb27-4dda-ab17-98d54cbd30aa/node-ca/0.log"
Sep 08 15:33:51.794500 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:51.794460    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3b1fd2bf-0e75-4f18-b224-db2a474aba0e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6tn6s\" (UID: \"3b1fd2bf-0e75-4f18-b224-db2a474aba0e\") " pod="openshift-insights/insights-runtime-extractor-6tn6s"
Sep 08 15:33:51.794667 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:51.794517    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3b1fd2bf-0e75-4f18-b224-db2a474aba0e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6tn6s\" (UID: \"3b1fd2bf-0e75-4f18-b224-db2a474aba0e\") " pod="openshift-insights/insights-runtime-extractor-6tn6s"
Sep 08 15:33:51.794667 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:33:51.794580    2549 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 08 15:33:51.794667 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:33:51.794625    2549 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 08 15:33:51.794667 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:33:51.794654    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3b1fd2bf-0e75-4f18-b224-db2a474aba0e-kube-rbac-proxy-cm podName:3b1fd2bf-0e75-4f18-b224-db2a474aba0e nodeName:}" failed. No retries permitted until 2026-09-08 15:33:55.794638361 +0000 UTC m=+73.884089433 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3b1fd2bf-0e75-4f18-b224-db2a474aba0e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6tn6s" (UID: "3b1fd2bf-0e75-4f18-b224-db2a474aba0e") : configmap "kube-rbac-proxy" not found
Sep 08 15:33:51.794811 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:33:51.794673    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b1fd2bf-0e75-4f18-b224-db2a474aba0e-insights-runtime-extractor-tls podName:3b1fd2bf-0e75-4f18-b224-db2a474aba0e nodeName:}" failed. No retries permitted until 2026-09-08 15:33:55.79466026 +0000 UTC m=+73.884111331 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3b1fd2bf-0e75-4f18-b224-db2a474aba0e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6tn6s" (UID: "3b1fd2bf-0e75-4f18-b224-db2a474aba0e") : secret "insights-runtime-extractor-tls" not found
Sep 08 15:33:52.031321 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:52.031291    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-tt5hr_0e985291-302f-4109-b4ba-8569a56235e4/serve-healthcheck-canary/0.log"
Sep 08 15:33:52.500619 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:52.500579    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0ecfb56c-9113-49a2-99c8-226c564c32d5-registry-tls\") pod \"image-registry-95fbf4c66-9k8sq\" (UID: \"0ecfb56c-9113-49a2-99c8-226c564c32d5\") " pod="openshift-image-registry/image-registry-95fbf4c66-9k8sq"
Sep 08 15:33:52.500619 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:52.500620    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c4ad30bb-757a-48cc-9d83-e788cf5cebb7-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-vhng9\" (UID: \"c4ad30bb-757a-48cc-9d83-e788cf5cebb7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-vhng9"
Sep 08 15:33:52.500887 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:33:52.500726    2549 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 08 15:33:52.500887 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:33:52.500741    2549 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 08 15:33:52.500887 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:33:52.500764    2549 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-95fbf4c66-9k8sq: secret "image-registry-tls" not found
Sep 08 15:33:52.500887 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:33:52.500777    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c4ad30bb-757a-48cc-9d83-e788cf5cebb7-samples-operator-tls podName:c4ad30bb-757a-48cc-9d83-e788cf5cebb7 nodeName:}" failed. No retries permitted until 2026-09-08 15:34:00.500763417 +0000 UTC m=+78.590214483 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c4ad30bb-757a-48cc-9d83-e788cf5cebb7-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-vhng9" (UID: "c4ad30bb-757a-48cc-9d83-e788cf5cebb7") : secret "samples-operator-tls" not found
Sep 08 15:33:52.500887 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:33:52.500815    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0ecfb56c-9113-49a2-99c8-226c564c32d5-registry-tls podName:0ecfb56c-9113-49a2-99c8-226c564c32d5 nodeName:}" failed. No retries permitted until 2026-09-08 15:34:00.500797521 +0000 UTC m=+78.590248606 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0ecfb56c-9113-49a2-99c8-226c564c32d5-registry-tls") pod "image-registry-95fbf4c66-9k8sq" (UID: "0ecfb56c-9113-49a2-99c8-226c564c32d5") : secret "image-registry-tls" not found
Sep 08 15:33:52.685746 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:52.685705    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-fw5jh" event={"ID":"84663ea3-1feb-4c87-8609-c580fa60416e","Type":"ContainerStarted","Data":"7c9ec1481e6f5571b9fae4a7524876756664d6b9e0d0e35b653e7b826480ecad"}
Sep 08 15:33:52.687060 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:52.687038    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-5vt6c" event={"ID":"3d62444d-76ba-4184-869c-496ba4da8d04","Type":"ContainerStarted","Data":"31aa6a4d4184149ded0f12da3ebb0c3b1c0c1a6b307ada81b1817c6c99c711c5"}
Sep 08 15:33:52.687189 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:52.687162    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-5vt6c"
Sep 08 15:33:52.704947 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:52.704904    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-fw5jh" podStartSLOduration=65.147555201 podStartE2EDuration="1m9.704889792s" totalImagesPullingTime="4.557334591s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 15:32:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:33:52.704377143 +0000 UTC m=+70.793828231" watchObservedRunningTime="2026-09-08 15:33:52.704889792 +0000 UTC m=+70.794340880"
Sep 08 15:33:54.991995 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:54.991959    2549 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-85b454874b-wpwdq"
Sep 08 15:33:54.992348 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:54.992007    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-wpwdq"
Sep 08 15:33:54.995543 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:54.992598    2549 scope.go:117] "RemoveContainer" containerID="fe0f67b144eef7d9a35b32180e0cd385f9acfefa04bb2af361dc8aa4561c3abb"
Sep 08 15:33:54.995543 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:33:54.992883    2549 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-wpwdq_openshift-console-operator(5f7b0f68-a8e3-4058-b032-15f419682b2a)\"" pod="openshift-console-operator/console-operator-85b454874b-wpwdq" podUID="5f7b0f68-a8e3-4058-b032-15f419682b2a"
Sep 08 15:33:55.826594 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:55.826558    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3b1fd2bf-0e75-4f18-b224-db2a474aba0e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6tn6s\" (UID: \"3b1fd2bf-0e75-4f18-b224-db2a474aba0e\") " pod="openshift-insights/insights-runtime-extractor-6tn6s"
Sep 08 15:33:55.826759 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:33:55.826609    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3b1fd2bf-0e75-4f18-b224-db2a474aba0e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6tn6s\" (UID: \"3b1fd2bf-0e75-4f18-b224-db2a474aba0e\") " pod="openshift-insights/insights-runtime-extractor-6tn6s"
Sep 08 15:33:55.826759 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:33:55.826668    2549 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 08 15:33:55.826759 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:33:55.826728    2549 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 08 15:33:55.826759 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:33:55.826738    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3b1fd2bf-0e75-4f18-b224-db2a474aba0e-kube-rbac-proxy-cm podName:3b1fd2bf-0e75-4f18-b224-db2a474aba0e nodeName:}" failed. No retries permitted until 2026-09-08 15:34:03.826724016 +0000 UTC m=+81.916175082 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3b1fd2bf-0e75-4f18-b224-db2a474aba0e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6tn6s" (UID: "3b1fd2bf-0e75-4f18-b224-db2a474aba0e") : configmap "kube-rbac-proxy" not found
Sep 08 15:33:55.826922 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:33:55.826776    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b1fd2bf-0e75-4f18-b224-db2a474aba0e-insights-runtime-extractor-tls podName:3b1fd2bf-0e75-4f18-b224-db2a474aba0e nodeName:}" failed. No retries permitted until 2026-09-08 15:34:03.826764951 +0000 UTC m=+81.916216022 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3b1fd2bf-0e75-4f18-b224-db2a474aba0e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6tn6s" (UID: "3b1fd2bf-0e75-4f18-b224-db2a474aba0e") : secret "insights-runtime-extractor-tls" not found
Sep 08 15:34:00.561463 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:00.561435    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0ecfb56c-9113-49a2-99c8-226c564c32d5-registry-tls\") pod \"image-registry-95fbf4c66-9k8sq\" (UID: \"0ecfb56c-9113-49a2-99c8-226c564c32d5\") " pod="openshift-image-registry/image-registry-95fbf4c66-9k8sq"
Sep 08 15:34:00.561463 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:00.561466    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c4ad30bb-757a-48cc-9d83-e788cf5cebb7-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-vhng9\" (UID: \"c4ad30bb-757a-48cc-9d83-e788cf5cebb7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-vhng9"
Sep 08 15:34:00.563610 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:00.563579    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c4ad30bb-757a-48cc-9d83-e788cf5cebb7-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-vhng9\" (UID: \"c4ad30bb-757a-48cc-9d83-e788cf5cebb7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-vhng9"
Sep 08 15:34:00.563610 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:00.563582    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0ecfb56c-9113-49a2-99c8-226c564c32d5-registry-tls\") pod \"image-registry-95fbf4c66-9k8sq\" (UID: \"0ecfb56c-9113-49a2-99c8-226c564c32d5\") " pod="openshift-image-registry/image-registry-95fbf4c66-9k8sq"
Sep 08 15:34:00.581468 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:00.581443    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-vhng9"
Sep 08 15:34:00.596301 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:00.596277    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-95fbf4c66-9k8sq"
Sep 08 15:34:00.850543 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:00.850436    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-5vt6c" podStartSLOduration=73.295470972 podStartE2EDuration="1m17.850420548s" totalImagesPullingTime="4.554949576s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:32:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:33:52.731357536 +0000 UTC m=+70.820808625" watchObservedRunningTime="2026-09-08 15:34:00.850420548 +0000 UTC m=+78.939871647"
Sep 08 15:34:00.851239 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:00.851217    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-vhng9"]
Sep 08 15:34:00.875069 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:00.875050    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-95fbf4c66-9k8sq"]
Sep 08 15:34:00.877140 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:34:00.877117    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ecfb56c_9113_49a2_99c8_226c564c32d5.slice/crio-de6afdcec9a5fac7c2c905f42733c3d7b57427554935d9624a097b1da504534c WatchSource:0}: Error finding container de6afdcec9a5fac7c2c905f42733c3d7b57427554935d9624a097b1da504534c: Status 404 returned error can't find the container with id de6afdcec9a5fac7c2c905f42733c3d7b57427554935d9624a097b1da504534c
Sep 08 15:34:01.715564 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:01.715526    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-95fbf4c66-9k8sq" event={"ID":"0ecfb56c-9113-49a2-99c8-226c564c32d5","Type":"ContainerStarted","Data":"e8a62c67fc534844b56bf1fda0dd9128412069cfeff388d1aa91df46bc47ba85"}
Sep 08 15:34:01.716011 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:01.715594    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-95fbf4c66-9k8sq"
Sep 08 15:34:01.716011 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:01.715615    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-95fbf4c66-9k8sq" event={"ID":"0ecfb56c-9113-49a2-99c8-226c564c32d5","Type":"ContainerStarted","Data":"de6afdcec9a5fac7c2c905f42733c3d7b57427554935d9624a097b1da504534c"}
Sep 08 15:34:01.716692 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:01.716668    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-vhng9" event={"ID":"c4ad30bb-757a-48cc-9d83-e788cf5cebb7","Type":"ContainerStarted","Data":"58895b6a571a9b91ef8666a01dcb26afa8caeb534c316524f0d702dfb4efbd78"}
Sep 08 15:34:01.738244 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:01.738199    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-95fbf4c66-9k8sq" podStartSLOduration=17.738185629 podStartE2EDuration="17.738185629s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:33:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:34:01.737556083 +0000 UTC m=+79.827007172" watchObservedRunningTime="2026-09-08 15:34:01.738185629 +0000 UTC m=+79.827636717"
Sep 08 15:34:03.726424 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:03.726390    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-vhng9" event={"ID":"c4ad30bb-757a-48cc-9d83-e788cf5cebb7","Type":"ContainerStarted","Data":"782e0b4238209aa3283b82c66a2f3abf3158a0dad81397a96da81c89a99cae8e"}
Sep 08 15:34:03.726928 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:03.726446    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-vhng9" event={"ID":"c4ad30bb-757a-48cc-9d83-e788cf5cebb7","Type":"ContainerStarted","Data":"693c586eff4f2b156c5e0b905ecf5b05725c88eac88fa55aaf23b176547edc13"}
Sep 08 15:34:03.743754 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:03.743709    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-vhng9" podStartSLOduration=17.793286107 podStartE2EDuration="19.743694338s" totalImagesPullingTime="1.950408231s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:33:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:34:03.742147242 +0000 UTC m=+81.831598331" watchObservedRunningTime="2026-09-08 15:34:03.743694338 +0000 UTC m=+81.833145426"
Sep 08 15:34:03.888909 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:03.888873    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3b1fd2bf-0e75-4f18-b224-db2a474aba0e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6tn6s\" (UID: \"3b1fd2bf-0e75-4f18-b224-db2a474aba0e\") " pod="openshift-insights/insights-runtime-extractor-6tn6s"
Sep 08 15:34:03.889013 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:03.888919    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3b1fd2bf-0e75-4f18-b224-db2a474aba0e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6tn6s\" (UID: \"3b1fd2bf-0e75-4f18-b224-db2a474aba0e\") " pod="openshift-insights/insights-runtime-extractor-6tn6s"
Sep 08 15:34:03.889448 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:03.889424    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3b1fd2bf-0e75-4f18-b224-db2a474aba0e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6tn6s\" (UID: \"3b1fd2bf-0e75-4f18-b224-db2a474aba0e\") " pod="openshift-insights/insights-runtime-extractor-6tn6s"
Sep 08 15:34:03.891198 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:03.891181    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3b1fd2bf-0e75-4f18-b224-db2a474aba0e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6tn6s\" (UID: \"3b1fd2bf-0e75-4f18-b224-db2a474aba0e\") " pod="openshift-insights/insights-runtime-extractor-6tn6s"
Sep 08 15:34:03.992693 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:03.992633    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-6tn6s"
Sep 08 15:34:04.263870 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:04.263710    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-6tn6s"]
Sep 08 15:34:04.266245 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:34:04.266216    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b1fd2bf_0e75_4f18_b224_db2a474aba0e.slice/crio-b14f12282009be313ff9c48b04fe9a10e8d8159bd2547e58d0656d5d40261074 WatchSource:0}: Error finding container b14f12282009be313ff9c48b04fe9a10e8d8159bd2547e58d0656d5d40261074: Status 404 returned error can't find the container with id b14f12282009be313ff9c48b04fe9a10e8d8159bd2547e58d0656d5d40261074
Sep 08 15:34:04.730374 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:04.730335    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6tn6s" event={"ID":"3b1fd2bf-0e75-4f18-b224-db2a474aba0e","Type":"ContainerStarted","Data":"27129dcf2a3ca4ad98efb566213964027f0912062c6c44ceea8f5855816a6e3a"}
Sep 08 15:34:04.730730 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:04.730433    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6tn6s" event={"ID":"3b1fd2bf-0e75-4f18-b224-db2a474aba0e","Type":"ContainerStarted","Data":"b14f12282009be313ff9c48b04fe9a10e8d8159bd2547e58d0656d5d40261074"}
Sep 08 15:34:05.737271 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:05.737239    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6tn6s" event={"ID":"3b1fd2bf-0e75-4f18-b224-db2a474aba0e","Type":"ContainerStarted","Data":"28625daf784fd00d09de9730a6709108bbef98df400930bd28b637a59558dcf1"}
Sep 08 15:34:06.742351 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:06.742317    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6tn6s" event={"ID":"3b1fd2bf-0e75-4f18-b224-db2a474aba0e","Type":"ContainerStarted","Data":"d4f5c6841b89519a896d2a74f2d8d925ad059595e8550393ba2e0dce1d194de3"}
Sep 08 15:34:06.761833 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:06.761788    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-6tn6s" podStartSLOduration=16.481188287 podStartE2EDuration="18.761773202s" totalImagesPullingTime="2.280584915s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 15:33:48 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:34:06.760960435 +0000 UTC m=+84.850411524" watchObservedRunningTime="2026-09-08 15:34:06.761773202 +0000 UTC m=+84.851224329"
Sep 08 15:34:08.394740 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:08.394709    2549 scope.go:117] "RemoveContainer" containerID="fe0f67b144eef7d9a35b32180e0cd385f9acfefa04bb2af361dc8aa4561c3abb"
Sep 08 15:34:08.751083 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:08.751017    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-wpwdq_5f7b0f68-a8e3-4058-b032-15f419682b2a/console-operator/2.log"
Sep 08 15:34:08.751427 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:08.751411    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-wpwdq_5f7b0f68-a8e3-4058-b032-15f419682b2a/console-operator/1.log"
Sep 08 15:34:08.751481 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:08.751444    2549 generic.go:356] "Generic (PLEG): container finished" podID="5f7b0f68-a8e3-4058-b032-15f419682b2a" containerID="aaf52d23e726e627052afdc3a46d710e3dc92dd6aa771f005236e578e6698d54" exitCode=255
Sep 08 15:34:08.751481 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:08.751473    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-wpwdq" event={"ID":"5f7b0f68-a8e3-4058-b032-15f419682b2a","Type":"ContainerDied","Data":"aaf52d23e726e627052afdc3a46d710e3dc92dd6aa771f005236e578e6698d54"}
Sep 08 15:34:08.751549 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:08.751515    2549 scope.go:117] "RemoveContainer" containerID="fe0f67b144eef7d9a35b32180e0cd385f9acfefa04bb2af361dc8aa4561c3abb"
Sep 08 15:34:08.751842 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:08.751818    2549 scope.go:117] "RemoveContainer" containerID="aaf52d23e726e627052afdc3a46d710e3dc92dd6aa771f005236e578e6698d54"
Sep 08 15:34:08.752042 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:34:08.752023    2549 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-85b454874b-wpwdq_openshift-console-operator(5f7b0f68-a8e3-4058-b032-15f419682b2a)\"" pod="openshift-console-operator/console-operator-85b454874b-wpwdq" podUID="5f7b0f68-a8e3-4058-b032-15f419682b2a"
Sep 08 15:34:09.756168 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:09.756135    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-wpwdq_5f7b0f68-a8e3-4058-b032-15f419682b2a/console-operator/2.log"
Sep 08 15:34:10.855543 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:10.855515    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7bf855885b-gqn4t"]
Sep 08 15:34:10.857952 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:10.857936    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7bf855885b-gqn4t"
Sep 08 15:34:10.861826 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:10.861803    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 08 15:34:10.861992 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:10.861870    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 08 15:34:10.861992 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:10.861902    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 08 15:34:10.861992 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:10.861922    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 08 15:34:10.869284 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:10.869260    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7bf855885b-gqn4t"]
Sep 08 15:34:10.922133 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:10.922110    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-95fbf4c66-9k8sq"]
Sep 08 15:34:10.937737 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:10.937716    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkrrp\" (UniqueName: \"kubernetes.io/projected/fe995ddd-5fdd-4996-a748-c5b9d8a1134f-kube-api-access-gkrrp\") pod \"klusterlet-addon-workmgr-7bf855885b-gqn4t\" (UID: \"fe995ddd-5fdd-4996-a748-c5b9d8a1134f\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7bf855885b-gqn4t"
Sep 08 15:34:10.937836 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:10.937760    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/fe995ddd-5fdd-4996-a748-c5b9d8a1134f-klusterlet-config\") pod \"klusterlet-addon-workmgr-7bf855885b-gqn4t\" (UID: \"fe995ddd-5fdd-4996-a748-c5b9d8a1134f\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7bf855885b-gqn4t"
Sep 08 15:34:10.937836 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:10.937784    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fe995ddd-5fdd-4996-a748-c5b9d8a1134f-tmp\") pod \"klusterlet-addon-workmgr-7bf855885b-gqn4t\" (UID: \"fe995ddd-5fdd-4996-a748-c5b9d8a1134f\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7bf855885b-gqn4t"
Sep 08 15:34:10.962818 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:10.962788    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5fc68db7dd-pl9pd"]
Sep 08 15:34:10.964559 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:10.964543    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5fc68db7dd-pl9pd"
Sep 08 15:34:10.976552 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:10.976534    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5fc68db7dd-pl9pd"]
Sep 08 15:34:11.038460 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:11.038438    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f95cc865-fd51-4726-9804-71f9b0b63ce7-registry-tls\") pod \"image-registry-5fc68db7dd-pl9pd\" (UID: \"f95cc865-fd51-4726-9804-71f9b0b63ce7\") " pod="openshift-image-registry/image-registry-5fc68db7dd-pl9pd"
Sep 08 15:34:11.038581 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:11.038480    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f95cc865-fd51-4726-9804-71f9b0b63ce7-installation-pull-secrets\") pod \"image-registry-5fc68db7dd-pl9pd\" (UID: \"f95cc865-fd51-4726-9804-71f9b0b63ce7\") " pod="openshift-image-registry/image-registry-5fc68db7dd-pl9pd"
Sep 08 15:34:11.038581 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:11.038502    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f95cc865-fd51-4726-9804-71f9b0b63ce7-registry-certificates\") pod \"image-registry-5fc68db7dd-pl9pd\" (UID: \"f95cc865-fd51-4726-9804-71f9b0b63ce7\") " pod="openshift-image-registry/image-registry-5fc68db7dd-pl9pd"
Sep 08 15:34:11.038581 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:11.038521    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f95cc865-fd51-4726-9804-71f9b0b63ce7-image-registry-private-configuration\") pod \"image-registry-5fc68db7dd-pl9pd\" (UID: \"f95cc865-fd51-4726-9804-71f9b0b63ce7\") " pod="openshift-image-registry/image-registry-5fc68db7dd-pl9pd"
Sep 08 15:34:11.038581 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:11.038561    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f95cc865-fd51-4726-9804-71f9b0b63ce7-trusted-ca\") pod \"image-registry-5fc68db7dd-pl9pd\" (UID: \"f95cc865-fd51-4726-9804-71f9b0b63ce7\") " pod="openshift-image-registry/image-registry-5fc68db7dd-pl9pd"
Sep 08 15:34:11.038721 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:11.038629    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkrrp\" (UniqueName: \"kubernetes.io/projected/fe995ddd-5fdd-4996-a748-c5b9d8a1134f-kube-api-access-gkrrp\") pod \"klusterlet-addon-workmgr-7bf855885b-gqn4t\" (UID: \"fe995ddd-5fdd-4996-a748-c5b9d8a1134f\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7bf855885b-gqn4t"
Sep 08 15:34:11.038721 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:11.038648    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f95cc865-fd51-4726-9804-71f9b0b63ce7-ca-trust-extracted\") pod \"image-registry-5fc68db7dd-pl9pd\" (UID: \"f95cc865-fd51-4726-9804-71f9b0b63ce7\") " pod="openshift-image-registry/image-registry-5fc68db7dd-pl9pd"
Sep 08 15:34:11.038721 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:11.038665    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f95cc865-fd51-4726-9804-71f9b0b63ce7-bound-sa-token\") pod \"image-registry-5fc68db7dd-pl9pd\" (UID: \"f95cc865-fd51-4726-9804-71f9b0b63ce7\") " pod="openshift-image-registry/image-registry-5fc68db7dd-pl9pd"
Sep 08 15:34:11.038721 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:11.038707    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svtf8\" (UniqueName: \"kubernetes.io/projected/f95cc865-fd51-4726-9804-71f9b0b63ce7-kube-api-access-svtf8\") pod \"image-registry-5fc68db7dd-pl9pd\" (UID: \"f95cc865-fd51-4726-9804-71f9b0b63ce7\") " pod="openshift-image-registry/image-registry-5fc68db7dd-pl9pd"
Sep 08 15:34:11.038887 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:11.038740    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/fe995ddd-5fdd-4996-a748-c5b9d8a1134f-klusterlet-config\") pod \"klusterlet-addon-workmgr-7bf855885b-gqn4t\" (UID: \"fe995ddd-5fdd-4996-a748-c5b9d8a1134f\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7bf855885b-gqn4t"
Sep 08 15:34:11.038887 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:11.038791    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fe995ddd-5fdd-4996-a748-c5b9d8a1134f-tmp\") pod \"klusterlet-addon-workmgr-7bf855885b-gqn4t\" (UID: \"fe995ddd-5fdd-4996-a748-c5b9d8a1134f\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7bf855885b-gqn4t"
Sep 08 15:34:11.039210 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:11.039192    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fe995ddd-5fdd-4996-a748-c5b9d8a1134f-tmp\") pod \"klusterlet-addon-workmgr-7bf855885b-gqn4t\" (UID: \"fe995ddd-5fdd-4996-a748-c5b9d8a1134f\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7bf855885b-gqn4t"
Sep 08 15:34:11.041159 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:11.041141    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/fe995ddd-5fdd-4996-a748-c5b9d8a1134f-klusterlet-config\") pod \"klusterlet-addon-workmgr-7bf855885b-gqn4t\" (UID: \"fe995ddd-5fdd-4996-a748-c5b9d8a1134f\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7bf855885b-gqn4t"
Sep 08 15:34:11.046729 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:11.046710    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkrrp\" (UniqueName: \"kubernetes.io/projected/fe995ddd-5fdd-4996-a748-c5b9d8a1134f-kube-api-access-gkrrp\") pod \"klusterlet-addon-workmgr-7bf855885b-gqn4t\" (UID: \"fe995ddd-5fdd-4996-a748-c5b9d8a1134f\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7bf855885b-gqn4t"
Sep 08 15:34:11.139886 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:11.139816    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svtf8\" (UniqueName: \"kubernetes.io/projected/f95cc865-fd51-4726-9804-71f9b0b63ce7-kube-api-access-svtf8\") pod \"image-registry-5fc68db7dd-pl9pd\" (UID: \"f95cc865-fd51-4726-9804-71f9b0b63ce7\") " pod="openshift-image-registry/image-registry-5fc68db7dd-pl9pd"
Sep 08 15:34:11.139886 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:11.139873    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f95cc865-fd51-4726-9804-71f9b0b63ce7-registry-tls\") pod \"image-registry-5fc68db7dd-pl9pd\" (UID: \"f95cc865-fd51-4726-9804-71f9b0b63ce7\") " pod="openshift-image-registry/image-registry-5fc68db7dd-pl9pd"
Sep 08 15:34:11.140092 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:11.139995    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f95cc865-fd51-4726-9804-71f9b0b63ce7-installation-pull-secrets\") pod \"image-registry-5fc68db7dd-pl9pd\" (UID: \"f95cc865-fd51-4726-9804-71f9b0b63ce7\") " pod="openshift-image-registry/image-registry-5fc68db7dd-pl9pd"
Sep 08 15:34:11.140092 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:11.140023    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f95cc865-fd51-4726-9804-71f9b0b63ce7-registry-certificates\") pod \"image-registry-5fc68db7dd-pl9pd\" (UID: \"f95cc865-fd51-4726-9804-71f9b0b63ce7\") " pod="openshift-image-registry/image-registry-5fc68db7dd-pl9pd"
Sep 08 15:34:11.140092 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:11.140045    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f95cc865-fd51-4726-9804-71f9b0b63ce7-image-registry-private-configuration\") pod \"image-registry-5fc68db7dd-pl9pd\" (UID: \"f95cc865-fd51-4726-9804-71f9b0b63ce7\") " pod="openshift-image-registry/image-registry-5fc68db7dd-pl9pd"
Sep 08 15:34:11.140092 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:11.140070    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f95cc865-fd51-4726-9804-71f9b0b63ce7-trusted-ca\") pod \"image-registry-5fc68db7dd-pl9pd\" (UID: \"f95cc865-fd51-4726-9804-71f9b0b63ce7\") " pod="openshift-image-registry/image-registry-5fc68db7dd-pl9pd"
Sep 08 15:34:11.140286 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:11.140101    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f95cc865-fd51-4726-9804-71f9b0b63ce7-ca-trust-extracted\") pod \"image-registry-5fc68db7dd-pl9pd\" (UID: \"f95cc865-fd51-4726-9804-71f9b0b63ce7\") " pod="openshift-image-registry/image-registry-5fc68db7dd-pl9pd"
Sep 08 15:34:11.140286 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:11.140232    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f95cc865-fd51-4726-9804-71f9b0b63ce7-bound-sa-token\") pod \"image-registry-5fc68db7dd-pl9pd\" (UID: \"f95cc865-fd51-4726-9804-71f9b0b63ce7\") " pod="openshift-image-registry/image-registry-5fc68db7dd-pl9pd"
Sep 08 15:34:11.140539 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:11.140506    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f95cc865-fd51-4726-9804-71f9b0b63ce7-ca-trust-extracted\") pod \"image-registry-5fc68db7dd-pl9pd\" (UID: \"f95cc865-fd51-4726-9804-71f9b0b63ce7\") " pod="openshift-image-registry/image-registry-5fc68db7dd-pl9pd"
Sep 08 15:34:11.140924 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:11.140900    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f95cc865-fd51-4726-9804-71f9b0b63ce7-registry-certificates\") pod \"image-registry-5fc68db7dd-pl9pd\" (UID: \"f95cc865-fd51-4726-9804-71f9b0b63ce7\") " pod="openshift-image-registry/image-registry-5fc68db7dd-pl9pd"
Sep 08 15:34:11.141051 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:11.140963    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f95cc865-fd51-4726-9804-71f9b0b63ce7-trusted-ca\") pod \"image-registry-5fc68db7dd-pl9pd\" (UID: \"f95cc865-fd51-4726-9804-71f9b0b63ce7\") " pod="openshift-image-registry/image-registry-5fc68db7dd-pl9pd"
Sep 08 15:34:11.142155 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:11.142134    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f95cc865-fd51-4726-9804-71f9b0b63ce7-registry-tls\") pod \"image-registry-5fc68db7dd-pl9pd\" (UID: \"f95cc865-fd51-4726-9804-71f9b0b63ce7\") " pod="openshift-image-registry/image-registry-5fc68db7dd-pl9pd"
Sep 08 15:34:11.142266 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:11.142246    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f95cc865-fd51-4726-9804-71f9b0b63ce7-image-registry-private-configuration\") pod \"image-registry-5fc68db7dd-pl9pd\" (UID: \"f95cc865-fd51-4726-9804-71f9b0b63ce7\") " pod="openshift-image-registry/image-registry-5fc68db7dd-pl9pd"
Sep 08 15:34:11.142512 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:11.142489    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f95cc865-fd51-4726-9804-71f9b0b63ce7-installation-pull-secrets\") pod \"image-registry-5fc68db7dd-pl9pd\" (UID: \"f95cc865-fd51-4726-9804-71f9b0b63ce7\") " pod="openshift-image-registry/image-registry-5fc68db7dd-pl9pd"
Sep 08 15:34:11.149675 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:11.149655    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-svtf8\" (UniqueName: \"kubernetes.io/projected/f95cc865-fd51-4726-9804-71f9b0b63ce7-kube-api-access-svtf8\") pod \"image-registry-5fc68db7dd-pl9pd\" (UID: \"f95cc865-fd51-4726-9804-71f9b0b63ce7\") " pod="openshift-image-registry/image-registry-5fc68db7dd-pl9pd"
Sep 08 15:34:11.150381 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:11.150359    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f95cc865-fd51-4726-9804-71f9b0b63ce7-bound-sa-token\") pod \"image-registry-5fc68db7dd-pl9pd\" (UID: \"f95cc865-fd51-4726-9804-71f9b0b63ce7\") " pod="openshift-image-registry/image-registry-5fc68db7dd-pl9pd"
Sep 08 15:34:11.167186 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:11.167162    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7bf855885b-gqn4t"
Sep 08 15:34:11.273010 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:11.272982    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5fc68db7dd-pl9pd"
Sep 08 15:34:11.443330 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:11.443305    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7bf855885b-gqn4t"]
Sep 08 15:34:11.446201 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:34:11.446175    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe995ddd_5fdd_4996_a748_c5b9d8a1134f.slice/crio-1948f4ced1add67ba0b5e01a181df2bcddf8dbc3a52727204044d384c5180017 WatchSource:0}: Error finding container 1948f4ced1add67ba0b5e01a181df2bcddf8dbc3a52727204044d384c5180017: Status 404 returned error can't find the container with id 1948f4ced1add67ba0b5e01a181df2bcddf8dbc3a52727204044d384c5180017
Sep 08 15:34:11.541950 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:11.541929    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5fc68db7dd-pl9pd"]
Sep 08 15:34:11.544059 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:34:11.544034    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf95cc865_fd51_4726_9804_71f9b0b63ce7.slice/crio-c33ff91ac122cdebe1829dfed804d91a41a1700fb59a0c360e616bb129a9065f WatchSource:0}: Error finding container c33ff91ac122cdebe1829dfed804d91a41a1700fb59a0c360e616bb129a9065f: Status 404 returned error can't find the container with id c33ff91ac122cdebe1829dfed804d91a41a1700fb59a0c360e616bb129a9065f
Sep 08 15:34:11.764228 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:11.764145    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7bf855885b-gqn4t" event={"ID":"fe995ddd-5fdd-4996-a748-c5b9d8a1134f","Type":"ContainerStarted","Data":"1948f4ced1add67ba0b5e01a181df2bcddf8dbc3a52727204044d384c5180017"}
Sep 08 15:34:11.765731 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:11.765707    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5fc68db7dd-pl9pd" event={"ID":"f95cc865-fd51-4726-9804-71f9b0b63ce7","Type":"ContainerStarted","Data":"af37344f0272c18b7b721fe7095fea3b31a145fa97b960e7b31f952ea415fb6e"}
Sep 08 15:34:11.765883 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:11.765749    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5fc68db7dd-pl9pd" event={"ID":"f95cc865-fd51-4726-9804-71f9b0b63ce7","Type":"ContainerStarted","Data":"c33ff91ac122cdebe1829dfed804d91a41a1700fb59a0c360e616bb129a9065f"}
Sep 08 15:34:11.765883 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:11.765828    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5fc68db7dd-pl9pd"
Sep 08 15:34:11.785774 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:11.785723    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5fc68db7dd-pl9pd" podStartSLOduration=1.785710082 podStartE2EDuration="1.785710082s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:34:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:34:11.784695931 +0000 UTC m=+89.874147021" watchObservedRunningTime="2026-09-08 15:34:11.785710082 +0000 UTC m=+89.875161161"
Sep 08 15:34:14.991980 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:14.991950    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-wpwdq"
Sep 08 15:34:14.992316 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:14.992008    2549 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-85b454874b-wpwdq"
Sep 08 15:34:14.992619 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:14.992382    2549 scope.go:117] "RemoveContainer" containerID="aaf52d23e726e627052afdc3a46d710e3dc92dd6aa771f005236e578e6698d54"
Sep 08 15:34:14.992619 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:34:14.992574    2549 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-85b454874b-wpwdq_openshift-console-operator(5f7b0f68-a8e3-4058-b032-15f419682b2a)\"" pod="openshift-console-operator/console-operator-85b454874b-wpwdq" podUID="5f7b0f68-a8e3-4058-b032-15f419682b2a"
Sep 08 15:34:15.781949 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:15.781910    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7bf855885b-gqn4t" event={"ID":"fe995ddd-5fdd-4996-a748-c5b9d8a1134f","Type":"ContainerStarted","Data":"4e3359729f01ab4a0789ded49d84990e913303e9da71fe4eabb24ef04a6885bd"}
Sep 08 15:34:15.782140 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:15.782102    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7bf855885b-gqn4t"
Sep 08 15:34:15.783779 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:15.783755    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7bf855885b-gqn4t"
Sep 08 15:34:15.801549 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:15.801503    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7bf855885b-gqn4t" podStartSLOduration=2.28936006 podStartE2EDuration="5.801488781s" totalImagesPullingTime="3.512128721s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:34:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:34:15.799372038 +0000 UTC m=+93.888823129" watchObservedRunningTime="2026-09-08 15:34:15.801488781 +0000 UTC m=+93.890939869"
Sep 08 15:34:20.108710 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:20.108677    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/57cef312-9005-43bc-9e2f-b59db9e864c3-metrics-tls\") pod \"dns-default-vhsng\" (UID: \"57cef312-9005-43bc-9e2f-b59db9e864c3\") " pod="openshift-dns/dns-default-vhsng"
Sep 08 15:34:20.110908 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:20.110888    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/57cef312-9005-43bc-9e2f-b59db9e864c3-metrics-tls\") pod \"dns-default-vhsng\" (UID: \"57cef312-9005-43bc-9e2f-b59db9e864c3\") " pod="openshift-dns/dns-default-vhsng"
Sep 08 15:34:20.174576 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:20.174548    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-5prt2\""
Sep 08 15:34:20.181552 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:20.181533    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-vhsng"
Sep 08 15:34:20.454405 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:20.454374    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-vhsng"]
Sep 08 15:34:20.457217 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:34:20.457190    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57cef312_9005_43bc_9e2f_b59db9e864c3.slice/crio-9c2390cf65d5750bffadea337340aa1563651bf8e5f32d609cb19c296cc911eb WatchSource:0}: Error finding container 9c2390cf65d5750bffadea337340aa1563651bf8e5f32d609cb19c296cc911eb: Status 404 returned error can't find the container with id 9c2390cf65d5750bffadea337340aa1563651bf8e5f32d609cb19c296cc911eb
Sep 08 15:34:20.797273 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:20.797193    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vhsng" event={"ID":"57cef312-9005-43bc-9e2f-b59db9e864c3","Type":"ContainerStarted","Data":"9c2390cf65d5750bffadea337340aa1563651bf8e5f32d609cb19c296cc911eb"}
Sep 08 15:34:20.927304 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:20.927276    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-95fbf4c66-9k8sq"
Sep 08 15:34:22.805912 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:22.805876    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vhsng" event={"ID":"57cef312-9005-43bc-9e2f-b59db9e864c3","Type":"ContainerStarted","Data":"5ed1b0a1eb187185d039462c590acf0dae4a41412e1da06382321abc25e48ecf"}
Sep 08 15:34:22.806278 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:22.805927    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vhsng" event={"ID":"57cef312-9005-43bc-9e2f-b59db9e864c3","Type":"ContainerStarted","Data":"69bd96d4ac49ff94d10ca5f53b0e8ed71bfdc359652b339f177ef6a405d6faaf"}
Sep 08 15:34:22.806278 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:22.806009    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-vhsng"
Sep 08 15:34:22.826370 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:22.826318    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-vhsng" podStartSLOduration=65.133551273 podStartE2EDuration="1m6.826305727s" totalImagesPullingTime="1.692754454s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:33:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:34:22.824456536 +0000 UTC m=+100.913907624" watchObservedRunningTime="2026-09-08 15:34:22.826305727 +0000 UTC m=+100.915756815"
Sep 08 15:34:23.692800 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:23.692770    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-5vt6c"
Sep 08 15:34:27.400874 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:27.400833    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-mpbc4"]
Sep 08 15:34:27.411157 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:27.411133    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-mpbc4"
Sep 08 15:34:27.416180 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:27.416157    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Sep 08 15:34:27.416325 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:27.416194    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-xv9lq\""
Sep 08 15:34:27.416325 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:27.416233    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Sep 08 15:34:27.416325 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:27.416238    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 08 15:34:27.416477 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:27.416462    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 08 15:34:27.416543 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:27.416527    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 08 15:34:27.422173 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:27.422152    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-mpbc4"]
Sep 08 15:34:27.441895 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:27.441869    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-4cgxb"]
Sep 08 15:34:27.443782 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:27.443766    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-phnv9"]
Sep 08 15:34:27.443916 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:27.443901    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-74cb57df7-4cgxb"
Sep 08 15:34:27.445622 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:27.445603    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-phnv9"
Sep 08 15:34:27.447089 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:27.447069    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Sep 08 15:34:27.447406 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:27.447386    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Sep 08 15:34:27.447577 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:27.447562    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Sep 08 15:34:27.448846 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:27.448827    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-szq2c\""
Sep 08 15:34:27.448932 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:27.448877    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 08 15:34:27.449733 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:27.449719    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 08 15:34:27.450080 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:27.449960    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-gfwpq\""
Sep 08 15:34:27.450080 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:27.450030    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 08 15:34:27.471407 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:27.471385    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-4cgxb"]
Sep 08 15:34:27.562930 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:27.562899    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6865c4b2-f146-4d9f-88cf-2eaa6460880a-node-exporter-textfile\") pod \"node-exporter-phnv9\" (UID: \"6865c4b2-f146-4d9f-88cf-2eaa6460880a\") " pod="openshift-monitoring/node-exporter-phnv9"
Sep 08 15:34:27.563080 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:27.562934    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/09d9c8c2-604a-481d-9e58-57bbb07b2e03-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-4cgxb\" (UID: \"09d9c8c2-604a-481d-9e58-57bbb07b2e03\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-4cgxb"
Sep 08 15:34:27.563080 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:27.562952    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6865c4b2-f146-4d9f-88cf-2eaa6460880a-metrics-client-ca\") pod \"node-exporter-phnv9\" (UID: \"6865c4b2-f146-4d9f-88cf-2eaa6460880a\") " pod="openshift-monitoring/node-exporter-phnv9"
Sep 08 15:34:27.563080 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:27.563006    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6865c4b2-f146-4d9f-88cf-2eaa6460880a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-phnv9\" (UID: \"6865c4b2-f146-4d9f-88cf-2eaa6460880a\") " pod="openshift-monitoring/node-exporter-phnv9"
Sep 08 15:34:27.563080 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:27.563038    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e942dd64-4a92-449d-b013-7ac71c351448-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-mpbc4\" (UID: \"e942dd64-4a92-449d-b013-7ac71c351448\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-mpbc4"
Sep 08 15:34:27.563219 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:27.563084    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/09d9c8c2-604a-481d-9e58-57bbb07b2e03-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-4cgxb\" (UID: \"09d9c8c2-604a-481d-9e58-57bbb07b2e03\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-4cgxb"
Sep 08 15:34:27.563219 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:27.563104    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6865c4b2-f146-4d9f-88cf-2eaa6460880a-node-exporter-wtmp\") pod \"node-exporter-phnv9\" (UID: \"6865c4b2-f146-4d9f-88cf-2eaa6460880a\") " pod="openshift-monitoring/node-exporter-phnv9"
Sep 08 15:34:27.563219 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:27.563132    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hn28f\" (UniqueName: \"kubernetes.io/projected/e942dd64-4a92-449d-b013-7ac71c351448-kube-api-access-hn28f\") pod \"openshift-state-metrics-657fd6c54-mpbc4\" (UID: \"e942dd64-4a92-449d-b013-7ac71c351448\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-mpbc4"
Sep 08 15:34:27.563219 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:27.563164    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/09d9c8c2-604a-481d-9e58-57bbb07b2e03-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-4cgxb\" (UID: \"09d9c8c2-604a-481d-9e58-57bbb07b2e03\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-4cgxb"
Sep 08 15:34:27.563219 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:27.563194    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e942dd64-4a92-449d-b013-7ac71c351448-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-mpbc4\" (UID: \"e942dd64-4a92-449d-b013-7ac71c351448\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-mpbc4"
Sep 08 15:34:27.563414 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:27.563223    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/09d9c8c2-604a-481d-9e58-57bbb07b2e03-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-4cgxb\" (UID: \"09d9c8c2-604a-481d-9e58-57bbb07b2e03\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-4cgxb"
Sep 08 15:34:27.563414 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:27.563241    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6865c4b2-f146-4d9f-88cf-2eaa6460880a-sys\") pod \"node-exporter-phnv9\" (UID: \"6865c4b2-f146-4d9f-88cf-2eaa6460880a\") " pod="openshift-monitoring/node-exporter-phnv9"
Sep 08 15:34:27.563414 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:27.563266    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6865c4b2-f146-4d9f-88cf-2eaa6460880a-node-exporter-accelerators-collector-config\") pod \"node-exporter-phnv9\" (UID: \"6865c4b2-f146-4d9f-88cf-2eaa6460880a\") " pod="openshift-monitoring/node-exporter-phnv9"
Sep 08 15:34:27.563414 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:27.563289    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2f94h\" (UniqueName: \"kubernetes.io/projected/6865c4b2-f146-4d9f-88cf-2eaa6460880a-kube-api-access-2f94h\") pod \"node-exporter-phnv9\" (UID: \"6865c4b2-f146-4d9f-88cf-2eaa6460880a\") " pod="openshift-monitoring/node-exporter-phnv9"
Sep 08 15:34:27.563414 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:27.563315    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8rct\" (UniqueName: \"kubernetes.io/projected/09d9c8c2-604a-481d-9e58-57bbb07b2e03-kube-api-access-l8rct\") pod \"kube-state-metrics-74cb57df7-4cgxb\" (UID: \"09d9c8c2-604a-481d-9e58-57bbb07b2e03\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-4cgxb"
Sep 08 15:34:27.563414 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:27.563347    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e942dd64-4a92-449d-b013-7ac71c351448-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-mpbc4\" (UID: \"e942dd64-4a92-449d-b013-7ac71c351448\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-mpbc4"
Sep 08 15:34:27.563414 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:27.563375    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/09d9c8c2-604a-481d-9e58-57bbb07b2e03-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-4cgxb\" (UID: \"09d9c8c2-604a-481d-9e58-57bbb07b2e03\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-4cgxb"
Sep 08 15:34:27.563414 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:27.563389    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6865c4b2-f146-4d9f-88cf-2eaa6460880a-root\") pod \"node-exporter-phnv9\" (UID: \"6865c4b2-f146-4d9f-88cf-2eaa6460880a\") " pod="openshift-monitoring/node-exporter-phnv9"
Sep 08 15:34:27.563651 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:27.563419    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6865c4b2-f146-4d9f-88cf-2eaa6460880a-node-exporter-tls\") pod \"node-exporter-phnv9\" (UID: \"6865c4b2-f146-4d9f-88cf-2eaa6460880a\") " pod="openshift-monitoring/node-exporter-phnv9"
Sep 08 15:34:27.664381 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:27.664299    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6865c4b2-f146-4d9f-88cf-2eaa6460880a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-phnv9\" (UID: \"6865c4b2-f146-4d9f-88cf-2eaa6460880a\") " pod="openshift-monitoring/node-exporter-phnv9"
Sep 08 15:34:27.664381 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:27.664341    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e942dd64-4a92-449d-b013-7ac71c351448-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-mpbc4\" (UID: \"e942dd64-4a92-449d-b013-7ac71c351448\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-mpbc4"
Sep 08 15:34:27.664381 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:27.664364    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/09d9c8c2-604a-481d-9e58-57bbb07b2e03-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-4cgxb\" (UID: \"09d9c8c2-604a-481d-9e58-57bbb07b2e03\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-4cgxb"
Sep 08 15:34:27.664608 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:27.664385    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6865c4b2-f146-4d9f-88cf-2eaa6460880a-node-exporter-wtmp\") pod \"node-exporter-phnv9\" (UID: \"6865c4b2-f146-4d9f-88cf-2eaa6460880a\") " pod="openshift-monitoring/node-exporter-phnv9"
Sep 08 15:34:27.664608 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:27.664408    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hn28f\" (UniqueName: \"kubernetes.io/projected/e942dd64-4a92-449d-b013-7ac71c351448-kube-api-access-hn28f\") pod \"openshift-state-metrics-657fd6c54-mpbc4\" (UID: \"e942dd64-4a92-449d-b013-7ac71c351448\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-mpbc4"
Sep 08 15:34:27.664608 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:27.664423    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/09d9c8c2-604a-481d-9e58-57bbb07b2e03-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-4cgxb\" (UID: \"09d9c8c2-604a-481d-9e58-57bbb07b2e03\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-4cgxb"
Sep 08 15:34:27.664608 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:27.664439    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e942dd64-4a92-449d-b013-7ac71c351448-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-mpbc4\" (UID: \"e942dd64-4a92-449d-b013-7ac71c351448\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-mpbc4"
Sep 08 15:34:27.664608 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:27.664474    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/09d9c8c2-604a-481d-9e58-57bbb07b2e03-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-4cgxb\" (UID: \"09d9c8c2-604a-481d-9e58-57bbb07b2e03\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-4cgxb"
Sep 08 15:34:27.664608 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:27.664517    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6865c4b2-f146-4d9f-88cf-2eaa6460880a-sys\") pod \"node-exporter-phnv9\" (UID: \"6865c4b2-f146-4d9f-88cf-2eaa6460880a\") " pod="openshift-monitoring/node-exporter-phnv9"
Sep 08 15:34:27.664608 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:27.664549    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6865c4b2-f146-4d9f-88cf-2eaa6460880a-node-exporter-accelerators-collector-config\") pod \"node-exporter-phnv9\" (UID: \"6865c4b2-f146-4d9f-88cf-2eaa6460880a\") " pod="openshift-monitoring/node-exporter-phnv9"
Sep 08 15:34:27.664608 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:27.664585    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2f94h\" (UniqueName: \"kubernetes.io/projected/6865c4b2-f146-4d9f-88cf-2eaa6460880a-kube-api-access-2f94h\") pod \"node-exporter-phnv9\" (UID: \"6865c4b2-f146-4d9f-88cf-2eaa6460880a\") " pod="openshift-monitoring/node-exporter-phnv9"
Sep 08 15:34:27.664999 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:27.664613    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8rct\" (UniqueName: \"kubernetes.io/projected/09d9c8c2-604a-481d-9e58-57bbb07b2e03-kube-api-access-l8rct\") pod \"kube-state-metrics-74cb57df7-4cgxb\" (UID: \"09d9c8c2-604a-481d-9e58-57bbb07b2e03\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-4cgxb"
Sep 08 15:34:27.664999 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:27.664619    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6865c4b2-f146-4d9f-88cf-2eaa6460880a-sys\") pod \"node-exporter-phnv9\" (UID: \"6865c4b2-f146-4d9f-88cf-2eaa6460880a\") " pod="openshift-monitoring/node-exporter-phnv9"
Sep 08 15:34:27.664999 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:27.664620    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6865c4b2-f146-4d9f-88cf-2eaa6460880a-node-exporter-wtmp\") pod \"node-exporter-phnv9\" (UID: \"6865c4b2-f146-4d9f-88cf-2eaa6460880a\") " pod="openshift-monitoring/node-exporter-phnv9"
Sep 08 15:34:27.664999 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:27.664637    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e942dd64-4a92-449d-b013-7ac71c351448-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-mpbc4\" (UID: \"e942dd64-4a92-449d-b013-7ac71c351448\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-mpbc4"
Sep 08 15:34:27.664999 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:27.664696    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/09d9c8c2-604a-481d-9e58-57bbb07b2e03-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-4cgxb\" (UID: \"09d9c8c2-604a-481d-9e58-57bbb07b2e03\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-4cgxb"
Sep 08 15:34:27.664999 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:27.664723    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6865c4b2-f146-4d9f-88cf-2eaa6460880a-root\") pod \"node-exporter-phnv9\" (UID: \"6865c4b2-f146-4d9f-88cf-2eaa6460880a\") " pod="openshift-monitoring/node-exporter-phnv9"
Sep 08 15:34:27.664999 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:27.664752    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6865c4b2-f146-4d9f-88cf-2eaa6460880a-node-exporter-tls\") pod \"node-exporter-phnv9\" (UID: \"6865c4b2-f146-4d9f-88cf-2eaa6460880a\") " pod="openshift-monitoring/node-exporter-phnv9"
Sep 08 15:34:27.664999 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:27.664785    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6865c4b2-f146-4d9f-88cf-2eaa6460880a-node-exporter-textfile\") pod \"node-exporter-phnv9\" (UID: \"6865c4b2-f146-4d9f-88cf-2eaa6460880a\") " pod="openshift-monitoring/node-exporter-phnv9"
Sep 08 15:34:27.664999 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:27.664817    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/09d9c8c2-604a-481d-9e58-57bbb07b2e03-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-4cgxb\" (UID: \"09d9c8c2-604a-481d-9e58-57bbb07b2e03\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-4cgxb"
Sep 08 15:34:27.664999 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:27.664841    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6865c4b2-f146-4d9f-88cf-2eaa6460880a-metrics-client-ca\") pod \"node-exporter-phnv9\" (UID: \"6865c4b2-f146-4d9f-88cf-2eaa6460880a\") " pod="openshift-monitoring/node-exporter-phnv9"
Sep 08 15:34:27.665489 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:27.665110    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6865c4b2-f146-4d9f-88cf-2eaa6460880a-root\") pod \"node-exporter-phnv9\" (UID: \"6865c4b2-f146-4d9f-88cf-2eaa6460880a\") " pod="openshift-monitoring/node-exporter-phnv9"
Sep 08 15:34:27.665489 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:27.665424    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6865c4b2-f146-4d9f-88cf-2eaa6460880a-node-exporter-textfile\") pod \"node-exporter-phnv9\" (UID: \"6865c4b2-f146-4d9f-88cf-2eaa6460880a\") " pod="openshift-monitoring/node-exporter-phnv9"
Sep 08 15:34:27.665594 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:27.665522    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e942dd64-4a92-449d-b013-7ac71c351448-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-mpbc4\" (UID: \"e942dd64-4a92-449d-b013-7ac71c351448\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-mpbc4"
Sep 08 15:34:27.666120 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:27.666055    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6865c4b2-f146-4d9f-88cf-2eaa6460880a-metrics-client-ca\") pod \"node-exporter-phnv9\" (UID: \"6865c4b2-f146-4d9f-88cf-2eaa6460880a\") " pod="openshift-monitoring/node-exporter-phnv9"
Sep 08 15:34:27.666120 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:27.666069    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/09d9c8c2-604a-481d-9e58-57bbb07b2e03-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-4cgxb\" (UID: \"09d9c8c2-604a-481d-9e58-57bbb07b2e03\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-4cgxb"
Sep 08 15:34:27.666350 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:27.666250    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/09d9c8c2-604a-481d-9e58-57bbb07b2e03-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-4cgxb\" (UID: \"09d9c8c2-604a-481d-9e58-57bbb07b2e03\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-4cgxb"
Sep 08 15:34:27.666803 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:27.666778    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/09d9c8c2-604a-481d-9e58-57bbb07b2e03-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-4cgxb\" (UID: \"09d9c8c2-604a-481d-9e58-57bbb07b2e03\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-4cgxb"
Sep 08 15:34:27.666975 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:27.666803    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6865c4b2-f146-4d9f-88cf-2eaa6460880a-node-exporter-accelerators-collector-config\") pod \"node-exporter-phnv9\" (UID: \"6865c4b2-f146-4d9f-88cf-2eaa6460880a\") " pod="openshift-monitoring/node-exporter-phnv9"
Sep 08 15:34:27.667602 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:27.667581    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6865c4b2-f146-4d9f-88cf-2eaa6460880a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-phnv9\" (UID: \"6865c4b2-f146-4d9f-88cf-2eaa6460880a\") " pod="openshift-monitoring/node-exporter-phnv9"
Sep 08 15:34:27.667721 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:27.667699    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e942dd64-4a92-449d-b013-7ac71c351448-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-mpbc4\" (UID: \"e942dd64-4a92-449d-b013-7ac71c351448\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-mpbc4"
Sep 08 15:34:27.667788 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:27.667741    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e942dd64-4a92-449d-b013-7ac71c351448-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-mpbc4\" (UID: \"e942dd64-4a92-449d-b013-7ac71c351448\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-mpbc4"
Sep 08 15:34:27.667788 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:27.667741    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6865c4b2-f146-4d9f-88cf-2eaa6460880a-node-exporter-tls\") pod \"node-exporter-phnv9\" (UID: \"6865c4b2-f146-4d9f-88cf-2eaa6460880a\") " pod="openshift-monitoring/node-exporter-phnv9"
Sep 08 15:34:27.667929 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:27.667842    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/09d9c8c2-604a-481d-9e58-57bbb07b2e03-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-4cgxb\" (UID: \"09d9c8c2-604a-481d-9e58-57bbb07b2e03\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-4cgxb"
Sep 08 15:34:27.668081 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:27.668062    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/09d9c8c2-604a-481d-9e58-57bbb07b2e03-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-4cgxb\" (UID: \"09d9c8c2-604a-481d-9e58-57bbb07b2e03\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-4cgxb"
Sep 08 15:34:27.673333 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:27.673312    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8rct\" (UniqueName: \"kubernetes.io/projected/09d9c8c2-604a-481d-9e58-57bbb07b2e03-kube-api-access-l8rct\") pod \"kube-state-metrics-74cb57df7-4cgxb\" (UID: \"09d9c8c2-604a-481d-9e58-57bbb07b2e03\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-4cgxb"
Sep 08 15:34:27.673930 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:27.673906    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2f94h\" (UniqueName: \"kubernetes.io/projected/6865c4b2-f146-4d9f-88cf-2eaa6460880a-kube-api-access-2f94h\") pod \"node-exporter-phnv9\" (UID: \"6865c4b2-f146-4d9f-88cf-2eaa6460880a\") " pod="openshift-monitoring/node-exporter-phnv9"
Sep 08 15:34:27.674008 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:27.673936    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hn28f\" (UniqueName: \"kubernetes.io/projected/e942dd64-4a92-449d-b013-7ac71c351448-kube-api-access-hn28f\") pod \"openshift-state-metrics-657fd6c54-mpbc4\" (UID: \"e942dd64-4a92-449d-b013-7ac71c351448\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-mpbc4"
Sep 08 15:34:27.720147 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:27.720120    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-mpbc4"
Sep 08 15:34:27.754015 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:27.753980    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-74cb57df7-4cgxb"
Sep 08 15:34:27.759022 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:27.758992    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-phnv9"
Sep 08 15:34:27.825738 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:27.825709    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-phnv9" event={"ID":"6865c4b2-f146-4d9f-88cf-2eaa6460880a","Type":"ContainerStarted","Data":"c9e4ddc9d5dde4197c2bd86c47e027883f7b5fc33c309c020c91fb7cfbc10d02"}
Sep 08 15:34:27.991701 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:27.991675    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-mpbc4"]
Sep 08 15:34:27.993922 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:34:27.993898    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode942dd64_4a92_449d_b013_7ac71c351448.slice/crio-d81e815d8099b970e544924a9fdaacefe82b4e8eb228026d73d06330d0a3d42e WatchSource:0}: Error finding container d81e815d8099b970e544924a9fdaacefe82b4e8eb228026d73d06330d0a3d42e: Status 404 returned error can't find the container with id d81e815d8099b970e544924a9fdaacefe82b4e8eb228026d73d06330d0a3d42e
Sep 08 15:34:28.029423 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:28.029401    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-4cgxb"]
Sep 08 15:34:28.031017 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:34:28.030995    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09d9c8c2_604a_481d_9e58_57bbb07b2e03.slice/crio-3966253525d4c74485c75dad8b3e87e03a6d653639591685c6d5957071f423e4 WatchSource:0}: Error finding container 3966253525d4c74485c75dad8b3e87e03a6d653639591685c6d5957071f423e4: Status 404 returned error can't find the container with id 3966253525d4c74485c75dad8b3e87e03a6d653639591685c6d5957071f423e4
Sep 08 15:34:28.395569 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:28.395538    2549 scope.go:117] "RemoveContainer" containerID="aaf52d23e726e627052afdc3a46d710e3dc92dd6aa771f005236e578e6698d54"
Sep 08 15:34:28.395748 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:34:28.395728    2549 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-85b454874b-wpwdq_openshift-console-operator(5f7b0f68-a8e3-4058-b032-15f419682b2a)\"" pod="openshift-console-operator/console-operator-85b454874b-wpwdq" podUID="5f7b0f68-a8e3-4058-b032-15f419682b2a"
Sep 08 15:34:28.831011 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:28.830981    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-4cgxb" event={"ID":"09d9c8c2-604a-481d-9e58-57bbb07b2e03","Type":"ContainerStarted","Data":"3966253525d4c74485c75dad8b3e87e03a6d653639591685c6d5957071f423e4"}
Sep 08 15:34:28.833304 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:28.833278    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-mpbc4" event={"ID":"e942dd64-4a92-449d-b013-7ac71c351448","Type":"ContainerStarted","Data":"ee85ee6f09da004b28941ead95fd75cb219e81a9370a3405d6d05c6fc71e13fb"}
Sep 08 15:34:28.833414 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:28.833340    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-mpbc4" event={"ID":"e942dd64-4a92-449d-b013-7ac71c351448","Type":"ContainerStarted","Data":"9cd3367556eae81890f227cc8c9e465162a60b6248cd04f8202439739aa8ec6e"}
Sep 08 15:34:28.833414 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:28.833371    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-mpbc4" event={"ID":"e942dd64-4a92-449d-b013-7ac71c351448","Type":"ContainerStarted","Data":"d81e815d8099b970e544924a9fdaacefe82b4e8eb228026d73d06330d0a3d42e"}
Sep 08 15:34:29.435875 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:29.435816    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-68f45db6d-mt86k"]
Sep 08 15:34:29.441497 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:29.441462    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-68f45db6d-mt86k"
Sep 08 15:34:29.444371 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:29.444343    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Sep 08 15:34:29.444508 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:29.444392    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Sep 08 15:34:29.444661 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:29.444639    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-ancp70la1m186\""
Sep 08 15:34:29.444735 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:29.444694    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Sep 08 15:34:29.444735 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:29.444649    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Sep 08 15:34:29.445051 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:29.444980    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Sep 08 15:34:29.445676 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:29.445658    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-bh6nc\""
Sep 08 15:34:29.452109 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:29.452082    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-68f45db6d-mt86k"]
Sep 08 15:34:29.584287 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:29.584263    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/87979b2e-c099-493e-b4eb-03264e005462-secret-grpc-tls\") pod \"thanos-querier-68f45db6d-mt86k\" (UID: \"87979b2e-c099-493e-b4eb-03264e005462\") " pod="openshift-monitoring/thanos-querier-68f45db6d-mt86k"
Sep 08 15:34:29.584387 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:29.584302    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/87979b2e-c099-493e-b4eb-03264e005462-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-68f45db6d-mt86k\" (UID: \"87979b2e-c099-493e-b4eb-03264e005462\") " pod="openshift-monitoring/thanos-querier-68f45db6d-mt86k"
Sep 08 15:34:29.584387 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:29.584321    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/87979b2e-c099-493e-b4eb-03264e005462-secret-thanos-querier-tls\") pod \"thanos-querier-68f45db6d-mt86k\" (UID: \"87979b2e-c099-493e-b4eb-03264e005462\") " pod="openshift-monitoring/thanos-querier-68f45db6d-mt86k"
Sep 08 15:34:29.584387 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:29.584343    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sb5r6\" (UniqueName: \"kubernetes.io/projected/87979b2e-c099-493e-b4eb-03264e005462-kube-api-access-sb5r6\") pod \"thanos-querier-68f45db6d-mt86k\" (UID: \"87979b2e-c099-493e-b4eb-03264e005462\") " pod="openshift-monitoring/thanos-querier-68f45db6d-mt86k"
Sep 08 15:34:29.584496 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:29.584473    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/87979b2e-c099-493e-b4eb-03264e005462-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-68f45db6d-mt86k\" (UID: \"87979b2e-c099-493e-b4eb-03264e005462\") " pod="openshift-monitoring/thanos-querier-68f45db6d-mt86k"
Sep 08 15:34:29.584530 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:29.584514    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/87979b2e-c099-493e-b4eb-03264e005462-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-68f45db6d-mt86k\" (UID: \"87979b2e-c099-493e-b4eb-03264e005462\") " pod="openshift-monitoring/thanos-querier-68f45db6d-mt86k"
Sep 08 15:34:29.584578 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:29.584563    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/87979b2e-c099-493e-b4eb-03264e005462-metrics-client-ca\") pod \"thanos-querier-68f45db6d-mt86k\" (UID: \"87979b2e-c099-493e-b4eb-03264e005462\") " pod="openshift-monitoring/thanos-querier-68f45db6d-mt86k"
Sep 08 15:34:29.584621 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:29.584607    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/87979b2e-c099-493e-b4eb-03264e005462-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-68f45db6d-mt86k\" (UID: \"87979b2e-c099-493e-b4eb-03264e005462\") " pod="openshift-monitoring/thanos-querier-68f45db6d-mt86k"
Sep 08 15:34:29.685646 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:29.685611    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/87979b2e-c099-493e-b4eb-03264e005462-secret-grpc-tls\") pod \"thanos-querier-68f45db6d-mt86k\" (UID: \"87979b2e-c099-493e-b4eb-03264e005462\") " pod="openshift-monitoring/thanos-querier-68f45db6d-mt86k"
Sep 08 15:34:29.685774 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:29.685677    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/87979b2e-c099-493e-b4eb-03264e005462-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-68f45db6d-mt86k\" (UID: \"87979b2e-c099-493e-b4eb-03264e005462\") " pod="openshift-monitoring/thanos-querier-68f45db6d-mt86k"
Sep 08 15:34:29.685774 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:29.685709    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/87979b2e-c099-493e-b4eb-03264e005462-secret-thanos-querier-tls\") pod \"thanos-querier-68f45db6d-mt86k\" (UID: \"87979b2e-c099-493e-b4eb-03264e005462\") " pod="openshift-monitoring/thanos-querier-68f45db6d-mt86k"
Sep 08 15:34:29.685774 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:29.685756    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sb5r6\" (UniqueName: \"kubernetes.io/projected/87979b2e-c099-493e-b4eb-03264e005462-kube-api-access-sb5r6\") pod \"thanos-querier-68f45db6d-mt86k\" (UID: \"87979b2e-c099-493e-b4eb-03264e005462\") " pod="openshift-monitoring/thanos-querier-68f45db6d-mt86k"
Sep 08 15:34:29.685984 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:29.685827    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/87979b2e-c099-493e-b4eb-03264e005462-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-68f45db6d-mt86k\" (UID: \"87979b2e-c099-493e-b4eb-03264e005462\") " pod="openshift-monitoring/thanos-querier-68f45db6d-mt86k"
Sep 08 15:34:29.685984 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:29.685868    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/87979b2e-c099-493e-b4eb-03264e005462-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-68f45db6d-mt86k\" (UID: \"87979b2e-c099-493e-b4eb-03264e005462\") " pod="openshift-monitoring/thanos-querier-68f45db6d-mt86k"
Sep 08 15:34:29.685984 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:29.685901    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/87979b2e-c099-493e-b4eb-03264e005462-metrics-client-ca\") pod \"thanos-querier-68f45db6d-mt86k\" (UID: \"87979b2e-c099-493e-b4eb-03264e005462\") " pod="openshift-monitoring/thanos-querier-68f45db6d-mt86k"
Sep 08 15:34:29.685984 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:29.685944    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/87979b2e-c099-493e-b4eb-03264e005462-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-68f45db6d-mt86k\" (UID: \"87979b2e-c099-493e-b4eb-03264e005462\") " pod="openshift-monitoring/thanos-querier-68f45db6d-mt86k"
Sep 08 15:34:29.687503 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:29.687185    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/87979b2e-c099-493e-b4eb-03264e005462-metrics-client-ca\") pod \"thanos-querier-68f45db6d-mt86k\" (UID: \"87979b2e-c099-493e-b4eb-03264e005462\") " pod="openshift-monitoring/thanos-querier-68f45db6d-mt86k"
Sep 08 15:34:29.691934 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:29.691888    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/87979b2e-c099-493e-b4eb-03264e005462-secret-grpc-tls\") pod \"thanos-querier-68f45db6d-mt86k\" (UID: \"87979b2e-c099-493e-b4eb-03264e005462\") " pod="openshift-monitoring/thanos-querier-68f45db6d-mt86k"
Sep 08 15:34:29.692618 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:29.692340    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/87979b2e-c099-493e-b4eb-03264e005462-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-68f45db6d-mt86k\" (UID: \"87979b2e-c099-493e-b4eb-03264e005462\") " pod="openshift-monitoring/thanos-querier-68f45db6d-mt86k"
Sep 08 15:34:29.692618 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:29.692482    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/87979b2e-c099-493e-b4eb-03264e005462-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-68f45db6d-mt86k\" (UID: \"87979b2e-c099-493e-b4eb-03264e005462\") " pod="openshift-monitoring/thanos-querier-68f45db6d-mt86k"
Sep 08 15:34:29.693295 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:29.692996    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/87979b2e-c099-493e-b4eb-03264e005462-secret-thanos-querier-tls\") pod \"thanos-querier-68f45db6d-mt86k\" (UID: \"87979b2e-c099-493e-b4eb-03264e005462\") " pod="openshift-monitoring/thanos-querier-68f45db6d-mt86k"
Sep 08 15:34:29.693295 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:29.693253    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/87979b2e-c099-493e-b4eb-03264e005462-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-68f45db6d-mt86k\" (UID: \"87979b2e-c099-493e-b4eb-03264e005462\") " pod="openshift-monitoring/thanos-querier-68f45db6d-mt86k"
Sep 08 15:34:29.694341 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:29.694309    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/87979b2e-c099-493e-b4eb-03264e005462-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-68f45db6d-mt86k\" (UID: \"87979b2e-c099-493e-b4eb-03264e005462\") " pod="openshift-monitoring/thanos-querier-68f45db6d-mt86k"
Sep 08 15:34:29.698719 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:29.698697    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sb5r6\" (UniqueName: \"kubernetes.io/projected/87979b2e-c099-493e-b4eb-03264e005462-kube-api-access-sb5r6\") pod \"thanos-querier-68f45db6d-mt86k\" (UID: \"87979b2e-c099-493e-b4eb-03264e005462\") " pod="openshift-monitoring/thanos-querier-68f45db6d-mt86k"
Sep 08 15:34:29.752963 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:29.752931    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-68f45db6d-mt86k"
Sep 08 15:34:29.840048 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:29.840019    2549 generic.go:356] "Generic (PLEG): container finished" podID="6865c4b2-f146-4d9f-88cf-2eaa6460880a" containerID="175bcbe1fdd1dcbac9c9da3bfeb07fee62aa653f7bc8105a83f6d2600c3e19b7" exitCode=0
Sep 08 15:34:29.840446 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:29.840057    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-phnv9" event={"ID":"6865c4b2-f146-4d9f-88cf-2eaa6460880a","Type":"ContainerDied","Data":"175bcbe1fdd1dcbac9c9da3bfeb07fee62aa653f7bc8105a83f6d2600c3e19b7"}
Sep 08 15:34:29.843001 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:29.842967    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-4cgxb" event={"ID":"09d9c8c2-604a-481d-9e58-57bbb07b2e03","Type":"ContainerStarted","Data":"00ce303660ea609064b253d15db595be78223e45b027b3bc2d76a20f435fc6fa"}
Sep 08 15:34:29.843105 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:29.843028    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-4cgxb" event={"ID":"09d9c8c2-604a-481d-9e58-57bbb07b2e03","Type":"ContainerStarted","Data":"61406dccfbd7b0a6b838f503cc162e6b6f3c5dab85749dff2311632d969efc9b"}
Sep 08 15:34:29.843105 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:29.843058    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-4cgxb" event={"ID":"09d9c8c2-604a-481d-9e58-57bbb07b2e03","Type":"ContainerStarted","Data":"47d08adaac8785729c3e3a51d9404538cafc723e1b545765df8b5556da906fbf"}
Sep 08 15:34:29.846179 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:29.846158    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-mpbc4" event={"ID":"e942dd64-4a92-449d-b013-7ac71c351448","Type":"ContainerStarted","Data":"97de221f77025b213782d77b9e5d79025229c8737bb7ea0ea878d430f7006106"}
Sep 08 15:34:29.888627 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:29.888584    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-74cb57df7-4cgxb" podStartSLOduration=1.394166216 podStartE2EDuration="2.888566722s" totalImagesPullingTime="1.494400506s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:34:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:34:29.888270512 +0000 UTC m=+107.977721600" watchObservedRunningTime="2026-09-08 15:34:29.888566722 +0000 UTC m=+107.978017810"
Sep 08 15:34:29.906908 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:29.906841    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-mpbc4" podStartSLOduration=1.49629491 podStartE2EDuration="2.906826167s" totalImagesPullingTime="1.410531257s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:34:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:34:29.906314912 +0000 UTC m=+107.995766003" watchObservedRunningTime="2026-09-08 15:34:29.906826167 +0000 UTC m=+107.996277256"
Sep 08 15:34:30.033645 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:30.033614    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-68f45db6d-mt86k"]
Sep 08 15:34:30.037214 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:34:30.037188    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87979b2e_c099_493e_b4eb_03264e005462.slice/crio-7966dbcde0c7e02601d23a63d7c24ef585e8fbfdff1107284984b992fc3585d0 WatchSource:0}: Error finding container 7966dbcde0c7e02601d23a63d7c24ef585e8fbfdff1107284984b992fc3585d0: Status 404 returned error can't find the container with id 7966dbcde0c7e02601d23a63d7c24ef585e8fbfdff1107284984b992fc3585d0
Sep 08 15:34:30.853261 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:30.853219    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-phnv9" event={"ID":"6865c4b2-f146-4d9f-88cf-2eaa6460880a","Type":"ContainerStarted","Data":"0aa3de71be6ab7f678d93f96fce89d84eee4d47a0fb9de57cf0fa8f33715453d"}
Sep 08 15:34:30.853806 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:30.853287    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-phnv9" event={"ID":"6865c4b2-f146-4d9f-88cf-2eaa6460880a","Type":"ContainerStarted","Data":"fa70f3650b28c289a6e4ed5ed0d33e8bbe1692da6ae6fe6fd02aa4c5a7e509cf"}
Sep 08 15:34:30.855013 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:30.854985    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-68f45db6d-mt86k" event={"ID":"87979b2e-c099-493e-b4eb-03264e005462","Type":"ContainerStarted","Data":"7966dbcde0c7e02601d23a63d7c24ef585e8fbfdff1107284984b992fc3585d0"}
Sep 08 15:34:30.881625 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:30.881583    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-phnv9" podStartSLOduration=2.794023171 podStartE2EDuration="3.881568978s" totalImagesPullingTime="1.017372566s" totalInitContainerRuntime="70.173241ms" isStatefulPod=true podCreationTimestamp="2026-09-08 15:34:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:34:30.880372422 +0000 UTC m=+108.969823511" watchObservedRunningTime="2026-09-08 15:34:30.881568978 +0000 UTC m=+108.971020066"
Sep 08 15:34:31.761844 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:31.761809    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-699f699998-xsc8g"]
Sep 08 15:34:31.764983 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:31.764962    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-699f699998-xsc8g"
Sep 08 15:34:31.769969 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:31.768235    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Sep 08 15:34:31.770246 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:31.770221    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Sep 08 15:34:31.770363 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:31.770305    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 08 15:34:31.770436 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:31.770370    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Sep 08 15:34:31.770436 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:31.770419    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-tmtgj\""
Sep 08 15:34:31.771029 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:31.770999    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-2rv4gabia9e6p\""
Sep 08 15:34:31.780988 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:31.780966    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-699f699998-xsc8g"]
Sep 08 15:34:31.909049 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:31.909017    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c03ad4e-85a7-4ca0-ad0a-389282c57543-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-699f699998-xsc8g\" (UID: \"7c03ad4e-85a7-4ca0-ad0a-389282c57543\") " pod="openshift-monitoring/metrics-server-699f699998-xsc8g"
Sep 08 15:34:31.909049 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:31.909055    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/7c03ad4e-85a7-4ca0-ad0a-389282c57543-secret-metrics-server-client-certs\") pod \"metrics-server-699f699998-xsc8g\" (UID: \"7c03ad4e-85a7-4ca0-ad0a-389282c57543\") " pod="openshift-monitoring/metrics-server-699f699998-xsc8g"
Sep 08 15:34:31.909465 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:31.909081    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/7c03ad4e-85a7-4ca0-ad0a-389282c57543-secret-metrics-server-tls\") pod \"metrics-server-699f699998-xsc8g\" (UID: \"7c03ad4e-85a7-4ca0-ad0a-389282c57543\") " pod="openshift-monitoring/metrics-server-699f699998-xsc8g"
Sep 08 15:34:31.909465 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:31.909176    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmsh9\" (UniqueName: \"kubernetes.io/projected/7c03ad4e-85a7-4ca0-ad0a-389282c57543-kube-api-access-rmsh9\") pod \"metrics-server-699f699998-xsc8g\" (UID: \"7c03ad4e-85a7-4ca0-ad0a-389282c57543\") " pod="openshift-monitoring/metrics-server-699f699998-xsc8g"
Sep 08 15:34:31.909465 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:31.909203    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/7c03ad4e-85a7-4ca0-ad0a-389282c57543-client-ca-bundle\") pod \"metrics-server-699f699998-xsc8g\" (UID: \"7c03ad4e-85a7-4ca0-ad0a-389282c57543\") " pod="openshift-monitoring/metrics-server-699f699998-xsc8g"
Sep 08 15:34:31.909465 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:31.909235    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/7c03ad4e-85a7-4ca0-ad0a-389282c57543-metrics-server-audit-profiles\") pod \"metrics-server-699f699998-xsc8g\" (UID: \"7c03ad4e-85a7-4ca0-ad0a-389282c57543\") " pod="openshift-monitoring/metrics-server-699f699998-xsc8g"
Sep 08 15:34:31.909465 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:31.909279    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/7c03ad4e-85a7-4ca0-ad0a-389282c57543-audit-log\") pod \"metrics-server-699f699998-xsc8g\" (UID: \"7c03ad4e-85a7-4ca0-ad0a-389282c57543\") " pod="openshift-monitoring/metrics-server-699f699998-xsc8g"
Sep 08 15:34:32.010205 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:32.010177    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/7c03ad4e-85a7-4ca0-ad0a-389282c57543-secret-metrics-server-client-certs\") pod \"metrics-server-699f699998-xsc8g\" (UID: \"7c03ad4e-85a7-4ca0-ad0a-389282c57543\") " pod="openshift-monitoring/metrics-server-699f699998-xsc8g"
Sep 08 15:34:32.010322 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:32.010228    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/7c03ad4e-85a7-4ca0-ad0a-389282c57543-secret-metrics-server-tls\") pod \"metrics-server-699f699998-xsc8g\" (UID: \"7c03ad4e-85a7-4ca0-ad0a-389282c57543\") " pod="openshift-monitoring/metrics-server-699f699998-xsc8g"
Sep 08 15:34:32.010322 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:32.010264    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmsh9\" (UniqueName: \"kubernetes.io/projected/7c03ad4e-85a7-4ca0-ad0a-389282c57543-kube-api-access-rmsh9\") pod \"metrics-server-699f699998-xsc8g\" (UID: \"7c03ad4e-85a7-4ca0-ad0a-389282c57543\") " pod="openshift-monitoring/metrics-server-699f699998-xsc8g"
Sep 08 15:34:32.010322 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:32.010285    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/7c03ad4e-85a7-4ca0-ad0a-389282c57543-client-ca-bundle\") pod \"metrics-server-699f699998-xsc8g\" (UID: \"7c03ad4e-85a7-4ca0-ad0a-389282c57543\") " pod="openshift-monitoring/metrics-server-699f699998-xsc8g"
Sep 08 15:34:32.010478 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:32.010320    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/7c03ad4e-85a7-4ca0-ad0a-389282c57543-metrics-server-audit-profiles\") pod \"metrics-server-699f699998-xsc8g\" (UID: \"7c03ad4e-85a7-4ca0-ad0a-389282c57543\") " pod="openshift-monitoring/metrics-server-699f699998-xsc8g"
Sep 08 15:34:32.010641 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:32.010561    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/7c03ad4e-85a7-4ca0-ad0a-389282c57543-audit-log\") pod \"metrics-server-699f699998-xsc8g\" (UID: \"7c03ad4e-85a7-4ca0-ad0a-389282c57543\") " pod="openshift-monitoring/metrics-server-699f699998-xsc8g"
Sep 08 15:34:32.010641 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:32.010627    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c03ad4e-85a7-4ca0-ad0a-389282c57543-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-699f699998-xsc8g\" (UID: \"7c03ad4e-85a7-4ca0-ad0a-389282c57543\") " pod="openshift-monitoring/metrics-server-699f699998-xsc8g"
Sep 08 15:34:32.011969 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:32.011013    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/7c03ad4e-85a7-4ca0-ad0a-389282c57543-audit-log\") pod \"metrics-server-699f699998-xsc8g\" (UID: \"7c03ad4e-85a7-4ca0-ad0a-389282c57543\") " pod="openshift-monitoring/metrics-server-699f699998-xsc8g"
Sep 08 15:34:32.011969 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:32.011266    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c03ad4e-85a7-4ca0-ad0a-389282c57543-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-699f699998-xsc8g\" (UID: \"7c03ad4e-85a7-4ca0-ad0a-389282c57543\") " pod="openshift-monitoring/metrics-server-699f699998-xsc8g"
Sep 08 15:34:32.011969 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:32.011808    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/7c03ad4e-85a7-4ca0-ad0a-389282c57543-metrics-server-audit-profiles\") pod \"metrics-server-699f699998-xsc8g\" (UID: \"7c03ad4e-85a7-4ca0-ad0a-389282c57543\") " pod="openshift-monitoring/metrics-server-699f699998-xsc8g"
Sep 08 15:34:32.013103 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:32.013086    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/7c03ad4e-85a7-4ca0-ad0a-389282c57543-client-ca-bundle\") pod \"metrics-server-699f699998-xsc8g\" (UID: \"7c03ad4e-85a7-4ca0-ad0a-389282c57543\") " pod="openshift-monitoring/metrics-server-699f699998-xsc8g"
Sep 08 15:34:32.013319 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:32.013296    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/7c03ad4e-85a7-4ca0-ad0a-389282c57543-secret-metrics-server-tls\") pod \"metrics-server-699f699998-xsc8g\" (UID: \"7c03ad4e-85a7-4ca0-ad0a-389282c57543\") " pod="openshift-monitoring/metrics-server-699f699998-xsc8g"
Sep 08 15:34:32.013359 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:32.013348    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/7c03ad4e-85a7-4ca0-ad0a-389282c57543-secret-metrics-server-client-certs\") pod \"metrics-server-699f699998-xsc8g\" (UID: \"7c03ad4e-85a7-4ca0-ad0a-389282c57543\") " pod="openshift-monitoring/metrics-server-699f699998-xsc8g"
Sep 08 15:34:32.022347 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:32.022319    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmsh9\" (UniqueName: \"kubernetes.io/projected/7c03ad4e-85a7-4ca0-ad0a-389282c57543-kube-api-access-rmsh9\") pod \"metrics-server-699f699998-xsc8g\" (UID: \"7c03ad4e-85a7-4ca0-ad0a-389282c57543\") " pod="openshift-monitoring/metrics-server-699f699998-xsc8g"
Sep 08 15:34:32.078731 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:32.078709    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-699f699998-xsc8g"
Sep 08 15:34:32.355821 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:32.355747    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-699f699998-xsc8g"]
Sep 08 15:34:32.359235 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:34:32.359209    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c03ad4e_85a7_4ca0_ad0a_389282c57543.slice/crio-695f23b7fd9652f7fdf8952caab507cb45c48a874a99b50987be19a38ec6387b WatchSource:0}: Error finding container 695f23b7fd9652f7fdf8952caab507cb45c48a874a99b50987be19a38ec6387b: Status 404 returned error can't find the container with id 695f23b7fd9652f7fdf8952caab507cb45c48a874a99b50987be19a38ec6387b
Sep 08 15:34:32.775641 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:32.775613    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5fc68db7dd-pl9pd"
Sep 08 15:34:32.810979 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:32.810954    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-vhsng"
Sep 08 15:34:32.874610 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:32.874570    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-68f45db6d-mt86k" event={"ID":"87979b2e-c099-493e-b4eb-03264e005462","Type":"ContainerStarted","Data":"8dc0a65eed1f987465da2cd33e5a684a8c5c3101a486cac6d7107e6565651d54"}
Sep 08 15:34:32.874781 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:32.874688    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-68f45db6d-mt86k" event={"ID":"87979b2e-c099-493e-b4eb-03264e005462","Type":"ContainerStarted","Data":"13449585943f6afdcf4cfcdd4b8aca79a332007c3dfcca71a816b24be493b6bb"}
Sep 08 15:34:32.874781 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:32.874730    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-68f45db6d-mt86k" event={"ID":"87979b2e-c099-493e-b4eb-03264e005462","Type":"ContainerStarted","Data":"2fd9bdfc49e042ab7146d4c3550d3ec1a59cdcf076f298719af153db9b0aab53"}
Sep 08 15:34:32.876410 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:32.876384    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-699f699998-xsc8g" event={"ID":"7c03ad4e-85a7-4ca0-ad0a-389282c57543","Type":"ContainerStarted","Data":"695f23b7fd9652f7fdf8952caab507cb45c48a874a99b50987be19a38ec6387b"}
Sep 08 15:34:33.883204 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:33.883172    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-68f45db6d-mt86k" event={"ID":"87979b2e-c099-493e-b4eb-03264e005462","Type":"ContainerStarted","Data":"24a0aedca6446c73ccd13a28ae15359c0a13352555b6c63cdb8a61a04231961d"}
Sep 08 15:34:33.883569 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:33.883244    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-68f45db6d-mt86k" event={"ID":"87979b2e-c099-493e-b4eb-03264e005462","Type":"ContainerStarted","Data":"01cb81dc063335545cc08a2ef27a140d1ac43d67a0c88916463edcdfb7e50f79"}
Sep 08 15:34:33.883569 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:33.883304    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-68f45db6d-mt86k" event={"ID":"87979b2e-c099-493e-b4eb-03264e005462","Type":"ContainerStarted","Data":"70a21b47119cdf4924a161b86daaa978c60fda8e20f4a6ea76bba8e266874213"}
Sep 08 15:34:33.883569 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:33.883353    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-68f45db6d-mt86k"
Sep 08 15:34:33.884960 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:33.884935    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-699f699998-xsc8g" event={"ID":"7c03ad4e-85a7-4ca0-ad0a-389282c57543","Type":"ContainerStarted","Data":"f73e35873009b4581a264ec896d8c2c9e656fa87ee5036d1876e62cf2ea3d04e"}
Sep 08 15:34:33.907451 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:33.907407    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-68f45db6d-mt86k" podStartSLOduration=2.110696879 podStartE2EDuration="4.907393232s" totalImagesPullingTime="2.796696353s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:34:29 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:34:33.905614006 +0000 UTC m=+111.995065096" watchObservedRunningTime="2026-09-08 15:34:33.907393232 +0000 UTC m=+111.996844390"
Sep 08 15:34:33.926551 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:33.926513    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-699f699998-xsc8g" podStartSLOduration=1.518790794 podStartE2EDuration="2.926500049s" totalImagesPullingTime="1.407709255s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:34:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:34:33.925708328 +0000 UTC m=+112.015159416" watchObservedRunningTime="2026-09-08 15:34:33.926500049 +0000 UTC m=+112.015951137"
Sep 08 15:34:35.940773 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:35.940730    2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-95fbf4c66-9k8sq" podUID="0ecfb56c-9113-49a2-99c8-226c564c32d5" containerName="registry" containerID="cri-o://e8a62c67fc534844b56bf1fda0dd9128412069cfeff388d1aa91df46bc47ba85" gracePeriod=30
Sep 08 15:34:36.173685 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:36.173661    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-95fbf4c66-9k8sq"
Sep 08 15:34:36.348712 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:36.348645    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0ecfb56c-9113-49a2-99c8-226c564c32d5-registry-tls\") pod \"0ecfb56c-9113-49a2-99c8-226c564c32d5\" (UID: \"0ecfb56c-9113-49a2-99c8-226c564c32d5\") "
Sep 08 15:34:36.348712 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:36.348677    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0ecfb56c-9113-49a2-99c8-226c564c32d5-image-registry-private-configuration\") pod \"0ecfb56c-9113-49a2-99c8-226c564c32d5\" (UID: \"0ecfb56c-9113-49a2-99c8-226c564c32d5\") "
Sep 08 15:34:36.348712 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:36.348698    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wm2zh\" (UniqueName: \"kubernetes.io/projected/0ecfb56c-9113-49a2-99c8-226c564c32d5-kube-api-access-wm2zh\") pod \"0ecfb56c-9113-49a2-99c8-226c564c32d5\" (UID: \"0ecfb56c-9113-49a2-99c8-226c564c32d5\") "
Sep 08 15:34:36.348974 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:36.348742    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0ecfb56c-9113-49a2-99c8-226c564c32d5-bound-sa-token\") pod \"0ecfb56c-9113-49a2-99c8-226c564c32d5\" (UID: \"0ecfb56c-9113-49a2-99c8-226c564c32d5\") "
Sep 08 15:34:36.348974 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:36.348766    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0ecfb56c-9113-49a2-99c8-226c564c32d5-registry-certificates\") pod \"0ecfb56c-9113-49a2-99c8-226c564c32d5\" (UID: \"0ecfb56c-9113-49a2-99c8-226c564c32d5\") "
Sep 08 15:34:36.348974 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:36.348790    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0ecfb56c-9113-49a2-99c8-226c564c32d5-trusted-ca\") pod \"0ecfb56c-9113-49a2-99c8-226c564c32d5\" (UID: \"0ecfb56c-9113-49a2-99c8-226c564c32d5\") "
Sep 08 15:34:36.348974 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:36.348816    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0ecfb56c-9113-49a2-99c8-226c564c32d5-installation-pull-secrets\") pod \"0ecfb56c-9113-49a2-99c8-226c564c32d5\" (UID: \"0ecfb56c-9113-49a2-99c8-226c564c32d5\") "
Sep 08 15:34:36.348974 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:36.348886    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0ecfb56c-9113-49a2-99c8-226c564c32d5-ca-trust-extracted\") pod \"0ecfb56c-9113-49a2-99c8-226c564c32d5\" (UID: \"0ecfb56c-9113-49a2-99c8-226c564c32d5\") "
Sep 08 15:34:36.349441 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:36.349274    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ecfb56c-9113-49a2-99c8-226c564c32d5-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "0ecfb56c-9113-49a2-99c8-226c564c32d5" (UID: "0ecfb56c-9113-49a2-99c8-226c564c32d5"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 15:34:36.349441 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:36.349404    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ecfb56c-9113-49a2-99c8-226c564c32d5-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "0ecfb56c-9113-49a2-99c8-226c564c32d5" (UID: "0ecfb56c-9113-49a2-99c8-226c564c32d5"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 15:34:36.351299 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:36.351274    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ecfb56c-9113-49a2-99c8-226c564c32d5-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "0ecfb56c-9113-49a2-99c8-226c564c32d5" (UID: "0ecfb56c-9113-49a2-99c8-226c564c32d5"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:34:36.351395 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:36.351350    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ecfb56c-9113-49a2-99c8-226c564c32d5-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "0ecfb56c-9113-49a2-99c8-226c564c32d5" (UID: "0ecfb56c-9113-49a2-99c8-226c564c32d5"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 15:34:36.351395 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:36.351372    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ecfb56c-9113-49a2-99c8-226c564c32d5-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "0ecfb56c-9113-49a2-99c8-226c564c32d5" (UID: "0ecfb56c-9113-49a2-99c8-226c564c32d5"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:34:36.351523 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:36.351500    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ecfb56c-9113-49a2-99c8-226c564c32d5-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "0ecfb56c-9113-49a2-99c8-226c564c32d5" (UID: "0ecfb56c-9113-49a2-99c8-226c564c32d5"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 15:34:36.351577 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:36.351562    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ecfb56c-9113-49a2-99c8-226c564c32d5-kube-api-access-wm2zh" (OuterVolumeSpecName: "kube-api-access-wm2zh") pod "0ecfb56c-9113-49a2-99c8-226c564c32d5" (UID: "0ecfb56c-9113-49a2-99c8-226c564c32d5"). InnerVolumeSpecName "kube-api-access-wm2zh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:34:36.357306 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:36.357287    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ecfb56c-9113-49a2-99c8-226c564c32d5-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "0ecfb56c-9113-49a2-99c8-226c564c32d5" (UID: "0ecfb56c-9113-49a2-99c8-226c564c32d5"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:34:36.450392 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:36.450372    2549 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0ecfb56c-9113-49a2-99c8-226c564c32d5-bound-sa-token\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:34:36.450392 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:36.450394    2549 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0ecfb56c-9113-49a2-99c8-226c564c32d5-registry-certificates\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:34:36.450508 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:36.450404    2549 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0ecfb56c-9113-49a2-99c8-226c564c32d5-trusted-ca\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:34:36.450508 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:36.450413    2549 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0ecfb56c-9113-49a2-99c8-226c564c32d5-installation-pull-secrets\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:34:36.450508 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:36.450421    2549 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0ecfb56c-9113-49a2-99c8-226c564c32d5-ca-trust-extracted\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:34:36.450508 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:36.450429    2549 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0ecfb56c-9113-49a2-99c8-226c564c32d5-registry-tls\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:34:36.450508 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:36.450436    2549 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0ecfb56c-9113-49a2-99c8-226c564c32d5-image-registry-private-configuration\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:34:36.450508 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:36.450444    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wm2zh\" (UniqueName: \"kubernetes.io/projected/0ecfb56c-9113-49a2-99c8-226c564c32d5-kube-api-access-wm2zh\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:34:36.897936 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:36.897903    2549 generic.go:356] "Generic (PLEG): container finished" podID="0ecfb56c-9113-49a2-99c8-226c564c32d5" containerID="e8a62c67fc534844b56bf1fda0dd9128412069cfeff388d1aa91df46bc47ba85" exitCode=0
Sep 08 15:34:36.898104 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:36.897961    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-95fbf4c66-9k8sq"
Sep 08 15:34:36.898104 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:36.897970    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-95fbf4c66-9k8sq" event={"ID":"0ecfb56c-9113-49a2-99c8-226c564c32d5","Type":"ContainerDied","Data":"e8a62c67fc534844b56bf1fda0dd9128412069cfeff388d1aa91df46bc47ba85"}
Sep 08 15:34:36.898104 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:36.898009    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-95fbf4c66-9k8sq" event={"ID":"0ecfb56c-9113-49a2-99c8-226c564c32d5","Type":"ContainerDied","Data":"de6afdcec9a5fac7c2c905f42733c3d7b57427554935d9624a097b1da504534c"}
Sep 08 15:34:36.898104 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:36.898028    2549 scope.go:117] "RemoveContainer" containerID="e8a62c67fc534844b56bf1fda0dd9128412069cfeff388d1aa91df46bc47ba85"
Sep 08 15:34:36.905709 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:36.905687    2549 scope.go:117] "RemoveContainer" containerID="e8a62c67fc534844b56bf1fda0dd9128412069cfeff388d1aa91df46bc47ba85"
Sep 08 15:34:36.906001 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:34:36.905981    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8a62c67fc534844b56bf1fda0dd9128412069cfeff388d1aa91df46bc47ba85\": container with ID starting with e8a62c67fc534844b56bf1fda0dd9128412069cfeff388d1aa91df46bc47ba85 not found: ID does not exist" containerID="e8a62c67fc534844b56bf1fda0dd9128412069cfeff388d1aa91df46bc47ba85"
Sep 08 15:34:36.906068 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:36.906013    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8a62c67fc534844b56bf1fda0dd9128412069cfeff388d1aa91df46bc47ba85"} err="failed to get container status \"e8a62c67fc534844b56bf1fda0dd9128412069cfeff388d1aa91df46bc47ba85\": rpc error: code = NotFound desc = could not find container \"e8a62c67fc534844b56bf1fda0dd9128412069cfeff388d1aa91df46bc47ba85\": container with ID starting with e8a62c67fc534844b56bf1fda0dd9128412069cfeff388d1aa91df46bc47ba85 not found: ID does not exist"
Sep 08 15:34:36.922000 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:36.921978    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-95fbf4c66-9k8sq"]
Sep 08 15:34:36.929366 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:36.929349    2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-95fbf4c66-9k8sq"]
Sep 08 15:34:38.399070 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:38.399031    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0ecfb56c-9113-49a2-99c8-226c564c32d5" path="/var/lib/kubelet/pods/0ecfb56c-9113-49a2-99c8-226c564c32d5/volumes"
Sep 08 15:34:39.895521 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:39.895496    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-68f45db6d-mt86k"
Sep 08 15:34:43.395190 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:43.395161    2549 scope.go:117] "RemoveContainer" containerID="aaf52d23e726e627052afdc3a46d710e3dc92dd6aa771f005236e578e6698d54"
Sep 08 15:34:43.925705 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:43.925680    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-wpwdq_5f7b0f68-a8e3-4058-b032-15f419682b2a/console-operator/2.log"
Sep 08 15:34:43.926202 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:43.926159    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-wpwdq" event={"ID":"5f7b0f68-a8e3-4058-b032-15f419682b2a","Type":"ContainerStarted","Data":"64df7e9907ba3071bd44ff637637a890dd30a03d9e45beac43f5ab42d90e69f5"}
Sep 08 15:34:43.926709 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:43.926657    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-wpwdq"
Sep 08 15:34:43.934947 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:43.934912    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-85b454874b-wpwdq"
Sep 08 15:34:43.944185 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:43.944127    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-85b454874b-wpwdq" podStartSLOduration=57.723212156 podStartE2EDuration="59.944109038s" totalImagesPullingTime="2.220896882s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:33:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:34:43.943253079 +0000 UTC m=+122.032704169" watchObservedRunningTime="2026-09-08 15:34:43.944109038 +0000 UTC m=+122.033560125"
Sep 08 15:34:44.087095 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:44.087061    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-85d98c998f-28wpz"]
Sep 08 15:34:44.087944 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:44.087920    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ecfb56c-9113-49a2-99c8-226c564c32d5" containerName="registry"
Sep 08 15:34:44.088105 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:44.088094    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ecfb56c-9113-49a2-99c8-226c564c32d5" containerName="registry"
Sep 08 15:34:44.088325 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:44.088314    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="0ecfb56c-9113-49a2-99c8-226c564c32d5" containerName="registry"
Sep 08 15:34:44.092089 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:44.092069    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-85d98c998f-28wpz"
Sep 08 15:34:44.094833 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:44.094814    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-8xmpq\""
Sep 08 15:34:44.094968 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:44.094850    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 08 15:34:44.095038 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:44.094959    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 08 15:34:44.097385 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:44.097365    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-85d98c998f-28wpz"]
Sep 08 15:34:44.113345 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:44.113325    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flscq\" (UniqueName: \"kubernetes.io/projected/2d27f249-a6aa-47a6-a5cb-a8dcc588390c-kube-api-access-flscq\") pod \"downloads-85d98c998f-28wpz\" (UID: \"2d27f249-a6aa-47a6-a5cb-a8dcc588390c\") " pod="openshift-console/downloads-85d98c998f-28wpz"
Sep 08 15:34:44.213816 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:44.213751    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flscq\" (UniqueName: \"kubernetes.io/projected/2d27f249-a6aa-47a6-a5cb-a8dcc588390c-kube-api-access-flscq\") pod \"downloads-85d98c998f-28wpz\" (UID: \"2d27f249-a6aa-47a6-a5cb-a8dcc588390c\") " pod="openshift-console/downloads-85d98c998f-28wpz"
Sep 08 15:34:44.222212 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:44.222186    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-flscq\" (UniqueName: \"kubernetes.io/projected/2d27f249-a6aa-47a6-a5cb-a8dcc588390c-kube-api-access-flscq\") pod \"downloads-85d98c998f-28wpz\" (UID: \"2d27f249-a6aa-47a6-a5cb-a8dcc588390c\") " pod="openshift-console/downloads-85d98c998f-28wpz"
Sep 08 15:34:44.401475 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:44.401449    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-85d98c998f-28wpz"
Sep 08 15:34:44.677203 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:44.677171    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-85d98c998f-28wpz"]
Sep 08 15:34:44.680278 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:34:44.680248    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d27f249_a6aa_47a6_a5cb_a8dcc588390c.slice/crio-ae5150b36596d82d75aba9ad10ca3db1a01d4809a6650162206cb72af198115d WatchSource:0}: Error finding container ae5150b36596d82d75aba9ad10ca3db1a01d4809a6650162206cb72af198115d: Status 404 returned error can't find the container with id ae5150b36596d82d75aba9ad10ca3db1a01d4809a6650162206cb72af198115d
Sep 08 15:34:44.931707 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:44.931629    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-85d98c998f-28wpz" event={"ID":"2d27f249-a6aa-47a6-a5cb-a8dcc588390c","Type":"ContainerStarted","Data":"ae5150b36596d82d75aba9ad10ca3db1a01d4809a6650162206cb72af198115d"}
Sep 08 15:34:51.178183 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:51.178148    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d4d759e-9d3c-497a-b546-462ae2b48785-metrics-certs\") pod \"network-metrics-daemon-h4cml\" (UID: \"7d4d759e-9d3c-497a-b546-462ae2b48785\") " pod="openshift-multus/network-metrics-daemon-h4cml"
Sep 08 15:34:51.180902 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:51.180850    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d4d759e-9d3c-497a-b546-462ae2b48785-metrics-certs\") pod \"network-metrics-daemon-h4cml\" (UID: \"7d4d759e-9d3c-497a-b546-462ae2b48785\") " pod="openshift-multus/network-metrics-daemon-h4cml"
Sep 08 15:34:51.309103 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:51.309070    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-xsqld\""
Sep 08 15:34:51.315835 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:51.315814    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h4cml"
Sep 08 15:34:51.591214 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:51.591190    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-h4cml"]
Sep 08 15:34:51.593534 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:34:51.593507    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d4d759e_9d3c_497a_b546_462ae2b48785.slice/crio-6056b094f9b659632f353a33fddbbae763cd9217f66a066aa2ba714db2a176ac WatchSource:0}: Error finding container 6056b094f9b659632f353a33fddbbae763cd9217f66a066aa2ba714db2a176ac: Status 404 returned error can't find the container with id 6056b094f9b659632f353a33fddbbae763cd9217f66a066aa2ba714db2a176ac
Sep 08 15:34:51.962149 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:51.962112    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-h4cml" event={"ID":"7d4d759e-9d3c-497a-b546-462ae2b48785","Type":"ContainerStarted","Data":"6056b094f9b659632f353a33fddbbae763cd9217f66a066aa2ba714db2a176ac"}
Sep 08 15:34:52.079827 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:52.079341    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-699f699998-xsc8g"
Sep 08 15:34:52.080078 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:52.080039    2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-699f699998-xsc8g"
Sep 08 15:34:52.970768 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:52.970728    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-h4cml" event={"ID":"7d4d759e-9d3c-497a-b546-462ae2b48785","Type":"ContainerStarted","Data":"a9de017224642ae9786d3297020d9ee2010c5d07cc2ed5ef6a985080bc60d48f"}
Sep 08 15:34:52.971242 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:52.970835    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-h4cml" event={"ID":"7d4d759e-9d3c-497a-b546-462ae2b48785","Type":"ContainerStarted","Data":"f15d535e6bc8a8dc292236b370a82e50b85156342cece558a21df4336d0e882e"}
Sep 08 15:34:52.991002 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:52.990953    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-h4cml" podStartSLOduration=128.997073442 podStartE2EDuration="2m9.990937711s" totalImagesPullingTime="993.864269ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:32:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:34:52.988412851 +0000 UTC m=+131.077863940" watchObservedRunningTime="2026-09-08 15:34:52.990937711 +0000 UTC m=+131.080388800"
Sep 08 15:34:53.456287 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:53.456254    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6f96d99b88-hs77c"]
Sep 08 15:34:53.459882 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:53.459847    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f96d99b88-hs77c"
Sep 08 15:34:53.462557 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:53.462533    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 08 15:34:53.462557 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:53.462556    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 08 15:34:53.462710 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:53.462676    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 08 15:34:53.464038 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:53.463972    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-7kl9k\""
Sep 08 15:34:53.464138 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:53.463972    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 08 15:34:53.464138 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:53.463980    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 08 15:34:53.471121 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:53.471098    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6f96d99b88-hs77c"]
Sep 08 15:34:53.499801 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:53.499777    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmpdf\" (UniqueName: \"kubernetes.io/projected/c7e4ba52-a21d-4a22-80e1-762d6b422107-kube-api-access-mmpdf\") pod \"console-6f96d99b88-hs77c\" (UID: \"c7e4ba52-a21d-4a22-80e1-762d6b422107\") " pod="openshift-console/console-6f96d99b88-hs77c"
Sep 08 15:34:53.499919 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:53.499820    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c7e4ba52-a21d-4a22-80e1-762d6b422107-service-ca\") pod \"console-6f96d99b88-hs77c\" (UID: \"c7e4ba52-a21d-4a22-80e1-762d6b422107\") " pod="openshift-console/console-6f96d99b88-hs77c"
Sep 08 15:34:53.499919 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:53.499843    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c7e4ba52-a21d-4a22-80e1-762d6b422107-oauth-serving-cert\") pod \"console-6f96d99b88-hs77c\" (UID: \"c7e4ba52-a21d-4a22-80e1-762d6b422107\") " pod="openshift-console/console-6f96d99b88-hs77c"
Sep 08 15:34:53.499919 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:53.499888    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c7e4ba52-a21d-4a22-80e1-762d6b422107-console-oauth-config\") pod \"console-6f96d99b88-hs77c\" (UID: \"c7e4ba52-a21d-4a22-80e1-762d6b422107\") " pod="openshift-console/console-6f96d99b88-hs77c"
Sep 08 15:34:53.499919 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:53.499918    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c7e4ba52-a21d-4a22-80e1-762d6b422107-console-config\") pod \"console-6f96d99b88-hs77c\" (UID: \"c7e4ba52-a21d-4a22-80e1-762d6b422107\") " pod="openshift-console/console-6f96d99b88-hs77c"
Sep 08 15:34:53.500079 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:53.499936    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c7e4ba52-a21d-4a22-80e1-762d6b422107-console-serving-cert\") pod \"console-6f96d99b88-hs77c\" (UID: \"c7e4ba52-a21d-4a22-80e1-762d6b422107\") " pod="openshift-console/console-6f96d99b88-hs77c"
Sep 08 15:34:53.601170 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:53.601130    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c7e4ba52-a21d-4a22-80e1-762d6b422107-service-ca\") pod \"console-6f96d99b88-hs77c\" (UID: \"c7e4ba52-a21d-4a22-80e1-762d6b422107\") " pod="openshift-console/console-6f96d99b88-hs77c"
Sep 08 15:34:53.601340 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:53.601174    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c7e4ba52-a21d-4a22-80e1-762d6b422107-oauth-serving-cert\") pod \"console-6f96d99b88-hs77c\" (UID: \"c7e4ba52-a21d-4a22-80e1-762d6b422107\") " pod="openshift-console/console-6f96d99b88-hs77c"
Sep 08 15:34:53.601340 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:53.601204    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c7e4ba52-a21d-4a22-80e1-762d6b422107-console-oauth-config\") pod \"console-6f96d99b88-hs77c\" (UID: \"c7e4ba52-a21d-4a22-80e1-762d6b422107\") " pod="openshift-console/console-6f96d99b88-hs77c"
Sep 08 15:34:53.601340 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:53.601239    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c7e4ba52-a21d-4a22-80e1-762d6b422107-console-config\") pod \"console-6f96d99b88-hs77c\" (UID: \"c7e4ba52-a21d-4a22-80e1-762d6b422107\") " pod="openshift-console/console-6f96d99b88-hs77c"
Sep 08 15:34:53.601340 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:53.601270    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c7e4ba52-a21d-4a22-80e1-762d6b422107-console-serving-cert\") pod \"console-6f96d99b88-hs77c\" (UID: \"c7e4ba52-a21d-4a22-80e1-762d6b422107\") " pod="openshift-console/console-6f96d99b88-hs77c"
Sep 08 15:34:53.601549 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:53.601345    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmpdf\" (UniqueName: \"kubernetes.io/projected/c7e4ba52-a21d-4a22-80e1-762d6b422107-kube-api-access-mmpdf\") pod \"console-6f96d99b88-hs77c\" (UID: \"c7e4ba52-a21d-4a22-80e1-762d6b422107\") " pod="openshift-console/console-6f96d99b88-hs77c"
Sep 08 15:34:53.602181 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:53.601973    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c7e4ba52-a21d-4a22-80e1-762d6b422107-service-ca\") pod \"console-6f96d99b88-hs77c\" (UID: \"c7e4ba52-a21d-4a22-80e1-762d6b422107\") " pod="openshift-console/console-6f96d99b88-hs77c"
Sep 08 15:34:53.602338 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:53.602310    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c7e4ba52-a21d-4a22-80e1-762d6b422107-console-config\") pod \"console-6f96d99b88-hs77c\" (UID: \"c7e4ba52-a21d-4a22-80e1-762d6b422107\") " pod="openshift-console/console-6f96d99b88-hs77c"
Sep 08 15:34:53.602423 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:53.602348    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c7e4ba52-a21d-4a22-80e1-762d6b422107-oauth-serving-cert\") pod \"console-6f96d99b88-hs77c\" (UID: \"c7e4ba52-a21d-4a22-80e1-762d6b422107\") " pod="openshift-console/console-6f96d99b88-hs77c"
Sep 08 15:34:53.603952 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:53.603927    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c7e4ba52-a21d-4a22-80e1-762d6b422107-console-oauth-config\") pod \"console-6f96d99b88-hs77c\" (UID: \"c7e4ba52-a21d-4a22-80e1-762d6b422107\") " pod="openshift-console/console-6f96d99b88-hs77c"
Sep 08 15:34:53.604071 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:53.603975    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c7e4ba52-a21d-4a22-80e1-762d6b422107-console-serving-cert\") pod \"console-6f96d99b88-hs77c\" (UID: \"c7e4ba52-a21d-4a22-80e1-762d6b422107\") " pod="openshift-console/console-6f96d99b88-hs77c"
Sep 08 15:34:53.610356 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:53.610337    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmpdf\" (UniqueName: \"kubernetes.io/projected/c7e4ba52-a21d-4a22-80e1-762d6b422107-kube-api-access-mmpdf\") pod \"console-6f96d99b88-hs77c\" (UID: \"c7e4ba52-a21d-4a22-80e1-762d6b422107\") " pod="openshift-console/console-6f96d99b88-hs77c"
Sep 08 15:34:53.770756 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:53.770666    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f96d99b88-hs77c"
Sep 08 15:34:54.052115 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:54.052031    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6f96d99b88-hs77c"]
Sep 08 15:34:54.055354 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:34:54.055320    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7e4ba52_a21d_4a22_80e1_762d6b422107.slice/crio-795cc493ab23c74cf885c2925447c1d206b34ccb5437e8a9f9446be7e1d0a378 WatchSource:0}: Error finding container 795cc493ab23c74cf885c2925447c1d206b34ccb5437e8a9f9446be7e1d0a378: Status 404 returned error can't find the container with id 795cc493ab23c74cf885c2925447c1d206b34ccb5437e8a9f9446be7e1d0a378
Sep 08 15:34:54.981910 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:54.981847    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f96d99b88-hs77c" event={"ID":"c7e4ba52-a21d-4a22-80e1-762d6b422107","Type":"ContainerStarted","Data":"795cc493ab23c74cf885c2925447c1d206b34ccb5437e8a9f9446be7e1d0a378"}
Sep 08 15:34:55.463661 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:55.463623    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-9ddf64b77-tr67g"]
Sep 08 15:34:55.468278 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:55.468244    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9ddf64b77-tr67g"
Sep 08 15:34:55.488054 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:55.488028    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 08 15:34:55.504015 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:55.503988    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-9ddf64b77-tr67g"]
Sep 08 15:34:55.515540 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:55.515473    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e2893fb4-81e8-4b0d-8188-5fc4d9993a23-oauth-serving-cert\") pod \"console-9ddf64b77-tr67g\" (UID: \"e2893fb4-81e8-4b0d-8188-5fc4d9993a23\") " pod="openshift-console/console-9ddf64b77-tr67g"
Sep 08 15:34:55.515677 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:55.515549    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e2893fb4-81e8-4b0d-8188-5fc4d9993a23-service-ca\") pod \"console-9ddf64b77-tr67g\" (UID: \"e2893fb4-81e8-4b0d-8188-5fc4d9993a23\") " pod="openshift-console/console-9ddf64b77-tr67g"
Sep 08 15:34:55.515677 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:55.515574    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e2893fb4-81e8-4b0d-8188-5fc4d9993a23-console-oauth-config\") pod \"console-9ddf64b77-tr67g\" (UID: \"e2893fb4-81e8-4b0d-8188-5fc4d9993a23\") " pod="openshift-console/console-9ddf64b77-tr67g"
Sep 08 15:34:55.515677 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:55.515604    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e2893fb4-81e8-4b0d-8188-5fc4d9993a23-console-serving-cert\") pod \"console-9ddf64b77-tr67g\" (UID: \"e2893fb4-81e8-4b0d-8188-5fc4d9993a23\") " pod="openshift-console/console-9ddf64b77-tr67g"
Sep 08 15:34:55.515677 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:55.515632    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e2893fb4-81e8-4b0d-8188-5fc4d9993a23-trusted-ca-bundle\") pod \"console-9ddf64b77-tr67g\" (UID: \"e2893fb4-81e8-4b0d-8188-5fc4d9993a23\") " pod="openshift-console/console-9ddf64b77-tr67g"
Sep 08 15:34:55.515677 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:55.515668    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qkhr\" (UniqueName: \"kubernetes.io/projected/e2893fb4-81e8-4b0d-8188-5fc4d9993a23-kube-api-access-9qkhr\") pod \"console-9ddf64b77-tr67g\" (UID: \"e2893fb4-81e8-4b0d-8188-5fc4d9993a23\") " pod="openshift-console/console-9ddf64b77-tr67g"
Sep 08 15:34:55.515960 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:55.515746    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e2893fb4-81e8-4b0d-8188-5fc4d9993a23-console-config\") pod \"console-9ddf64b77-tr67g\" (UID: \"e2893fb4-81e8-4b0d-8188-5fc4d9993a23\") " pod="openshift-console/console-9ddf64b77-tr67g"
Sep 08 15:34:55.616498 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:55.616463    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e2893fb4-81e8-4b0d-8188-5fc4d9993a23-console-config\") pod \"console-9ddf64b77-tr67g\" (UID: \"e2893fb4-81e8-4b0d-8188-5fc4d9993a23\") " pod="openshift-console/console-9ddf64b77-tr67g"
Sep 08 15:34:55.616680 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:55.616525    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e2893fb4-81e8-4b0d-8188-5fc4d9993a23-oauth-serving-cert\") pod \"console-9ddf64b77-tr67g\" (UID: \"e2893fb4-81e8-4b0d-8188-5fc4d9993a23\") " pod="openshift-console/console-9ddf64b77-tr67g"
Sep 08 15:34:55.616680 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:55.616558    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e2893fb4-81e8-4b0d-8188-5fc4d9993a23-service-ca\") pod \"console-9ddf64b77-tr67g\" (UID: \"e2893fb4-81e8-4b0d-8188-5fc4d9993a23\") " pod="openshift-console/console-9ddf64b77-tr67g"
Sep 08 15:34:55.616680 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:55.616581    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e2893fb4-81e8-4b0d-8188-5fc4d9993a23-console-oauth-config\") pod \"console-9ddf64b77-tr67g\" (UID: \"e2893fb4-81e8-4b0d-8188-5fc4d9993a23\") " pod="openshift-console/console-9ddf64b77-tr67g"
Sep 08 15:34:55.616680 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:55.616614    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e2893fb4-81e8-4b0d-8188-5fc4d9993a23-console-serving-cert\") pod \"console-9ddf64b77-tr67g\" (UID: \"e2893fb4-81e8-4b0d-8188-5fc4d9993a23\") " pod="openshift-console/console-9ddf64b77-tr67g"
Sep 08 15:34:55.616680 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:55.616640    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e2893fb4-81e8-4b0d-8188-5fc4d9993a23-trusted-ca-bundle\") pod \"console-9ddf64b77-tr67g\" (UID: \"e2893fb4-81e8-4b0d-8188-5fc4d9993a23\") " pod="openshift-console/console-9ddf64b77-tr67g"
Sep 08 15:34:55.616680 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:55.616670    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qkhr\" (UniqueName: \"kubernetes.io/projected/e2893fb4-81e8-4b0d-8188-5fc4d9993a23-kube-api-access-9qkhr\") pod \"console-9ddf64b77-tr67g\" (UID: \"e2893fb4-81e8-4b0d-8188-5fc4d9993a23\") " pod="openshift-console/console-9ddf64b77-tr67g"
Sep 08 15:34:55.617243 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:55.617217    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e2893fb4-81e8-4b0d-8188-5fc4d9993a23-oauth-serving-cert\") pod \"console-9ddf64b77-tr67g\" (UID: \"e2893fb4-81e8-4b0d-8188-5fc4d9993a23\") " pod="openshift-console/console-9ddf64b77-tr67g"
Sep 08 15:34:55.617381 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:55.617305    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e2893fb4-81e8-4b0d-8188-5fc4d9993a23-console-config\") pod \"console-9ddf64b77-tr67g\" (UID: \"e2893fb4-81e8-4b0d-8188-5fc4d9993a23\") " pod="openshift-console/console-9ddf64b77-tr67g"
Sep 08 15:34:55.617782 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:55.617741    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e2893fb4-81e8-4b0d-8188-5fc4d9993a23-service-ca\") pod \"console-9ddf64b77-tr67g\" (UID: \"e2893fb4-81e8-4b0d-8188-5fc4d9993a23\") " pod="openshift-console/console-9ddf64b77-tr67g"
Sep 08 15:34:55.618066 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:55.618035    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e2893fb4-81e8-4b0d-8188-5fc4d9993a23-trusted-ca-bundle\") pod \"console-9ddf64b77-tr67g\" (UID: \"e2893fb4-81e8-4b0d-8188-5fc4d9993a23\") " pod="openshift-console/console-9ddf64b77-tr67g"
Sep 08 15:34:55.619661 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:55.619636    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e2893fb4-81e8-4b0d-8188-5fc4d9993a23-console-oauth-config\") pod \"console-9ddf64b77-tr67g\" (UID: \"e2893fb4-81e8-4b0d-8188-5fc4d9993a23\") " pod="openshift-console/console-9ddf64b77-tr67g"
Sep 08 15:34:55.620344 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:55.620324    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e2893fb4-81e8-4b0d-8188-5fc4d9993a23-console-serving-cert\") pod \"console-9ddf64b77-tr67g\" (UID: \"e2893fb4-81e8-4b0d-8188-5fc4d9993a23\") " pod="openshift-console/console-9ddf64b77-tr67g"
Sep 08 15:34:55.626633 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:55.626590    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qkhr\" (UniqueName: \"kubernetes.io/projected/e2893fb4-81e8-4b0d-8188-5fc4d9993a23-kube-api-access-9qkhr\") pod \"console-9ddf64b77-tr67g\" (UID: \"e2893fb4-81e8-4b0d-8188-5fc4d9993a23\") " pod="openshift-console/console-9ddf64b77-tr67g"
Sep 08 15:34:55.781113 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:34:55.781024    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9ddf64b77-tr67g"
Sep 08 15:35:03.502787 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:35:03.502709    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-9ddf64b77-tr67g"]
Sep 08 15:35:03.506260 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:35:03.506231    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2893fb4_81e8_4b0d_8188_5fc4d9993a23.slice/crio-0ba53c1c0a89c57067d9339bfb5ec2a3aa55cbe3cfbbf46e6d0ab5980e18d7d6 WatchSource:0}: Error finding container 0ba53c1c0a89c57067d9339bfb5ec2a3aa55cbe3cfbbf46e6d0ab5980e18d7d6: Status 404 returned error can't find the container with id 0ba53c1c0a89c57067d9339bfb5ec2a3aa55cbe3cfbbf46e6d0ab5980e18d7d6
Sep 08 15:35:04.023799 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:35:04.023546    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9ddf64b77-tr67g" event={"ID":"e2893fb4-81e8-4b0d-8188-5fc4d9993a23","Type":"ContainerStarted","Data":"060497b8cb71aaa91dde8bdcd6e2a3002d32408ac061e790cef5161a7ef265fa"}
Sep 08 15:35:04.023799 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:35:04.023619    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9ddf64b77-tr67g" event={"ID":"e2893fb4-81e8-4b0d-8188-5fc4d9993a23","Type":"ContainerStarted","Data":"0ba53c1c0a89c57067d9339bfb5ec2a3aa55cbe3cfbbf46e6d0ab5980e18d7d6"}
Sep 08 15:35:04.025590 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:35:04.025555    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-85d98c998f-28wpz" event={"ID":"2d27f249-a6aa-47a6-a5cb-a8dcc588390c","Type":"ContainerStarted","Data":"a36c1e39119fa606364a9f31e75514f73b20ec9ebeeb71077071c136c0db6767"}
Sep 08 15:35:04.025766 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:35:04.025745    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-85d98c998f-28wpz"
Sep 08 15:35:04.028298 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:35:04.028270    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f96d99b88-hs77c" event={"ID":"c7e4ba52-a21d-4a22-80e1-762d6b422107","Type":"ContainerStarted","Data":"1674b38445f069834f47506bb36986b7e5aa8bfffa28924d7fb97064ff7f1690"}
Sep 08 15:35:04.039480 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:35:04.039453    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-85d98c998f-28wpz"
Sep 08 15:35:04.045163 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:35:04.045112    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-9ddf64b77-tr67g" podStartSLOduration=9.045092158 podStartE2EDuration="9.045092158s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:34:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:35:04.042796477 +0000 UTC m=+142.132247568" watchObservedRunningTime="2026-09-08 15:35:04.045092158 +0000 UTC m=+142.134543249"
Sep 08 15:35:04.060454 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:35:04.060399    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-85d98c998f-28wpz" podStartSLOduration=1.511781992 podStartE2EDuration="20.060379346s" totalImagesPullingTime="18.548597354s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:34:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:35:04.059653753 +0000 UTC m=+142.149104844" watchObservedRunningTime="2026-09-08 15:35:04.060379346 +0000 UTC m=+142.149830436"
Sep 08 15:35:04.082534 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:35:04.082476    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6f96d99b88-hs77c" podStartSLOduration=1.9026979179999999 podStartE2EDuration="11.082461048s" totalImagesPullingTime="9.17976313s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:34:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:35:04.080219303 +0000 UTC m=+142.169670397" watchObservedRunningTime="2026-09-08 15:35:04.082461048 +0000 UTC m=+142.171912138"
Sep 08 15:35:05.781849 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:35:05.781806    2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-9ddf64b77-tr67g"
Sep 08 15:35:05.782304 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:35:05.781887    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-9ddf64b77-tr67g"
Sep 08 15:35:05.787253 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:35:05.787228    2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-9ddf64b77-tr67g"
Sep 08 15:35:06.046524 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:35:06.046438    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-9ddf64b77-tr67g"
Sep 08 15:35:06.122798 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:35:06.122765    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6f96d99b88-hs77c"]
Sep 08 15:35:09.052845 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:35:09.052808    2549 generic.go:356] "Generic (PLEG): container finished" podID="d1d7e194-2f46-486c-aa08-f87ef377f54f" containerID="1d9b3fda97b5f50803637ff4e564b84647d5f2f39ece21b97c6fb0895619b65c" exitCode=0
Sep 08 15:35:09.053322 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:35:09.052894    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-7mfs7" event={"ID":"d1d7e194-2f46-486c-aa08-f87ef377f54f","Type":"ContainerDied","Data":"1d9b3fda97b5f50803637ff4e564b84647d5f2f39ece21b97c6fb0895619b65c"}
Sep 08 15:35:09.053322 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:35:09.053272    2549 scope.go:117] "RemoveContainer" containerID="1d9b3fda97b5f50803637ff4e564b84647d5f2f39ece21b97c6fb0895619b65c"
Sep 08 15:35:10.060601 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:35:10.060561    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-7mfs7" event={"ID":"d1d7e194-2f46-486c-aa08-f87ef377f54f","Type":"ContainerStarted","Data":"90f2ae39f586bb321692bd2fa0e5400dead334100c96955a4720465d3c143c28"}
Sep 08 15:35:12.085082 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:35:12.085055    2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-699f699998-xsc8g"
Sep 08 15:35:12.090014 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:35:12.089983    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-699f699998-xsc8g"
Sep 08 15:35:13.771406 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:35:13.771368    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6f96d99b88-hs77c"
Sep 08 15:35:31.148258 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:35:31.148196    2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6f96d99b88-hs77c" podUID="c7e4ba52-a21d-4a22-80e1-762d6b422107" containerName="console" containerID="cri-o://1674b38445f069834f47506bb36986b7e5aa8bfffa28924d7fb97064ff7f1690" gracePeriod=15
Sep 08 15:35:31.462309 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:35:31.461979    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f96d99b88-hs77c"
Sep 08 15:35:31.516123 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:35:31.515264    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c7e4ba52-a21d-4a22-80e1-762d6b422107-service-ca\") pod \"c7e4ba52-a21d-4a22-80e1-762d6b422107\" (UID: \"c7e4ba52-a21d-4a22-80e1-762d6b422107\") "
Sep 08 15:35:31.516123 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:35:31.515342    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mmpdf\" (UniqueName: \"kubernetes.io/projected/c7e4ba52-a21d-4a22-80e1-762d6b422107-kube-api-access-mmpdf\") pod \"c7e4ba52-a21d-4a22-80e1-762d6b422107\" (UID: \"c7e4ba52-a21d-4a22-80e1-762d6b422107\") "
Sep 08 15:35:31.516123 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:35:31.515378    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c7e4ba52-a21d-4a22-80e1-762d6b422107-console-config\") pod \"c7e4ba52-a21d-4a22-80e1-762d6b422107\" (UID: \"c7e4ba52-a21d-4a22-80e1-762d6b422107\") "
Sep 08 15:35:31.516123 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:35:31.515409    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c7e4ba52-a21d-4a22-80e1-762d6b422107-oauth-serving-cert\") pod \"c7e4ba52-a21d-4a22-80e1-762d6b422107\" (UID: \"c7e4ba52-a21d-4a22-80e1-762d6b422107\") "
Sep 08 15:35:31.516123 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:35:31.515498    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c7e4ba52-a21d-4a22-80e1-762d6b422107-console-oauth-config\") pod \"c7e4ba52-a21d-4a22-80e1-762d6b422107\" (UID: \"c7e4ba52-a21d-4a22-80e1-762d6b422107\") "
Sep 08 15:35:31.516123 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:35:31.515537    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c7e4ba52-a21d-4a22-80e1-762d6b422107-console-serving-cert\") pod \"c7e4ba52-a21d-4a22-80e1-762d6b422107\" (UID: \"c7e4ba52-a21d-4a22-80e1-762d6b422107\") "
Sep 08 15:35:31.517211 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:35:31.516957    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7e4ba52-a21d-4a22-80e1-762d6b422107-console-config" (OuterVolumeSpecName: "console-config") pod "c7e4ba52-a21d-4a22-80e1-762d6b422107" (UID: "c7e4ba52-a21d-4a22-80e1-762d6b422107"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 15:35:31.517473 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:35:31.517453    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7e4ba52-a21d-4a22-80e1-762d6b422107-service-ca" (OuterVolumeSpecName: "service-ca") pod "c7e4ba52-a21d-4a22-80e1-762d6b422107" (UID: "c7e4ba52-a21d-4a22-80e1-762d6b422107"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 15:35:31.519267 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:35:31.519234    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7e4ba52-a21d-4a22-80e1-762d6b422107-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "c7e4ba52-a21d-4a22-80e1-762d6b422107" (UID: "c7e4ba52-a21d-4a22-80e1-762d6b422107"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 15:35:31.522375 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:35:31.522336    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7e4ba52-a21d-4a22-80e1-762d6b422107-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "c7e4ba52-a21d-4a22-80e1-762d6b422107" (UID: "c7e4ba52-a21d-4a22-80e1-762d6b422107"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 15:35:31.522932 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:35:31.522896    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7e4ba52-a21d-4a22-80e1-762d6b422107-kube-api-access-mmpdf" (OuterVolumeSpecName: "kube-api-access-mmpdf") pod "c7e4ba52-a21d-4a22-80e1-762d6b422107" (UID: "c7e4ba52-a21d-4a22-80e1-762d6b422107"). InnerVolumeSpecName "kube-api-access-mmpdf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:35:31.523081 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:35:31.522935    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7e4ba52-a21d-4a22-80e1-762d6b422107-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "c7e4ba52-a21d-4a22-80e1-762d6b422107" (UID: "c7e4ba52-a21d-4a22-80e1-762d6b422107"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 15:35:31.617171 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:35:31.617034    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmpdf\" (UniqueName: \"kubernetes.io/projected/c7e4ba52-a21d-4a22-80e1-762d6b422107-kube-api-access-mmpdf\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:35:31.617171 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:35:31.617073    2549 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c7e4ba52-a21d-4a22-80e1-762d6b422107-console-config\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:35:31.617171 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:35:31.617089    2549 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c7e4ba52-a21d-4a22-80e1-762d6b422107-oauth-serving-cert\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:35:31.617171 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:35:31.617103    2549 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c7e4ba52-a21d-4a22-80e1-762d6b422107-console-oauth-config\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:35:31.617171 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:35:31.617117    2549 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c7e4ba52-a21d-4a22-80e1-762d6b422107-console-serving-cert\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:35:31.617171 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:35:31.617133    2549 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c7e4ba52-a21d-4a22-80e1-762d6b422107-service-ca\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:35:32.153882 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:35:32.150338    2549 generic.go:356] "Generic (PLEG): container finished" podID="c7e4ba52-a21d-4a22-80e1-762d6b422107" containerID="1674b38445f069834f47506bb36986b7e5aa8bfffa28924d7fb97064ff7f1690" exitCode=0
Sep 08 15:35:32.153882 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:35:32.150471    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f96d99b88-hs77c" event={"ID":"c7e4ba52-a21d-4a22-80e1-762d6b422107","Type":"ContainerDied","Data":"1674b38445f069834f47506bb36986b7e5aa8bfffa28924d7fb97064ff7f1690"}
Sep 08 15:35:32.153882 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:35:32.150524    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f96d99b88-hs77c" event={"ID":"c7e4ba52-a21d-4a22-80e1-762d6b422107","Type":"ContainerDied","Data":"795cc493ab23c74cf885c2925447c1d206b34ccb5437e8a9f9446be7e1d0a378"}
Sep 08 15:35:32.153882 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:35:32.150556    2549 scope.go:117] "RemoveContainer" containerID="1674b38445f069834f47506bb36986b7e5aa8bfffa28924d7fb97064ff7f1690"
Sep 08 15:35:32.153882 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:35:32.150730    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f96d99b88-hs77c"
Sep 08 15:35:32.173813 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:35:32.173785    2549 scope.go:117] "RemoveContainer" containerID="1674b38445f069834f47506bb36986b7e5aa8bfffa28924d7fb97064ff7f1690"
Sep 08 15:35:32.178969 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:35:32.178942    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1674b38445f069834f47506bb36986b7e5aa8bfffa28924d7fb97064ff7f1690\": container with ID starting with 1674b38445f069834f47506bb36986b7e5aa8bfffa28924d7fb97064ff7f1690 not found: ID does not exist" containerID="1674b38445f069834f47506bb36986b7e5aa8bfffa28924d7fb97064ff7f1690"
Sep 08 15:35:32.179144 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:35:32.179108    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1674b38445f069834f47506bb36986b7e5aa8bfffa28924d7fb97064ff7f1690"} err="failed to get container status \"1674b38445f069834f47506bb36986b7e5aa8bfffa28924d7fb97064ff7f1690\": rpc error: code = NotFound desc = could not find container \"1674b38445f069834f47506bb36986b7e5aa8bfffa28924d7fb97064ff7f1690\": container with ID starting with 1674b38445f069834f47506bb36986b7e5aa8bfffa28924d7fb97064ff7f1690 not found: ID does not exist"
Sep 08 15:35:32.181940 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:35:32.181902    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6f96d99b88-hs77c"]
Sep 08 15:35:32.185188 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:35:32.185163    2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6f96d99b88-hs77c"]
Sep 08 15:35:32.400009 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:35:32.399977    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c7e4ba52-a21d-4a22-80e1-762d6b422107" path="/var/lib/kubelet/pods/c7e4ba52-a21d-4a22-80e1-762d6b422107/volumes"
Sep 08 15:35:44.090366 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:35:44.090332    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-79fcdbdb96-lcm82"]
Sep 08 15:35:44.090720 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:35:44.090678    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7e4ba52-a21d-4a22-80e1-762d6b422107" containerName="console"
Sep 08 15:35:44.090720 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:35:44.090691    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7e4ba52-a21d-4a22-80e1-762d6b422107" containerName="console"
Sep 08 15:35:44.090791 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:35:44.090753    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="c7e4ba52-a21d-4a22-80e1-762d6b422107" containerName="console"
Sep 08 15:35:44.123594 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:35:44.123567    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-79fcdbdb96-lcm82"]
Sep 08 15:35:44.123752 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:35:44.123661    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-79fcdbdb96-lcm82"
Sep 08 15:35:44.211809 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:35:44.211783    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7dab304e-5c53-4fe2-8e5b-ef40fa074c3d-service-ca\") pod \"console-79fcdbdb96-lcm82\" (UID: \"7dab304e-5c53-4fe2-8e5b-ef40fa074c3d\") " pod="openshift-console/console-79fcdbdb96-lcm82"
Sep 08 15:35:44.211990 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:35:44.211820    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7dab304e-5c53-4fe2-8e5b-ef40fa074c3d-trusted-ca-bundle\") pod \"console-79fcdbdb96-lcm82\" (UID: \"7dab304e-5c53-4fe2-8e5b-ef40fa074c3d\") " pod="openshift-console/console-79fcdbdb96-lcm82"
Sep 08 15:35:44.211990 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:35:44.211844    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7dab304e-5c53-4fe2-8e5b-ef40fa074c3d-oauth-serving-cert\") pod \"console-79fcdbdb96-lcm82\" (UID: \"7dab304e-5c53-4fe2-8e5b-ef40fa074c3d\") " pod="openshift-console/console-79fcdbdb96-lcm82"
Sep 08 15:35:44.211990 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:35:44.211887    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nppz\" (UniqueName: \"kubernetes.io/projected/7dab304e-5c53-4fe2-8e5b-ef40fa074c3d-kube-api-access-8nppz\") pod \"console-79fcdbdb96-lcm82\" (UID: \"7dab304e-5c53-4fe2-8e5b-ef40fa074c3d\") " pod="openshift-console/console-79fcdbdb96-lcm82"
Sep 08 15:35:44.211990 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:35:44.211969    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7dab304e-5c53-4fe2-8e5b-ef40fa074c3d-console-oauth-config\") pod \"console-79fcdbdb96-lcm82\" (UID: \"7dab304e-5c53-4fe2-8e5b-ef40fa074c3d\") " pod="openshift-console/console-79fcdbdb96-lcm82"
Sep 08 15:35:44.212180 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:35:44.212017    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7dab304e-5c53-4fe2-8e5b-ef40fa074c3d-console-serving-cert\") pod \"console-79fcdbdb96-lcm82\" (UID: \"7dab304e-5c53-4fe2-8e5b-ef40fa074c3d\") " pod="openshift-console/console-79fcdbdb96-lcm82"
Sep 08 15:35:44.212180 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:35:44.212069    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7dab304e-5c53-4fe2-8e5b-ef40fa074c3d-console-config\") pod \"console-79fcdbdb96-lcm82\" (UID: \"7dab304e-5c53-4fe2-8e5b-ef40fa074c3d\") " pod="openshift-console/console-79fcdbdb96-lcm82"
Sep 08 15:35:44.312500 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:35:44.312468    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7dab304e-5c53-4fe2-8e5b-ef40fa074c3d-oauth-serving-cert\") pod \"console-79fcdbdb96-lcm82\" (UID: \"7dab304e-5c53-4fe2-8e5b-ef40fa074c3d\") " pod="openshift-console/console-79fcdbdb96-lcm82"
Sep 08 15:35:44.312647 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:35:44.312504    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nppz\" (UniqueName: \"kubernetes.io/projected/7dab304e-5c53-4fe2-8e5b-ef40fa074c3d-kube-api-access-8nppz\") pod \"console-79fcdbdb96-lcm82\" (UID: \"7dab304e-5c53-4fe2-8e5b-ef40fa074c3d\") " pod="openshift-console/console-79fcdbdb96-lcm82"
Sep 08 15:35:44.312647 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:35:44.312529    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7dab304e-5c53-4fe2-8e5b-ef40fa074c3d-console-oauth-config\") pod \"console-79fcdbdb96-lcm82\" (UID: \"7dab304e-5c53-4fe2-8e5b-ef40fa074c3d\") " pod="openshift-console/console-79fcdbdb96-lcm82"
Sep 08 15:35:44.312647 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:35:44.312551    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7dab304e-5c53-4fe2-8e5b-ef40fa074c3d-console-serving-cert\") pod \"console-79fcdbdb96-lcm82\" (UID: \"7dab304e-5c53-4fe2-8e5b-ef40fa074c3d\") " pod="openshift-console/console-79fcdbdb96-lcm82"
Sep 08 15:35:44.312808 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:35:44.312734    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7dab304e-5c53-4fe2-8e5b-ef40fa074c3d-console-config\") pod \"console-79fcdbdb96-lcm82\" (UID: \"7dab304e-5c53-4fe2-8e5b-ef40fa074c3d\") " pod="openshift-console/console-79fcdbdb96-lcm82"
Sep 08 15:35:44.312808 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:35:44.312772    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7dab304e-5c53-4fe2-8e5b-ef40fa074c3d-service-ca\") pod \"console-79fcdbdb96-lcm82\" (UID: \"7dab304e-5c53-4fe2-8e5b-ef40fa074c3d\") " pod="openshift-console/console-79fcdbdb96-lcm82"
Sep 08 15:35:44.312808 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:35:44.312798    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7dab304e-5c53-4fe2-8e5b-ef40fa074c3d-trusted-ca-bundle\") pod \"console-79fcdbdb96-lcm82\" (UID: \"7dab304e-5c53-4fe2-8e5b-ef40fa074c3d\") " pod="openshift-console/console-79fcdbdb96-lcm82"
Sep 08 15:35:44.313351 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:35:44.313324    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7dab304e-5c53-4fe2-8e5b-ef40fa074c3d-oauth-serving-cert\") pod \"console-79fcdbdb96-lcm82\" (UID: \"7dab304e-5c53-4fe2-8e5b-ef40fa074c3d\") " pod="openshift-console/console-79fcdbdb96-lcm82"
Sep 08 15:35:44.313483 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:35:44.313411    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7dab304e-5c53-4fe2-8e5b-ef40fa074c3d-service-ca\") pod \"console-79fcdbdb96-lcm82\" (UID: \"7dab304e-5c53-4fe2-8e5b-ef40fa074c3d\") " pod="openshift-console/console-79fcdbdb96-lcm82"
Sep 08 15:35:44.313554 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:35:44.313477    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7dab304e-5c53-4fe2-8e5b-ef40fa074c3d-console-config\") pod \"console-79fcdbdb96-lcm82\" (UID: \"7dab304e-5c53-4fe2-8e5b-ef40fa074c3d\") " pod="openshift-console/console-79fcdbdb96-lcm82"
Sep 08 15:35:44.313614 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:35:44.313597    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7dab304e-5c53-4fe2-8e5b-ef40fa074c3d-trusted-ca-bundle\") pod \"console-79fcdbdb96-lcm82\" (UID: \"7dab304e-5c53-4fe2-8e5b-ef40fa074c3d\") " pod="openshift-console/console-79fcdbdb96-lcm82"
Sep 08 15:35:44.314982 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:35:44.314956    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7dab304e-5c53-4fe2-8e5b-ef40fa074c3d-console-oauth-config\") pod \"console-79fcdbdb96-lcm82\" (UID: \"7dab304e-5c53-4fe2-8e5b-ef40fa074c3d\") " pod="openshift-console/console-79fcdbdb96-lcm82"
Sep 08 15:35:44.315071 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:35:44.315043    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7dab304e-5c53-4fe2-8e5b-ef40fa074c3d-console-serving-cert\") pod \"console-79fcdbdb96-lcm82\" (UID: \"7dab304e-5c53-4fe2-8e5b-ef40fa074c3d\") " pod="openshift-console/console-79fcdbdb96-lcm82"
Sep 08 15:35:44.321470 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:35:44.321452    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nppz\" (UniqueName: \"kubernetes.io/projected/7dab304e-5c53-4fe2-8e5b-ef40fa074c3d-kube-api-access-8nppz\") pod \"console-79fcdbdb96-lcm82\" (UID: \"7dab304e-5c53-4fe2-8e5b-ef40fa074c3d\") " pod="openshift-console/console-79fcdbdb96-lcm82"
Sep 08 15:35:44.432457 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:35:44.432430    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-79fcdbdb96-lcm82"
Sep 08 15:35:44.730195 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:35:44.730169    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-79fcdbdb96-lcm82"]
Sep 08 15:35:44.732433 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:35:44.732407    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7dab304e_5c53_4fe2_8e5b_ef40fa074c3d.slice/crio-12fe0396eb3b99786baeb0118637cdc082880d33ca590eba30976ab31e5c3b8c WatchSource:0}: Error finding container 12fe0396eb3b99786baeb0118637cdc082880d33ca590eba30976ab31e5c3b8c: Status 404 returned error can't find the container with id 12fe0396eb3b99786baeb0118637cdc082880d33ca590eba30976ab31e5c3b8c
Sep 08 15:35:45.206106 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:35:45.206073    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-79fcdbdb96-lcm82" event={"ID":"7dab304e-5c53-4fe2-8e5b-ef40fa074c3d","Type":"ContainerStarted","Data":"01a1ee7fef3a4d504a621e6799c5c96a33805cb757d5b0c36fd247bbdf008888"}
Sep 08 15:35:45.206443 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:35:45.206121    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-79fcdbdb96-lcm82" event={"ID":"7dab304e-5c53-4fe2-8e5b-ef40fa074c3d","Type":"ContainerStarted","Data":"12fe0396eb3b99786baeb0118637cdc082880d33ca590eba30976ab31e5c3b8c"}
Sep 08 15:35:45.224273 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:35:45.224222    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-79fcdbdb96-lcm82" podStartSLOduration=1.224206406 podStartE2EDuration="1.224206406s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:35:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:35:45.2237163 +0000 UTC m=+183.313167391" watchObservedRunningTime="2026-09-08 15:35:45.224206406 +0000 UTC m=+183.313657495"
Sep 08 15:35:54.433563 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:35:54.433525    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-79fcdbdb96-lcm82"
Sep 08 15:35:54.434055 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:35:54.433586    2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-79fcdbdb96-lcm82"
Sep 08 15:35:54.438057 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:35:54.438037    2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-79fcdbdb96-lcm82"
Sep 08 15:35:55.248758 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:35:55.248722    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-79fcdbdb96-lcm82"
Sep 08 15:35:55.308321 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:35:55.308291    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-9ddf64b77-tr67g"]
Sep 08 15:36:20.328441 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:36:20.328379    2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-9ddf64b77-tr67g" podUID="e2893fb4-81e8-4b0d-8188-5fc4d9993a23" containerName="console" containerID="cri-o://060497b8cb71aaa91dde8bdcd6e2a3002d32408ac061e790cef5161a7ef265fa" gracePeriod=15
Sep 08 15:36:20.562669 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:36:20.562644    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9ddf64b77-tr67g"
Sep 08 15:36:20.692545 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:36:20.692517    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e2893fb4-81e8-4b0d-8188-5fc4d9993a23-console-oauth-config\") pod \"e2893fb4-81e8-4b0d-8188-5fc4d9993a23\" (UID: \"e2893fb4-81e8-4b0d-8188-5fc4d9993a23\") "
Sep 08 15:36:20.692687 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:36:20.692591    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e2893fb4-81e8-4b0d-8188-5fc4d9993a23-service-ca\") pod \"e2893fb4-81e8-4b0d-8188-5fc4d9993a23\" (UID: \"e2893fb4-81e8-4b0d-8188-5fc4d9993a23\") "
Sep 08 15:36:20.692732 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:36:20.692684    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9qkhr\" (UniqueName: \"kubernetes.io/projected/e2893fb4-81e8-4b0d-8188-5fc4d9993a23-kube-api-access-9qkhr\") pod \"e2893fb4-81e8-4b0d-8188-5fc4d9993a23\" (UID: \"e2893fb4-81e8-4b0d-8188-5fc4d9993a23\") "
Sep 08 15:36:20.692779 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:36:20.692740    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e2893fb4-81e8-4b0d-8188-5fc4d9993a23-oauth-serving-cert\") pod \"e2893fb4-81e8-4b0d-8188-5fc4d9993a23\" (UID: \"e2893fb4-81e8-4b0d-8188-5fc4d9993a23\") "
Sep 08 15:36:20.692822 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:36:20.692790    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e2893fb4-81e8-4b0d-8188-5fc4d9993a23-console-config\") pod \"e2893fb4-81e8-4b0d-8188-5fc4d9993a23\" (UID: \"e2893fb4-81e8-4b0d-8188-5fc4d9993a23\") "
Sep 08 15:36:20.692822 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:36:20.692818    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e2893fb4-81e8-4b0d-8188-5fc4d9993a23-trusted-ca-bundle\") pod \"e2893fb4-81e8-4b0d-8188-5fc4d9993a23\" (UID: \"e2893fb4-81e8-4b0d-8188-5fc4d9993a23\") "
Sep 08 15:36:20.692953 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:36:20.692849    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e2893fb4-81e8-4b0d-8188-5fc4d9993a23-console-serving-cert\") pod \"e2893fb4-81e8-4b0d-8188-5fc4d9993a23\" (UID: \"e2893fb4-81e8-4b0d-8188-5fc4d9993a23\") "
Sep 08 15:36:20.692953 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:36:20.692942    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2893fb4-81e8-4b0d-8188-5fc4d9993a23-service-ca" (OuterVolumeSpecName: "service-ca") pod "e2893fb4-81e8-4b0d-8188-5fc4d9993a23" (UID: "e2893fb4-81e8-4b0d-8188-5fc4d9993a23"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 15:36:20.693138 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:36:20.693119    2549 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e2893fb4-81e8-4b0d-8188-5fc4d9993a23-service-ca\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:36:20.693197 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:36:20.693165    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2893fb4-81e8-4b0d-8188-5fc4d9993a23-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "e2893fb4-81e8-4b0d-8188-5fc4d9993a23" (UID: "e2893fb4-81e8-4b0d-8188-5fc4d9993a23"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 15:36:20.693197 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:36:20.693177    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2893fb4-81e8-4b0d-8188-5fc4d9993a23-console-config" (OuterVolumeSpecName: "console-config") pod "e2893fb4-81e8-4b0d-8188-5fc4d9993a23" (UID: "e2893fb4-81e8-4b0d-8188-5fc4d9993a23"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 15:36:20.693316 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:36:20.693290    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2893fb4-81e8-4b0d-8188-5fc4d9993a23-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "e2893fb4-81e8-4b0d-8188-5fc4d9993a23" (UID: "e2893fb4-81e8-4b0d-8188-5fc4d9993a23"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 15:36:20.694803 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:36:20.694776    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2893fb4-81e8-4b0d-8188-5fc4d9993a23-kube-api-access-9qkhr" (OuterVolumeSpecName: "kube-api-access-9qkhr") pod "e2893fb4-81e8-4b0d-8188-5fc4d9993a23" (UID: "e2893fb4-81e8-4b0d-8188-5fc4d9993a23"). InnerVolumeSpecName "kube-api-access-9qkhr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:36:20.694939 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:36:20.694776    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2893fb4-81e8-4b0d-8188-5fc4d9993a23-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "e2893fb4-81e8-4b0d-8188-5fc4d9993a23" (UID: "e2893fb4-81e8-4b0d-8188-5fc4d9993a23"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 15:36:20.694939 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:36:20.694811    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2893fb4-81e8-4b0d-8188-5fc4d9993a23-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "e2893fb4-81e8-4b0d-8188-5fc4d9993a23" (UID: "e2893fb4-81e8-4b0d-8188-5fc4d9993a23"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 15:36:20.794154 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:36:20.794132    2549 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e2893fb4-81e8-4b0d-8188-5fc4d9993a23-console-config\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:36:20.794154 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:36:20.794152    2549 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e2893fb4-81e8-4b0d-8188-5fc4d9993a23-trusted-ca-bundle\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:36:20.794300 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:36:20.794161    2549 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e2893fb4-81e8-4b0d-8188-5fc4d9993a23-console-serving-cert\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:36:20.794300 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:36:20.794170    2549 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e2893fb4-81e8-4b0d-8188-5fc4d9993a23-console-oauth-config\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:36:20.794300 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:36:20.794180    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9qkhr\" (UniqueName: \"kubernetes.io/projected/e2893fb4-81e8-4b0d-8188-5fc4d9993a23-kube-api-access-9qkhr\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:36:20.794300 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:36:20.794189    2549 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e2893fb4-81e8-4b0d-8188-5fc4d9993a23-oauth-serving-cert\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:36:21.341768 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:36:21.341736    2549 generic.go:356] "Generic (PLEG): container finished" podID="e2893fb4-81e8-4b0d-8188-5fc4d9993a23" containerID="060497b8cb71aaa91dde8bdcd6e2a3002d32408ac061e790cef5161a7ef265fa" exitCode=0
Sep 08 15:36:21.342142 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:36:21.341780    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9ddf64b77-tr67g"
Sep 08 15:36:21.342142 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:36:21.341824    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9ddf64b77-tr67g" event={"ID":"e2893fb4-81e8-4b0d-8188-5fc4d9993a23","Type":"ContainerDied","Data":"060497b8cb71aaa91dde8bdcd6e2a3002d32408ac061e790cef5161a7ef265fa"}
Sep 08 15:36:21.342142 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:36:21.341893    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9ddf64b77-tr67g" event={"ID":"e2893fb4-81e8-4b0d-8188-5fc4d9993a23","Type":"ContainerDied","Data":"0ba53c1c0a89c57067d9339bfb5ec2a3aa55cbe3cfbbf46e6d0ab5980e18d7d6"}
Sep 08 15:36:21.342142 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:36:21.341914    2549 scope.go:117] "RemoveContainer" containerID="060497b8cb71aaa91dde8bdcd6e2a3002d32408ac061e790cef5161a7ef265fa"
Sep 08 15:36:21.354186 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:36:21.354167    2549 scope.go:117] "RemoveContainer" containerID="060497b8cb71aaa91dde8bdcd6e2a3002d32408ac061e790cef5161a7ef265fa"
Sep 08 15:36:21.354459 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:36:21.354439    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"060497b8cb71aaa91dde8bdcd6e2a3002d32408ac061e790cef5161a7ef265fa\": container with ID starting with 060497b8cb71aaa91dde8bdcd6e2a3002d32408ac061e790cef5161a7ef265fa not found: ID does not exist" containerID="060497b8cb71aaa91dde8bdcd6e2a3002d32408ac061e790cef5161a7ef265fa"
Sep 08 15:36:21.354517 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:36:21.354467    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"060497b8cb71aaa91dde8bdcd6e2a3002d32408ac061e790cef5161a7ef265fa"} err="failed to get container status \"060497b8cb71aaa91dde8bdcd6e2a3002d32408ac061e790cef5161a7ef265fa\": rpc error: code = NotFound desc = could not find container \"060497b8cb71aaa91dde8bdcd6e2a3002d32408ac061e790cef5161a7ef265fa\": container with ID starting with 060497b8cb71aaa91dde8bdcd6e2a3002d32408ac061e790cef5161a7ef265fa not found: ID does not exist"
Sep 08 15:36:21.365249 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:36:21.365227    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-9ddf64b77-tr67g"]
Sep 08 15:36:21.370679 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:36:21.370659    2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-9ddf64b77-tr67g"]
Sep 08 15:36:22.399285 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:36:22.399252    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e2893fb4-81e8-4b0d-8188-5fc4d9993a23" path="/var/lib/kubelet/pods/e2893fb4-81e8-4b0d-8188-5fc4d9993a23/volumes"
Sep 08 15:37:16.106179 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:37:16.106142    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsb4jr"]
Sep 08 15:37:16.106666 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:37:16.106463    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2893fb4-81e8-4b0d-8188-5fc4d9993a23" containerName="console"
Sep 08 15:37:16.106666 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:37:16.106474    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2893fb4-81e8-4b0d-8188-5fc4d9993a23" containerName="console"
Sep 08 15:37:16.106666 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:37:16.106551    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="e2893fb4-81e8-4b0d-8188-5fc4d9993a23" containerName="console"
Sep 08 15:37:16.108409 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:37:16.108393    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsb4jr"
Sep 08 15:37:16.111318 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:37:16.111292    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 08 15:37:16.112640 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:37:16.112621    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-qc6rj\""
Sep 08 15:37:16.112725 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:37:16.112637    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 08 15:37:16.121354 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:37:16.121332    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsb4jr"]
Sep 08 15:37:16.193096 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:37:16.193068    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e4bc5022-b507-44c2-bf1b-2a8c5900ac32-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsb4jr\" (UID: \"e4bc5022-b507-44c2-bf1b-2a8c5900ac32\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsb4jr"
Sep 08 15:37:16.193253 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:37:16.193104    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e4bc5022-b507-44c2-bf1b-2a8c5900ac32-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsb4jr\" (UID: \"e4bc5022-b507-44c2-bf1b-2a8c5900ac32\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsb4jr"
Sep 08 15:37:16.193253 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:37:16.193133    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wvbv\" (UniqueName: \"kubernetes.io/projected/e4bc5022-b507-44c2-bf1b-2a8c5900ac32-kube-api-access-7wvbv\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsb4jr\" (UID: \"e4bc5022-b507-44c2-bf1b-2a8c5900ac32\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsb4jr"
Sep 08 15:37:16.293657 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:37:16.293617    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wvbv\" (UniqueName: \"kubernetes.io/projected/e4bc5022-b507-44c2-bf1b-2a8c5900ac32-kube-api-access-7wvbv\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsb4jr\" (UID: \"e4bc5022-b507-44c2-bf1b-2a8c5900ac32\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsb4jr"
Sep 08 15:37:16.293792 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:37:16.293713    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e4bc5022-b507-44c2-bf1b-2a8c5900ac32-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsb4jr\" (UID: \"e4bc5022-b507-44c2-bf1b-2a8c5900ac32\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsb4jr"
Sep 08 15:37:16.293792 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:37:16.293751    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e4bc5022-b507-44c2-bf1b-2a8c5900ac32-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsb4jr\" (UID: \"e4bc5022-b507-44c2-bf1b-2a8c5900ac32\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsb4jr"
Sep 08 15:37:16.294099 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:37:16.294077    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e4bc5022-b507-44c2-bf1b-2a8c5900ac32-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsb4jr\" (UID: \"e4bc5022-b507-44c2-bf1b-2a8c5900ac32\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsb4jr"
Sep 08 15:37:16.294170 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:37:16.294103    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e4bc5022-b507-44c2-bf1b-2a8c5900ac32-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsb4jr\" (UID: \"e4bc5022-b507-44c2-bf1b-2a8c5900ac32\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsb4jr"
Sep 08 15:37:16.301861 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:37:16.301828    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wvbv\" (UniqueName: \"kubernetes.io/projected/e4bc5022-b507-44c2-bf1b-2a8c5900ac32-kube-api-access-7wvbv\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsb4jr\" (UID: \"e4bc5022-b507-44c2-bf1b-2a8c5900ac32\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsb4jr"
Sep 08 15:37:16.417700 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:37:16.417677    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsb4jr"
Sep 08 15:37:16.691613 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:37:16.691592    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsb4jr"]
Sep 08 15:37:16.694413 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:37:16.694389    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4bc5022_b507_44c2_bf1b_2a8c5900ac32.slice/crio-2be990971c94ae5a4261d8be5a81123b54646753f542f31e74a8b538b22763a0 WatchSource:0}: Error finding container 2be990971c94ae5a4261d8be5a81123b54646753f542f31e74a8b538b22763a0: Status 404 returned error can't find the container with id 2be990971c94ae5a4261d8be5a81123b54646753f542f31e74a8b538b22763a0
Sep 08 15:37:17.548848 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:37:17.548802    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsb4jr" event={"ID":"e4bc5022-b507-44c2-bf1b-2a8c5900ac32","Type":"ContainerStarted","Data":"2be990971c94ae5a4261d8be5a81123b54646753f542f31e74a8b538b22763a0"}
Sep 08 15:37:22.017712 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:37:22.017675    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-2mfzt"]
Sep 08 15:37:22.020891 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:37:22.020842    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-2mfzt"
Sep 08 15:37:22.023789 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:37:22.023746    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 08 15:37:22.025008 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:37:22.024875    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 08 15:37:22.025153 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:37:22.024874    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-d7zf5\""
Sep 08 15:37:22.030005 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:37:22.029978    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-2mfzt"]
Sep 08 15:37:22.145726 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:37:22.145692    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-972ps\" (UniqueName: \"kubernetes.io/projected/2b0fc6da-8d9f-49b2-88c5-9668607e2cb1-kube-api-access-972ps\") pod \"cert-manager-cainjector-559cdd8c87-2mfzt\" (UID: \"2b0fc6da-8d9f-49b2-88c5-9668607e2cb1\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-2mfzt"
Sep 08 15:37:22.145726 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:37:22.145728    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2b0fc6da-8d9f-49b2-88c5-9668607e2cb1-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-2mfzt\" (UID: \"2b0fc6da-8d9f-49b2-88c5-9668607e2cb1\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-2mfzt"
Sep 08 15:37:22.246777 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:37:22.246738    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-972ps\" (UniqueName: \"kubernetes.io/projected/2b0fc6da-8d9f-49b2-88c5-9668607e2cb1-kube-api-access-972ps\") pod \"cert-manager-cainjector-559cdd8c87-2mfzt\" (UID: \"2b0fc6da-8d9f-49b2-88c5-9668607e2cb1\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-2mfzt"
Sep 08 15:37:22.246777 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:37:22.246776    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2b0fc6da-8d9f-49b2-88c5-9668607e2cb1-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-2mfzt\" (UID: \"2b0fc6da-8d9f-49b2-88c5-9668607e2cb1\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-2mfzt"
Sep 08 15:37:22.254897 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:37:22.254875    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2b0fc6da-8d9f-49b2-88c5-9668607e2cb1-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-2mfzt\" (UID: \"2b0fc6da-8d9f-49b2-88c5-9668607e2cb1\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-2mfzt"
Sep 08 15:37:22.255193 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:37:22.255171    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-972ps\" (UniqueName: \"kubernetes.io/projected/2b0fc6da-8d9f-49b2-88c5-9668607e2cb1-kube-api-access-972ps\") pod \"cert-manager-cainjector-559cdd8c87-2mfzt\" (UID: \"2b0fc6da-8d9f-49b2-88c5-9668607e2cb1\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-2mfzt"
Sep 08 15:37:22.343534 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:37:22.343510    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-2mfzt"
Sep 08 15:37:22.574958 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:37:22.574875    2549 generic.go:356] "Generic (PLEG): container finished" podID="e4bc5022-b507-44c2-bf1b-2a8c5900ac32" containerID="67618c690b0350b01e8fc5833a73e472e9022acd01ebc5262e335261ff606fb5" exitCode=0
Sep 08 15:37:22.575112 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:37:22.574951    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsb4jr" event={"ID":"e4bc5022-b507-44c2-bf1b-2a8c5900ac32","Type":"ContainerDied","Data":"67618c690b0350b01e8fc5833a73e472e9022acd01ebc5262e335261ff606fb5"}
Sep 08 15:37:22.617787 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:37:22.617762    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-2mfzt"]
Sep 08 15:37:22.621021 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:37:22.620999    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2b0fc6da_8d9f_49b2_88c5_9668607e2cb1.slice/crio-bf28663d71a0da4e8ed55ce26ea13f4d05addaa5231b4977b8854586018eafee WatchSource:0}: Error finding container bf28663d71a0da4e8ed55ce26ea13f4d05addaa5231b4977b8854586018eafee: Status 404 returned error can't find the container with id bf28663d71a0da4e8ed55ce26ea13f4d05addaa5231b4977b8854586018eafee
Sep 08 15:37:23.582172 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:37:23.582082    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-2mfzt" event={"ID":"2b0fc6da-8d9f-49b2-88c5-9668607e2cb1","Type":"ContainerStarted","Data":"bf28663d71a0da4e8ed55ce26ea13f4d05addaa5231b4977b8854586018eafee"}
Sep 08 15:37:26.597096 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:37:26.597009    2549 generic.go:356] "Generic (PLEG): container finished" podID="e4bc5022-b507-44c2-bf1b-2a8c5900ac32" containerID="9118366851be2017380d50ef6168c9b60187ccc2658b9e6b8d2dd48d1d64d8fc" exitCode=0
Sep 08 15:37:26.597533 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:37:26.597098    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsb4jr" event={"ID":"e4bc5022-b507-44c2-bf1b-2a8c5900ac32","Type":"ContainerDied","Data":"9118366851be2017380d50ef6168c9b60187ccc2658b9e6b8d2dd48d1d64d8fc"}
Sep 08 15:37:26.598822 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:37:26.598800    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-2mfzt" event={"ID":"2b0fc6da-8d9f-49b2-88c5-9668607e2cb1","Type":"ContainerStarted","Data":"de9da18d86f11f7467836bf24602d600c86b3caed6fc467df2a170e63452c829"}
Sep 08 15:37:26.637036 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:37:26.636992    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-2mfzt" podStartSLOduration=0.928359603 podStartE2EDuration="4.636977393s" totalImagesPullingTime="3.70861779s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:37:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:37:26.636517716 +0000 UTC m=+284.725968809" watchObservedRunningTime="2026-09-08 15:37:26.636977393 +0000 UTC m=+284.726428482"
Sep 08 15:37:33.630193 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:37:33.630106    2549 generic.go:356] "Generic (PLEG): container finished" podID="e4bc5022-b507-44c2-bf1b-2a8c5900ac32" containerID="49ab573d2b23d4b44660d18b979c6525bdbbb3483ce3f91715b667da1bc38867" exitCode=0
Sep 08 15:37:33.630541 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:37:33.630216    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsb4jr" event={"ID":"e4bc5022-b507-44c2-bf1b-2a8c5900ac32","Type":"ContainerDied","Data":"49ab573d2b23d4b44660d18b979c6525bdbbb3483ce3f91715b667da1bc38867"}
Sep 08 15:37:34.756650 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:37:34.756625    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsb4jr"
Sep 08 15:37:34.853643 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:37:34.853613    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e4bc5022-b507-44c2-bf1b-2a8c5900ac32-util\") pod \"e4bc5022-b507-44c2-bf1b-2a8c5900ac32\" (UID: \"e4bc5022-b507-44c2-bf1b-2a8c5900ac32\") "
Sep 08 15:37:34.853777 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:37:34.853666    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7wvbv\" (UniqueName: \"kubernetes.io/projected/e4bc5022-b507-44c2-bf1b-2a8c5900ac32-kube-api-access-7wvbv\") pod \"e4bc5022-b507-44c2-bf1b-2a8c5900ac32\" (UID: \"e4bc5022-b507-44c2-bf1b-2a8c5900ac32\") "
Sep 08 15:37:34.853777 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:37:34.853747    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e4bc5022-b507-44c2-bf1b-2a8c5900ac32-bundle\") pod \"e4bc5022-b507-44c2-bf1b-2a8c5900ac32\" (UID: \"e4bc5022-b507-44c2-bf1b-2a8c5900ac32\") "
Sep 08 15:37:34.854129 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:37:34.854106    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4bc5022-b507-44c2-bf1b-2a8c5900ac32-bundle" (OuterVolumeSpecName: "bundle") pod "e4bc5022-b507-44c2-bf1b-2a8c5900ac32" (UID: "e4bc5022-b507-44c2-bf1b-2a8c5900ac32"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:37:34.855828 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:37:34.855801    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4bc5022-b507-44c2-bf1b-2a8c5900ac32-kube-api-access-7wvbv" (OuterVolumeSpecName: "kube-api-access-7wvbv") pod "e4bc5022-b507-44c2-bf1b-2a8c5900ac32" (UID: "e4bc5022-b507-44c2-bf1b-2a8c5900ac32"). InnerVolumeSpecName "kube-api-access-7wvbv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:37:34.857530 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:37:34.857507    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4bc5022-b507-44c2-bf1b-2a8c5900ac32-util" (OuterVolumeSpecName: "util") pod "e4bc5022-b507-44c2-bf1b-2a8c5900ac32" (UID: "e4bc5022-b507-44c2-bf1b-2a8c5900ac32"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:37:34.954358 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:37:34.954335    2549 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e4bc5022-b507-44c2-bf1b-2a8c5900ac32-util\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:37:34.954358 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:37:34.954355    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7wvbv\" (UniqueName: \"kubernetes.io/projected/e4bc5022-b507-44c2-bf1b-2a8c5900ac32-kube-api-access-7wvbv\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:37:34.954481 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:37:34.954364    2549 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e4bc5022-b507-44c2-bf1b-2a8c5900ac32-bundle\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:37:35.640224 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:37:35.640195    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsb4jr"
Sep 08 15:37:35.640224 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:37:35.640214    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsb4jr" event={"ID":"e4bc5022-b507-44c2-bf1b-2a8c5900ac32","Type":"ContainerDied","Data":"2be990971c94ae5a4261d8be5a81123b54646753f542f31e74a8b538b22763a0"}
Sep 08 15:37:35.640423 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:37:35.640263    2549 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2be990971c94ae5a4261d8be5a81123b54646753f542f31e74a8b538b22763a0"
Sep 08 15:37:35.814389 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:37:35.814357    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-dfjj9"]
Sep 08 15:37:35.814705 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:37:35.814632    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4bc5022-b507-44c2-bf1b-2a8c5900ac32" containerName="extract"
Sep 08 15:37:35.814705 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:37:35.814642    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4bc5022-b507-44c2-bf1b-2a8c5900ac32" containerName="extract"
Sep 08 15:37:35.814705 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:37:35.814654    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4bc5022-b507-44c2-bf1b-2a8c5900ac32" containerName="util"
Sep 08 15:37:35.814705 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:37:35.814660    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4bc5022-b507-44c2-bf1b-2a8c5900ac32" containerName="util"
Sep 08 15:37:35.814705 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:37:35.814678    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4bc5022-b507-44c2-bf1b-2a8c5900ac32" containerName="pull"
Sep 08 15:37:35.814705 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:37:35.814684    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4bc5022-b507-44c2-bf1b-2a8c5900ac32" containerName="pull"
Sep 08 15:37:35.814928 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:37:35.814732    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4bc5022-b507-44c2-bf1b-2a8c5900ac32" containerName="extract"
Sep 08 15:37:35.817582 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:37:35.817561    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-dfjj9"
Sep 08 15:37:35.820220 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:37:35.820198    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-jcwns\""
Sep 08 15:37:35.828250 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:37:35.828227    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-dfjj9"]
Sep 08 15:37:35.962592 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:37:35.962568    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98fc9\" (UniqueName: \"kubernetes.io/projected/e6be3497-874f-479c-a655-580edc7a5c28-kube-api-access-98fc9\") pod \"cert-manager-55b4d954d5-dfjj9\" (UID: \"e6be3497-874f-479c-a655-580edc7a5c28\") " pod="cert-manager/cert-manager-55b4d954d5-dfjj9"
Sep 08 15:37:35.962690 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:37:35.962608    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e6be3497-874f-479c-a655-580edc7a5c28-bound-sa-token\") pod \"cert-manager-55b4d954d5-dfjj9\" (UID: \"e6be3497-874f-479c-a655-580edc7a5c28\") " pod="cert-manager/cert-manager-55b4d954d5-dfjj9"
Sep 08 15:37:36.063798 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:37:36.063761    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98fc9\" (UniqueName: \"kubernetes.io/projected/e6be3497-874f-479c-a655-580edc7a5c28-kube-api-access-98fc9\") pod \"cert-manager-55b4d954d5-dfjj9\" (UID: \"e6be3497-874f-479c-a655-580edc7a5c28\") " pod="cert-manager/cert-manager-55b4d954d5-dfjj9"
Sep 08 15:37:36.063926 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:37:36.063817    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e6be3497-874f-479c-a655-580edc7a5c28-bound-sa-token\") pod \"cert-manager-55b4d954d5-dfjj9\" (UID: \"e6be3497-874f-479c-a655-580edc7a5c28\") " pod="cert-manager/cert-manager-55b4d954d5-dfjj9"
Sep 08 15:37:36.073729 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:37:36.073702    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e6be3497-874f-479c-a655-580edc7a5c28-bound-sa-token\") pod \"cert-manager-55b4d954d5-dfjj9\" (UID: \"e6be3497-874f-479c-a655-580edc7a5c28\") " pod="cert-manager/cert-manager-55b4d954d5-dfjj9"
Sep 08 15:37:36.074010 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:37:36.073988    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-98fc9\" (UniqueName: \"kubernetes.io/projected/e6be3497-874f-479c-a655-580edc7a5c28-kube-api-access-98fc9\") pod \"cert-manager-55b4d954d5-dfjj9\" (UID: \"e6be3497-874f-479c-a655-580edc7a5c28\") " pod="cert-manager/cert-manager-55b4d954d5-dfjj9"
Sep 08 15:37:36.126760 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:37:36.126735    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-dfjj9"
Sep 08 15:37:36.396733 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:37:36.396704    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode6be3497_874f_479c_a655_580edc7a5c28.slice/crio-545678ade8ccb899eb6a41098f1a1329fea6b6fbf8ebfe93dea243ee9ca4e5ce WatchSource:0}: Error finding container 545678ade8ccb899eb6a41098f1a1329fea6b6fbf8ebfe93dea243ee9ca4e5ce: Status 404 returned error can't find the container with id 545678ade8ccb899eb6a41098f1a1329fea6b6fbf8ebfe93dea243ee9ca4e5ce
Sep 08 15:37:36.400316 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:37:36.400289    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-dfjj9"]
Sep 08 15:37:36.645715 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:37:36.645678    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-dfjj9" event={"ID":"e6be3497-874f-479c-a655-580edc7a5c28","Type":"ContainerStarted","Data":"43cbb2349e50e3b9c0c1ac88cadaa2ff7d806f23f3d79e419394f17cfd9c184b"}
Sep 08 15:37:36.645930 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:37:36.645719    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-dfjj9" event={"ID":"e6be3497-874f-479c-a655-580edc7a5c28","Type":"ContainerStarted","Data":"545678ade8ccb899eb6a41098f1a1329fea6b6fbf8ebfe93dea243ee9ca4e5ce"}
Sep 08 15:37:36.665228 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:37:36.665185    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-dfjj9" podStartSLOduration=1.665168849 podStartE2EDuration="1.665168849s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:37:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:37:36.663678225 +0000 UTC m=+294.753129316" watchObservedRunningTime="2026-09-08 15:37:36.665168849 +0000 UTC m=+294.754619938"
Sep 08 15:37:42.297526 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:37:42.297491    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-wpwdq_5f7b0f68-a8e3-4058-b032-15f419682b2a/console-operator/2.log"
Sep 08 15:37:42.298090 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:37:42.297704    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-wpwdq_5f7b0f68-a8e3-4058-b032-15f419682b2a/console-operator/2.log"
Sep 08 15:37:42.304744 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:37:42.304721    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-f4vvt_195f1a74-19bc-49c9-aada-fa1c13488480/ovn-acl-logging/0.log"
Sep 08 15:37:42.304889 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:37:42.304720    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-f4vvt_195f1a74-19bc-49c9-aada-fa1c13488480/ovn-acl-logging/0.log"
Sep 08 15:37:42.308889 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:37:42.308873    2549 kubelet.go:1648] "Image garbage collection succeeded"
Sep 08 15:38:02.851771 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:38:02.851734    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-84d887cf8-6rlx5"]
Sep 08 15:38:02.854095 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:38:02.854077    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-84d887cf8-6rlx5"
Sep 08 15:38:02.859977 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:38:02.859954    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 08 15:38:02.860108 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:38:02.860066    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 08 15:38:02.860108 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:38:02.860065    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4fe4fa15-773c-4e7d-a5d9-c5ee9d7755e4-webhook-cert\") pod \"opendatahub-operator-controller-manager-84d887cf8-6rlx5\" (UID: \"4fe4fa15-773c-4e7d-a5d9-c5ee9d7755e4\") " pod="opendatahub/opendatahub-operator-controller-manager-84d887cf8-6rlx5"
Sep 08 15:38:02.860260 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:38:02.860132    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/4fe4fa15-773c-4e7d-a5d9-c5ee9d7755e4-apiservice-cert\") pod \"opendatahub-operator-controller-manager-84d887cf8-6rlx5\" (UID: \"4fe4fa15-773c-4e7d-a5d9-c5ee9d7755e4\") " pod="opendatahub/opendatahub-operator-controller-manager-84d887cf8-6rlx5"
Sep 08 15:38:02.860260 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:38:02.860170    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/4fe4fa15-773c-4e7d-a5d9-c5ee9d7755e4-manifests\") pod \"opendatahub-operator-controller-manager-84d887cf8-6rlx5\" (UID: \"4fe4fa15-773c-4e7d-a5d9-c5ee9d7755e4\") " pod="opendatahub/opendatahub-operator-controller-manager-84d887cf8-6rlx5"
Sep 08 15:38:02.860260 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:38:02.860203    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vkgx\" (UniqueName: \"kubernetes.io/projected/4fe4fa15-773c-4e7d-a5d9-c5ee9d7755e4-kube-api-access-9vkgx\") pod \"opendatahub-operator-controller-manager-84d887cf8-6rlx5\" (UID: \"4fe4fa15-773c-4e7d-a5d9-c5ee9d7755e4\") " pod="opendatahub/opendatahub-operator-controller-manager-84d887cf8-6rlx5"
Sep 08 15:38:02.860832 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:38:02.860267    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4fe4fa15-773c-4e7d-a5d9-c5ee9d7755e4-tmp\") pod \"opendatahub-operator-controller-manager-84d887cf8-6rlx5\" (UID: \"4fe4fa15-773c-4e7d-a5d9-c5ee9d7755e4\") " pod="opendatahub/opendatahub-operator-controller-manager-84d887cf8-6rlx5"
Sep 08 15:38:02.860832 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:38:02.860518    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 08 15:38:02.860832 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:38:02.860527    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 08 15:38:02.860832 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:38:02.860551    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-pvhm4\""
Sep 08 15:38:02.890987 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:38:02.890958    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-84d887cf8-6rlx5"]
Sep 08 15:38:02.961177 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:38:02.961139    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vkgx\" (UniqueName: \"kubernetes.io/projected/4fe4fa15-773c-4e7d-a5d9-c5ee9d7755e4-kube-api-access-9vkgx\") pod \"opendatahub-operator-controller-manager-84d887cf8-6rlx5\" (UID: \"4fe4fa15-773c-4e7d-a5d9-c5ee9d7755e4\") " pod="opendatahub/opendatahub-operator-controller-manager-84d887cf8-6rlx5"
Sep 08 15:38:02.961347 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:38:02.961189    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4fe4fa15-773c-4e7d-a5d9-c5ee9d7755e4-tmp\") pod \"opendatahub-operator-controller-manager-84d887cf8-6rlx5\" (UID: \"4fe4fa15-773c-4e7d-a5d9-c5ee9d7755e4\") " pod="opendatahub/opendatahub-operator-controller-manager-84d887cf8-6rlx5"
Sep 08 15:38:02.961347 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:38:02.961227    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4fe4fa15-773c-4e7d-a5d9-c5ee9d7755e4-webhook-cert\") pod \"opendatahub-operator-controller-manager-84d887cf8-6rlx5\" (UID: \"4fe4fa15-773c-4e7d-a5d9-c5ee9d7755e4\") " pod="opendatahub/opendatahub-operator-controller-manager-84d887cf8-6rlx5"
Sep 08 15:38:02.961347 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:38:02.961252    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/4fe4fa15-773c-4e7d-a5d9-c5ee9d7755e4-apiservice-cert\") pod \"opendatahub-operator-controller-manager-84d887cf8-6rlx5\" (UID: \"4fe4fa15-773c-4e7d-a5d9-c5ee9d7755e4\") " pod="opendatahub/opendatahub-operator-controller-manager-84d887cf8-6rlx5"
Sep 08 15:38:02.961347 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:38:02.961273    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/4fe4fa15-773c-4e7d-a5d9-c5ee9d7755e4-manifests\") pod \"opendatahub-operator-controller-manager-84d887cf8-6rlx5\" (UID: \"4fe4fa15-773c-4e7d-a5d9-c5ee9d7755e4\") " pod="opendatahub/opendatahub-operator-controller-manager-84d887cf8-6rlx5"
Sep 08 15:38:02.961600 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:38:02.961583    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/4fe4fa15-773c-4e7d-a5d9-c5ee9d7755e4-manifests\") pod \"opendatahub-operator-controller-manager-84d887cf8-6rlx5\" (UID: \"4fe4fa15-773c-4e7d-a5d9-c5ee9d7755e4\") " pod="opendatahub/opendatahub-operator-controller-manager-84d887cf8-6rlx5"
Sep 08 15:38:02.963610 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:38:02.963577    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4fe4fa15-773c-4e7d-a5d9-c5ee9d7755e4-tmp\") pod \"opendatahub-operator-controller-manager-84d887cf8-6rlx5\" (UID: \"4fe4fa15-773c-4e7d-a5d9-c5ee9d7755e4\") " pod="opendatahub/opendatahub-operator-controller-manager-84d887cf8-6rlx5"
Sep 08 15:38:02.963971 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:38:02.963948    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4fe4fa15-773c-4e7d-a5d9-c5ee9d7755e4-webhook-cert\") pod \"opendatahub-operator-controller-manager-84d887cf8-6rlx5\" (UID: \"4fe4fa15-773c-4e7d-a5d9-c5ee9d7755e4\") " pod="opendatahub/opendatahub-operator-controller-manager-84d887cf8-6rlx5"
Sep 08 15:38:02.964065 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:38:02.964009    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/4fe4fa15-773c-4e7d-a5d9-c5ee9d7755e4-apiservice-cert\") pod \"opendatahub-operator-controller-manager-84d887cf8-6rlx5\" (UID: \"4fe4fa15-773c-4e7d-a5d9-c5ee9d7755e4\") " pod="opendatahub/opendatahub-operator-controller-manager-84d887cf8-6rlx5"
Sep 08 15:38:02.985665 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:38:02.985640    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vkgx\" (UniqueName: \"kubernetes.io/projected/4fe4fa15-773c-4e7d-a5d9-c5ee9d7755e4-kube-api-access-9vkgx\") pod \"opendatahub-operator-controller-manager-84d887cf8-6rlx5\" (UID: \"4fe4fa15-773c-4e7d-a5d9-c5ee9d7755e4\") " pod="opendatahub/opendatahub-operator-controller-manager-84d887cf8-6rlx5"
Sep 08 15:38:03.163905 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:38:03.163871    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-84d887cf8-6rlx5"
Sep 08 15:38:03.455714 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:38:03.455626    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-84d887cf8-6rlx5"]
Sep 08 15:38:03.459238 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:38:03.459208    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fe4fa15_773c_4e7d_a5d9_c5ee9d7755e4.slice/crio-39af66aa7dd9db9bc5c81504142bb7d4d75ba1a9cc5b8eb389d8bd30fc6c25b6 WatchSource:0}: Error finding container 39af66aa7dd9db9bc5c81504142bb7d4d75ba1a9cc5b8eb389d8bd30fc6c25b6: Status 404 returned error can't find the container with id 39af66aa7dd9db9bc5c81504142bb7d4d75ba1a9cc5b8eb389d8bd30fc6c25b6
Sep 08 15:38:03.461067 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:38:03.461047    2549 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 08 15:38:03.775739 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:38:03.775658    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-84d887cf8-6rlx5" event={"ID":"4fe4fa15-773c-4e7d-a5d9-c5ee9d7755e4","Type":"ContainerStarted","Data":"39af66aa7dd9db9bc5c81504142bb7d4d75ba1a9cc5b8eb389d8bd30fc6c25b6"}
Sep 08 15:38:06.791595 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:38:06.791500    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-84d887cf8-6rlx5" event={"ID":"4fe4fa15-773c-4e7d-a5d9-c5ee9d7755e4","Type":"ContainerStarted","Data":"a67240d91af22232594cc84e6547cbbc1fa9ff548cfda805d91d6e0c012b0ff6"}
Sep 08 15:38:07.797690 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:38:07.797649    2549 generic.go:356] "Generic (PLEG): container finished" podID="4fe4fa15-773c-4e7d-a5d9-c5ee9d7755e4" containerID="a67240d91af22232594cc84e6547cbbc1fa9ff548cfda805d91d6e0c012b0ff6" exitCode=0
Sep 08 15:38:07.798184 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:38:07.797729    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-84d887cf8-6rlx5" event={"ID":"4fe4fa15-773c-4e7d-a5d9-c5ee9d7755e4","Type":"ContainerDied","Data":"a67240d91af22232594cc84e6547cbbc1fa9ff548cfda805d91d6e0c012b0ff6"}
Sep 08 15:38:09.808618 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:38:09.808586    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-84d887cf8-6rlx5" event={"ID":"4fe4fa15-773c-4e7d-a5d9-c5ee9d7755e4","Type":"ContainerStarted","Data":"068400f8ed8254bdf14180499161147ba6eaa3f0cd7d17067a6bf2906b0d67a5"}
Sep 08 15:38:09.809066 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:38:09.808694    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-84d887cf8-6rlx5"
Sep 08 15:38:09.840460 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:38:09.840413    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-84d887cf8-6rlx5" podStartSLOduration=1.9220944599999998 podStartE2EDuration="7.840397042s" totalImagesPullingTime="4.835630757s" totalInitContainerRuntime="1.082671825s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:38:02 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:38:09.837528087 +0000 UTC m=+327.926979175" watchObservedRunningTime="2026-09-08 15:38:09.840397042 +0000 UTC m=+327.929848130"
Sep 08 15:38:20.815689 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:38:20.815647    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-84d887cf8-6rlx5"
Sep 08 15:38:35.216343 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:38:35.216256    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-7c6995f664-vbm7s"]
Sep 08 15:38:35.219358 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:38:35.219339    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7c6995f664-vbm7s"
Sep 08 15:38:35.223923 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:38:35.223902    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 08 15:38:35.225218 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:38:35.225196    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 08 15:38:35.225337 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:38:35.225210    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 08 15:38:35.245273 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:38:35.245234    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 08 15:38:35.274809 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:38:35.274773    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7c6995f664-vbm7s"]
Sep 08 15:38:35.329249 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:38:35.329210    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1ddcc598-ce1e-427c-9bd2-7bb888ba47a7-tmp\") pod \"kube-auth-proxy-7c6995f664-vbm7s\" (UID: \"1ddcc598-ce1e-427c-9bd2-7bb888ba47a7\") " pod="openshift-ingress/kube-auth-proxy-7c6995f664-vbm7s"
Sep 08 15:38:35.329249 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:38:35.329252    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-br5ks\" (UniqueName: \"kubernetes.io/projected/1ddcc598-ce1e-427c-9bd2-7bb888ba47a7-kube-api-access-br5ks\") pod \"kube-auth-proxy-7c6995f664-vbm7s\" (UID: \"1ddcc598-ce1e-427c-9bd2-7bb888ba47a7\") " pod="openshift-ingress/kube-auth-proxy-7c6995f664-vbm7s"
Sep 08 15:38:35.329476 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:38:35.329331    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ddcc598-ce1e-427c-9bd2-7bb888ba47a7-tls-certs\") pod \"kube-auth-proxy-7c6995f664-vbm7s\" (UID: \"1ddcc598-ce1e-427c-9bd2-7bb888ba47a7\") " pod="openshift-ingress/kube-auth-proxy-7c6995f664-vbm7s"
Sep 08 15:38:35.430721 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:38:35.430683    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ddcc598-ce1e-427c-9bd2-7bb888ba47a7-tls-certs\") pod \"kube-auth-proxy-7c6995f664-vbm7s\" (UID: \"1ddcc598-ce1e-427c-9bd2-7bb888ba47a7\") " pod="openshift-ingress/kube-auth-proxy-7c6995f664-vbm7s"
Sep 08 15:38:35.430915 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:38:35.430733    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1ddcc598-ce1e-427c-9bd2-7bb888ba47a7-tmp\") pod \"kube-auth-proxy-7c6995f664-vbm7s\" (UID: \"1ddcc598-ce1e-427c-9bd2-7bb888ba47a7\") " pod="openshift-ingress/kube-auth-proxy-7c6995f664-vbm7s"
Sep 08 15:38:35.430915 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:38:35.430753    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-br5ks\" (UniqueName: \"kubernetes.io/projected/1ddcc598-ce1e-427c-9bd2-7bb888ba47a7-kube-api-access-br5ks\") pod \"kube-auth-proxy-7c6995f664-vbm7s\" (UID: \"1ddcc598-ce1e-427c-9bd2-7bb888ba47a7\") " pod="openshift-ingress/kube-auth-proxy-7c6995f664-vbm7s"
Sep 08 15:38:35.432989 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:38:35.432958    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1ddcc598-ce1e-427c-9bd2-7bb888ba47a7-tmp\") pod \"kube-auth-proxy-7c6995f664-vbm7s\" (UID: \"1ddcc598-ce1e-427c-9bd2-7bb888ba47a7\") " pod="openshift-ingress/kube-auth-proxy-7c6995f664-vbm7s"
Sep 08 15:38:35.433242 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:38:35.433222    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ddcc598-ce1e-427c-9bd2-7bb888ba47a7-tls-certs\") pod \"kube-auth-proxy-7c6995f664-vbm7s\" (UID: \"1ddcc598-ce1e-427c-9bd2-7bb888ba47a7\") " pod="openshift-ingress/kube-auth-proxy-7c6995f664-vbm7s"
Sep 08 15:38:35.453984 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:38:35.453950    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-br5ks\" (UniqueName: \"kubernetes.io/projected/1ddcc598-ce1e-427c-9bd2-7bb888ba47a7-kube-api-access-br5ks\") pod \"kube-auth-proxy-7c6995f664-vbm7s\" (UID: \"1ddcc598-ce1e-427c-9bd2-7bb888ba47a7\") " pod="openshift-ingress/kube-auth-proxy-7c6995f664-vbm7s"
Sep 08 15:38:35.529167 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:38:35.529090    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7c6995f664-vbm7s"
Sep 08 15:38:35.810053 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:38:35.810019    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7c6995f664-vbm7s"]
Sep 08 15:38:35.812013 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:38:35.811982    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ddcc598_ce1e_427c_9bd2_7bb888ba47a7.slice/crio-4882294310d15a0c838ff4e1e25c3e8750e69800074e38678595f280eef5570c WatchSource:0}: Error finding container 4882294310d15a0c838ff4e1e25c3e8750e69800074e38678595f280eef5570c: Status 404 returned error can't find the container with id 4882294310d15a0c838ff4e1e25c3e8750e69800074e38678595f280eef5570c
Sep 08 15:38:35.920444 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:38:35.920405    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7c6995f664-vbm7s" event={"ID":"1ddcc598-ce1e-427c-9bd2-7bb888ba47a7","Type":"ContainerStarted","Data":"4882294310d15a0c838ff4e1e25c3e8750e69800074e38678595f280eef5570c"}
Sep 08 15:38:36.061435 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:38:36.061349    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-r8vkh"]
Sep 08 15:38:36.065732 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:38:36.065702    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-r8vkh"
Sep 08 15:38:36.069834 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:38:36.069808    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Sep 08 15:38:36.080459 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:38:36.080428    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-r8vkh"]
Sep 08 15:38:36.137255 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:38:36.137223    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjxdf\" (UniqueName: \"kubernetes.io/projected/a1b207e9-fabc-4090-8658-040c165b446b-kube-api-access-mjxdf\") pod \"model-serving-api-54d89bd5fc-r8vkh\" (UID: \"a1b207e9-fabc-4090-8658-040c165b446b\") " pod="opendatahub/model-serving-api-54d89bd5fc-r8vkh"
Sep 08 15:38:36.137431 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:38:36.137271    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a1b207e9-fabc-4090-8658-040c165b446b-tls-certs\") pod \"model-serving-api-54d89bd5fc-r8vkh\" (UID: \"a1b207e9-fabc-4090-8658-040c165b446b\") " pod="opendatahub/model-serving-api-54d89bd5fc-r8vkh"
Sep 08 15:38:36.237808 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:38:36.237767    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjxdf\" (UniqueName: \"kubernetes.io/projected/a1b207e9-fabc-4090-8658-040c165b446b-kube-api-access-mjxdf\") pod \"model-serving-api-54d89bd5fc-r8vkh\" (UID: \"a1b207e9-fabc-4090-8658-040c165b446b\") " pod="opendatahub/model-serving-api-54d89bd5fc-r8vkh"
Sep 08 15:38:36.238229 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:38:36.237817    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a1b207e9-fabc-4090-8658-040c165b446b-tls-certs\") pod \"model-serving-api-54d89bd5fc-r8vkh\" (UID: \"a1b207e9-fabc-4090-8658-040c165b446b\") " pod="opendatahub/model-serving-api-54d89bd5fc-r8vkh"
Sep 08 15:38:36.240228 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:38:36.240207    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a1b207e9-fabc-4090-8658-040c165b446b-tls-certs\") pod \"model-serving-api-54d89bd5fc-r8vkh\" (UID: \"a1b207e9-fabc-4090-8658-040c165b446b\") " pod="opendatahub/model-serving-api-54d89bd5fc-r8vkh"
Sep 08 15:38:36.251028 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:38:36.251002    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjxdf\" (UniqueName: \"kubernetes.io/projected/a1b207e9-fabc-4090-8658-040c165b446b-kube-api-access-mjxdf\") pod \"model-serving-api-54d89bd5fc-r8vkh\" (UID: \"a1b207e9-fabc-4090-8658-040c165b446b\") " pod="opendatahub/model-serving-api-54d89bd5fc-r8vkh"
Sep 08 15:38:36.379288 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:38:36.378970    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-r8vkh"
Sep 08 15:38:36.678013 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:38:36.677828    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-r8vkh"]
Sep 08 15:38:36.689800 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:38:36.686737    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1b207e9_fabc_4090_8658_040c165b446b.slice/crio-127942e9519e273375fc5b3d86f084c82068a818994f52900c25483ae09ab82e WatchSource:0}: Error finding container 127942e9519e273375fc5b3d86f084c82068a818994f52900c25483ae09ab82e: Status 404 returned error can't find the container with id 127942e9519e273375fc5b3d86f084c82068a818994f52900c25483ae09ab82e
Sep 08 15:38:36.928454 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:38:36.928356    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-r8vkh" event={"ID":"a1b207e9-fabc-4090-8658-040c165b446b","Type":"ContainerStarted","Data":"127942e9519e273375fc5b3d86f084c82068a818994f52900c25483ae09ab82e"}
Sep 08 15:38:39.945080 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:38:39.945042    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7c6995f664-vbm7s" event={"ID":"1ddcc598-ce1e-427c-9bd2-7bb888ba47a7","Type":"ContainerStarted","Data":"4708ce4dc5f6d1223749fd414e0759d6d337db119c5f4813fa6d9e22a20ae8ee"}
Sep 08 15:38:39.946567 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:38:39.946537    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-r8vkh" event={"ID":"a1b207e9-fabc-4090-8658-040c165b446b","Type":"ContainerStarted","Data":"59fe8d265b3af6fe7da6cd85b154d393a6637638c1725b62331d4188e2ee59c5"}
Sep 08 15:38:39.946690 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:38:39.946637    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-r8vkh"
Sep 08 15:38:39.972780 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:38:39.967286    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-7c6995f664-vbm7s" podStartSLOduration=1.930591699 podStartE2EDuration="4.967263719s" totalImagesPullingTime="3.03667202s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:38:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:38:39.965390353 +0000 UTC m=+358.054841464" watchObservedRunningTime="2026-09-08 15:38:39.967263719 +0000 UTC m=+358.056714811"
Sep 08 15:38:39.988535 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:38:39.988471    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-r8vkh" podStartSLOduration=0.832372489 podStartE2EDuration="3.988448059s" totalImagesPullingTime="3.15607557s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:38:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:38:39.987255002 +0000 UTC m=+358.076706093" watchObservedRunningTime="2026-09-08 15:38:39.988448059 +0000 UTC m=+358.077899149"
Sep 08 15:38:43.522919 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:38:43.522884    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-trp8s"]
Sep 08 15:38:43.527443 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:38:43.527424    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-trp8s"
Sep 08 15:38:43.540229 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:38:43.540207    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Sep 08 15:38:43.541566 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:38:43.541551    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Sep 08 15:38:43.541694 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:38:43.541682    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 08 15:38:43.545333 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:38:43.545316    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-krbl7\""
Sep 08 15:38:43.545979 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:38:43.545965    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Sep 08 15:38:43.568512 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:38:43.568488    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-trp8s"]
Sep 08 15:38:43.707377 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:38:43.707340    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b22c4ca4-806f-4ab6-8b4c-d934e28a619f-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-trp8s\" (UID: \"b22c4ca4-806f-4ab6-8b4c-d934e28a619f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-trp8s"
Sep 08 15:38:43.707377 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:38:43.707379    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b22c4ca4-806f-4ab6-8b4c-d934e28a619f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-trp8s\" (UID: \"b22c4ca4-806f-4ab6-8b4c-d934e28a619f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-trp8s"
Sep 08 15:38:43.707630 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:38:43.707423    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b22c4ca4-806f-4ab6-8b4c-d934e28a619f-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-trp8s\" (UID: \"b22c4ca4-806f-4ab6-8b4c-d934e28a619f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-trp8s"
Sep 08 15:38:43.707630 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:38:43.707487    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b22c4ca4-806f-4ab6-8b4c-d934e28a619f-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-trp8s\" (UID: \"b22c4ca4-806f-4ab6-8b4c-d934e28a619f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-trp8s"
Sep 08 15:38:43.707630 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:38:43.707557    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qssm\" (UniqueName: \"kubernetes.io/projected/b22c4ca4-806f-4ab6-8b4c-d934e28a619f-kube-api-access-7qssm\") pod \"istiod-openshift-gateway-5677ccb49b-trp8s\" (UID: \"b22c4ca4-806f-4ab6-8b4c-d934e28a619f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-trp8s"
Sep 08 15:38:43.707630 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:38:43.707603    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b22c4ca4-806f-4ab6-8b4c-d934e28a619f-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-trp8s\" (UID: \"b22c4ca4-806f-4ab6-8b4c-d934e28a619f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-trp8s"
Sep 08 15:38:43.707783 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:38:43.707632    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b22c4ca4-806f-4ab6-8b4c-d934e28a619f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-trp8s\" (UID: \"b22c4ca4-806f-4ab6-8b4c-d934e28a619f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-trp8s"
Sep 08 15:38:43.808233 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:38:43.808142    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qssm\" (UniqueName: \"kubernetes.io/projected/b22c4ca4-806f-4ab6-8b4c-d934e28a619f-kube-api-access-7qssm\") pod \"istiod-openshift-gateway-5677ccb49b-trp8s\" (UID: \"b22c4ca4-806f-4ab6-8b4c-d934e28a619f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-trp8s"
Sep 08 15:38:43.808233 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:38:43.808185    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b22c4ca4-806f-4ab6-8b4c-d934e28a619f-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-trp8s\" (UID: \"b22c4ca4-806f-4ab6-8b4c-d934e28a619f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-trp8s"
Sep 08 15:38:43.808233 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:38:43.808203    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b22c4ca4-806f-4ab6-8b4c-d934e28a619f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-trp8s\" (UID: \"b22c4ca4-806f-4ab6-8b4c-d934e28a619f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-trp8s"
Sep 08 15:38:43.808233 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:38:43.808227    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b22c4ca4-806f-4ab6-8b4c-d934e28a619f-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-trp8s\" (UID: \"b22c4ca4-806f-4ab6-8b4c-d934e28a619f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-trp8s"
Sep 08 15:38:43.808555 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:38:43.808275    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b22c4ca4-806f-4ab6-8b4c-d934e28a619f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-trp8s\" (UID: \"b22c4ca4-806f-4ab6-8b4c-d934e28a619f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-trp8s"
Sep 08 15:38:43.808555 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:38:43.808347    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b22c4ca4-806f-4ab6-8b4c-d934e28a619f-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-trp8s\" (UID: \"b22c4ca4-806f-4ab6-8b4c-d934e28a619f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-trp8s"
Sep 08 15:38:43.808555 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:38:43.808377    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b22c4ca4-806f-4ab6-8b4c-d934e28a619f-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-trp8s\" (UID: \"b22c4ca4-806f-4ab6-8b4c-d934e28a619f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-trp8s"
Sep 08 15:38:43.809081 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:38:43.809049    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b22c4ca4-806f-4ab6-8b4c-d934e28a619f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-trp8s\" (UID: \"b22c4ca4-806f-4ab6-8b4c-d934e28a619f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-trp8s"
Sep 08 15:38:43.810596 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:38:43.810573    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b22c4ca4-806f-4ab6-8b4c-d934e28a619f-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-trp8s\" (UID: \"b22c4ca4-806f-4ab6-8b4c-d934e28a619f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-trp8s"
Sep 08 15:38:43.810673 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:38:43.810598    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b22c4ca4-806f-4ab6-8b4c-d934e28a619f-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-trp8s\" (UID: \"b22c4ca4-806f-4ab6-8b4c-d934e28a619f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-trp8s"
Sep 08 15:38:43.810785 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:38:43.810766    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b22c4ca4-806f-4ab6-8b4c-d934e28a619f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-trp8s\" (UID: \"b22c4ca4-806f-4ab6-8b4c-d934e28a619f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-trp8s"
Sep 08 15:38:43.810828 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:38:43.810774    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b22c4ca4-806f-4ab6-8b4c-d934e28a619f-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-trp8s\" (UID: \"b22c4ca4-806f-4ab6-8b4c-d934e28a619f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-trp8s"
Sep 08 15:38:43.820825 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:38:43.820797    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b22c4ca4-806f-4ab6-8b4c-d934e28a619f-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-trp8s\" (UID: \"b22c4ca4-806f-4ab6-8b4c-d934e28a619f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-trp8s"
Sep 08 15:38:43.820938 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:38:43.820822    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qssm\" (UniqueName: \"kubernetes.io/projected/b22c4ca4-806f-4ab6-8b4c-d934e28a619f-kube-api-access-7qssm\") pod \"istiod-openshift-gateway-5677ccb49b-trp8s\" (UID: \"b22c4ca4-806f-4ab6-8b4c-d934e28a619f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-trp8s"
Sep 08 15:38:43.835917 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:38:43.835897    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-trp8s"
Sep 08 15:38:44.162502 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:38:44.162467    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-trp8s"]
Sep 08 15:38:44.168413 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:38:44.168380    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb22c4ca4_806f_4ab6_8b4c_d934e28a619f.slice/crio-28a70e6cbbaa73bcc081f13a5ba926d3e7da3cc81e293e84d60895cdf204e3b3 WatchSource:0}: Error finding container 28a70e6cbbaa73bcc081f13a5ba926d3e7da3cc81e293e84d60895cdf204e3b3: Status 404 returned error can't find the container with id 28a70e6cbbaa73bcc081f13a5ba926d3e7da3cc81e293e84d60895cdf204e3b3
Sep 08 15:38:44.977283 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:38:44.977232    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-trp8s" event={"ID":"b22c4ca4-806f-4ab6-8b4c-d934e28a619f","Type":"ContainerStarted","Data":"28a70e6cbbaa73bcc081f13a5ba926d3e7da3cc81e293e84d60895cdf204e3b3"}
Sep 08 15:38:46.117310 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:38:46.117273    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-66d9b677c7-dhthn"]
Sep 08 15:38:46.124657 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:38:46.124631    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-66d9b677c7-dhthn"
Sep 08 15:38:46.130468 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:38:46.130437    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-b8fc8\""
Sep 08 15:38:46.130623 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:38:46.130491    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 08 15:38:46.130782 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:38:46.130763    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 08 15:38:46.130782 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:38:46.130778    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 08 15:38:46.131092 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:38:46.131074    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 08 15:38:46.131953 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:38:46.131934    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 08 15:38:46.162055 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:38:46.162018    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-66d9b677c7-dhthn"]
Sep 08 15:38:46.230273 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:38:46.230228    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/4de71f3c-7786-45f7-b075-765ecbdaa643-metrics-cert\") pod \"lws-controller-manager-66d9b677c7-dhthn\" (UID: \"4de71f3c-7786-45f7-b075-765ecbdaa643\") " pod="openshift-lws-operator/lws-controller-manager-66d9b677c7-dhthn"
Sep 08 15:38:46.230459 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:38:46.230320    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4de71f3c-7786-45f7-b075-765ecbdaa643-cert\") pod \"lws-controller-manager-66d9b677c7-dhthn\" (UID: \"4de71f3c-7786-45f7-b075-765ecbdaa643\") " pod="openshift-lws-operator/lws-controller-manager-66d9b677c7-dhthn"
Sep 08 15:38:46.230459 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:38:46.230359    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/4de71f3c-7786-45f7-b075-765ecbdaa643-manager-config\") pod \"lws-controller-manager-66d9b677c7-dhthn\" (UID: \"4de71f3c-7786-45f7-b075-765ecbdaa643\") " pod="openshift-lws-operator/lws-controller-manager-66d9b677c7-dhthn"
Sep 08 15:38:46.230459 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:38:46.230395    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mz54\" (UniqueName: \"kubernetes.io/projected/4de71f3c-7786-45f7-b075-765ecbdaa643-kube-api-access-2mz54\") pod \"lws-controller-manager-66d9b677c7-dhthn\" (UID: \"4de71f3c-7786-45f7-b075-765ecbdaa643\") " pod="openshift-lws-operator/lws-controller-manager-66d9b677c7-dhthn"
Sep 08 15:38:46.331713 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:38:46.331601    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/4de71f3c-7786-45f7-b075-765ecbdaa643-metrics-cert\") pod \"lws-controller-manager-66d9b677c7-dhthn\" (UID: \"4de71f3c-7786-45f7-b075-765ecbdaa643\") " pod="openshift-lws-operator/lws-controller-manager-66d9b677c7-dhthn"
Sep 08 15:38:46.331713 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:38:46.331685    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4de71f3c-7786-45f7-b075-765ecbdaa643-cert\") pod \"lws-controller-manager-66d9b677c7-dhthn\" (UID: \"4de71f3c-7786-45f7-b075-765ecbdaa643\") " pod="openshift-lws-operator/lws-controller-manager-66d9b677c7-dhthn"
Sep 08 15:38:46.331713 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:38:46.331720    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/4de71f3c-7786-45f7-b075-765ecbdaa643-manager-config\") pod \"lws-controller-manager-66d9b677c7-dhthn\" (UID: \"4de71f3c-7786-45f7-b075-765ecbdaa643\") " pod="openshift-lws-operator/lws-controller-manager-66d9b677c7-dhthn"
Sep 08 15:38:46.332124 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:38:46.331750    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mz54\" (UniqueName: \"kubernetes.io/projected/4de71f3c-7786-45f7-b075-765ecbdaa643-kube-api-access-2mz54\") pod \"lws-controller-manager-66d9b677c7-dhthn\" (UID: \"4de71f3c-7786-45f7-b075-765ecbdaa643\") " pod="openshift-lws-operator/lws-controller-manager-66d9b677c7-dhthn"
Sep 08 15:38:46.332659 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:38:46.332634    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/4de71f3c-7786-45f7-b075-765ecbdaa643-manager-config\") pod \"lws-controller-manager-66d9b677c7-dhthn\" (UID: \"4de71f3c-7786-45f7-b075-765ecbdaa643\") " pod="openshift-lws-operator/lws-controller-manager-66d9b677c7-dhthn"
Sep 08 15:38:46.334509 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:38:46.334482    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/4de71f3c-7786-45f7-b075-765ecbdaa643-metrics-cert\") pod \"lws-controller-manager-66d9b677c7-dhthn\" (UID: \"4de71f3c-7786-45f7-b075-765ecbdaa643\") " pod="openshift-lws-operator/lws-controller-manager-66d9b677c7-dhthn"
Sep 08 15:38:46.334598 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:38:46.334508    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4de71f3c-7786-45f7-b075-765ecbdaa643-cert\") pod \"lws-controller-manager-66d9b677c7-dhthn\" (UID: \"4de71f3c-7786-45f7-b075-765ecbdaa643\") " pod="openshift-lws-operator/lws-controller-manager-66d9b677c7-dhthn"
Sep 08 15:38:46.366527 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:38:46.366460    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mz54\" (UniqueName: \"kubernetes.io/projected/4de71f3c-7786-45f7-b075-765ecbdaa643-kube-api-access-2mz54\") pod \"lws-controller-manager-66d9b677c7-dhthn\" (UID: \"4de71f3c-7786-45f7-b075-765ecbdaa643\") " pod="openshift-lws-operator/lws-controller-manager-66d9b677c7-dhthn"
Sep 08 15:38:46.439435 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:38:46.439392    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-66d9b677c7-dhthn"
Sep 08 15:38:46.563563 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:38:46.563452    2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 08 15:38:46.563708 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:38:46.563622    2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 08 15:38:46.823243 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:38:46.823214    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-66d9b677c7-dhthn"]
Sep 08 15:38:46.825823 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:38:46.825793    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4de71f3c_7786_45f7_b075_765ecbdaa643.slice/crio-9b48b7026241381847ae09eeba862f7669cd00cb8ed868dfcbbfc92a6ecfdc12 WatchSource:0}: Error finding container 9b48b7026241381847ae09eeba862f7669cd00cb8ed868dfcbbfc92a6ecfdc12: Status 404 returned error can't find the container with id 9b48b7026241381847ae09eeba862f7669cd00cb8ed868dfcbbfc92a6ecfdc12
Sep 08 15:38:46.988494 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:38:46.988462    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-trp8s" event={"ID":"b22c4ca4-806f-4ab6-8b4c-d934e28a619f","Type":"ContainerStarted","Data":"2317bcb87bdc865fadd20a56d47f1e5d4ffd93b418b83a1d33eb6eb0e3688fca"}
Sep 08 15:38:46.988697 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:38:46.988538    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-trp8s"
Sep 08 15:38:46.989876 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:38:46.989824    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-66d9b677c7-dhthn" event={"ID":"4de71f3c-7786-45f7-b075-765ecbdaa643","Type":"ContainerStarted","Data":"9b48b7026241381847ae09eeba862f7669cd00cb8ed868dfcbbfc92a6ecfdc12"}
Sep 08 15:38:47.160576 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:38:47.160517    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-trp8s" podStartSLOduration=1.767976381 podStartE2EDuration="4.160499752s" totalImagesPullingTime="2.392523371s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:38:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:38:47.152153133 +0000 UTC m=+365.241604223" watchObservedRunningTime="2026-09-08 15:38:47.160499752 +0000 UTC m=+365.249950841"
Sep 08 15:38:48.002006 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:38:48.000926    2549 patch_prober.go:28] interesting pod/istiod-openshift-gateway-5677ccb49b-trp8s container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 08 15:38:48.002006 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:38:48.000984    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-trp8s" podUID="b22c4ca4-806f-4ab6-8b4c-d934e28a619f" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 08 15:38:49.004132 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:38:49.004098    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-66d9b677c7-dhthn" event={"ID":"4de71f3c-7786-45f7-b075-765ecbdaa643","Type":"ContainerStarted","Data":"605c061b573b1b60f1e8f2213dbe35f4d70f5122eede50155f7c169a748dcfc9"}
Sep 08 15:38:49.004489 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:38:49.004161    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-66d9b677c7-dhthn"
Sep 08 15:38:49.049713 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:38:49.049657    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-66d9b677c7-dhthn" podStartSLOduration=1.585553468 podStartE2EDuration="3.049638845s" totalImagesPullingTime="1.464085377s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:38:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:38:49.049273156 +0000 UTC m=+367.138724245" watchObservedRunningTime="2026-09-08 15:38:49.049638845 +0000 UTC m=+367.139089934"
Sep 08 15:38:50.955965 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:38:50.955929    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-r8vkh"
Sep 08 15:38:50.999730 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:38:50.999697    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-trp8s"
Sep 08 15:39:00.010547 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:39:00.010516    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-66d9b677c7-dhthn"
Sep 08 15:40:07.136960 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:40:07.136918    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-79fcdbdb96-lcm82"]
Sep 08 15:40:32.157275 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:40:32.157218    2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-79fcdbdb96-lcm82" podUID="7dab304e-5c53-4fe2-8e5b-ef40fa074c3d" containerName="console" containerID="cri-o://01a1ee7fef3a4d504a621e6799c5c96a33805cb757d5b0c36fd247bbdf008888" gracePeriod=15
Sep 08 15:40:32.405335 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:40:32.405308    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-79fcdbdb96-lcm82"
Sep 08 15:40:32.449803 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:40:32.449714    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8nppz\" (UniqueName: \"kubernetes.io/projected/7dab304e-5c53-4fe2-8e5b-ef40fa074c3d-kube-api-access-8nppz\") pod \"7dab304e-5c53-4fe2-8e5b-ef40fa074c3d\" (UID: \"7dab304e-5c53-4fe2-8e5b-ef40fa074c3d\") "
Sep 08 15:40:32.449803 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:40:32.449759    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7dab304e-5c53-4fe2-8e5b-ef40fa074c3d-oauth-serving-cert\") pod \"7dab304e-5c53-4fe2-8e5b-ef40fa074c3d\" (UID: \"7dab304e-5c53-4fe2-8e5b-ef40fa074c3d\") "
Sep 08 15:40:32.449803 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:40:32.449794    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7dab304e-5c53-4fe2-8e5b-ef40fa074c3d-service-ca\") pod \"7dab304e-5c53-4fe2-8e5b-ef40fa074c3d\" (UID: \"7dab304e-5c53-4fe2-8e5b-ef40fa074c3d\") "
Sep 08 15:40:32.450095 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:40:32.449876    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7dab304e-5c53-4fe2-8e5b-ef40fa074c3d-console-serving-cert\") pod \"7dab304e-5c53-4fe2-8e5b-ef40fa074c3d\" (UID: \"7dab304e-5c53-4fe2-8e5b-ef40fa074c3d\") "
Sep 08 15:40:32.450095 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:40:32.449903    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7dab304e-5c53-4fe2-8e5b-ef40fa074c3d-trusted-ca-bundle\") pod \"7dab304e-5c53-4fe2-8e5b-ef40fa074c3d\" (UID: \"7dab304e-5c53-4fe2-8e5b-ef40fa074c3d\") "
Sep 08 15:40:32.450095 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:40:32.449917    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7dab304e-5c53-4fe2-8e5b-ef40fa074c3d-console-oauth-config\") pod \"7dab304e-5c53-4fe2-8e5b-ef40fa074c3d\" (UID: \"7dab304e-5c53-4fe2-8e5b-ef40fa074c3d\") "
Sep 08 15:40:32.450095 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:40:32.449945    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7dab304e-5c53-4fe2-8e5b-ef40fa074c3d-console-config\") pod \"7dab304e-5c53-4fe2-8e5b-ef40fa074c3d\" (UID: \"7dab304e-5c53-4fe2-8e5b-ef40fa074c3d\") "
Sep 08 15:40:32.450298 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:40:32.450268    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7dab304e-5c53-4fe2-8e5b-ef40fa074c3d-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "7dab304e-5c53-4fe2-8e5b-ef40fa074c3d" (UID: "7dab304e-5c53-4fe2-8e5b-ef40fa074c3d"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 15:40:32.450398 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:40:32.450312    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7dab304e-5c53-4fe2-8e5b-ef40fa074c3d-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "7dab304e-5c53-4fe2-8e5b-ef40fa074c3d" (UID: "7dab304e-5c53-4fe2-8e5b-ef40fa074c3d"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 15:40:32.450466 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:40:32.450395    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7dab304e-5c53-4fe2-8e5b-ef40fa074c3d-service-ca" (OuterVolumeSpecName: "service-ca") pod "7dab304e-5c53-4fe2-8e5b-ef40fa074c3d" (UID: "7dab304e-5c53-4fe2-8e5b-ef40fa074c3d"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 15:40:32.450526 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:40:32.450467    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7dab304e-5c53-4fe2-8e5b-ef40fa074c3d-console-config" (OuterVolumeSpecName: "console-config") pod "7dab304e-5c53-4fe2-8e5b-ef40fa074c3d" (UID: "7dab304e-5c53-4fe2-8e5b-ef40fa074c3d"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 15:40:32.452321 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:40:32.452293    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7dab304e-5c53-4fe2-8e5b-ef40fa074c3d-kube-api-access-8nppz" (OuterVolumeSpecName: "kube-api-access-8nppz") pod "7dab304e-5c53-4fe2-8e5b-ef40fa074c3d" (UID: "7dab304e-5c53-4fe2-8e5b-ef40fa074c3d"). InnerVolumeSpecName "kube-api-access-8nppz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:40:32.452441 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:40:32.452350    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7dab304e-5c53-4fe2-8e5b-ef40fa074c3d-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "7dab304e-5c53-4fe2-8e5b-ef40fa074c3d" (UID: "7dab304e-5c53-4fe2-8e5b-ef40fa074c3d"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 15:40:32.452510 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:40:32.452473    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7dab304e-5c53-4fe2-8e5b-ef40fa074c3d-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "7dab304e-5c53-4fe2-8e5b-ef40fa074c3d" (UID: "7dab304e-5c53-4fe2-8e5b-ef40fa074c3d"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 15:40:32.474107 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:40:32.474084    2549 generic.go:356] "Generic (PLEG): container finished" podID="7dab304e-5c53-4fe2-8e5b-ef40fa074c3d" containerID="01a1ee7fef3a4d504a621e6799c5c96a33805cb757d5b0c36fd247bbdf008888" exitCode=0
Sep 08 15:40:32.474193 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:40:32.474123    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-79fcdbdb96-lcm82"
Sep 08 15:40:32.474193 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:40:32.474160    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-79fcdbdb96-lcm82" event={"ID":"7dab304e-5c53-4fe2-8e5b-ef40fa074c3d","Type":"ContainerDied","Data":"01a1ee7fef3a4d504a621e6799c5c96a33805cb757d5b0c36fd247bbdf008888"}
Sep 08 15:40:32.474268 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:40:32.474210    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-79fcdbdb96-lcm82" event={"ID":"7dab304e-5c53-4fe2-8e5b-ef40fa074c3d","Type":"ContainerDied","Data":"12fe0396eb3b99786baeb0118637cdc082880d33ca590eba30976ab31e5c3b8c"}
Sep 08 15:40:32.474268 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:40:32.474230    2549 scope.go:117] "RemoveContainer" containerID="01a1ee7fef3a4d504a621e6799c5c96a33805cb757d5b0c36fd247bbdf008888"
Sep 08 15:40:32.482464 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:40:32.482447    2549 scope.go:117] "RemoveContainer" containerID="01a1ee7fef3a4d504a621e6799c5c96a33805cb757d5b0c36fd247bbdf008888"
Sep 08 15:40:32.482741 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:40:32.482717    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01a1ee7fef3a4d504a621e6799c5c96a33805cb757d5b0c36fd247bbdf008888\": container with ID starting with 01a1ee7fef3a4d504a621e6799c5c96a33805cb757d5b0c36fd247bbdf008888 not found: ID does not exist" containerID="01a1ee7fef3a4d504a621e6799c5c96a33805cb757d5b0c36fd247bbdf008888"
Sep 08 15:40:32.482801 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:40:32.482742    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01a1ee7fef3a4d504a621e6799c5c96a33805cb757d5b0c36fd247bbdf008888"} err="failed to get container status \"01a1ee7fef3a4d504a621e6799c5c96a33805cb757d5b0c36fd247bbdf008888\": rpc error: code = NotFound desc = could not find container \"01a1ee7fef3a4d504a621e6799c5c96a33805cb757d5b0c36fd247bbdf008888\": container with ID starting with 01a1ee7fef3a4d504a621e6799c5c96a33805cb757d5b0c36fd247bbdf008888 not found: ID does not exist"
Sep 08 15:40:32.494883 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:40:32.494840    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-79fcdbdb96-lcm82"]
Sep 08 15:40:32.497432 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:40:32.497409    2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-79fcdbdb96-lcm82"]
Sep 08 15:40:32.551066 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:40:32.551035    2549 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7dab304e-5c53-4fe2-8e5b-ef40fa074c3d-oauth-serving-cert\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:40:32.551066 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:40:32.551060    2549 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7dab304e-5c53-4fe2-8e5b-ef40fa074c3d-service-ca\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:40:32.551066 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:40:32.551071    2549 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7dab304e-5c53-4fe2-8e5b-ef40fa074c3d-console-serving-cert\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:40:32.551272 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:40:32.551080    2549 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7dab304e-5c53-4fe2-8e5b-ef40fa074c3d-trusted-ca-bundle\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:40:32.551272 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:40:32.551089    2549 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7dab304e-5c53-4fe2-8e5b-ef40fa074c3d-console-oauth-config\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:40:32.551272 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:40:32.551097    2549 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7dab304e-5c53-4fe2-8e5b-ef40fa074c3d-console-config\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:40:32.551272 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:40:32.551105    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8nppz\" (UniqueName: \"kubernetes.io/projected/7dab304e-5c53-4fe2-8e5b-ef40fa074c3d-kube-api-access-8nppz\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:40:34.400357 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:40:34.400322    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7dab304e-5c53-4fe2-8e5b-ef40fa074c3d" path="/var/lib/kubelet/pods/7dab304e-5c53-4fe2-8e5b-ef40fa074c3d/volumes"
Sep 08 15:42:42.326475 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:42:42.326443    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-wpwdq_5f7b0f68-a8e3-4058-b032-15f419682b2a/console-operator/2.log"
Sep 08 15:42:42.327263 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:42:42.327240    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-wpwdq_5f7b0f68-a8e3-4058-b032-15f419682b2a/console-operator/2.log"
Sep 08 15:42:42.332887 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:42:42.332864    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-f4vvt_195f1a74-19bc-49c9-aada-fa1c13488480/ovn-acl-logging/0.log"
Sep 08 15:42:42.333585 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:42:42.333562    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-f4vvt_195f1a74-19bc-49c9-aada-fa1c13488480/ovn-acl-logging/0.log"
Sep 08 15:45:27.188413 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:45:27.188380    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7790-139920034489648"]
Sep 08 15:45:27.188892 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:45:27.188724    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7dab304e-5c53-4fe2-8e5b-ef40fa074c3d" containerName="console"
Sep 08 15:45:27.188892 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:45:27.188736    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="7dab304e-5c53-4fe2-8e5b-ef40fa074c3d" containerName="console"
Sep 08 15:45:27.188892 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:45:27.188798    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="7dab304e-5c53-4fe2-8e5b-ef40fa074c3d" containerName="console"
Sep 08 15:45:27.191764 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:45:27.191746    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7790-139920034489648"
Sep 08 15:45:27.194292 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:45:27.194271    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-6hc5f\""
Sep 08 15:45:27.197621 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:45:27.197598    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7790-139920034489648"]
Sep 08 15:45:27.341130 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:45:27.341099    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68shx\" (UniqueName: \"kubernetes.io/projected/ad0f0113-dc28-432a-8a36-e8d4e8bf6ede-kube-api-access-68shx\") pod \"test-curl-7790-139920034489648\" (UID: \"ad0f0113-dc28-432a-8a36-e8d4e8bf6ede\") " pod="opendatahub/test-curl-7790-139920034489648"
Sep 08 15:45:27.441679 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:45:27.441599    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68shx\" (UniqueName: \"kubernetes.io/projected/ad0f0113-dc28-432a-8a36-e8d4e8bf6ede-kube-api-access-68shx\") pod \"test-curl-7790-139920034489648\" (UID: \"ad0f0113-dc28-432a-8a36-e8d4e8bf6ede\") " pod="opendatahub/test-curl-7790-139920034489648"
Sep 08 15:45:27.450450 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:45:27.450422    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-68shx\" (UniqueName: \"kubernetes.io/projected/ad0f0113-dc28-432a-8a36-e8d4e8bf6ede-kube-api-access-68shx\") pod \"test-curl-7790-139920034489648\" (UID: \"ad0f0113-dc28-432a-8a36-e8d4e8bf6ede\") " pod="opendatahub/test-curl-7790-139920034489648"
Sep 08 15:45:27.503406 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:45:27.503371    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7790-139920034489648"
Sep 08 15:45:27.776149 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:45:27.776122    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7790-139920034489648"]
Sep 08 15:45:27.778490 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:45:27.778457    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podad0f0113_dc28_432a_8a36_e8d4e8bf6ede.slice/crio-b4ea483fc55de0c636187c8f868def0cf49f1f4814a5533fb5274d863590ce87 WatchSource:0}: Error finding container b4ea483fc55de0c636187c8f868def0cf49f1f4814a5533fb5274d863590ce87: Status 404 returned error can't find the container with id b4ea483fc55de0c636187c8f868def0cf49f1f4814a5533fb5274d863590ce87
Sep 08 15:45:27.779966 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:45:27.779944    2549 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 08 15:45:27.801283 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:45:27.801253    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7790-139920034489648" event={"ID":"ad0f0113-dc28-432a-8a36-e8d4e8bf6ede","Type":"ContainerStarted","Data":"b4ea483fc55de0c636187c8f868def0cf49f1f4814a5533fb5274d863590ce87"}
Sep 08 15:45:30.816264 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:45:30.816225    2549 generic.go:356] "Generic (PLEG): container finished" podID="ad0f0113-dc28-432a-8a36-e8d4e8bf6ede" containerID="a1ef4d2c36550237e6c54889274a44fcff7b8608efb2367ec5af40898ff121b0" exitCode=0
Sep 08 15:45:30.816670 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:45:30.816291    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7790-139920034489648" event={"ID":"ad0f0113-dc28-432a-8a36-e8d4e8bf6ede","Type":"ContainerDied","Data":"a1ef4d2c36550237e6c54889274a44fcff7b8608efb2367ec5af40898ff121b0"}
Sep 08 15:45:31.956068 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:45:31.956044    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7790-139920034489648"
Sep 08 15:45:31.988474 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:45:31.988443    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7790-139920034489648_ad0f0113-dc28-432a-8a36-e8d4e8bf6ede/test-curl-7790-139920034489648/0.log"
Sep 08 15:45:32.009460 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:45:32.009425    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7790-139920034489648"]
Sep 08 15:45:32.012235 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:45:32.012211    2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7790-139920034489648"]
Sep 08 15:45:32.080614 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:45:32.080543    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-68shx\" (UniqueName: \"kubernetes.io/projected/ad0f0113-dc28-432a-8a36-e8d4e8bf6ede-kube-api-access-68shx\") pod \"ad0f0113-dc28-432a-8a36-e8d4e8bf6ede\" (UID: \"ad0f0113-dc28-432a-8a36-e8d4e8bf6ede\") "
Sep 08 15:45:32.082738 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:45:32.082710    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad0f0113-dc28-432a-8a36-e8d4e8bf6ede-kube-api-access-68shx" (OuterVolumeSpecName: "kube-api-access-68shx") pod "ad0f0113-dc28-432a-8a36-e8d4e8bf6ede" (UID: "ad0f0113-dc28-432a-8a36-e8d4e8bf6ede"). InnerVolumeSpecName "kube-api-access-68shx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:45:32.181876 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:45:32.181825    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68shx\" (UniqueName: \"kubernetes.io/projected/ad0f0113-dc28-432a-8a36-e8d4e8bf6ede-kube-api-access-68shx\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:45:32.400622 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:45:32.400580    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ad0f0113-dc28-432a-8a36-e8d4e8bf6ede" path="/var/lib/kubelet/pods/ad0f0113-dc28-432a-8a36-e8d4e8bf6ede/volumes"
Sep 08 15:45:32.826445 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:45:32.826371    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7790-139920034489648"
Sep 08 15:45:32.826445 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:45:32.826430    2549 scope.go:117] "RemoveContainer" containerID="a1ef4d2c36550237e6c54889274a44fcff7b8608efb2367ec5af40898ff121b0"
Sep 08 15:45:36.851106 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:45:36.851060    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-36c51d-7bb79986f8-6zjxg"]
Sep 08 15:45:36.851582 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:45:36.851401    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad0f0113-dc28-432a-8a36-e8d4e8bf6ede" containerName="test-curl-7790-139920034489648"
Sep 08 15:45:36.851582 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:45:36.851413    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad0f0113-dc28-432a-8a36-e8d4e8bf6ede" containerName="test-curl-7790-139920034489648"
Sep 08 15:45:36.851582 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:45:36.851470    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad0f0113-dc28-432a-8a36-e8d4e8bf6ede" containerName="test-curl-7790-139920034489648"
Sep 08 15:45:36.855908 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:45:36.855885    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-36c51d-7bb79986f8-6zjxg"
Sep 08 15:45:36.859096 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:45:36.859063    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-36c51d-serving-cert\""
Sep 08 15:45:36.859209 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:45:36.859177    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 08 15:45:36.860477 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:45:36.860456    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 08 15:45:36.860477 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:45:36.860471    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-tq4f4\""
Sep 08 15:45:36.860643 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:45:36.860519    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 08 15:45:36.868933 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:45:36.868911    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-36c51d-7bb79986f8-6zjxg"]
Sep 08 15:45:37.028233 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:45:37.028195    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b14b149c-07c6-4108-a054-a47c6b2c6091-maas-api-tls\") pod \"maas-api-e2e-autores-36c51d-7bb79986f8-6zjxg\" (UID: \"b14b149c-07c6-4108-a054-a47c6b2c6091\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-36c51d-7bb79986f8-6zjxg"
Sep 08 15:45:37.028233 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:45:37.028232    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmz8p\" (UniqueName: \"kubernetes.io/projected/b14b149c-07c6-4108-a054-a47c6b2c6091-kube-api-access-qmz8p\") pod \"maas-api-e2e-autores-36c51d-7bb79986f8-6zjxg\" (UID: \"b14b149c-07c6-4108-a054-a47c6b2c6091\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-36c51d-7bb79986f8-6zjxg"
Sep 08 15:45:37.028446 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:45:37.028267    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b14b149c-07c6-4108-a054-a47c6b2c6091-metrics-certs\") pod \"maas-api-e2e-autores-36c51d-7bb79986f8-6zjxg\" (UID: \"b14b149c-07c6-4108-a054-a47c6b2c6091\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-36c51d-7bb79986f8-6zjxg"
Sep 08 15:45:37.028446 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:45:37.028325    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b14b149c-07c6-4108-a054-a47c6b2c6091-openshift-service-ca\") pod \"maas-api-e2e-autores-36c51d-7bb79986f8-6zjxg\" (UID: \"b14b149c-07c6-4108-a054-a47c6b2c6091\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-36c51d-7bb79986f8-6zjxg"
Sep 08 15:45:37.129671 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:45:37.129575    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b14b149c-07c6-4108-a054-a47c6b2c6091-metrics-certs\") pod \"maas-api-e2e-autores-36c51d-7bb79986f8-6zjxg\" (UID: \"b14b149c-07c6-4108-a054-a47c6b2c6091\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-36c51d-7bb79986f8-6zjxg"
Sep 08 15:45:37.129671 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:45:37.129634    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b14b149c-07c6-4108-a054-a47c6b2c6091-openshift-service-ca\") pod \"maas-api-e2e-autores-36c51d-7bb79986f8-6zjxg\" (UID: \"b14b149c-07c6-4108-a054-a47c6b2c6091\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-36c51d-7bb79986f8-6zjxg"
Sep 08 15:45:37.129946 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:45:37.129699    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b14b149c-07c6-4108-a054-a47c6b2c6091-maas-api-tls\") pod \"maas-api-e2e-autores-36c51d-7bb79986f8-6zjxg\" (UID: \"b14b149c-07c6-4108-a054-a47c6b2c6091\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-36c51d-7bb79986f8-6zjxg"
Sep 08 15:45:37.129946 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:45:37.129721    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmz8p\" (UniqueName: \"kubernetes.io/projected/b14b149c-07c6-4108-a054-a47c6b2c6091-kube-api-access-qmz8p\") pod \"maas-api-e2e-autores-36c51d-7bb79986f8-6zjxg\" (UID: \"b14b149c-07c6-4108-a054-a47c6b2c6091\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-36c51d-7bb79986f8-6zjxg"
Sep 08 15:45:37.129946 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:45:37.129801    2549 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-autores-36c51d-serving-cert: secret "maas-api-e2e-autores-36c51d-serving-cert" not found
Sep 08 15:45:37.129946 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:45:37.129898    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b14b149c-07c6-4108-a054-a47c6b2c6091-maas-api-tls podName:b14b149c-07c6-4108-a054-a47c6b2c6091 nodeName:}" failed. No retries permitted until 2026-09-08 15:45:37.629880804 +0000 UTC m=+775.719331875 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b14b149c-07c6-4108-a054-a47c6b2c6091-maas-api-tls") pod "maas-api-e2e-autores-36c51d-7bb79986f8-6zjxg" (UID: "b14b149c-07c6-4108-a054-a47c6b2c6091") : secret "maas-api-e2e-autores-36c51d-serving-cert" not found
Sep 08 15:45:37.130401 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:45:37.130383    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b14b149c-07c6-4108-a054-a47c6b2c6091-openshift-service-ca\") pod \"maas-api-e2e-autores-36c51d-7bb79986f8-6zjxg\" (UID: \"b14b149c-07c6-4108-a054-a47c6b2c6091\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-36c51d-7bb79986f8-6zjxg"
Sep 08 15:45:37.132198 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:45:37.132176    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b14b149c-07c6-4108-a054-a47c6b2c6091-metrics-certs\") pod \"maas-api-e2e-autores-36c51d-7bb79986f8-6zjxg\" (UID: \"b14b149c-07c6-4108-a054-a47c6b2c6091\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-36c51d-7bb79986f8-6zjxg"
Sep 08 15:45:37.140796 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:45:37.140769    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmz8p\" (UniqueName: \"kubernetes.io/projected/b14b149c-07c6-4108-a054-a47c6b2c6091-kube-api-access-qmz8p\") pod \"maas-api-e2e-autores-36c51d-7bb79986f8-6zjxg\" (UID: \"b14b149c-07c6-4108-a054-a47c6b2c6091\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-36c51d-7bb79986f8-6zjxg"
Sep 08 15:45:37.635377 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:45:37.635343    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b14b149c-07c6-4108-a054-a47c6b2c6091-maas-api-tls\") pod \"maas-api-e2e-autores-36c51d-7bb79986f8-6zjxg\" (UID: \"b14b149c-07c6-4108-a054-a47c6b2c6091\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-36c51d-7bb79986f8-6zjxg"
Sep 08 15:45:37.637696 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:45:37.637666    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b14b149c-07c6-4108-a054-a47c6b2c6091-maas-api-tls\") pod \"maas-api-e2e-autores-36c51d-7bb79986f8-6zjxg\" (UID: \"b14b149c-07c6-4108-a054-a47c6b2c6091\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-36c51d-7bb79986f8-6zjxg"
Sep 08 15:45:37.769271 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:45:37.769223    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-36c51d-7bb79986f8-6zjxg"
Sep 08 15:45:38.045175 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:45:38.045151    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-36c51d-7bb79986f8-6zjxg"]
Sep 08 15:45:38.047314 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:45:38.047290    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb14b149c_07c6_4108_a054_a47c6b2c6091.slice/crio-defd94ee532bd7c84f0fdd956e23aae7e6049627e74189c4457ecae41b6a1129 WatchSource:0}: Error finding container defd94ee532bd7c84f0fdd956e23aae7e6049627e74189c4457ecae41b6a1129: Status 404 returned error can't find the container with id defd94ee532bd7c84f0fdd956e23aae7e6049627e74189c4457ecae41b6a1129
Sep 08 15:45:38.863274 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:45:38.863237    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-36c51d-7bb79986f8-6zjxg" event={"ID":"b14b149c-07c6-4108-a054-a47c6b2c6091","Type":"ContainerStarted","Data":"defd94ee532bd7c84f0fdd956e23aae7e6049627e74189c4457ecae41b6a1129"}
Sep 08 15:45:40.874561 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:45:40.874525    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-36c51d-7bb79986f8-6zjxg" event={"ID":"b14b149c-07c6-4108-a054-a47c6b2c6091","Type":"ContainerStarted","Data":"a0405d12a3fb5857d0bad49f5f6b826e22ce6b7d3fafabf59aef135600b4aa0a"}
Sep 08 15:45:40.874973 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:45:40.874597    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-36c51d-7bb79986f8-6zjxg"
Sep 08 15:45:40.907734 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:45:40.907683    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-36c51d-7bb79986f8-6zjxg" podStartSLOduration=2.829707932 podStartE2EDuration="4.907666525s" totalImagesPullingTime="2.077958593s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:45:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:45:40.906372849 +0000 UTC m=+778.995823929" watchObservedRunningTime="2026-09-08 15:45:40.907666525 +0000 UTC m=+778.997117614"
Sep 08 15:45:46.884953 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:45:46.884919    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-36c51d-7bb79986f8-6zjxg"
Sep 08 15:45:51.232642 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:45:51.229991    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-a057c5-openshift-default-5f564fb8cd-2mflt"]
Sep 08 15:45:51.236385 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:45:51.236356    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-a057c5-openshift-default-5f564fb8cd-2mflt"
Sep 08 15:45:51.239569 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:45:51.239549    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-a057c5-openshift-default-dockercfg-nkps9\""
Sep 08 15:45:51.244291 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:45:51.244267    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-a057c5-openshift-default-5f564fb8cd-2mflt"]
Sep 08 15:45:51.251996 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:45:51.251974    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9hvl\" (UniqueName: \"kubernetes.io/projected/55c30a1d-0943-4733-9a53-ed49d3d4184d-kube-api-access-n9hvl\") pod \"e2e-inf-b-a057c5-openshift-default-5f564fb8cd-2mflt\" (UID: \"55c30a1d-0943-4733-9a53-ed49d3d4184d\") " pod="openshift-ingress/e2e-inf-b-a057c5-openshift-default-5f564fb8cd-2mflt"
Sep 08 15:45:51.252076 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:45:51.252008    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55c30a1d-0943-4733-9a53-ed49d3d4184d-workload-socket\") pod \"e2e-inf-b-a057c5-openshift-default-5f564fb8cd-2mflt\" (UID: \"55c30a1d-0943-4733-9a53-ed49d3d4184d\") " pod="openshift-ingress/e2e-inf-b-a057c5-openshift-default-5f564fb8cd-2mflt"
Sep 08 15:45:51.252076 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:45:51.252030    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55c30a1d-0943-4733-9a53-ed49d3d4184d-credential-socket\") pod \"e2e-inf-b-a057c5-openshift-default-5f564fb8cd-2mflt\" (UID: \"55c30a1d-0943-4733-9a53-ed49d3d4184d\") " pod="openshift-ingress/e2e-inf-b-a057c5-openshift-default-5f564fb8cd-2mflt"
Sep 08 15:45:51.252076 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:45:51.252053    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55c30a1d-0943-4733-9a53-ed49d3d4184d-istio-token\") pod \"e2e-inf-b-a057c5-openshift-default-5f564fb8cd-2mflt\" (UID: \"55c30a1d-0943-4733-9a53-ed49d3d4184d\") " pod="openshift-ingress/e2e-inf-b-a057c5-openshift-default-5f564fb8cd-2mflt"
Sep 08 15:45:51.252185 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:45:51.252079    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55c30a1d-0943-4733-9a53-ed49d3d4184d-istio-podinfo\") pod \"e2e-inf-b-a057c5-openshift-default-5f564fb8cd-2mflt\" (UID: \"55c30a1d-0943-4733-9a53-ed49d3d4184d\") " pod="openshift-ingress/e2e-inf-b-a057c5-openshift-default-5f564fb8cd-2mflt"
Sep 08 15:45:51.252185 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:45:51.252098    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55c30a1d-0943-4733-9a53-ed49d3d4184d-istio-envoy\") pod \"e2e-inf-b-a057c5-openshift-default-5f564fb8cd-2mflt\" (UID: \"55c30a1d-0943-4733-9a53-ed49d3d4184d\") " pod="openshift-ingress/e2e-inf-b-a057c5-openshift-default-5f564fb8cd-2mflt"
Sep 08 15:45:51.252185 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:45:51.252112    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55c30a1d-0943-4733-9a53-ed49d3d4184d-istiod-ca-cert\") pod \"e2e-inf-b-a057c5-openshift-default-5f564fb8cd-2mflt\" (UID: \"55c30a1d-0943-4733-9a53-ed49d3d4184d\") " pod="openshift-ingress/e2e-inf-b-a057c5-openshift-default-5f564fb8cd-2mflt"
Sep 08 15:45:51.252276 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:45:51.252195    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55c30a1d-0943-4733-9a53-ed49d3d4184d-istio-data\") pod \"e2e-inf-b-a057c5-openshift-default-5f564fb8cd-2mflt\" (UID: \"55c30a1d-0943-4733-9a53-ed49d3d4184d\") " pod="openshift-ingress/e2e-inf-b-a057c5-openshift-default-5f564fb8cd-2mflt"
Sep 08 15:45:51.252276 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:45:51.252233    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55c30a1d-0943-4733-9a53-ed49d3d4184d-workload-certs\") pod \"e2e-inf-b-a057c5-openshift-default-5f564fb8cd-2mflt\" (UID: \"55c30a1d-0943-4733-9a53-ed49d3d4184d\") " pod="openshift-ingress/e2e-inf-b-a057c5-openshift-default-5f564fb8cd-2mflt"
Sep 08 15:45:51.352595 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:45:51.352557    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55c30a1d-0943-4733-9a53-ed49d3d4184d-credential-socket\") pod \"e2e-inf-b-a057c5-openshift-default-5f564fb8cd-2mflt\" (UID: \"55c30a1d-0943-4733-9a53-ed49d3d4184d\") " pod="openshift-ingress/e2e-inf-b-a057c5-openshift-default-5f564fb8cd-2mflt"
Sep 08 15:45:51.352595 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:45:51.352595    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55c30a1d-0943-4733-9a53-ed49d3d4184d-istio-token\") pod \"e2e-inf-b-a057c5-openshift-default-5f564fb8cd-2mflt\" (UID: \"55c30a1d-0943-4733-9a53-ed49d3d4184d\") " pod="openshift-ingress/e2e-inf-b-a057c5-openshift-default-5f564fb8cd-2mflt"
Sep 08 15:45:51.352832 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:45:51.352619    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55c30a1d-0943-4733-9a53-ed49d3d4184d-istio-podinfo\") pod \"e2e-inf-b-a057c5-openshift-default-5f564fb8cd-2mflt\" (UID: \"55c30a1d-0943-4733-9a53-ed49d3d4184d\") " pod="openshift-ingress/e2e-inf-b-a057c5-openshift-default-5f564fb8cd-2mflt"
Sep 08 15:45:51.352832 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:45:51.352649    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55c30a1d-0943-4733-9a53-ed49d3d4184d-istio-envoy\") pod \"e2e-inf-b-a057c5-openshift-default-5f564fb8cd-2mflt\" (UID: \"55c30a1d-0943-4733-9a53-ed49d3d4184d\") " pod="openshift-ingress/e2e-inf-b-a057c5-openshift-default-5f564fb8cd-2mflt"
Sep 08 15:45:51.352832 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:45:51.352673    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55c30a1d-0943-4733-9a53-ed49d3d4184d-istiod-ca-cert\") pod \"e2e-inf-b-a057c5-openshift-default-5f564fb8cd-2mflt\" (UID: \"55c30a1d-0943-4733-9a53-ed49d3d4184d\") " pod="openshift-ingress/e2e-inf-b-a057c5-openshift-default-5f564fb8cd-2mflt"
Sep 08 15:45:51.352832 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:45:51.352728    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55c30a1d-0943-4733-9a53-ed49d3d4184d-istio-data\") pod \"e2e-inf-b-a057c5-openshift-default-5f564fb8cd-2mflt\" (UID: \"55c30a1d-0943-4733-9a53-ed49d3d4184d\") " pod="openshift-ingress/e2e-inf-b-a057c5-openshift-default-5f564fb8cd-2mflt"
Sep 08 15:45:51.353069 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:45:51.352893    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55c30a1d-0943-4733-9a53-ed49d3d4184d-workload-certs\") pod \"e2e-inf-b-a057c5-openshift-default-5f564fb8cd-2mflt\" (UID: \"55c30a1d-0943-4733-9a53-ed49d3d4184d\") " pod="openshift-ingress/e2e-inf-b-a057c5-openshift-default-5f564fb8cd-2mflt"
Sep 08 15:45:51.353069 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:45:51.352955    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9hvl\" (UniqueName: \"kubernetes.io/projected/55c30a1d-0943-4733-9a53-ed49d3d4184d-kube-api-access-n9hvl\") pod \"e2e-inf-b-a057c5-openshift-default-5f564fb8cd-2mflt\" (UID: \"55c30a1d-0943-4733-9a53-ed49d3d4184d\") " pod="openshift-ingress/e2e-inf-b-a057c5-openshift-default-5f564fb8cd-2mflt"
Sep 08 15:45:51.353069 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:45:51.352987    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55c30a1d-0943-4733-9a53-ed49d3d4184d-workload-socket\") pod \"e2e-inf-b-a057c5-openshift-default-5f564fb8cd-2mflt\" (UID: \"55c30a1d-0943-4733-9a53-ed49d3d4184d\") " pod="openshift-ingress/e2e-inf-b-a057c5-openshift-default-5f564fb8cd-2mflt"
Sep 08 15:45:51.353069 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:45:51.353033    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55c30a1d-0943-4733-9a53-ed49d3d4184d-credential-socket\") pod \"e2e-inf-b-a057c5-openshift-default-5f564fb8cd-2mflt\" (UID: \"55c30a1d-0943-4733-9a53-ed49d3d4184d\") " pod="openshift-ingress/e2e-inf-b-a057c5-openshift-default-5f564fb8cd-2mflt"
Sep 08 15:45:51.353268 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:45:51.353078    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55c30a1d-0943-4733-9a53-ed49d3d4184d-istio-data\") pod \"e2e-inf-b-a057c5-openshift-default-5f564fb8cd-2mflt\" (UID: \"55c30a1d-0943-4733-9a53-ed49d3d4184d\") " pod="openshift-ingress/e2e-inf-b-a057c5-openshift-default-5f564fb8cd-2mflt"
Sep 08 15:45:51.353323 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:45:51.353304    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55c30a1d-0943-4733-9a53-ed49d3d4184d-workload-socket\") pod \"e2e-inf-b-a057c5-openshift-default-5f564fb8cd-2mflt\" (UID: \"55c30a1d-0943-4733-9a53-ed49d3d4184d\") " pod="openshift-ingress/e2e-inf-b-a057c5-openshift-default-5f564fb8cd-2mflt"
Sep 08 15:45:51.353381 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:45:51.353304    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55c30a1d-0943-4733-9a53-ed49d3d4184d-workload-certs\") pod \"e2e-inf-b-a057c5-openshift-default-5f564fb8cd-2mflt\" (UID: \"55c30a1d-0943-4733-9a53-ed49d3d4184d\") " pod="openshift-ingress/e2e-inf-b-a057c5-openshift-default-5f564fb8cd-2mflt"
Sep 08 15:45:51.353447 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:45:51.353427    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55c30a1d-0943-4733-9a53-ed49d3d4184d-istiod-ca-cert\") pod \"e2e-inf-b-a057c5-openshift-default-5f564fb8cd-2mflt\" (UID: \"55c30a1d-0943-4733-9a53-ed49d3d4184d\") " pod="openshift-ingress/e2e-inf-b-a057c5-openshift-default-5f564fb8cd-2mflt"
Sep 08 15:45:51.354914 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:45:51.354893    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55c30a1d-0943-4733-9a53-ed49d3d4184d-istio-envoy\") pod \"e2e-inf-b-a057c5-openshift-default-5f564fb8cd-2mflt\" (UID: \"55c30a1d-0943-4733-9a53-ed49d3d4184d\") " pod="openshift-ingress/e2e-inf-b-a057c5-openshift-default-5f564fb8cd-2mflt"
Sep 08 15:45:51.355364 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:45:51.355343    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55c30a1d-0943-4733-9a53-ed49d3d4184d-istio-podinfo\") pod \"e2e-inf-b-a057c5-openshift-default-5f564fb8cd-2mflt\" (UID: \"55c30a1d-0943-4733-9a53-ed49d3d4184d\") " pod="openshift-ingress/e2e-inf-b-a057c5-openshift-default-5f564fb8cd-2mflt"
Sep 08 15:45:51.360594 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:45:51.360576    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55c30a1d-0943-4733-9a53-ed49d3d4184d-istio-token\") pod \"e2e-inf-b-a057c5-openshift-default-5f564fb8cd-2mflt\" (UID: \"55c30a1d-0943-4733-9a53-ed49d3d4184d\") " pod="openshift-ingress/e2e-inf-b-a057c5-openshift-default-5f564fb8cd-2mflt"
Sep 08 15:45:51.360929 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:45:51.360909    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9hvl\" (UniqueName: \"kubernetes.io/projected/55c30a1d-0943-4733-9a53-ed49d3d4184d-kube-api-access-n9hvl\") pod \"e2e-inf-b-a057c5-openshift-default-5f564fb8cd-2mflt\" (UID: \"55c30a1d-0943-4733-9a53-ed49d3d4184d\") " pod="openshift-ingress/e2e-inf-b-a057c5-openshift-default-5f564fb8cd-2mflt"
Sep 08 15:45:51.549095 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:45:51.549008    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-a057c5-openshift-default-5f564fb8cd-2mflt"
Sep 08 15:45:51.822905 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:45:51.822880    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-a057c5-openshift-default-5f564fb8cd-2mflt"]
Sep 08 15:45:51.824466 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:45:51.824437    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55c30a1d_0943_4733_9a53_ed49d3d4184d.slice/crio-5a583b0657349218bfe3e99dfa19f4313c70be437013aa4f00d9aa227710ae87 WatchSource:0}: Error finding container 5a583b0657349218bfe3e99dfa19f4313c70be437013aa4f00d9aa227710ae87: Status 404 returned error can't find the container with id 5a583b0657349218bfe3e99dfa19f4313c70be437013aa4f00d9aa227710ae87
Sep 08 15:45:51.928461 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:45:51.928426    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a057c5-openshift-default-5f564fb8cd-2mflt" event={"ID":"55c30a1d-0943-4733-9a53-ed49d3d4184d","Type":"ContainerStarted","Data":"5a583b0657349218bfe3e99dfa19f4313c70be437013aa4f00d9aa227710ae87"}
Sep 08 15:45:54.940375 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:45:54.940332    2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 08 15:45:54.940775 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:45:54.940403    2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 08 15:45:54.940775 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:45:54.940432    2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 08 15:45:55.950109 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:45:55.950073    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a057c5-openshift-default-5f564fb8cd-2mflt" event={"ID":"55c30a1d-0943-4733-9a53-ed49d3d4184d","Type":"ContainerStarted","Data":"b6b9f975886b8fc8a2cf5c649c9d94b841ee530352f5b76a999fecfcd62d1073"}
Sep 08 15:45:55.980911 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:45:55.980851    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-a057c5-openshift-default-5f564fb8cd-2mflt" podStartSLOduration=1.8674211509999998 podStartE2EDuration="4.980834142s" totalImagesPullingTime="3.113412991s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:45:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:45:55.979834601 +0000 UTC m=+794.069285690" watchObservedRunningTime="2026-09-08 15:45:55.980834142 +0000 UTC m=+794.070285231"
Sep 08 15:45:56.549979 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:45:56.549938    2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-a057c5-openshift-default-5f564fb8cd-2mflt"
Sep 08 15:45:56.555517 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:45:56.555493    2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-a057c5-openshift-default-5f564fb8cd-2mflt"
Sep 08 15:45:56.955661 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:45:56.955634    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-a057c5-openshift-default-5f564fb8cd-2mflt"
Sep 08 15:45:56.956411 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:45:56.956392    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-a057c5-openshift-default-5f564fb8cd-2mflt"
Sep 08 15:46:00.541173 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:46:00.541139    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-a057c5-78b7f5fbfd-kdgcz"]
Sep 08 15:46:00.545406 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:46:00.545386    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a057c5-78b7f5fbfd-kdgcz"
Sep 08 15:46:00.548098 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:46:00.548070    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-a057c5-serving-cert\""
Sep 08 15:46:00.554825 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:46:00.554800    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-a057c5-78b7f5fbfd-kdgcz"]
Sep 08 15:46:00.629449 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:46:00.629419    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c67f9b6-d044-475a-a032-c869b4a8c158-maas-api-tls\") pod \"maas-api-e2e-inf-b-a057c5-78b7f5fbfd-kdgcz\" (UID: \"6c67f9b6-d044-475a-a032-c869b4a8c158\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a057c5-78b7f5fbfd-kdgcz"
Sep 08 15:46:00.629600 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:46:00.629453    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c67f9b6-d044-475a-a032-c869b4a8c158-openshift-service-ca\") pod \"maas-api-e2e-inf-b-a057c5-78b7f5fbfd-kdgcz\" (UID: \"6c67f9b6-d044-475a-a032-c869b4a8c158\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a057c5-78b7f5fbfd-kdgcz"
Sep 08 15:46:00.629600 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:46:00.629562    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stpk2\" (UniqueName: \"kubernetes.io/projected/6c67f9b6-d044-475a-a032-c869b4a8c158-kube-api-access-stpk2\") pod \"maas-api-e2e-inf-b-a057c5-78b7f5fbfd-kdgcz\" (UID: \"6c67f9b6-d044-475a-a032-c869b4a8c158\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a057c5-78b7f5fbfd-kdgcz"
Sep 08 15:46:00.629712 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:46:00.629599    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c67f9b6-d044-475a-a032-c869b4a8c158-metrics-certs\") pod \"maas-api-e2e-inf-b-a057c5-78b7f5fbfd-kdgcz\" (UID: \"6c67f9b6-d044-475a-a032-c869b4a8c158\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a057c5-78b7f5fbfd-kdgcz"
Sep 08 15:46:00.730484 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:46:00.730451    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c67f9b6-d044-475a-a032-c869b4a8c158-maas-api-tls\") pod \"maas-api-e2e-inf-b-a057c5-78b7f5fbfd-kdgcz\" (UID: \"6c67f9b6-d044-475a-a032-c869b4a8c158\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a057c5-78b7f5fbfd-kdgcz"
Sep 08 15:46:00.730662 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:46:00.730492    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c67f9b6-d044-475a-a032-c869b4a8c158-openshift-service-ca\") pod \"maas-api-e2e-inf-b-a057c5-78b7f5fbfd-kdgcz\" (UID: \"6c67f9b6-d044-475a-a032-c869b4a8c158\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a057c5-78b7f5fbfd-kdgcz"
Sep 08 15:46:00.730662 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:46:00.730551    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stpk2\" (UniqueName: \"kubernetes.io/projected/6c67f9b6-d044-475a-a032-c869b4a8c158-kube-api-access-stpk2\") pod \"maas-api-e2e-inf-b-a057c5-78b7f5fbfd-kdgcz\" (UID: \"6c67f9b6-d044-475a-a032-c869b4a8c158\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a057c5-78b7f5fbfd-kdgcz"
Sep 08 15:46:00.730662 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:46:00.730575    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c67f9b6-d044-475a-a032-c869b4a8c158-metrics-certs\") pod \"maas-api-e2e-inf-b-a057c5-78b7f5fbfd-kdgcz\" (UID: \"6c67f9b6-d044-475a-a032-c869b4a8c158\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a057c5-78b7f5fbfd-kdgcz"
Sep 08 15:46:00.730662 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:46:00.730606    2549 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-a057c5-serving-cert: secret "maas-api-e2e-inf-b-a057c5-serving-cert" not found
Sep 08 15:46:00.730898 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:46:00.730685    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c67f9b6-d044-475a-a032-c869b4a8c158-maas-api-tls podName:6c67f9b6-d044-475a-a032-c869b4a8c158 nodeName:}" failed. No retries permitted until 2026-09-08 15:46:01.230665405 +0000 UTC m=+799.320116487 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6c67f9b6-d044-475a-a032-c869b4a8c158-maas-api-tls") pod "maas-api-e2e-inf-b-a057c5-78b7f5fbfd-kdgcz" (UID: "6c67f9b6-d044-475a-a032-c869b4a8c158") : secret "maas-api-e2e-inf-b-a057c5-serving-cert" not found
Sep 08 15:46:00.731224 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:46:00.731203    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c67f9b6-d044-475a-a032-c869b4a8c158-openshift-service-ca\") pod \"maas-api-e2e-inf-b-a057c5-78b7f5fbfd-kdgcz\" (UID: \"6c67f9b6-d044-475a-a032-c869b4a8c158\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a057c5-78b7f5fbfd-kdgcz"
Sep 08 15:46:00.733127 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:46:00.733096    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c67f9b6-d044-475a-a032-c869b4a8c158-metrics-certs\") pod \"maas-api-e2e-inf-b-a057c5-78b7f5fbfd-kdgcz\" (UID: \"6c67f9b6-d044-475a-a032-c869b4a8c158\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a057c5-78b7f5fbfd-kdgcz"
Sep 08 15:46:00.739830 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:46:00.739809    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-stpk2\" (UniqueName: \"kubernetes.io/projected/6c67f9b6-d044-475a-a032-c869b4a8c158-kube-api-access-stpk2\") pod \"maas-api-e2e-inf-b-a057c5-78b7f5fbfd-kdgcz\" (UID: \"6c67f9b6-d044-475a-a032-c869b4a8c158\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a057c5-78b7f5fbfd-kdgcz"
Sep 08 15:46:01.235679 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:46:01.235645    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c67f9b6-d044-475a-a032-c869b4a8c158-maas-api-tls\") pod \"maas-api-e2e-inf-b-a057c5-78b7f5fbfd-kdgcz\" (UID: \"6c67f9b6-d044-475a-a032-c869b4a8c158\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a057c5-78b7f5fbfd-kdgcz"
Sep 08 15:46:01.238714 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:46:01.238682    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c67f9b6-d044-475a-a032-c869b4a8c158-maas-api-tls\") pod \"maas-api-e2e-inf-b-a057c5-78b7f5fbfd-kdgcz\" (UID: \"6c67f9b6-d044-475a-a032-c869b4a8c158\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a057c5-78b7f5fbfd-kdgcz"
Sep 08 15:46:01.458926 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:46:01.458892    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a057c5-78b7f5fbfd-kdgcz"
Sep 08 15:46:01.730154 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:46:01.730096    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-a057c5-78b7f5fbfd-kdgcz"]
Sep 08 15:46:01.732349 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:46:01.732317    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c67f9b6_d044_475a_a032_c869b4a8c158.slice/crio-6d5b4dad8892ec5a2a59f1e1c8b0673674f9282e07e0a29ebec635dd2caed906 WatchSource:0}: Error finding container 6d5b4dad8892ec5a2a59f1e1c8b0673674f9282e07e0a29ebec635dd2caed906: Status 404 returned error can't find the container with id 6d5b4dad8892ec5a2a59f1e1c8b0673674f9282e07e0a29ebec635dd2caed906
Sep 08 15:46:01.980623 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:46:01.980590    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a057c5-78b7f5fbfd-kdgcz" event={"ID":"6c67f9b6-d044-475a-a032-c869b4a8c158","Type":"ContainerStarted","Data":"6d5b4dad8892ec5a2a59f1e1c8b0673674f9282e07e0a29ebec635dd2caed906"}
Sep 08 15:46:03.991376 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:46:03.991347    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a057c5-78b7f5fbfd-kdgcz" event={"ID":"6c67f9b6-d044-475a-a032-c869b4a8c158","Type":"ContainerStarted","Data":"7359d0ca2f80d4ff3e01841cdc8466be566b583fc4c30ddc38f4c40e3f964630"}
Sep 08 15:46:03.991742 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:46:03.991490    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a057c5-78b7f5fbfd-kdgcz"
Sep 08 15:46:04.010159 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:46:04.010113    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a057c5-78b7f5fbfd-kdgcz" podStartSLOduration=1.869360349 podStartE2EDuration="4.010097576s" totalImagesPullingTime="2.140737227s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:46:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:46:04.009283865 +0000 UTC m=+802.098734956" watchObservedRunningTime="2026-09-08 15:46:04.010097576 +0000 UTC m=+802.099548664"
Sep 08 15:46:10.002236 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:46:10.002205    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a057c5-78b7f5fbfd-kdgcz"
Sep 08 15:46:13.607339 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:46:13.607302    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-conflict-a-a97eab05-kserve-5c4fd789f-mdtbg"]
Sep 08 15:46:13.610707 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:46:13.610684    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-a97eab05-kserve-5c4fd789f-mdtbg"
Sep 08 15:46:13.614683 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:46:13.614656    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-przhh\""
Sep 08 15:46:13.614803 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:46:13.614724    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 08 15:46:13.614803 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:46:13.614771    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-conflict-a-a97eab05-kserve-self-signed-certs\""
Sep 08 15:46:13.614803 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:46:13.614780    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 08 15:46:13.621534 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:46:13.621490    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-a-a97eab05-kserve-5c4fd789f-mdtbg"]
Sep 08 15:46:13.749883 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:46:13.747839    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2af2595b-8e90-4149-b1e2-c39fb945c775-tmp-dir\") pod \"e2e-conflict-a-a97eab05-kserve-5c4fd789f-mdtbg\" (UID: \"2af2595b-8e90-4149-b1e2-c39fb945c775\") " pod="llm/e2e-conflict-a-a97eab05-kserve-5c4fd789f-mdtbg"
Sep 08 15:46:13.749883 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:46:13.747976    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2af2595b-8e90-4149-b1e2-c39fb945c775-tls-certs\") pod \"e2e-conflict-a-a97eab05-kserve-5c4fd789f-mdtbg\" (UID: \"2af2595b-8e90-4149-b1e2-c39fb945c775\") " pod="llm/e2e-conflict-a-a97eab05-kserve-5c4fd789f-mdtbg"
Sep 08 15:46:13.749883 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:46:13.748024    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zs6ln\" (UniqueName: \"kubernetes.io/projected/2af2595b-8e90-4149-b1e2-c39fb945c775-kube-api-access-zs6ln\") pod \"e2e-conflict-a-a97eab05-kserve-5c4fd789f-mdtbg\" (UID: \"2af2595b-8e90-4149-b1e2-c39fb945c775\") " pod="llm/e2e-conflict-a-a97eab05-kserve-5c4fd789f-mdtbg"
Sep 08 15:46:13.749883 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:46:13.748094    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2af2595b-8e90-4149-b1e2-c39fb945c775-home\") pod \"e2e-conflict-a-a97eab05-kserve-5c4fd789f-mdtbg\" (UID: \"2af2595b-8e90-4149-b1e2-c39fb945c775\") " pod="llm/e2e-conflict-a-a97eab05-kserve-5c4fd789f-mdtbg"
Sep 08 15:46:13.749883 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:46:13.748117    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2af2595b-8e90-4149-b1e2-c39fb945c775-dshm\") pod \"e2e-conflict-a-a97eab05-kserve-5c4fd789f-mdtbg\" (UID: \"2af2595b-8e90-4149-b1e2-c39fb945c775\") " pod="llm/e2e-conflict-a-a97eab05-kserve-5c4fd789f-mdtbg"
Sep 08 15:46:13.749883 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:46:13.748146    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2af2595b-8e90-4149-b1e2-c39fb945c775-model-cache\") pod \"e2e-conflict-a-a97eab05-kserve-5c4fd789f-mdtbg\" (UID: \"2af2595b-8e90-4149-b1e2-c39fb945c775\") " pod="llm/e2e-conflict-a-a97eab05-kserve-5c4fd789f-mdtbg"
Sep 08 15:46:13.849347 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:46:13.849269    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2af2595b-8e90-4149-b1e2-c39fb945c775-home\") pod \"e2e-conflict-a-a97eab05-kserve-5c4fd789f-mdtbg\" (UID: \"2af2595b-8e90-4149-b1e2-c39fb945c775\") " pod="llm/e2e-conflict-a-a97eab05-kserve-5c4fd789f-mdtbg"
Sep 08 15:46:13.849347 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:46:13.849305    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2af2595b-8e90-4149-b1e2-c39fb945c775-dshm\") pod \"e2e-conflict-a-a97eab05-kserve-5c4fd789f-mdtbg\" (UID: \"2af2595b-8e90-4149-b1e2-c39fb945c775\") " pod="llm/e2e-conflict-a-a97eab05-kserve-5c4fd789f-mdtbg"
Sep 08 15:46:13.849347 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:46:13.849321    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2af2595b-8e90-4149-b1e2-c39fb945c775-model-cache\") pod \"e2e-conflict-a-a97eab05-kserve-5c4fd789f-mdtbg\" (UID: \"2af2595b-8e90-4149-b1e2-c39fb945c775\") " pod="llm/e2e-conflict-a-a97eab05-kserve-5c4fd789f-mdtbg"
Sep 08 15:46:13.849598 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:46:13.849480    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2af2595b-8e90-4149-b1e2-c39fb945c775-tmp-dir\") pod \"e2e-conflict-a-a97eab05-kserve-5c4fd789f-mdtbg\" (UID: \"2af2595b-8e90-4149-b1e2-c39fb945c775\") " pod="llm/e2e-conflict-a-a97eab05-kserve-5c4fd789f-mdtbg"
Sep 08 15:46:13.849598 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:46:13.849585    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2af2595b-8e90-4149-b1e2-c39fb945c775-tls-certs\") pod \"e2e-conflict-a-a97eab05-kserve-5c4fd789f-mdtbg\" (UID: \"2af2595b-8e90-4149-b1e2-c39fb945c775\") " pod="llm/e2e-conflict-a-a97eab05-kserve-5c4fd789f-mdtbg"
Sep 08 15:46:13.849699 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:46:13.849613    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zs6ln\" (UniqueName: \"kubernetes.io/projected/2af2595b-8e90-4149-b1e2-c39fb945c775-kube-api-access-zs6ln\") pod \"e2e-conflict-a-a97eab05-kserve-5c4fd789f-mdtbg\" (UID: \"2af2595b-8e90-4149-b1e2-c39fb945c775\") " pod="llm/e2e-conflict-a-a97eab05-kserve-5c4fd789f-mdtbg"
Sep 08 15:46:13.849750 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:46:13.849698    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2af2595b-8e90-4149-b1e2-c39fb945c775-model-cache\") pod \"e2e-conflict-a-a97eab05-kserve-5c4fd789f-mdtbg\" (UID: \"2af2595b-8e90-4149-b1e2-c39fb945c775\") " pod="llm/e2e-conflict-a-a97eab05-kserve-5c4fd789f-mdtbg"
Sep 08 15:46:13.849807 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:46:13.849750    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2af2595b-8e90-4149-b1e2-c39fb945c775-home\") pod \"e2e-conflict-a-a97eab05-kserve-5c4fd789f-mdtbg\" (UID: \"2af2595b-8e90-4149-b1e2-c39fb945c775\") " pod="llm/e2e-conflict-a-a97eab05-kserve-5c4fd789f-mdtbg"
Sep 08 15:46:13.849924 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:46:13.849900    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2af2595b-8e90-4149-b1e2-c39fb945c775-tmp-dir\") pod \"e2e-conflict-a-a97eab05-kserve-5c4fd789f-mdtbg\" (UID: \"2af2595b-8e90-4149-b1e2-c39fb945c775\") " pod="llm/e2e-conflict-a-a97eab05-kserve-5c4fd789f-mdtbg"
Sep 08 15:46:13.851628 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:46:13.851603    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2af2595b-8e90-4149-b1e2-c39fb945c775-dshm\") pod \"e2e-conflict-a-a97eab05-kserve-5c4fd789f-mdtbg\" (UID: \"2af2595b-8e90-4149-b1e2-c39fb945c775\") " pod="llm/e2e-conflict-a-a97eab05-kserve-5c4fd789f-mdtbg"
Sep 08 15:46:13.851966 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:46:13.851947    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2af2595b-8e90-4149-b1e2-c39fb945c775-tls-certs\") pod \"e2e-conflict-a-a97eab05-kserve-5c4fd789f-mdtbg\" (UID: \"2af2595b-8e90-4149-b1e2-c39fb945c775\") " pod="llm/e2e-conflict-a-a97eab05-kserve-5c4fd789f-mdtbg"
Sep 08 15:46:13.857514 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:46:13.857490    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zs6ln\" (UniqueName: \"kubernetes.io/projected/2af2595b-8e90-4149-b1e2-c39fb945c775-kube-api-access-zs6ln\") pod \"e2e-conflict-a-a97eab05-kserve-5c4fd789f-mdtbg\" (UID: \"2af2595b-8e90-4149-b1e2-c39fb945c775\") " pod="llm/e2e-conflict-a-a97eab05-kserve-5c4fd789f-mdtbg"
Sep 08 15:46:13.921377 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:46:13.921347    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-a97eab05-kserve-5c4fd789f-mdtbg"
Sep 08 15:46:14.201216 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:46:14.201192    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-a-a97eab05-kserve-5c4fd789f-mdtbg"]
Sep 08 15:46:14.203796 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:46:14.203765    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2af2595b_8e90_4149_b1e2_c39fb945c775.slice/crio-6c2b8195eba4db60fea0de90dbd9260faf88db3b97ca47389460b9dc62f86327 WatchSource:0}: Error finding container 6c2b8195eba4db60fea0de90dbd9260faf88db3b97ca47389460b9dc62f86327: Status 404 returned error can't find the container with id 6c2b8195eba4db60fea0de90dbd9260faf88db3b97ca47389460b9dc62f86327
Sep 08 15:46:15.050679 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:46:15.050634    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-a97eab05-kserve-5c4fd789f-mdtbg" event={"ID":"2af2595b-8e90-4149-b1e2-c39fb945c775","Type":"ContainerStarted","Data":"6c2b8195eba4db60fea0de90dbd9260faf88db3b97ca47389460b9dc62f86327"}
Sep 08 15:46:16.057139 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:46:16.057103    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-a97eab05-kserve-5c4fd789f-mdtbg" event={"ID":"2af2595b-8e90-4149-b1e2-c39fb945c775","Type":"ContainerStarted","Data":"8737834fb50dd5fa3bee61d1d45e235fb0afe52f10ce3fdbee659ec42f3edb74"}
Sep 08 15:46:16.075667 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:46:16.075617    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-conflict-a-a97eab05-kserve-5c4fd789f-mdtbg" podStartSLOduration=1.9263025040000001 podStartE2EDuration="3.075601582s" totalImagesPullingTime="1.149299078s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:46:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:46:16.074482469 +0000 UTC m=+814.163933580" watchObservedRunningTime="2026-09-08 15:46:16.075601582 +0000 UTC m=+814.165052671"
Sep 08 15:46:17.063519 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:46:17.063440    2549 generic.go:356] "Generic (PLEG): container finished" podID="55c30a1d-0943-4733-9a53-ed49d3d4184d" containerID="b6b9f975886b8fc8a2cf5c649c9d94b841ee530352f5b76a999fecfcd62d1073" exitCode=0
Sep 08 15:46:17.063921 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:46:17.063514    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a057c5-openshift-default-5f564fb8cd-2mflt" event={"ID":"55c30a1d-0943-4733-9a53-ed49d3d4184d","Type":"ContainerDied","Data":"b6b9f975886b8fc8a2cf5c649c9d94b841ee530352f5b76a999fecfcd62d1073"}
Sep 08 15:46:17.063964 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:46:17.063946    2549 scope.go:117] "RemoveContainer" containerID="b6b9f975886b8fc8a2cf5c649c9d94b841ee530352f5b76a999fecfcd62d1073"
Sep 08 15:46:17.064987 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:46:17.064960    2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 08 15:46:17.065046 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:46:17.065028    2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 08 15:46:17.065077 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:46:17.065059    2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 08 15:46:18.081382 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:46:18.081338    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a057c5-openshift-default-5f564fb8cd-2mflt" event={"ID":"55c30a1d-0943-4733-9a53-ed49d3d4184d","Type":"ContainerStarted","Data":"2783d7dbbc01842411bd24980061e88bd7f0392b36ddd78aa81fb76724d495c3"}
Sep 08 15:46:18.549586 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:46:18.549550    2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-a057c5-openshift-default-5f564fb8cd-2mflt"
Sep 08 15:46:18.549895 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:46:18.549606    2549 patch_prober.go:28] interesting pod/e2e-inf-b-a057c5-openshift-default-5f564fb8cd-2mflt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body=
Sep 08 15:46:18.550067 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:46:18.550044    2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a057c5-openshift-default-5f564fb8cd-2mflt" podUID="55c30a1d-0943-4733-9a53-ed49d3d4184d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused"
Sep 08 15:46:19.549816 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:46:19.549776    2549 patch_prober.go:28] interesting pod/e2e-inf-b-a057c5-openshift-default-5f564fb8cd-2mflt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body=
Sep 08 15:46:19.550282 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:46:19.549873    2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a057c5-openshift-default-5f564fb8cd-2mflt" podUID="55c30a1d-0943-4733-9a53-ed49d3d4184d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused"
Sep 08 15:46:20.550140 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:46:20.549680    2549 patch_prober.go:28] interesting pod/e2e-inf-b-a057c5-openshift-default-5f564fb8cd-2mflt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body=
Sep 08 15:46:20.550140 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:46:20.549741    2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a057c5-openshift-default-5f564fb8cd-2mflt" podUID="55c30a1d-0943-4733-9a53-ed49d3d4184d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused"
Sep 08 15:46:21.550354 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:46:21.550307    2549 patch_prober.go:28] interesting pod/e2e-inf-b-a057c5-openshift-default-5f564fb8cd-2mflt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body=
Sep 08 15:46:21.550890 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:46:21.550393    2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a057c5-openshift-default-5f564fb8cd-2mflt" podUID="55c30a1d-0943-4733-9a53-ed49d3d4184d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused"
Sep 08 15:46:22.561284 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:46:22.561242    2549 patch_prober.go:28] interesting pod/e2e-inf-b-a057c5-openshift-default-5f564fb8cd-2mflt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body=
Sep 08 15:46:22.561877 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:46:22.561829    2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a057c5-openshift-default-5f564fb8cd-2mflt" podUID="55c30a1d-0943-4733-9a53-ed49d3d4184d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused"
Sep 08 15:46:23.553557 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:46:23.553525    2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-a057c5-openshift-default-5f564fb8cd-2mflt"
Sep 08 15:46:23.553940 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:46:23.553917    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-a057c5-openshift-default-5f564fb8cd-2mflt"
Sep 08 15:46:23.554614 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:46:23.554591    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-a057c5-openshift-default-5f564fb8cd-2mflt"
Sep 08 15:46:23.922362 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:46:23.922332    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-conflict-a-a97eab05-kserve-5c4fd789f-mdtbg"
Sep 08 15:46:23.922747 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:46:23.922380    2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-conflict-a-a97eab05-kserve-5c4fd789f-mdtbg"
Sep 08 15:46:23.935362 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:46:23.935335    2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-conflict-a-a97eab05-kserve-5c4fd789f-mdtbg"
Sep 08 15:46:24.123274 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:46:24.123246    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-conflict-a-a97eab05-kserve-5c4fd789f-mdtbg"
Sep 08 15:46:49.013734 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:46:49.013693    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ddd2c784-b78c8dc7f-r2pcx"]
Sep 08 15:46:49.018025 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:46:49.017999    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ddd2c784-b78c8dc7f-r2pcx"
Sep 08 15:46:49.020974 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:46:49.020951    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-ddd2c784-serving-cert\""
Sep 08 15:46:49.026620 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:46:49.026594    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ddd2c784-b78c8dc7f-r2pcx"]
Sep 08 15:46:49.051842 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:46:49.051812    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c79d20a3-b64b-4249-991e-8a36df9a6d3f-openshift-service-ca\") pod \"maas-api-e2e-ait-ddd2c784-b78c8dc7f-r2pcx\" (UID: \"c79d20a3-b64b-4249-991e-8a36df9a6d3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ddd2c784-b78c8dc7f-r2pcx"
Sep 08 15:46:49.052001 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:46:49.051981    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c79d20a3-b64b-4249-991e-8a36df9a6d3f-maas-api-tls\") pod \"maas-api-e2e-ait-ddd2c784-b78c8dc7f-r2pcx\" (UID: \"c79d20a3-b64b-4249-991e-8a36df9a6d3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ddd2c784-b78c8dc7f-r2pcx"
Sep 08 15:46:49.052057 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:46:49.052011    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c79d20a3-b64b-4249-991e-8a36df9a6d3f-metrics-certs\") pod \"maas-api-e2e-ait-ddd2c784-b78c8dc7f-r2pcx\" (UID: \"c79d20a3-b64b-4249-991e-8a36df9a6d3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ddd2c784-b78c8dc7f-r2pcx"
Sep 08 15:46:49.052104 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:46:49.052090    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqmh6\" (UniqueName: \"kubernetes.io/projected/c79d20a3-b64b-4249-991e-8a36df9a6d3f-kube-api-access-tqmh6\") pod \"maas-api-e2e-ait-ddd2c784-b78c8dc7f-r2pcx\" (UID: \"c79d20a3-b64b-4249-991e-8a36df9a6d3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ddd2c784-b78c8dc7f-r2pcx"
Sep 08 15:46:49.152673 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:46:49.152637    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqmh6\" (UniqueName: \"kubernetes.io/projected/c79d20a3-b64b-4249-991e-8a36df9a6d3f-kube-api-access-tqmh6\") pod \"maas-api-e2e-ait-ddd2c784-b78c8dc7f-r2pcx\" (UID: \"c79d20a3-b64b-4249-991e-8a36df9a6d3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ddd2c784-b78c8dc7f-r2pcx"
Sep 08 15:46:49.152824 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:46:49.152772    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c79d20a3-b64b-4249-991e-8a36df9a6d3f-openshift-service-ca\") pod \"maas-api-e2e-ait-ddd2c784-b78c8dc7f-r2pcx\" (UID: \"c79d20a3-b64b-4249-991e-8a36df9a6d3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ddd2c784-b78c8dc7f-r2pcx"
Sep 08 15:46:49.152923 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:46:49.152909    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c79d20a3-b64b-4249-991e-8a36df9a6d3f-maas-api-tls\") pod \"maas-api-e2e-ait-ddd2c784-b78c8dc7f-r2pcx\" (UID: \"c79d20a3-b64b-4249-991e-8a36df9a6d3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ddd2c784-b78c8dc7f-r2pcx"
Sep 08 15:46:49.152968 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:46:49.152935    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c79d20a3-b64b-4249-991e-8a36df9a6d3f-metrics-certs\") pod \"maas-api-e2e-ait-ddd2c784-b78c8dc7f-r2pcx\" (UID: \"c79d20a3-b64b-4249-991e-8a36df9a6d3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ddd2c784-b78c8dc7f-r2pcx"
Sep 08 15:46:49.153070 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:46:49.153051    2549 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-ddd2c784-serving-cert: secret "maas-api-e2e-ait-ddd2c784-serving-cert" not found
Sep 08 15:46:49.153137 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:46:49.153128    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c79d20a3-b64b-4249-991e-8a36df9a6d3f-maas-api-tls podName:c79d20a3-b64b-4249-991e-8a36df9a6d3f nodeName:}" failed. No retries permitted until 2026-09-08 15:46:49.653106129 +0000 UTC m=+847.742557211 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c79d20a3-b64b-4249-991e-8a36df9a6d3f-maas-api-tls") pod "maas-api-e2e-ait-ddd2c784-b78c8dc7f-r2pcx" (UID: "c79d20a3-b64b-4249-991e-8a36df9a6d3f") : secret "maas-api-e2e-ait-ddd2c784-serving-cert" not found
Sep 08 15:46:49.153413 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:46:49.153395    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c79d20a3-b64b-4249-991e-8a36df9a6d3f-openshift-service-ca\") pod \"maas-api-e2e-ait-ddd2c784-b78c8dc7f-r2pcx\" (UID: \"c79d20a3-b64b-4249-991e-8a36df9a6d3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ddd2c784-b78c8dc7f-r2pcx"
Sep 08 15:46:49.155294 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:46:49.155266    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c79d20a3-b64b-4249-991e-8a36df9a6d3f-metrics-certs\") pod \"maas-api-e2e-ait-ddd2c784-b78c8dc7f-r2pcx\" (UID: \"c79d20a3-b64b-4249-991e-8a36df9a6d3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ddd2c784-b78c8dc7f-r2pcx"
Sep 08 15:46:49.163367 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:46:49.163341    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqmh6\" (UniqueName: \"kubernetes.io/projected/c79d20a3-b64b-4249-991e-8a36df9a6d3f-kube-api-access-tqmh6\") pod \"maas-api-e2e-ait-ddd2c784-b78c8dc7f-r2pcx\" (UID: \"c79d20a3-b64b-4249-991e-8a36df9a6d3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ddd2c784-b78c8dc7f-r2pcx"
Sep 08 15:46:49.658239 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:46:49.658205    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c79d20a3-b64b-4249-991e-8a36df9a6d3f-maas-api-tls\") pod \"maas-api-e2e-ait-ddd2c784-b78c8dc7f-r2pcx\" (UID: \"c79d20a3-b64b-4249-991e-8a36df9a6d3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ddd2c784-b78c8dc7f-r2pcx"
Sep 08 15:46:49.660379 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:46:49.660352    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c79d20a3-b64b-4249-991e-8a36df9a6d3f-maas-api-tls\") pod \"maas-api-e2e-ait-ddd2c784-b78c8dc7f-r2pcx\" (UID: \"c79d20a3-b64b-4249-991e-8a36df9a6d3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ddd2c784-b78c8dc7f-r2pcx"
Sep 08 15:46:49.930215 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:46:49.930131    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ddd2c784-b78c8dc7f-r2pcx"
Sep 08 15:46:50.201784 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:46:50.201754    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ddd2c784-b78c8dc7f-r2pcx"]
Sep 08 15:46:50.203170 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:46:50.203131    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc79d20a3_b64b_4249_991e_8a36df9a6d3f.slice/crio-1c256f493142e508966a66bd91994d8e39ed7dfeb6bf884be7bdc537324666d0 WatchSource:0}: Error finding container 1c256f493142e508966a66bd91994d8e39ed7dfeb6bf884be7bdc537324666d0: Status 404 returned error can't find the container with id 1c256f493142e508966a66bd91994d8e39ed7dfeb6bf884be7bdc537324666d0
Sep 08 15:46:50.247183 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:46:50.247147    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ddd2c784-b78c8dc7f-r2pcx" event={"ID":"c79d20a3-b64b-4249-991e-8a36df9a6d3f","Type":"ContainerStarted","Data":"1c256f493142e508966a66bd91994d8e39ed7dfeb6bf884be7bdc537324666d0"}
Sep 08 15:46:51.253551 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:46:51.253512    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ddd2c784-b78c8dc7f-r2pcx" event={"ID":"c79d20a3-b64b-4249-991e-8a36df9a6d3f","Type":"ContainerStarted","Data":"4bbd7f8f0607561724dfa2780e1394a4fa3be8ea8d5739ca2d3af2625c699fc5"}
Sep 08 15:46:51.253945 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:46:51.253666    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ddd2c784-b78c8dc7f-r2pcx"
Sep 08 15:46:51.271055 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:46:51.270997    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ddd2c784-b78c8dc7f-r2pcx" podStartSLOduration=2.780496278 podStartE2EDuration="3.270978124s" totalImagesPullingTime="490.481846ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:46:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:46:51.270697209 +0000 UTC m=+849.360148299" watchObservedRunningTime="2026-09-08 15:46:51.270978124 +0000 UTC m=+849.360429216"
Sep 08 15:46:57.263928 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:46:57.263898    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ddd2c784-b78c8dc7f-r2pcx"
Sep 08 15:47:13.743297 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:47:13.743264    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-36c51d/autores-model-36c51d-kserve-666c949bcf-gdll4"]
Sep 08 15:47:13.746767 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:47:13.746743    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-36c51d/autores-model-36c51d-kserve-666c949bcf-gdll4"
Sep 08 15:47:13.750710 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:47:13.750680    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-36c51d\"/\"autores-model-36c51d-kserve-self-signed-certs\""
Sep 08 15:47:13.750823 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:47:13.750748    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-36c51d\"/\"openshift-service-ca.crt\""
Sep 08 15:47:13.750929 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:47:13.750913    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-36c51d\"/\"kube-root-ca.crt\""
Sep 08 15:47:13.751000 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:47:13.750966    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-36c51d\"/\"default-dockercfg-tlmww\""
Sep 08 15:47:13.757069 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:47:13.757042    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-36c51d/autores-model-36c51d-kserve-666c949bcf-gdll4"]
Sep 08 15:47:13.854181 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:47:13.854152    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39ddfd23-702d-47e2-ad5a-c46dc797606c-tmp-dir\") pod \"autores-model-36c51d-kserve-666c949bcf-gdll4\" (UID: \"39ddfd23-702d-47e2-ad5a-c46dc797606c\") " pod="ai-tenant-e2e-autores-36c51d/autores-model-36c51d-kserve-666c949bcf-gdll4"
Sep 08 15:47:13.854338 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:47:13.854235    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5ggb\" (UniqueName: \"kubernetes.io/projected/39ddfd23-702d-47e2-ad5a-c46dc797606c-kube-api-access-p5ggb\") pod \"autores-model-36c51d-kserve-666c949bcf-gdll4\" (UID: \"39ddfd23-702d-47e2-ad5a-c46dc797606c\") " pod="ai-tenant-e2e-autores-36c51d/autores-model-36c51d-kserve-666c949bcf-gdll4"
Sep 08 15:47:13.854338 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:47:13.854277    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39ddfd23-702d-47e2-ad5a-c46dc797606c-home\") pod \"autores-model-36c51d-kserve-666c949bcf-gdll4\" (UID: \"39ddfd23-702d-47e2-ad5a-c46dc797606c\") " pod="ai-tenant-e2e-autores-36c51d/autores-model-36c51d-kserve-666c949bcf-gdll4"
Sep 08 15:47:13.854434 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:47:13.854340    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39ddfd23-702d-47e2-ad5a-c46dc797606c-dshm\") pod \"autores-model-36c51d-kserve-666c949bcf-gdll4\" (UID: \"39ddfd23-702d-47e2-ad5a-c46dc797606c\") " pod="ai-tenant-e2e-autores-36c51d/autores-model-36c51d-kserve-666c949bcf-gdll4"
Sep 08 15:47:13.854434 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:47:13.854411    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39ddfd23-702d-47e2-ad5a-c46dc797606c-tls-certs\") pod \"autores-model-36c51d-kserve-666c949bcf-gdll4\" (UID: \"39ddfd23-702d-47e2-ad5a-c46dc797606c\") " pod="ai-tenant-e2e-autores-36c51d/autores-model-36c51d-kserve-666c949bcf-gdll4"
Sep 08 15:47:13.854516 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:47:13.854469    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39ddfd23-702d-47e2-ad5a-c46dc797606c-model-cache\") pod \"autores-model-36c51d-kserve-666c949bcf-gdll4\" (UID: \"39ddfd23-702d-47e2-ad5a-c46dc797606c\") " pod="ai-tenant-e2e-autores-36c51d/autores-model-36c51d-kserve-666c949bcf-gdll4"
Sep 08 15:47:13.955316 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:47:13.955282    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39ddfd23-702d-47e2-ad5a-c46dc797606c-model-cache\") pod \"autores-model-36c51d-kserve-666c949bcf-gdll4\" (UID: \"39ddfd23-702d-47e2-ad5a-c46dc797606c\") " pod="ai-tenant-e2e-autores-36c51d/autores-model-36c51d-kserve-666c949bcf-gdll4"
Sep 08 15:47:13.955462 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:47:13.955328    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39ddfd23-702d-47e2-ad5a-c46dc797606c-tmp-dir\") pod \"autores-model-36c51d-kserve-666c949bcf-gdll4\" (UID: \"39ddfd23-702d-47e2-ad5a-c46dc797606c\") " pod="ai-tenant-e2e-autores-36c51d/autores-model-36c51d-kserve-666c949bcf-gdll4"
Sep 08 15:47:13.955462 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:47:13.955379    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5ggb\" (UniqueName: \"kubernetes.io/projected/39ddfd23-702d-47e2-ad5a-c46dc797606c-kube-api-access-p5ggb\") pod \"autores-model-36c51d-kserve-666c949bcf-gdll4\" (UID: \"39ddfd23-702d-47e2-ad5a-c46dc797606c\") " pod="ai-tenant-e2e-autores-36c51d/autores-model-36c51d-kserve-666c949bcf-gdll4"
Sep 08 15:47:13.955462 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:47:13.955421    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39ddfd23-702d-47e2-ad5a-c46dc797606c-home\") pod \"autores-model-36c51d-kserve-666c949bcf-gdll4\" (UID: \"39ddfd23-702d-47e2-ad5a-c46dc797606c\") " pod="ai-tenant-e2e-autores-36c51d/autores-model-36c51d-kserve-666c949bcf-gdll4"
Sep 08 15:47:13.955462 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:47:13.955449    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39ddfd23-702d-47e2-ad5a-c46dc797606c-dshm\") pod \"autores-model-36c51d-kserve-666c949bcf-gdll4\" (UID: \"39ddfd23-702d-47e2-ad5a-c46dc797606c\") " pod="ai-tenant-e2e-autores-36c51d/autores-model-36c51d-kserve-666c949bcf-gdll4"
Sep 08 15:47:13.955663 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:47:13.955464    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39ddfd23-702d-47e2-ad5a-c46dc797606c-tls-certs\") pod \"autores-model-36c51d-kserve-666c949bcf-gdll4\" (UID: \"39ddfd23-702d-47e2-ad5a-c46dc797606c\") " pod="ai-tenant-e2e-autores-36c51d/autores-model-36c51d-kserve-666c949bcf-gdll4"
Sep 08 15:47:13.955774 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:47:13.955747    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39ddfd23-702d-47e2-ad5a-c46dc797606c-model-cache\") pod \"autores-model-36c51d-kserve-666c949bcf-gdll4\" (UID: \"39ddfd23-702d-47e2-ad5a-c46dc797606c\") " pod="ai-tenant-e2e-autores-36c51d/autores-model-36c51d-kserve-666c949bcf-gdll4"
Sep 08 15:47:13.955774 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:47:13.955766    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39ddfd23-702d-47e2-ad5a-c46dc797606c-home\") pod \"autores-model-36c51d-kserve-666c949bcf-gdll4\" (UID: \"39ddfd23-702d-47e2-ad5a-c46dc797606c\") " pod="ai-tenant-e2e-autores-36c51d/autores-model-36c51d-kserve-666c949bcf-gdll4"
Sep 08 15:47:13.955922 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:47:13.955809    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39ddfd23-702d-47e2-ad5a-c46dc797606c-tmp-dir\") pod \"autores-model-36c51d-kserve-666c949bcf-gdll4\" (UID: \"39ddfd23-702d-47e2-ad5a-c46dc797606c\") " pod="ai-tenant-e2e-autores-36c51d/autores-model-36c51d-kserve-666c949bcf-gdll4"
Sep 08 15:47:13.957669 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:47:13.957645    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39ddfd23-702d-47e2-ad5a-c46dc797606c-dshm\") pod \"autores-model-36c51d-kserve-666c949bcf-gdll4\" (UID: \"39ddfd23-702d-47e2-ad5a-c46dc797606c\") " pod="ai-tenant-e2e-autores-36c51d/autores-model-36c51d-kserve-666c949bcf-gdll4"
Sep 08 15:47:13.957786 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:47:13.957769    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39ddfd23-702d-47e2-ad5a-c46dc797606c-tls-certs\") pod \"autores-model-36c51d-kserve-666c949bcf-gdll4\" (UID: \"39ddfd23-702d-47e2-ad5a-c46dc797606c\") " pod="ai-tenant-e2e-autores-36c51d/autores-model-36c51d-kserve-666c949bcf-gdll4"
Sep 08 15:47:13.963983 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:47:13.963959    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5ggb\" (UniqueName: \"kubernetes.io/projected/39ddfd23-702d-47e2-ad5a-c46dc797606c-kube-api-access-p5ggb\") pod \"autores-model-36c51d-kserve-666c949bcf-gdll4\" (UID: \"39ddfd23-702d-47e2-ad5a-c46dc797606c\") " pod="ai-tenant-e2e-autores-36c51d/autores-model-36c51d-kserve-666c949bcf-gdll4"
Sep 08 15:47:14.056532 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:47:14.056470    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-36c51d/autores-model-36c51d-kserve-666c949bcf-gdll4"
Sep 08 15:47:14.325080 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:47:14.325056    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-36c51d/autores-model-36c51d-kserve-666c949bcf-gdll4"]
Sep 08 15:47:14.327569 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:47:14.327541    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod39ddfd23_702d_47e2_ad5a_c46dc797606c.slice/crio-cff74a8d4cbf7757d73446ec382c37beb4404ed9222001ac155cf8cd403f2082 WatchSource:0}: Error finding container cff74a8d4cbf7757d73446ec382c37beb4404ed9222001ac155cf8cd403f2082: Status 404 returned error can't find the container with id cff74a8d4cbf7757d73446ec382c37beb4404ed9222001ac155cf8cd403f2082
Sep 08 15:47:14.363372 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:47:14.363338    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-36c51d/autores-model-36c51d-kserve-666c949bcf-gdll4" event={"ID":"39ddfd23-702d-47e2-ad5a-c46dc797606c","Type":"ContainerStarted","Data":"cff74a8d4cbf7757d73446ec382c37beb4404ed9222001ac155cf8cd403f2082"}
Sep 08 15:47:15.369576 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:47:15.369543    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-36c51d/autores-model-36c51d-kserve-666c949bcf-gdll4" event={"ID":"39ddfd23-702d-47e2-ad5a-c46dc797606c","Type":"ContainerStarted","Data":"ca00714dd3bf823183666b441609c5219df5e0cdf7cde8af5d3be5a9a83d09f6"}
Sep 08 15:47:15.392974 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:47:15.387673    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-autores-36c51d/autores-model-36c51d-kserve-666c949bcf-gdll4" podStartSLOduration=2.387654058 podStartE2EDuration="2.387654058s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:47:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:47:15.385274503 +0000 UTC m=+873.474725591" watchObservedRunningTime="2026-09-08 15:47:15.387654058 +0000 UTC m=+873.477105150"
Sep 08 15:47:19.556457 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:47:19.556419    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ddd2c784-b78c8dc7f-r2pcx"]
Sep 08 15:47:19.556940 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:47:19.556637    2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ddd2c784-b78c8dc7f-r2pcx" podUID="c79d20a3-b64b-4249-991e-8a36df9a6d3f" containerName="maas-api" containerID="cri-o://4bbd7f8f0607561724dfa2780e1394a4fa3be8ea8d5739ca2d3af2625c699fc5" gracePeriod=30
Sep 08 15:47:19.800813 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:47:19.800787    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ddd2c784-b78c8dc7f-r2pcx"
Sep 08 15:47:19.910193 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:47:19.910169    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tqmh6\" (UniqueName: \"kubernetes.io/projected/c79d20a3-b64b-4249-991e-8a36df9a6d3f-kube-api-access-tqmh6\") pod \"c79d20a3-b64b-4249-991e-8a36df9a6d3f\" (UID: \"c79d20a3-b64b-4249-991e-8a36df9a6d3f\") "
Sep 08 15:47:19.910362 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:47:19.910266    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c79d20a3-b64b-4249-991e-8a36df9a6d3f-maas-api-tls\") pod \"c79d20a3-b64b-4249-991e-8a36df9a6d3f\" (UID: \"c79d20a3-b64b-4249-991e-8a36df9a6d3f\") "
Sep 08 15:47:19.910362 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:47:19.910293    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c79d20a3-b64b-4249-991e-8a36df9a6d3f-openshift-service-ca\") pod \"c79d20a3-b64b-4249-991e-8a36df9a6d3f\" (UID: \"c79d20a3-b64b-4249-991e-8a36df9a6d3f\") "
Sep 08 15:47:19.910362 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:47:19.910325    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c79d20a3-b64b-4249-991e-8a36df9a6d3f-metrics-certs\") pod \"c79d20a3-b64b-4249-991e-8a36df9a6d3f\" (UID: \"c79d20a3-b64b-4249-991e-8a36df9a6d3f\") "
Sep 08 15:47:19.910665 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:47:19.910633    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c79d20a3-b64b-4249-991e-8a36df9a6d3f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c79d20a3-b64b-4249-991e-8a36df9a6d3f" (UID: "c79d20a3-b64b-4249-991e-8a36df9a6d3f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 15:47:19.912495 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:47:19.912470    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c79d20a3-b64b-4249-991e-8a36df9a6d3f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c79d20a3-b64b-4249-991e-8a36df9a6d3f" (UID: "c79d20a3-b64b-4249-991e-8a36df9a6d3f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 15:47:19.912597 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:47:19.912536    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c79d20a3-b64b-4249-991e-8a36df9a6d3f-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "c79d20a3-b64b-4249-991e-8a36df9a6d3f" (UID: "c79d20a3-b64b-4249-991e-8a36df9a6d3f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 15:47:19.912597 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:47:19.912563    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c79d20a3-b64b-4249-991e-8a36df9a6d3f-kube-api-access-tqmh6" (OuterVolumeSpecName: "kube-api-access-tqmh6") pod "c79d20a3-b64b-4249-991e-8a36df9a6d3f" (UID: "c79d20a3-b64b-4249-991e-8a36df9a6d3f"). InnerVolumeSpecName "kube-api-access-tqmh6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:47:20.011774 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:47:20.011744    2549 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c79d20a3-b64b-4249-991e-8a36df9a6d3f-maas-api-tls\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:47:20.011774 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:47:20.011768    2549 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c79d20a3-b64b-4249-991e-8a36df9a6d3f-openshift-service-ca\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:47:20.011774 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:47:20.011777    2549 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c79d20a3-b64b-4249-991e-8a36df9a6d3f-metrics-certs\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:47:20.012017 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:47:20.011786    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tqmh6\" (UniqueName: \"kubernetes.io/projected/c79d20a3-b64b-4249-991e-8a36df9a6d3f-kube-api-access-tqmh6\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:47:20.396650 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:47:20.396600    2549 generic.go:356] "Generic (PLEG): container finished" podID="c79d20a3-b64b-4249-991e-8a36df9a6d3f" containerID="4bbd7f8f0607561724dfa2780e1394a4fa3be8ea8d5739ca2d3af2625c699fc5" exitCode=0
Sep 08 15:47:20.396836 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:47:20.396709    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ddd2c784-b78c8dc7f-r2pcx"
Sep 08 15:47:20.403651 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:47:20.403621    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ddd2c784-b78c8dc7f-r2pcx" event={"ID":"c79d20a3-b64b-4249-991e-8a36df9a6d3f","Type":"ContainerDied","Data":"4bbd7f8f0607561724dfa2780e1394a4fa3be8ea8d5739ca2d3af2625c699fc5"}
Sep 08 15:47:20.403799 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:47:20.403664    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ddd2c784-b78c8dc7f-r2pcx" event={"ID":"c79d20a3-b64b-4249-991e-8a36df9a6d3f","Type":"ContainerDied","Data":"1c256f493142e508966a66bd91994d8e39ed7dfeb6bf884be7bdc537324666d0"}
Sep 08 15:47:20.403799 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:47:20.403687    2549 scope.go:117] "RemoveContainer" containerID="4bbd7f8f0607561724dfa2780e1394a4fa3be8ea8d5739ca2d3af2625c699fc5"
Sep 08 15:47:20.414953 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:47:20.414930    2549 scope.go:117] "RemoveContainer" containerID="4bbd7f8f0607561724dfa2780e1394a4fa3be8ea8d5739ca2d3af2625c699fc5"
Sep 08 15:47:20.415321 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:47:20.415283    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4bbd7f8f0607561724dfa2780e1394a4fa3be8ea8d5739ca2d3af2625c699fc5\": container with ID starting with 4bbd7f8f0607561724dfa2780e1394a4fa3be8ea8d5739ca2d3af2625c699fc5 not found: ID does not exist" containerID="4bbd7f8f0607561724dfa2780e1394a4fa3be8ea8d5739ca2d3af2625c699fc5"
Sep 08 15:47:20.415496 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:47:20.415327    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4bbd7f8f0607561724dfa2780e1394a4fa3be8ea8d5739ca2d3af2625c699fc5"} err="failed to get container status \"4bbd7f8f0607561724dfa2780e1394a4fa3be8ea8d5739ca2d3af2625c699fc5\": rpc error: code = NotFound desc = could not find container \"4bbd7f8f0607561724dfa2780e1394a4fa3be8ea8d5739ca2d3af2625c699fc5\": container with ID starting with 4bbd7f8f0607561724dfa2780e1394a4fa3be8ea8d5739ca2d3af2625c699fc5 not found: ID does not exist"
Sep 08 15:47:20.423874 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:47:20.423833    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ddd2c784-b78c8dc7f-r2pcx"]
Sep 08 15:47:20.428598 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:47:20.428577    2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ddd2c784-b78c8dc7f-r2pcx"]
Sep 08 15:47:22.400952 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:47:22.400914    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c79d20a3-b64b-4249-991e-8a36df9a6d3f" path="/var/lib/kubelet/pods/c79d20a3-b64b-4249-991e-8a36df9a6d3f/volumes"
Sep 08 15:47:24.057396 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:47:24.057350    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-36c51d/autores-model-36c51d-kserve-666c949bcf-gdll4"
Sep 08 15:47:24.057885 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:47:24.057425    2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-36c51d/autores-model-36c51d-kserve-666c949bcf-gdll4"
Sep 08 15:47:24.070463 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:47:24.070439    2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-36c51d/autores-model-36c51d-kserve-666c949bcf-gdll4"
Sep 08 15:47:24.427048 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:47:24.427017    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-36c51d/autores-model-36c51d-kserve-666c949bcf-gdll4"
Sep 08 15:47:25.153358 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:47:25.153321    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-a82afc10-5b77cdc7cb-5rmrk"]
Sep 08 15:47:25.153768 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:47:25.153671    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c79d20a3-b64b-4249-991e-8a36df9a6d3f" containerName="maas-api"
Sep 08 15:47:25.153768 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:47:25.153684    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="c79d20a3-b64b-4249-991e-8a36df9a6d3f" containerName="maas-api"
Sep 08 15:47:25.153768 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:47:25.153753    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="c79d20a3-b64b-4249-991e-8a36df9a6d3f" containerName="maas-api"
Sep 08 15:47:25.158152 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:47:25.158132    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a82afc10-5b77cdc7cb-5rmrk"
Sep 08 15:47:25.160863 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:47:25.160840    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-a82afc10-serving-cert\""
Sep 08 15:47:25.166982 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:47:25.166958    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-a82afc10-5b77cdc7cb-5rmrk"]
Sep 08 15:47:25.256593 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:47:25.256564    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krtq7\" (UniqueName: \"kubernetes.io/projected/00dfe99f-6c36-4d0b-ab40-466d0a31480f-kube-api-access-krtq7\") pod \"maas-api-e2e-derive-a82afc10-5b77cdc7cb-5rmrk\" (UID: \"00dfe99f-6c36-4d0b-ab40-466d0a31480f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a82afc10-5b77cdc7cb-5rmrk"
Sep 08 15:47:25.256739 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:47:25.256599    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00dfe99f-6c36-4d0b-ab40-466d0a31480f-openshift-service-ca\") pod \"maas-api-e2e-derive-a82afc10-5b77cdc7cb-5rmrk\" (UID: \"00dfe99f-6c36-4d0b-ab40-466d0a31480f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a82afc10-5b77cdc7cb-5rmrk"
Sep 08 15:47:25.256739 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:47:25.256677    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00dfe99f-6c36-4d0b-ab40-466d0a31480f-metrics-certs\") pod \"maas-api-e2e-derive-a82afc10-5b77cdc7cb-5rmrk\" (UID: \"00dfe99f-6c36-4d0b-ab40-466d0a31480f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a82afc10-5b77cdc7cb-5rmrk"
Sep 08 15:47:25.256739 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:47:25.256727    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00dfe99f-6c36-4d0b-ab40-466d0a31480f-maas-api-tls\") pod \"maas-api-e2e-derive-a82afc10-5b77cdc7cb-5rmrk\" (UID: \"00dfe99f-6c36-4d0b-ab40-466d0a31480f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a82afc10-5b77cdc7cb-5rmrk"
Sep 08 15:47:25.357674 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:47:25.357639    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krtq7\" (UniqueName: \"kubernetes.io/projected/00dfe99f-6c36-4d0b-ab40-466d0a31480f-kube-api-access-krtq7\") pod \"maas-api-e2e-derive-a82afc10-5b77cdc7cb-5rmrk\" (UID: \"00dfe99f-6c36-4d0b-ab40-466d0a31480f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a82afc10-5b77cdc7cb-5rmrk"
Sep 08 15:47:25.357674 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:47:25.357676    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00dfe99f-6c36-4d0b-ab40-466d0a31480f-openshift-service-ca\") pod \"maas-api-e2e-derive-a82afc10-5b77cdc7cb-5rmrk\" (UID: \"00dfe99f-6c36-4d0b-ab40-466d0a31480f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a82afc10-5b77cdc7cb-5rmrk"
Sep 08 15:47:25.357901 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:47:25.357716    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00dfe99f-6c36-4d0b-ab40-466d0a31480f-metrics-certs\") pod \"maas-api-e2e-derive-a82afc10-5b77cdc7cb-5rmrk\" (UID: \"00dfe99f-6c36-4d0b-ab40-466d0a31480f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a82afc10-5b77cdc7cb-5rmrk"
Sep 08 15:47:25.357901 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:47:25.357749    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00dfe99f-6c36-4d0b-ab40-466d0a31480f-maas-api-tls\") pod \"maas-api-e2e-derive-a82afc10-5b77cdc7cb-5rmrk\" (UID: \"00dfe99f-6c36-4d0b-ab40-466d0a31480f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a82afc10-5b77cdc7cb-5rmrk"
Sep 08 15:47:25.357901 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:47:25.357884    2549 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-a82afc10-serving-cert: secret "maas-api-e2e-derive-a82afc10-serving-cert" not found
Sep 08 15:47:25.358004 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:47:25.357950    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00dfe99f-6c36-4d0b-ab40-466d0a31480f-maas-api-tls podName:00dfe99f-6c36-4d0b-ab40-466d0a31480f nodeName:}" failed. No retries permitted until 2026-09-08 15:47:25.857929391 +0000 UTC m=+883.947380458 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/00dfe99f-6c36-4d0b-ab40-466d0a31480f-maas-api-tls") pod "maas-api-e2e-derive-a82afc10-5b77cdc7cb-5rmrk" (UID: "00dfe99f-6c36-4d0b-ab40-466d0a31480f") : secret "maas-api-e2e-derive-a82afc10-serving-cert" not found
Sep 08 15:47:25.358407 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:47:25.358384    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00dfe99f-6c36-4d0b-ab40-466d0a31480f-openshift-service-ca\") pod \"maas-api-e2e-derive-a82afc10-5b77cdc7cb-5rmrk\" (UID: \"00dfe99f-6c36-4d0b-ab40-466d0a31480f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a82afc10-5b77cdc7cb-5rmrk"
Sep 08 15:47:25.360086 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:47:25.360061    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00dfe99f-6c36-4d0b-ab40-466d0a31480f-metrics-certs\") pod \"maas-api-e2e-derive-a82afc10-5b77cdc7cb-5rmrk\" (UID: \"00dfe99f-6c36-4d0b-ab40-466d0a31480f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a82afc10-5b77cdc7cb-5rmrk"
Sep 08 15:47:25.366211 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:47:25.366191    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-krtq7\" (UniqueName: \"kubernetes.io/projected/00dfe99f-6c36-4d0b-ab40-466d0a31480f-kube-api-access-krtq7\") pod \"maas-api-e2e-derive-a82afc10-5b77cdc7cb-5rmrk\" (UID: \"00dfe99f-6c36-4d0b-ab40-466d0a31480f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a82afc10-5b77cdc7cb-5rmrk"
Sep 08 15:47:25.862132 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:47:25.862091    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00dfe99f-6c36-4d0b-ab40-466d0a31480f-maas-api-tls\") pod \"maas-api-e2e-derive-a82afc10-5b77cdc7cb-5rmrk\" (UID: \"00dfe99f-6c36-4d0b-ab40-466d0a31480f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a82afc10-5b77cdc7cb-5rmrk"
Sep 08 15:47:25.864641 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:47:25.864617    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00dfe99f-6c36-4d0b-ab40-466d0a31480f-maas-api-tls\") pod \"maas-api-e2e-derive-a82afc10-5b77cdc7cb-5rmrk\" (UID: \"00dfe99f-6c36-4d0b-ab40-466d0a31480f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a82afc10-5b77cdc7cb-5rmrk"
Sep 08 15:47:26.070323 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:47:26.070294    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a82afc10-5b77cdc7cb-5rmrk"
Sep 08 15:47:26.342658 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:47:26.342629    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-a82afc10-5b77cdc7cb-5rmrk"]
Sep 08 15:47:26.344472 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:47:26.344444    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00dfe99f_6c36_4d0b_ab40_466d0a31480f.slice/crio-ce687a174f89cc3fbca29f91c5e54ad80f9bd1ddf3ede9588207d524d46f7511 WatchSource:0}: Error finding container ce687a174f89cc3fbca29f91c5e54ad80f9bd1ddf3ede9588207d524d46f7511: Status 404 returned error can't find the container with id ce687a174f89cc3fbca29f91c5e54ad80f9bd1ddf3ede9588207d524d46f7511
Sep 08 15:47:26.432215 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:47:26.432181    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a82afc10-5b77cdc7cb-5rmrk" event={"ID":"00dfe99f-6c36-4d0b-ab40-466d0a31480f","Type":"ContainerStarted","Data":"ce687a174f89cc3fbca29f91c5e54ad80f9bd1ddf3ede9588207d524d46f7511"}
Sep 08 15:47:27.438265 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:47:27.438181    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a82afc10-5b77cdc7cb-5rmrk" event={"ID":"00dfe99f-6c36-4d0b-ab40-466d0a31480f","Type":"ContainerStarted","Data":"73afc09a3f1b7fe8f122430cfcc5fc1ed99a27ea289af93c0bef1fecc90937ce"}
Sep 08 15:47:27.438643 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:47:27.438283    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a82afc10-5b77cdc7cb-5rmrk"
Sep 08 15:47:27.456193 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:47:27.456150    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a82afc10-5b77cdc7cb-5rmrk" podStartSLOduration=1.6138685050000001 podStartE2EDuration="2.456135739s" totalImagesPullingTime="842.267234ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:47:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:47:27.454534403 +0000 UTC m=+885.543985493" watchObservedRunningTime="2026-09-08 15:47:27.456135739 +0000 UTC m=+885.545586828"
Sep 08 15:47:33.447814 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:47:33.447782    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a82afc10-5b77cdc7cb-5rmrk"
Sep 08 15:47:42.357054 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:47:42.356984    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-wpwdq_5f7b0f68-a8e3-4058-b032-15f419682b2a/console-operator/2.log"
Sep 08 15:47:42.358868 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:47:42.358825    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-wpwdq_5f7b0f68-a8e3-4058-b032-15f419682b2a/console-operator/2.log"
Sep 08 15:47:42.363777 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:47:42.363756    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-f4vvt_195f1a74-19bc-49c9-aada-fa1c13488480/ovn-acl-logging/0.log"
Sep 08 15:47:42.365549 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:47:42.365531    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-f4vvt_195f1a74-19bc-49c9-aada-fa1c13488480/ovn-acl-logging/0.log"
Sep 08 15:47:55.871460 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:47:55.871420    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-a82afc10-5b77cdc7cb-5rmrk"]
Sep 08 15:47:55.872057 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:47:55.871667    2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a82afc10-5b77cdc7cb-5rmrk" podUID="00dfe99f-6c36-4d0b-ab40-466d0a31480f" containerName="maas-api" containerID="cri-o://73afc09a3f1b7fe8f122430cfcc5fc1ed99a27ea289af93c0bef1fecc90937ce" gracePeriod=30
Sep 08 15:47:56.119929 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:47:56.119906    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a82afc10-5b77cdc7cb-5rmrk"
Sep 08 15:47:56.228078 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:47:56.227895    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00dfe99f-6c36-4d0b-ab40-466d0a31480f-metrics-certs\") pod \"00dfe99f-6c36-4d0b-ab40-466d0a31480f\" (UID: \"00dfe99f-6c36-4d0b-ab40-466d0a31480f\") "
Sep 08 15:47:56.228078 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:47:56.228018    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00dfe99f-6c36-4d0b-ab40-466d0a31480f-openshift-service-ca\") pod \"00dfe99f-6c36-4d0b-ab40-466d0a31480f\" (UID: \"00dfe99f-6c36-4d0b-ab40-466d0a31480f\") "
Sep 08 15:47:56.228383 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:47:56.228343    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00dfe99f-6c36-4d0b-ab40-466d0a31480f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "00dfe99f-6c36-4d0b-ab40-466d0a31480f" (UID: "00dfe99f-6c36-4d0b-ab40-466d0a31480f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 15:47:56.228461 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:47:56.228417    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00dfe99f-6c36-4d0b-ab40-466d0a31480f-maas-api-tls\") pod \"00dfe99f-6c36-4d0b-ab40-466d0a31480f\" (UID: \"00dfe99f-6c36-4d0b-ab40-466d0a31480f\") "
Sep 08 15:47:56.228782 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:47:56.228760    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-krtq7\" (UniqueName: \"kubernetes.io/projected/00dfe99f-6c36-4d0b-ab40-466d0a31480f-kube-api-access-krtq7\") pod \"00dfe99f-6c36-4d0b-ab40-466d0a31480f\" (UID: \"00dfe99f-6c36-4d0b-ab40-466d0a31480f\") "
Sep 08 15:47:56.229096 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:47:56.229075    2549 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00dfe99f-6c36-4d0b-ab40-466d0a31480f-openshift-service-ca\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:47:56.231028 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:47:56.231000    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00dfe99f-6c36-4d0b-ab40-466d0a31480f-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "00dfe99f-6c36-4d0b-ab40-466d0a31480f" (UID: "00dfe99f-6c36-4d0b-ab40-466d0a31480f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 15:47:56.233083 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:47:56.233060    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00dfe99f-6c36-4d0b-ab40-466d0a31480f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "00dfe99f-6c36-4d0b-ab40-466d0a31480f" (UID: "00dfe99f-6c36-4d0b-ab40-466d0a31480f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 15:47:56.234992 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:47:56.234966    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00dfe99f-6c36-4d0b-ab40-466d0a31480f-kube-api-access-krtq7" (OuterVolumeSpecName: "kube-api-access-krtq7") pod "00dfe99f-6c36-4d0b-ab40-466d0a31480f" (UID: "00dfe99f-6c36-4d0b-ab40-466d0a31480f"). InnerVolumeSpecName "kube-api-access-krtq7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:47:56.330458 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:47:56.330418    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-krtq7\" (UniqueName: \"kubernetes.io/projected/00dfe99f-6c36-4d0b-ab40-466d0a31480f-kube-api-access-krtq7\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:47:56.330458 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:47:56.330450    2549 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00dfe99f-6c36-4d0b-ab40-466d0a31480f-metrics-certs\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:47:56.330458 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:47:56.330463    2549 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00dfe99f-6c36-4d0b-ab40-466d0a31480f-maas-api-tls\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:47:56.573828 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:47:56.573738    2549 generic.go:356] "Generic (PLEG): container finished" podID="00dfe99f-6c36-4d0b-ab40-466d0a31480f" containerID="73afc09a3f1b7fe8f122430cfcc5fc1ed99a27ea289af93c0bef1fecc90937ce" exitCode=0
Sep 08 15:47:56.573828 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:47:56.573791    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a82afc10-5b77cdc7cb-5rmrk"
Sep 08 15:47:56.573828 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:47:56.573801    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a82afc10-5b77cdc7cb-5rmrk" event={"ID":"00dfe99f-6c36-4d0b-ab40-466d0a31480f","Type":"ContainerDied","Data":"73afc09a3f1b7fe8f122430cfcc5fc1ed99a27ea289af93c0bef1fecc90937ce"}
Sep 08 15:47:56.574124 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:47:56.573870    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a82afc10-5b77cdc7cb-5rmrk" event={"ID":"00dfe99f-6c36-4d0b-ab40-466d0a31480f","Type":"ContainerDied","Data":"ce687a174f89cc3fbca29f91c5e54ad80f9bd1ddf3ede9588207d524d46f7511"}
Sep 08 15:47:56.574124 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:47:56.573902    2549 scope.go:117] "RemoveContainer" containerID="73afc09a3f1b7fe8f122430cfcc5fc1ed99a27ea289af93c0bef1fecc90937ce"
Sep 08 15:47:56.582677 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:47:56.582655    2549 scope.go:117] "RemoveContainer" containerID="73afc09a3f1b7fe8f122430cfcc5fc1ed99a27ea289af93c0bef1fecc90937ce"
Sep 08 15:47:56.582975 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:47:56.582953    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73afc09a3f1b7fe8f122430cfcc5fc1ed99a27ea289af93c0bef1fecc90937ce\": container with ID starting with 73afc09a3f1b7fe8f122430cfcc5fc1ed99a27ea289af93c0bef1fecc90937ce not found: ID does not exist" containerID="73afc09a3f1b7fe8f122430cfcc5fc1ed99a27ea289af93c0bef1fecc90937ce"
Sep 08 15:47:56.583058 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:47:56.582985    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73afc09a3f1b7fe8f122430cfcc5fc1ed99a27ea289af93c0bef1fecc90937ce"} err="failed to get container status \"73afc09a3f1b7fe8f122430cfcc5fc1ed99a27ea289af93c0bef1fecc90937ce\": rpc error: code = NotFound desc = could not find container \"73afc09a3f1b7fe8f122430cfcc5fc1ed99a27ea289af93c0bef1fecc90937ce\": container with ID starting with 73afc09a3f1b7fe8f122430cfcc5fc1ed99a27ea289af93c0bef1fecc90937ce not found: ID does not exist"
Sep 08 15:47:56.592640 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:47:56.592618    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-a82afc10-5b77cdc7cb-5rmrk"]
Sep 08 15:47:56.596827 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:47:56.596806    2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-a82afc10-5b77cdc7cb-5rmrk"]
Sep 08 15:47:58.400144 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:47:58.400112    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="00dfe99f-6c36-4d0b-ab40-466d0a31480f" path="/var/lib/kubelet/pods/00dfe99f-6c36-4d0b-ab40-466d0a31480f/volumes"
Sep 08 15:48:00.184875 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:00.184833    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-36c51d-7bb79986f8-6zjxg"]
Sep 08 15:48:00.185262 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:00.185031    2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-36c51d-7bb79986f8-6zjxg" podUID="b14b149c-07c6-4108-a054-a47c6b2c6091" containerName="maas-api" containerID="cri-o://a0405d12a3fb5857d0bad49f5f6b826e22ce6b7d3fafabf59aef135600b4aa0a" gracePeriod=30
Sep 08 15:48:00.437366 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:00.437303    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-36c51d-7bb79986f8-6zjxg"
Sep 08 15:48:00.564588 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:00.564559    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b14b149c-07c6-4108-a054-a47c6b2c6091-openshift-service-ca\") pod \"b14b149c-07c6-4108-a054-a47c6b2c6091\" (UID: \"b14b149c-07c6-4108-a054-a47c6b2c6091\") "
Sep 08 15:48:00.564764 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:00.564625    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b14b149c-07c6-4108-a054-a47c6b2c6091-maas-api-tls\") pod \"b14b149c-07c6-4108-a054-a47c6b2c6091\" (UID: \"b14b149c-07c6-4108-a054-a47c6b2c6091\") "
Sep 08 15:48:00.564764 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:00.564668    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b14b149c-07c6-4108-a054-a47c6b2c6091-metrics-certs\") pod \"b14b149c-07c6-4108-a054-a47c6b2c6091\" (UID: \"b14b149c-07c6-4108-a054-a47c6b2c6091\") "
Sep 08 15:48:00.564764 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:00.564732    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qmz8p\" (UniqueName: \"kubernetes.io/projected/b14b149c-07c6-4108-a054-a47c6b2c6091-kube-api-access-qmz8p\") pod \"b14b149c-07c6-4108-a054-a47c6b2c6091\" (UID: \"b14b149c-07c6-4108-a054-a47c6b2c6091\") "
Sep 08 15:48:00.564953 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:00.564915    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b14b149c-07c6-4108-a054-a47c6b2c6091-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b14b149c-07c6-4108-a054-a47c6b2c6091" (UID: "b14b149c-07c6-4108-a054-a47c6b2c6091"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 15:48:00.565059 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:00.565043    2549 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b14b149c-07c6-4108-a054-a47c6b2c6091-openshift-service-ca\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:48:00.566694 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:00.566660    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b14b149c-07c6-4108-a054-a47c6b2c6091-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b14b149c-07c6-4108-a054-a47c6b2c6091" (UID: "b14b149c-07c6-4108-a054-a47c6b2c6091"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 15:48:00.566821 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:00.566763    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b14b149c-07c6-4108-a054-a47c6b2c6091-kube-api-access-qmz8p" (OuterVolumeSpecName: "kube-api-access-qmz8p") pod "b14b149c-07c6-4108-a054-a47c6b2c6091" (UID: "b14b149c-07c6-4108-a054-a47c6b2c6091"). InnerVolumeSpecName "kube-api-access-qmz8p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:48:00.566821 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:00.566769    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b14b149c-07c6-4108-a054-a47c6b2c6091-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "b14b149c-07c6-4108-a054-a47c6b2c6091" (UID: "b14b149c-07c6-4108-a054-a47c6b2c6091"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 15:48:00.595035 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:00.595002    2549 generic.go:356] "Generic (PLEG): container finished" podID="b14b149c-07c6-4108-a054-a47c6b2c6091" containerID="a0405d12a3fb5857d0bad49f5f6b826e22ce6b7d3fafabf59aef135600b4aa0a" exitCode=0
Sep 08 15:48:00.595161 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:00.595053    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-36c51d-7bb79986f8-6zjxg"
Sep 08 15:48:00.595161 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:00.595086    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-36c51d-7bb79986f8-6zjxg" event={"ID":"b14b149c-07c6-4108-a054-a47c6b2c6091","Type":"ContainerDied","Data":"a0405d12a3fb5857d0bad49f5f6b826e22ce6b7d3fafabf59aef135600b4aa0a"}
Sep 08 15:48:00.595161 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:00.595147    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-36c51d-7bb79986f8-6zjxg" event={"ID":"b14b149c-07c6-4108-a054-a47c6b2c6091","Type":"ContainerDied","Data":"defd94ee532bd7c84f0fdd956e23aae7e6049627e74189c4457ecae41b6a1129"}
Sep 08 15:48:00.595273 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:00.595175    2549 scope.go:117] "RemoveContainer" containerID="a0405d12a3fb5857d0bad49f5f6b826e22ce6b7d3fafabf59aef135600b4aa0a"
Sep 08 15:48:00.604461 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:00.604445    2549 scope.go:117] "RemoveContainer" containerID="a0405d12a3fb5857d0bad49f5f6b826e22ce6b7d3fafabf59aef135600b4aa0a"
Sep 08 15:48:00.604737 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:48:00.604703    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0405d12a3fb5857d0bad49f5f6b826e22ce6b7d3fafabf59aef135600b4aa0a\": container with ID starting with a0405d12a3fb5857d0bad49f5f6b826e22ce6b7d3fafabf59aef135600b4aa0a not found: ID does not exist" containerID="a0405d12a3fb5857d0bad49f5f6b826e22ce6b7d3fafabf59aef135600b4aa0a"
Sep 08 15:48:00.604798 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:00.604749    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0405d12a3fb5857d0bad49f5f6b826e22ce6b7d3fafabf59aef135600b4aa0a"} err="failed to get container status \"a0405d12a3fb5857d0bad49f5f6b826e22ce6b7d3fafabf59aef135600b4aa0a\": rpc error: code = NotFound desc = could not find container \"a0405d12a3fb5857d0bad49f5f6b826e22ce6b7d3fafabf59aef135600b4aa0a\": container with ID starting with a0405d12a3fb5857d0bad49f5f6b826e22ce6b7d3fafabf59aef135600b4aa0a not found: ID does not exist"
Sep 08 15:48:00.618752 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:00.618731    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-36c51d-7bb79986f8-6zjxg"]
Sep 08 15:48:00.624558 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:00.624536    2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-36c51d-7bb79986f8-6zjxg"]
Sep 08 15:48:00.666270 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:00.666250    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmz8p\" (UniqueName: \"kubernetes.io/projected/b14b149c-07c6-4108-a054-a47c6b2c6091-kube-api-access-qmz8p\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:48:00.666270 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:00.666269    2549 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b14b149c-07c6-4108-a054-a47c6b2c6091-maas-api-tls\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:48:00.666387 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:00.666279    2549 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b14b149c-07c6-4108-a054-a47c6b2c6091-metrics-certs\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:48:02.406333 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:02.406287    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b14b149c-07c6-4108-a054-a47c6b2c6091" path="/var/lib/kubelet/pods/b14b149c-07c6-4108-a054-a47c6b2c6091/volumes"
Sep 08 15:48:05.822647 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:05.822617    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-36c51d/autores-model-36c51d-kserve-666c949bcf-gdll4"]
Sep 08 15:48:11.506889 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:11.506835    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-conflict-b-a97eab05-kserve-6bd84bbfc-gtrrr"]
Sep 08 15:48:11.507389 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:11.507368    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b14b149c-07c6-4108-a054-a47c6b2c6091" containerName="maas-api"
Sep 08 15:48:11.507472 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:11.507393    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="b14b149c-07c6-4108-a054-a47c6b2c6091" containerName="maas-api"
Sep 08 15:48:11.507472 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:11.507415    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00dfe99f-6c36-4d0b-ab40-466d0a31480f" containerName="maas-api"
Sep 08 15:48:11.507472 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:11.507425    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="00dfe99f-6c36-4d0b-ab40-466d0a31480f" containerName="maas-api"
Sep 08 15:48:11.507613 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:11.507513    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="b14b149c-07c6-4108-a054-a47c6b2c6091" containerName="maas-api"
Sep 08 15:48:11.507613 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:11.507527    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="00dfe99f-6c36-4d0b-ab40-466d0a31480f" containerName="maas-api"
Sep 08 15:48:11.510469 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:11.510446    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-a97eab05-kserve-6bd84bbfc-gtrrr"
Sep 08 15:48:11.512996 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:11.512976    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-conflict-b-a97eab05-kserve-self-signed-certs\""
Sep 08 15:48:11.518074 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:11.518051    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-b-a97eab05-kserve-6bd84bbfc-gtrrr"]
Sep 08 15:48:11.660825 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:11.660796    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d28f4210-fc5a-46c0-bf94-b317e2d1c05d-tls-certs\") pod \"e2e-conflict-b-a97eab05-kserve-6bd84bbfc-gtrrr\" (UID: \"d28f4210-fc5a-46c0-bf94-b317e2d1c05d\") " pod="llm/e2e-conflict-b-a97eab05-kserve-6bd84bbfc-gtrrr"
Sep 08 15:48:11.660825 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:11.660826    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d28f4210-fc5a-46c0-bf94-b317e2d1c05d-home\") pod \"e2e-conflict-b-a97eab05-kserve-6bd84bbfc-gtrrr\" (UID: \"d28f4210-fc5a-46c0-bf94-b317e2d1c05d\") " pod="llm/e2e-conflict-b-a97eab05-kserve-6bd84bbfc-gtrrr"
Sep 08 15:48:11.661082 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:11.660953    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d28f4210-fc5a-46c0-bf94-b317e2d1c05d-tmp-dir\") pod \"e2e-conflict-b-a97eab05-kserve-6bd84bbfc-gtrrr\" (UID: \"d28f4210-fc5a-46c0-bf94-b317e2d1c05d\") " pod="llm/e2e-conflict-b-a97eab05-kserve-6bd84bbfc-gtrrr"
Sep 08 15:48:11.661082 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:11.660996    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d28f4210-fc5a-46c0-bf94-b317e2d1c05d-model-cache\") pod \"e2e-conflict-b-a97eab05-kserve-6bd84bbfc-gtrrr\" (UID: \"d28f4210-fc5a-46c0-bf94-b317e2d1c05d\") " pod="llm/e2e-conflict-b-a97eab05-kserve-6bd84bbfc-gtrrr"
Sep 08 15:48:11.661082 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:11.661025    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kp5n\" (UniqueName: \"kubernetes.io/projected/d28f4210-fc5a-46c0-bf94-b317e2d1c05d-kube-api-access-4kp5n\") pod \"e2e-conflict-b-a97eab05-kserve-6bd84bbfc-gtrrr\" (UID: \"d28f4210-fc5a-46c0-bf94-b317e2d1c05d\") " pod="llm/e2e-conflict-b-a97eab05-kserve-6bd84bbfc-gtrrr"
Sep 08 15:48:11.661216 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:11.661094    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d28f4210-fc5a-46c0-bf94-b317e2d1c05d-dshm\") pod \"e2e-conflict-b-a97eab05-kserve-6bd84bbfc-gtrrr\" (UID: \"d28f4210-fc5a-46c0-bf94-b317e2d1c05d\") " pod="llm/e2e-conflict-b-a97eab05-kserve-6bd84bbfc-gtrrr"
Sep 08 15:48:11.761982 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:11.761913    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d28f4210-fc5a-46c0-bf94-b317e2d1c05d-tmp-dir\") pod \"e2e-conflict-b-a97eab05-kserve-6bd84bbfc-gtrrr\" (UID: \"d28f4210-fc5a-46c0-bf94-b317e2d1c05d\") " pod="llm/e2e-conflict-b-a97eab05-kserve-6bd84bbfc-gtrrr"
Sep 08 15:48:11.761982 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:11.761946    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d28f4210-fc5a-46c0-bf94-b317e2d1c05d-model-cache\") pod \"e2e-conflict-b-a97eab05-kserve-6bd84bbfc-gtrrr\" (UID: \"d28f4210-fc5a-46c0-bf94-b317e2d1c05d\") " pod="llm/e2e-conflict-b-a97eab05-kserve-6bd84bbfc-gtrrr"
Sep 08 15:48:11.761982 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:11.761965    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kp5n\" (UniqueName: \"kubernetes.io/projected/d28f4210-fc5a-46c0-bf94-b317e2d1c05d-kube-api-access-4kp5n\") pod \"e2e-conflict-b-a97eab05-kserve-6bd84bbfc-gtrrr\" (UID: \"d28f4210-fc5a-46c0-bf94-b317e2d1c05d\") " pod="llm/e2e-conflict-b-a97eab05-kserve-6bd84bbfc-gtrrr"
Sep 08 15:48:11.762180 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:11.762084    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d28f4210-fc5a-46c0-bf94-b317e2d1c05d-dshm\") pod \"e2e-conflict-b-a97eab05-kserve-6bd84bbfc-gtrrr\" (UID: \"d28f4210-fc5a-46c0-bf94-b317e2d1c05d\") " pod="llm/e2e-conflict-b-a97eab05-kserve-6bd84bbfc-gtrrr"
Sep 08 15:48:11.762180 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:11.762142    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d28f4210-fc5a-46c0-bf94-b317e2d1c05d-tls-certs\") pod \"e2e-conflict-b-a97eab05-kserve-6bd84bbfc-gtrrr\" (UID: \"d28f4210-fc5a-46c0-bf94-b317e2d1c05d\") " pod="llm/e2e-conflict-b-a97eab05-kserve-6bd84bbfc-gtrrr"
Sep 08 15:48:11.762180 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:11.762158    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d28f4210-fc5a-46c0-bf94-b317e2d1c05d-home\") pod \"e2e-conflict-b-a97eab05-kserve-6bd84bbfc-gtrrr\" (UID: \"d28f4210-fc5a-46c0-bf94-b317e2d1c05d\") " pod="llm/e2e-conflict-b-a97eab05-kserve-6bd84bbfc-gtrrr"
Sep 08 15:48:11.762401 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:11.762376    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d28f4210-fc5a-46c0-bf94-b317e2d1c05d-tmp-dir\") pod \"e2e-conflict-b-a97eab05-kserve-6bd84bbfc-gtrrr\" (UID: \"d28f4210-fc5a-46c0-bf94-b317e2d1c05d\") " pod="llm/e2e-conflict-b-a97eab05-kserve-6bd84bbfc-gtrrr"
Sep 08 15:48:11.762464 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:11.762402    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d28f4210-fc5a-46c0-bf94-b317e2d1c05d-model-cache\") pod \"e2e-conflict-b-a97eab05-kserve-6bd84bbfc-gtrrr\" (UID: \"d28f4210-fc5a-46c0-bf94-b317e2d1c05d\") " pod="llm/e2e-conflict-b-a97eab05-kserve-6bd84bbfc-gtrrr"
Sep 08 15:48:11.762541 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:11.762524    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d28f4210-fc5a-46c0-bf94-b317e2d1c05d-home\") pod \"e2e-conflict-b-a97eab05-kserve-6bd84bbfc-gtrrr\" (UID: \"d28f4210-fc5a-46c0-bf94-b317e2d1c05d\") " pod="llm/e2e-conflict-b-a97eab05-kserve-6bd84bbfc-gtrrr"
Sep 08 15:48:11.764198 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:11.764178    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d28f4210-fc5a-46c0-bf94-b317e2d1c05d-dshm\") pod \"e2e-conflict-b-a97eab05-kserve-6bd84bbfc-gtrrr\" (UID: \"d28f4210-fc5a-46c0-bf94-b317e2d1c05d\") " pod="llm/e2e-conflict-b-a97eab05-kserve-6bd84bbfc-gtrrr"
Sep 08 15:48:11.764532 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:11.764512    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d28f4210-fc5a-46c0-bf94-b317e2d1c05d-tls-certs\") pod \"e2e-conflict-b-a97eab05-kserve-6bd84bbfc-gtrrr\" (UID: \"d28f4210-fc5a-46c0-bf94-b317e2d1c05d\") " pod="llm/e2e-conflict-b-a97eab05-kserve-6bd84bbfc-gtrrr"
Sep 08 15:48:11.769458 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:11.769437    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kp5n\" (UniqueName: \"kubernetes.io/projected/d28f4210-fc5a-46c0-bf94-b317e2d1c05d-kube-api-access-4kp5n\") pod \"e2e-conflict-b-a97eab05-kserve-6bd84bbfc-gtrrr\" (UID: \"d28f4210-fc5a-46c0-bf94-b317e2d1c05d\") " pod="llm/e2e-conflict-b-a97eab05-kserve-6bd84bbfc-gtrrr"
Sep 08 15:48:11.819522 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:11.819500    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-a97eab05-kserve-6bd84bbfc-gtrrr"
Sep 08 15:48:12.087907 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:12.087875    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-b-a97eab05-kserve-6bd84bbfc-gtrrr"]
Sep 08 15:48:12.089503 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:48:12.089470    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd28f4210_fc5a_46c0_bf94_b317e2d1c05d.slice/crio-a4a580861b5c4164805fb1a53f45660f9b7e081793c1c097de87bf4cad31b6ed WatchSource:0}: Error finding container a4a580861b5c4164805fb1a53f45660f9b7e081793c1c097de87bf4cad31b6ed: Status 404 returned error can't find the container with id a4a580861b5c4164805fb1a53f45660f9b7e081793c1c097de87bf4cad31b6ed
Sep 08 15:48:12.655285 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:12.655250    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-a97eab05-kserve-6bd84bbfc-gtrrr" event={"ID":"d28f4210-fc5a-46c0-bf94-b317e2d1c05d","Type":"ContainerStarted","Data":"0f3e2dde700e744e36dba1d74d992daddd2236dd59b52e653e854747ad453fcc"}
Sep 08 15:48:12.655713 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:12.655299    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-a97eab05-kserve-6bd84bbfc-gtrrr" event={"ID":"d28f4210-fc5a-46c0-bf94-b317e2d1c05d","Type":"ContainerStarted","Data":"a4a580861b5c4164805fb1a53f45660f9b7e081793c1c097de87bf4cad31b6ed"}
Sep 08 15:48:12.674233 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:12.674184    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-conflict-b-a97eab05-kserve-6bd84bbfc-gtrrr" podStartSLOduration=1.674169214 podStartE2EDuration="1.674169214s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:48:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:48:12.672989865 +0000 UTC m=+930.762440960" watchObservedRunningTime="2026-09-08 15:48:12.674169214 +0000 UTC m=+930.763620337"
Sep 08 15:48:20.842820 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:20.842733    2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-autores-36c51d/autores-model-36c51d-kserve-666c949bcf-gdll4" podUID="39ddfd23-702d-47e2-ad5a-c46dc797606c" containerName="main" containerID="cri-o://ca00714dd3bf823183666b441609c5219df5e0cdf7cde8af5d3be5a9a83d09f6" gracePeriod=45
Sep 08 15:48:21.090439 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:21.090415    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-36c51d/autores-model-36c51d-kserve-666c949bcf-gdll4"
Sep 08 15:48:21.239217 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:21.239185    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p5ggb\" (UniqueName: \"kubernetes.io/projected/39ddfd23-702d-47e2-ad5a-c46dc797606c-kube-api-access-p5ggb\") pod \"39ddfd23-702d-47e2-ad5a-c46dc797606c\" (UID: \"39ddfd23-702d-47e2-ad5a-c46dc797606c\") "
Sep 08 15:48:21.239401 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:21.239225    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39ddfd23-702d-47e2-ad5a-c46dc797606c-home\") pod \"39ddfd23-702d-47e2-ad5a-c46dc797606c\" (UID: \"39ddfd23-702d-47e2-ad5a-c46dc797606c\") "
Sep 08 15:48:21.239401 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:21.239259    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39ddfd23-702d-47e2-ad5a-c46dc797606c-model-cache\") pod \"39ddfd23-702d-47e2-ad5a-c46dc797606c\" (UID: \"39ddfd23-702d-47e2-ad5a-c46dc797606c\") "
Sep 08 15:48:21.239401 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:21.239309    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39ddfd23-702d-47e2-ad5a-c46dc797606c-tmp-dir\") pod \"39ddfd23-702d-47e2-ad5a-c46dc797606c\" (UID: \"39ddfd23-702d-47e2-ad5a-c46dc797606c\") "
Sep 08 15:48:21.239401 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:21.239345    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39ddfd23-702d-47e2-ad5a-c46dc797606c-tls-certs\") pod \"39ddfd23-702d-47e2-ad5a-c46dc797606c\" (UID: \"39ddfd23-702d-47e2-ad5a-c46dc797606c\") "
Sep 08 15:48:21.239401 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:21.239377    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39ddfd23-702d-47e2-ad5a-c46dc797606c-dshm\") pod \"39ddfd23-702d-47e2-ad5a-c46dc797606c\" (UID: \"39ddfd23-702d-47e2-ad5a-c46dc797606c\") "
Sep 08 15:48:21.239619 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:21.239478    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39ddfd23-702d-47e2-ad5a-c46dc797606c-model-cache" (OuterVolumeSpecName: "model-cache") pod "39ddfd23-702d-47e2-ad5a-c46dc797606c" (UID: "39ddfd23-702d-47e2-ad5a-c46dc797606c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:48:21.239619 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:21.239493    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39ddfd23-702d-47e2-ad5a-c46dc797606c-home" (OuterVolumeSpecName: "home") pod "39ddfd23-702d-47e2-ad5a-c46dc797606c" (UID: "39ddfd23-702d-47e2-ad5a-c46dc797606c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:48:21.239619 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:21.239557    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39ddfd23-702d-47e2-ad5a-c46dc797606c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "39ddfd23-702d-47e2-ad5a-c46dc797606c" (UID: "39ddfd23-702d-47e2-ad5a-c46dc797606c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:48:21.239731 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:21.239714    2549 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39ddfd23-702d-47e2-ad5a-c46dc797606c-tmp-dir\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:48:21.239802 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:21.239735    2549 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39ddfd23-702d-47e2-ad5a-c46dc797606c-home\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:48:21.239802 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:21.239744    2549 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39ddfd23-702d-47e2-ad5a-c46dc797606c-model-cache\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:48:21.241481 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:21.241450    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39ddfd23-702d-47e2-ad5a-c46dc797606c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "39ddfd23-702d-47e2-ad5a-c46dc797606c" (UID: "39ddfd23-702d-47e2-ad5a-c46dc797606c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 15:48:21.241578 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:21.241479    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39ddfd23-702d-47e2-ad5a-c46dc797606c-dshm" (OuterVolumeSpecName: "dshm") pod "39ddfd23-702d-47e2-ad5a-c46dc797606c" (UID: "39ddfd23-702d-47e2-ad5a-c46dc797606c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:48:21.241578 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:21.241485    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39ddfd23-702d-47e2-ad5a-c46dc797606c-kube-api-access-p5ggb" (OuterVolumeSpecName: "kube-api-access-p5ggb") pod "39ddfd23-702d-47e2-ad5a-c46dc797606c" (UID: "39ddfd23-702d-47e2-ad5a-c46dc797606c"). InnerVolumeSpecName "kube-api-access-p5ggb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:48:21.341003 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:21.340977    2549 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39ddfd23-702d-47e2-ad5a-c46dc797606c-tls-certs\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:48:21.341003 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:21.340999    2549 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39ddfd23-702d-47e2-ad5a-c46dc797606c-dshm\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:48:21.341149 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:21.341009    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5ggb\" (UniqueName: \"kubernetes.io/projected/39ddfd23-702d-47e2-ad5a-c46dc797606c-kube-api-access-p5ggb\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:48:21.622348 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:21.622268    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-28e012/test-model-28e012-kserve-78f4c6b865-rt9hj"]
Sep 08 15:48:21.622643 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:21.622630    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39ddfd23-702d-47e2-ad5a-c46dc797606c" containerName="main"
Sep 08 15:48:21.622686 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:21.622644    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="39ddfd23-702d-47e2-ad5a-c46dc797606c" containerName="main"
Sep 08 15:48:21.622729 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:21.622720    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="39ddfd23-702d-47e2-ad5a-c46dc797606c" containerName="main"
Sep 08 15:48:21.625884 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:21.625849    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-28e012/test-model-28e012-kserve-78f4c6b865-rt9hj"
Sep 08 15:48:21.628755 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:21.628732    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-28e012\"/\"default-dockercfg-btcrj\""
Sep 08 15:48:21.629804 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:21.629782    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-28e012\"/\"test-model-28e012-kserve-self-signed-certs\""
Sep 08 15:48:21.629900 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:21.629784    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-28e012\"/\"openshift-service-ca.crt\""
Sep 08 15:48:21.629900 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:21.629836    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-28e012\"/\"kube-root-ca.crt\""
Sep 08 15:48:21.636491 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:21.636469    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-28e012/test-model-28e012-kserve-78f4c6b865-rt9hj"]
Sep 08 15:48:21.699519 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:21.699488    2549 generic.go:356] "Generic (PLEG): container finished" podID="39ddfd23-702d-47e2-ad5a-c46dc797606c" containerID="ca00714dd3bf823183666b441609c5219df5e0cdf7cde8af5d3be5a9a83d09f6" exitCode=0
Sep 08 15:48:21.699673 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:21.699541    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-36c51d/autores-model-36c51d-kserve-666c949bcf-gdll4"
Sep 08 15:48:21.699673 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:21.699566    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-36c51d/autores-model-36c51d-kserve-666c949bcf-gdll4" event={"ID":"39ddfd23-702d-47e2-ad5a-c46dc797606c","Type":"ContainerDied","Data":"ca00714dd3bf823183666b441609c5219df5e0cdf7cde8af5d3be5a9a83d09f6"}
Sep 08 15:48:21.699673 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:21.699603    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-36c51d/autores-model-36c51d-kserve-666c949bcf-gdll4" event={"ID":"39ddfd23-702d-47e2-ad5a-c46dc797606c","Type":"ContainerDied","Data":"cff74a8d4cbf7757d73446ec382c37beb4404ed9222001ac155cf8cd403f2082"}
Sep 08 15:48:21.699673 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:21.699625    2549 scope.go:117] "RemoveContainer" containerID="ca00714dd3bf823183666b441609c5219df5e0cdf7cde8af5d3be5a9a83d09f6"
Sep 08 15:48:21.708593 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:21.708578    2549 scope.go:117] "RemoveContainer" containerID="ca00714dd3bf823183666b441609c5219df5e0cdf7cde8af5d3be5a9a83d09f6"
Sep 08 15:48:21.708850 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:48:21.708833    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca00714dd3bf823183666b441609c5219df5e0cdf7cde8af5d3be5a9a83d09f6\": container with ID starting with ca00714dd3bf823183666b441609c5219df5e0cdf7cde8af5d3be5a9a83d09f6 not found: ID does not exist" containerID="ca00714dd3bf823183666b441609c5219df5e0cdf7cde8af5d3be5a9a83d09f6"
Sep 08 15:48:21.708975 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:21.708884    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca00714dd3bf823183666b441609c5219df5e0cdf7cde8af5d3be5a9a83d09f6"} err="failed to get container status \"ca00714dd3bf823183666b441609c5219df5e0cdf7cde8af5d3be5a9a83d09f6\": rpc error: code = NotFound desc = could not find container \"ca00714dd3bf823183666b441609c5219df5e0cdf7cde8af5d3be5a9a83d09f6\": container with ID starting with ca00714dd3bf823183666b441609c5219df5e0cdf7cde8af5d3be5a9a83d09f6 not found: ID does not exist"
Sep 08 15:48:21.721257 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:21.721233    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-36c51d/autores-model-36c51d-kserve-666c949bcf-gdll4"]
Sep 08 15:48:21.727076 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:21.727053    2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-36c51d/autores-model-36c51d-kserve-666c949bcf-gdll4"]
Sep 08 15:48:21.744587 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:21.744563    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed1cf67d-6716-497c-9400-2f061528d436-model-cache\") pod \"test-model-28e012-kserve-78f4c6b865-rt9hj\" (UID: \"ed1cf67d-6716-497c-9400-2f061528d436\") " pod="ai-tenant-e2e-inf-a-28e012/test-model-28e012-kserve-78f4c6b865-rt9hj"
Sep 08 15:48:21.744657 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:21.744605    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdl58\" (UniqueName: \"kubernetes.io/projected/ed1cf67d-6716-497c-9400-2f061528d436-kube-api-access-pdl58\") pod \"test-model-28e012-kserve-78f4c6b865-rt9hj\" (UID: \"ed1cf67d-6716-497c-9400-2f061528d436\") " pod="ai-tenant-e2e-inf-a-28e012/test-model-28e012-kserve-78f4c6b865-rt9hj"
Sep 08 15:48:21.744657 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:21.744623    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed1cf67d-6716-497c-9400-2f061528d436-tmp-dir\") pod \"test-model-28e012-kserve-78f4c6b865-rt9hj\" (UID: \"ed1cf67d-6716-497c-9400-2f061528d436\") " pod="ai-tenant-e2e-inf-a-28e012/test-model-28e012-kserve-78f4c6b865-rt9hj"
Sep 08 15:48:21.744736 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:21.744693    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed1cf67d-6716-497c-9400-2f061528d436-tls-certs\") pod \"test-model-28e012-kserve-78f4c6b865-rt9hj\" (UID: \"ed1cf67d-6716-497c-9400-2f061528d436\") " pod="ai-tenant-e2e-inf-a-28e012/test-model-28e012-kserve-78f4c6b865-rt9hj"
Sep 08 15:48:21.744736 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:21.744724    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed1cf67d-6716-497c-9400-2f061528d436-dshm\") pod \"test-model-28e012-kserve-78f4c6b865-rt9hj\" (UID: \"ed1cf67d-6716-497c-9400-2f061528d436\") " pod="ai-tenant-e2e-inf-a-28e012/test-model-28e012-kserve-78f4c6b865-rt9hj"
Sep 08 15:48:21.744807 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:21.744774    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed1cf67d-6716-497c-9400-2f061528d436-home\") pod \"test-model-28e012-kserve-78f4c6b865-rt9hj\" (UID: \"ed1cf67d-6716-497c-9400-2f061528d436\") " pod="ai-tenant-e2e-inf-a-28e012/test-model-28e012-kserve-78f4c6b865-rt9hj"
Sep 08 15:48:21.820407 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:21.820379    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-conflict-b-a97eab05-kserve-6bd84bbfc-gtrrr"
Sep 08 15:48:21.820560 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:21.820421    2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-conflict-b-a97eab05-kserve-6bd84bbfc-gtrrr"
Sep 08 15:48:21.833175 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:21.833144    2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-conflict-b-a97eab05-kserve-6bd84bbfc-gtrrr"
Sep 08 15:48:21.845922 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:21.845904    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdl58\" (UniqueName: \"kubernetes.io/projected/ed1cf67d-6716-497c-9400-2f061528d436-kube-api-access-pdl58\") pod \"test-model-28e012-kserve-78f4c6b865-rt9hj\" (UID: \"ed1cf67d-6716-497c-9400-2f061528d436\") " pod="ai-tenant-e2e-inf-a-28e012/test-model-28e012-kserve-78f4c6b865-rt9hj"
Sep 08 15:48:21.846269 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:21.845932    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed1cf67d-6716-497c-9400-2f061528d436-tmp-dir\") pod \"test-model-28e012-kserve-78f4c6b865-rt9hj\" (UID: \"ed1cf67d-6716-497c-9400-2f061528d436\") " pod="ai-tenant-e2e-inf-a-28e012/test-model-28e012-kserve-78f4c6b865-rt9hj"
Sep 08 15:48:21.846269 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:21.845958    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed1cf67d-6716-497c-9400-2f061528d436-tls-certs\") pod \"test-model-28e012-kserve-78f4c6b865-rt9hj\" (UID: \"ed1cf67d-6716-497c-9400-2f061528d436\") " pod="ai-tenant-e2e-inf-a-28e012/test-model-28e012-kserve-78f4c6b865-rt9hj"
Sep 08 15:48:21.846269 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:21.846087    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed1cf67d-6716-497c-9400-2f061528d436-dshm\") pod \"test-model-28e012-kserve-78f4c6b865-rt9hj\" (UID: \"ed1cf67d-6716-497c-9400-2f061528d436\") " pod="ai-tenant-e2e-inf-a-28e012/test-model-28e012-kserve-78f4c6b865-rt9hj"
Sep 08 15:48:21.846269 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:21.846165    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed1cf67d-6716-497c-9400-2f061528d436-home\") pod \"test-model-28e012-kserve-78f4c6b865-rt9hj\" (UID: \"ed1cf67d-6716-497c-9400-2f061528d436\") " pod="ai-tenant-e2e-inf-a-28e012/test-model-28e012-kserve-78f4c6b865-rt9hj"
Sep 08 15:48:21.846269 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:21.846228    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed1cf67d-6716-497c-9400-2f061528d436-model-cache\") pod \"test-model-28e012-kserve-78f4c6b865-rt9hj\" (UID: \"ed1cf67d-6716-497c-9400-2f061528d436\") " pod="ai-tenant-e2e-inf-a-28e012/test-model-28e012-kserve-78f4c6b865-rt9hj"
Sep 08 15:48:21.846443 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:21.846273    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed1cf67d-6716-497c-9400-2f061528d436-tmp-dir\") pod \"test-model-28e012-kserve-78f4c6b865-rt9hj\" (UID: \"ed1cf67d-6716-497c-9400-2f061528d436\") " pod="ai-tenant-e2e-inf-a-28e012/test-model-28e012-kserve-78f4c6b865-rt9hj"
Sep 08 15:48:21.846524 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:21.846505    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed1cf67d-6716-497c-9400-2f061528d436-model-cache\") pod \"test-model-28e012-kserve-78f4c6b865-rt9hj\" (UID: \"ed1cf67d-6716-497c-9400-2f061528d436\") " pod="ai-tenant-e2e-inf-a-28e012/test-model-28e012-kserve-78f4c6b865-rt9hj"
Sep 08 15:48:21.846561 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:21.846533    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed1cf67d-6716-497c-9400-2f061528d436-home\") pod \"test-model-28e012-kserve-78f4c6b865-rt9hj\" (UID: \"ed1cf67d-6716-497c-9400-2f061528d436\") " pod="ai-tenant-e2e-inf-a-28e012/test-model-28e012-kserve-78f4c6b865-rt9hj"
Sep 08 15:48:21.848753 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:21.848708    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed1cf67d-6716-497c-9400-2f061528d436-tls-certs\") pod \"test-model-28e012-kserve-78f4c6b865-rt9hj\" (UID: \"ed1cf67d-6716-497c-9400-2f061528d436\") " pod="ai-tenant-e2e-inf-a-28e012/test-model-28e012-kserve-78f4c6b865-rt9hj"
Sep 08 15:48:21.848895 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:21.848761    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed1cf67d-6716-497c-9400-2f061528d436-dshm\") pod \"test-model-28e012-kserve-78f4c6b865-rt9hj\" (UID: \"ed1cf67d-6716-497c-9400-2f061528d436\") " pod="ai-tenant-e2e-inf-a-28e012/test-model-28e012-kserve-78f4c6b865-rt9hj"
Sep 08 15:48:21.854411 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:21.854390    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdl58\" (UniqueName: \"kubernetes.io/projected/ed1cf67d-6716-497c-9400-2f061528d436-kube-api-access-pdl58\") pod \"test-model-28e012-kserve-78f4c6b865-rt9hj\" (UID: \"ed1cf67d-6716-497c-9400-2f061528d436\") " pod="ai-tenant-e2e-inf-a-28e012/test-model-28e012-kserve-78f4c6b865-rt9hj"
Sep 08 15:48:21.936807 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:21.936780    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-28e012/test-model-28e012-kserve-78f4c6b865-rt9hj"
Sep 08 15:48:22.210697 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:22.210665    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-28e012/test-model-28e012-kserve-78f4c6b865-rt9hj"]
Sep 08 15:48:22.212505 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:48:22.212474    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poded1cf67d_6716_497c_9400_2f061528d436.slice/crio-fb7565d0cf4dac24e490c8c2c8a9ae5acacacf4cc92baaeca25a5da004e3ffe8 WatchSource:0}: Error finding container fb7565d0cf4dac24e490c8c2c8a9ae5acacacf4cc92baaeca25a5da004e3ffe8: Status 404 returned error can't find the container with id fb7565d0cf4dac24e490c8c2c8a9ae5acacacf4cc92baaeca25a5da004e3ffe8
Sep 08 15:48:22.401026 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:22.400992    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="39ddfd23-702d-47e2-ad5a-c46dc797606c" path="/var/lib/kubelet/pods/39ddfd23-702d-47e2-ad5a-c46dc797606c/volumes"
Sep 08 15:48:22.706360 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:22.706321    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-28e012/test-model-28e012-kserve-78f4c6b865-rt9hj" event={"ID":"ed1cf67d-6716-497c-9400-2f061528d436","Type":"ContainerStarted","Data":"91164caed0c542f8d05ef4b20fbf860222f7189d93fae027039daa5d9c1b9822"}
Sep 08 15:48:22.706553 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:22.706366    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-28e012/test-model-28e012-kserve-78f4c6b865-rt9hj" event={"ID":"ed1cf67d-6716-497c-9400-2f061528d436","Type":"ContainerStarted","Data":"fb7565d0cf4dac24e490c8c2c8a9ae5acacacf4cc92baaeca25a5da004e3ffe8"}
Sep 08 15:48:22.717424 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:22.717400    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-conflict-b-a97eab05-kserve-6bd84bbfc-gtrrr"
Sep 08 15:48:22.727497 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:22.727455    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-28e012/test-model-28e012-kserve-78f4c6b865-rt9hj" podStartSLOduration=1.727440933 podStartE2EDuration="1.727440933s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:48:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:48:22.725174252 +0000 UTC m=+940.814625369" watchObservedRunningTime="2026-09-08 15:48:22.727440933 +0000 UTC m=+940.816892022"
Sep 08 15:48:26.607668 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:26.607626    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-b-a97eab05-kserve-6bd84bbfc-gtrrr"]
Sep 08 15:48:30.505376 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:30.505342    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-a-a97eab05-kserve-5c4fd789f-mdtbg"]
Sep 08 15:48:31.937017 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:31.936982    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-28e012/test-model-28e012-kserve-78f4c6b865-rt9hj"
Sep 08 15:48:31.937379 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:31.937033    2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-28e012/test-model-28e012-kserve-78f4c6b865-rt9hj"
Sep 08 15:48:31.949589 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:31.949567    2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-28e012/test-model-28e012-kserve-78f4c6b865-rt9hj"
Sep 08 15:48:32.769179 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:32.769150    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-28e012/test-model-28e012-kserve-78f4c6b865-rt9hj"
Sep 08 15:48:41.627016 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:41.626979    2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-conflict-b-a97eab05-kserve-6bd84bbfc-gtrrr" podUID="d28f4210-fc5a-46c0-bf94-b317e2d1c05d" containerName="main" containerID="cri-o://0f3e2dde700e744e36dba1d74d992daddd2236dd59b52e653e854747ad453fcc" gracePeriod=45
Sep 08 15:48:41.804970 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:41.804934    2549 generic.go:356] "Generic (PLEG): container finished" podID="d28f4210-fc5a-46c0-bf94-b317e2d1c05d" containerID="0f3e2dde700e744e36dba1d74d992daddd2236dd59b52e653e854747ad453fcc" exitCode=0
Sep 08 15:48:41.805095 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:41.805044    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-a97eab05-kserve-6bd84bbfc-gtrrr" event={"ID":"d28f4210-fc5a-46c0-bf94-b317e2d1c05d","Type":"ContainerDied","Data":"0f3e2dde700e744e36dba1d74d992daddd2236dd59b52e653e854747ad453fcc"}
Sep 08 15:48:41.864695 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:41.864671    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-a97eab05-kserve-6bd84bbfc-gtrrr"
Sep 08 15:48:41.903165 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:41.903140    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d28f4210-fc5a-46c0-bf94-b317e2d1c05d-dshm\") pod \"d28f4210-fc5a-46c0-bf94-b317e2d1c05d\" (UID: \"d28f4210-fc5a-46c0-bf94-b317e2d1c05d\") "
Sep 08 15:48:41.903302 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:41.903187    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d28f4210-fc5a-46c0-bf94-b317e2d1c05d-tls-certs\") pod \"d28f4210-fc5a-46c0-bf94-b317e2d1c05d\" (UID: \"d28f4210-fc5a-46c0-bf94-b317e2d1c05d\") "
Sep 08 15:48:41.903302 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:41.903227    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d28f4210-fc5a-46c0-bf94-b317e2d1c05d-tmp-dir\") pod \"d28f4210-fc5a-46c0-bf94-b317e2d1c05d\" (UID: \"d28f4210-fc5a-46c0-bf94-b317e2d1c05d\") "
Sep 08 15:48:41.903397 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:41.903305    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d28f4210-fc5a-46c0-bf94-b317e2d1c05d-model-cache\") pod \"d28f4210-fc5a-46c0-bf94-b317e2d1c05d\" (UID: \"d28f4210-fc5a-46c0-bf94-b317e2d1c05d\") "
Sep 08 15:48:41.903397 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:41.903341    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4kp5n\" (UniqueName: \"kubernetes.io/projected/d28f4210-fc5a-46c0-bf94-b317e2d1c05d-kube-api-access-4kp5n\") pod \"d28f4210-fc5a-46c0-bf94-b317e2d1c05d\" (UID: \"d28f4210-fc5a-46c0-bf94-b317e2d1c05d\") "
Sep 08 15:48:41.903397 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:41.903384    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d28f4210-fc5a-46c0-bf94-b317e2d1c05d-home\") pod \"d28f4210-fc5a-46c0-bf94-b317e2d1c05d\" (UID: \"d28f4210-fc5a-46c0-bf94-b317e2d1c05d\") "
Sep 08 15:48:41.903797 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:41.903584    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d28f4210-fc5a-46c0-bf94-b317e2d1c05d-model-cache" (OuterVolumeSpecName: "model-cache") pod "d28f4210-fc5a-46c0-bf94-b317e2d1c05d" (UID: "d28f4210-fc5a-46c0-bf94-b317e2d1c05d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:48:41.903797 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:41.903608    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d28f4210-fc5a-46c0-bf94-b317e2d1c05d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d28f4210-fc5a-46c0-bf94-b317e2d1c05d" (UID: "d28f4210-fc5a-46c0-bf94-b317e2d1c05d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:48:41.903797 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:41.903749    2549 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d28f4210-fc5a-46c0-bf94-b317e2d1c05d-tmp-dir\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:48:41.903797 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:41.903770    2549 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d28f4210-fc5a-46c0-bf94-b317e2d1c05d-model-cache\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:48:41.904032 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:41.903820    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d28f4210-fc5a-46c0-bf94-b317e2d1c05d-home" (OuterVolumeSpecName: "home") pod "d28f4210-fc5a-46c0-bf94-b317e2d1c05d" (UID: "d28f4210-fc5a-46c0-bf94-b317e2d1c05d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:48:41.905371 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:41.905345    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d28f4210-fc5a-46c0-bf94-b317e2d1c05d-dshm" (OuterVolumeSpecName: "dshm") pod "d28f4210-fc5a-46c0-bf94-b317e2d1c05d" (UID: "d28f4210-fc5a-46c0-bf94-b317e2d1c05d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:48:41.905768 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:41.905743    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d28f4210-fc5a-46c0-bf94-b317e2d1c05d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d28f4210-fc5a-46c0-bf94-b317e2d1c05d" (UID: "d28f4210-fc5a-46c0-bf94-b317e2d1c05d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 15:48:41.905876 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:41.905745    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d28f4210-fc5a-46c0-bf94-b317e2d1c05d-kube-api-access-4kp5n" (OuterVolumeSpecName: "kube-api-access-4kp5n") pod "d28f4210-fc5a-46c0-bf94-b317e2d1c05d" (UID: "d28f4210-fc5a-46c0-bf94-b317e2d1c05d"). InnerVolumeSpecName "kube-api-access-4kp5n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:48:42.004893 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:42.004837    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4kp5n\" (UniqueName: \"kubernetes.io/projected/d28f4210-fc5a-46c0-bf94-b317e2d1c05d-kube-api-access-4kp5n\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:48:42.004893 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:42.004891    2549 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d28f4210-fc5a-46c0-bf94-b317e2d1c05d-home\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:48:42.004893 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:42.004900    2549 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d28f4210-fc5a-46c0-bf94-b317e2d1c05d-dshm\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:48:42.005109 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:42.004909    2549 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d28f4210-fc5a-46c0-bf94-b317e2d1c05d-tls-certs\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:48:42.810628 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:42.810546    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-a97eab05-kserve-6bd84bbfc-gtrrr" event={"ID":"d28f4210-fc5a-46c0-bf94-b317e2d1c05d","Type":"ContainerDied","Data":"a4a580861b5c4164805fb1a53f45660f9b7e081793c1c097de87bf4cad31b6ed"}
Sep 08 15:48:42.810628 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:42.810598    2549 scope.go:117] "RemoveContainer" containerID="0f3e2dde700e744e36dba1d74d992daddd2236dd59b52e653e854747ad453fcc"
Sep 08 15:48:42.811131 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:42.810559    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-a97eab05-kserve-6bd84bbfc-gtrrr"
Sep 08 15:48:42.828067 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:42.828041    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-b-a97eab05-kserve-6bd84bbfc-gtrrr"]
Sep 08 15:48:42.832263 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:42.832240    2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-conflict-b-a97eab05-kserve-6bd84bbfc-gtrrr"]
Sep 08 15:48:44.401815 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:44.401780    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d28f4210-fc5a-46c0-bf94-b317e2d1c05d" path="/var/lib/kubelet/pods/d28f4210-fc5a-46c0-bf94-b317e2d1c05d/volumes"
Sep 08 15:48:45.525591 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:45.525547    2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-conflict-a-a97eab05-kserve-5c4fd789f-mdtbg" podUID="2af2595b-8e90-4149-b1e2-c39fb945c775" containerName="main" containerID="cri-o://8737834fb50dd5fa3bee61d1d45e235fb0afe52f10ce3fdbee659ec42f3edb74" gracePeriod=45
Sep 08 15:48:45.776417 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:45.776358    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-a97eab05-kserve-5c4fd789f-mdtbg"
Sep 08 15:48:45.827413 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:45.827385    2549 generic.go:356] "Generic (PLEG): container finished" podID="2af2595b-8e90-4149-b1e2-c39fb945c775" containerID="8737834fb50dd5fa3bee61d1d45e235fb0afe52f10ce3fdbee659ec42f3edb74" exitCode=0
Sep 08 15:48:45.827576 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:45.827431    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-a97eab05-kserve-5c4fd789f-mdtbg"
Sep 08 15:48:45.827576 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:45.827479    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-a97eab05-kserve-5c4fd789f-mdtbg" event={"ID":"2af2595b-8e90-4149-b1e2-c39fb945c775","Type":"ContainerDied","Data":"8737834fb50dd5fa3bee61d1d45e235fb0afe52f10ce3fdbee659ec42f3edb74"}
Sep 08 15:48:45.827576 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:45.827527    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-a97eab05-kserve-5c4fd789f-mdtbg" event={"ID":"2af2595b-8e90-4149-b1e2-c39fb945c775","Type":"ContainerDied","Data":"6c2b8195eba4db60fea0de90dbd9260faf88db3b97ca47389460b9dc62f86327"}
Sep 08 15:48:45.827576 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:45.827549    2549 scope.go:117] "RemoveContainer" containerID="8737834fb50dd5fa3bee61d1d45e235fb0afe52f10ce3fdbee659ec42f3edb74"
Sep 08 15:48:45.834233 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:45.834214    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2af2595b-8e90-4149-b1e2-c39fb945c775-dshm\") pod \"2af2595b-8e90-4149-b1e2-c39fb945c775\" (UID: \"2af2595b-8e90-4149-b1e2-c39fb945c775\") "
Sep 08 15:48:45.834346 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:45.834249    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2af2595b-8e90-4149-b1e2-c39fb945c775-model-cache\") pod \"2af2595b-8e90-4149-b1e2-c39fb945c775\" (UID: \"2af2595b-8e90-4149-b1e2-c39fb945c775\") "
Sep 08 15:48:45.834346 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:45.834268    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2af2595b-8e90-4149-b1e2-c39fb945c775-tmp-dir\") pod \"2af2595b-8e90-4149-b1e2-c39fb945c775\" (UID: \"2af2595b-8e90-4149-b1e2-c39fb945c775\") "
Sep 08 15:48:45.834346 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:45.834294    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2af2595b-8e90-4149-b1e2-c39fb945c775-tls-certs\") pod \"2af2595b-8e90-4149-b1e2-c39fb945c775\" (UID: \"2af2595b-8e90-4149-b1e2-c39fb945c775\") "
Sep 08 15:48:45.834346 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:45.834315    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zs6ln\" (UniqueName: \"kubernetes.io/projected/2af2595b-8e90-4149-b1e2-c39fb945c775-kube-api-access-zs6ln\") pod \"2af2595b-8e90-4149-b1e2-c39fb945c775\" (UID: \"2af2595b-8e90-4149-b1e2-c39fb945c775\") "
Sep 08 15:48:45.834562 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:45.834405    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2af2595b-8e90-4149-b1e2-c39fb945c775-home\") pod \"2af2595b-8e90-4149-b1e2-c39fb945c775\" (UID: \"2af2595b-8e90-4149-b1e2-c39fb945c775\") "
Sep 08 15:48:45.834562 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:45.834507    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2af2595b-8e90-4149-b1e2-c39fb945c775-model-cache" (OuterVolumeSpecName: "model-cache") pod "2af2595b-8e90-4149-b1e2-c39fb945c775" (UID: "2af2595b-8e90-4149-b1e2-c39fb945c775"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:48:45.834681 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:45.834590    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2af2595b-8e90-4149-b1e2-c39fb945c775-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2af2595b-8e90-4149-b1e2-c39fb945c775" (UID: "2af2595b-8e90-4149-b1e2-c39fb945c775"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:48:45.834681 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:45.834671    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2af2595b-8e90-4149-b1e2-c39fb945c775-home" (OuterVolumeSpecName: "home") pod "2af2595b-8e90-4149-b1e2-c39fb945c775" (UID: "2af2595b-8e90-4149-b1e2-c39fb945c775"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:48:45.834781 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:45.834767    2549 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2af2595b-8e90-4149-b1e2-c39fb945c775-home\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:48:45.834836 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:45.834781    2549 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2af2595b-8e90-4149-b1e2-c39fb945c775-model-cache\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:48:45.834836 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:45.834791    2549 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2af2595b-8e90-4149-b1e2-c39fb945c775-tmp-dir\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:48:45.836157 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:45.836139    2549 scope.go:117] "RemoveContainer" containerID="8737834fb50dd5fa3bee61d1d45e235fb0afe52f10ce3fdbee659ec42f3edb74"
Sep 08 15:48:45.836321 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:45.836295    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2af2595b-8e90-4149-b1e2-c39fb945c775-dshm" (OuterVolumeSpecName: "dshm") pod "2af2595b-8e90-4149-b1e2-c39fb945c775" (UID: "2af2595b-8e90-4149-b1e2-c39fb945c775"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:48:45.836509 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:48:45.836489    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8737834fb50dd5fa3bee61d1d45e235fb0afe52f10ce3fdbee659ec42f3edb74\": container with ID starting with 8737834fb50dd5fa3bee61d1d45e235fb0afe52f10ce3fdbee659ec42f3edb74 not found: ID does not exist" containerID="8737834fb50dd5fa3bee61d1d45e235fb0afe52f10ce3fdbee659ec42f3edb74"
Sep 08 15:48:45.836564 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:45.836518    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8737834fb50dd5fa3bee61d1d45e235fb0afe52f10ce3fdbee659ec42f3edb74"} err="failed to get container status \"8737834fb50dd5fa3bee61d1d45e235fb0afe52f10ce3fdbee659ec42f3edb74\": rpc error: code = NotFound desc = could not find container \"8737834fb50dd5fa3bee61d1d45e235fb0afe52f10ce3fdbee659ec42f3edb74\": container with ID starting with 8737834fb50dd5fa3bee61d1d45e235fb0afe52f10ce3fdbee659ec42f3edb74 not found: ID does not exist"
Sep 08 15:48:45.836663 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:45.836648    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2af2595b-8e90-4149-b1e2-c39fb945c775-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2af2595b-8e90-4149-b1e2-c39fb945c775" (UID: "2af2595b-8e90-4149-b1e2-c39fb945c775"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 15:48:45.836971 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:45.836953    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2af2595b-8e90-4149-b1e2-c39fb945c775-kube-api-access-zs6ln" (OuterVolumeSpecName: "kube-api-access-zs6ln") pod "2af2595b-8e90-4149-b1e2-c39fb945c775" (UID: "2af2595b-8e90-4149-b1e2-c39fb945c775"). InnerVolumeSpecName "kube-api-access-zs6ln". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:48:45.935296 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:45.935268    2549 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2af2595b-8e90-4149-b1e2-c39fb945c775-dshm\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:48:45.935296 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:45.935291    2549 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2af2595b-8e90-4149-b1e2-c39fb945c775-tls-certs\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:48:45.935438 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:45.935302    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zs6ln\" (UniqueName: \"kubernetes.io/projected/2af2595b-8e90-4149-b1e2-c39fb945c775-kube-api-access-zs6ln\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:48:46.151399 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:46.151373    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-a-a97eab05-kserve-5c4fd789f-mdtbg"]
Sep 08 15:48:46.155177 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:46.155152    2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-conflict-a-a97eab05-kserve-5c4fd789f-mdtbg"]
Sep 08 15:48:46.400035 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:46.400004    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2af2595b-8e90-4149-b1e2-c39fb945c775" path="/var/lib/kubelet/pods/2af2595b-8e90-4149-b1e2-c39fb945c775/volumes"
Sep 08 15:48:48.347834 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:48.347792    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-bb8457-7b7fdf489c-w7ch8"]
Sep 08 15:48:48.348364 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:48.348346    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d28f4210-fc5a-46c0-bf94-b317e2d1c05d" containerName="main"
Sep 08 15:48:48.348420 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:48.348367    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="d28f4210-fc5a-46c0-bf94-b317e2d1c05d" containerName="main"
Sep 08 15:48:48.348420 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:48.348393    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2af2595b-8e90-4149-b1e2-c39fb945c775" containerName="main"
Sep 08 15:48:48.348420 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:48.348402    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="2af2595b-8e90-4149-b1e2-c39fb945c775" containerName="main"
Sep 08 15:48:48.348527 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:48.348477    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="d28f4210-fc5a-46c0-bf94-b317e2d1c05d" containerName="main"
Sep 08 15:48:48.348527 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:48.348489    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="2af2595b-8e90-4149-b1e2-c39fb945c775" containerName="main"
Sep 08 15:48:48.353179 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:48.353159    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-bb8457-7b7fdf489c-w7ch8"
Sep 08 15:48:48.355892 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:48.355846    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-bb8457-serving-cert\""
Sep 08 15:48:48.362499 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:48.362473    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-bb8457-7b7fdf489c-w7ch8"]
Sep 08 15:48:48.456716 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:48.456673    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df4f9d7c-5c3d-4655-9d92-0d5a1e1aa5b4-maas-api-tls\") pod \"maas-api-e2e-explicit-bb8457-7b7fdf489c-w7ch8\" (UID: \"df4f9d7c-5c3d-4655-9d92-0d5a1e1aa5b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-bb8457-7b7fdf489c-w7ch8"
Sep 08 15:48:48.456908 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:48.456729    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df4f9d7c-5c3d-4655-9d92-0d5a1e1aa5b4-openshift-service-ca\") pod \"maas-api-e2e-explicit-bb8457-7b7fdf489c-w7ch8\" (UID: \"df4f9d7c-5c3d-4655-9d92-0d5a1e1aa5b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-bb8457-7b7fdf489c-w7ch8"
Sep 08 15:48:48.456908 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:48.456801    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df4f9d7c-5c3d-4655-9d92-0d5a1e1aa5b4-metrics-certs\") pod \"maas-api-e2e-explicit-bb8457-7b7fdf489c-w7ch8\" (UID: \"df4f9d7c-5c3d-4655-9d92-0d5a1e1aa5b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-bb8457-7b7fdf489c-w7ch8"
Sep 08 15:48:48.456908 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:48.456847    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgk95\" (UniqueName: \"kubernetes.io/projected/df4f9d7c-5c3d-4655-9d92-0d5a1e1aa5b4-kube-api-access-sgk95\") pod \"maas-api-e2e-explicit-bb8457-7b7fdf489c-w7ch8\" (UID: \"df4f9d7c-5c3d-4655-9d92-0d5a1e1aa5b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-bb8457-7b7fdf489c-w7ch8"
Sep 08 15:48:48.558201 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:48.558163    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df4f9d7c-5c3d-4655-9d92-0d5a1e1aa5b4-openshift-service-ca\") pod \"maas-api-e2e-explicit-bb8457-7b7fdf489c-w7ch8\" (UID: \"df4f9d7c-5c3d-4655-9d92-0d5a1e1aa5b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-bb8457-7b7fdf489c-w7ch8"
Sep 08 15:48:48.558380 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:48.558213    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df4f9d7c-5c3d-4655-9d92-0d5a1e1aa5b4-metrics-certs\") pod \"maas-api-e2e-explicit-bb8457-7b7fdf489c-w7ch8\" (UID: \"df4f9d7c-5c3d-4655-9d92-0d5a1e1aa5b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-bb8457-7b7fdf489c-w7ch8"
Sep 08 15:48:48.558380 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:48.558248    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgk95\" (UniqueName: \"kubernetes.io/projected/df4f9d7c-5c3d-4655-9d92-0d5a1e1aa5b4-kube-api-access-sgk95\") pod \"maas-api-e2e-explicit-bb8457-7b7fdf489c-w7ch8\" (UID: \"df4f9d7c-5c3d-4655-9d92-0d5a1e1aa5b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-bb8457-7b7fdf489c-w7ch8"
Sep 08 15:48:48.558380 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:48.558296    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df4f9d7c-5c3d-4655-9d92-0d5a1e1aa5b4-maas-api-tls\") pod \"maas-api-e2e-explicit-bb8457-7b7fdf489c-w7ch8\" (UID: \"df4f9d7c-5c3d-4655-9d92-0d5a1e1aa5b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-bb8457-7b7fdf489c-w7ch8"
Sep 08 15:48:48.558993 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:48.558965    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df4f9d7c-5c3d-4655-9d92-0d5a1e1aa5b4-openshift-service-ca\") pod \"maas-api-e2e-explicit-bb8457-7b7fdf489c-w7ch8\" (UID: \"df4f9d7c-5c3d-4655-9d92-0d5a1e1aa5b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-bb8457-7b7fdf489c-w7ch8"
Sep 08 15:48:48.560702 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:48.560671    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df4f9d7c-5c3d-4655-9d92-0d5a1e1aa5b4-metrics-certs\") pod \"maas-api-e2e-explicit-bb8457-7b7fdf489c-w7ch8\" (UID: \"df4f9d7c-5c3d-4655-9d92-0d5a1e1aa5b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-bb8457-7b7fdf489c-w7ch8"
Sep 08 15:48:48.560702 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:48.560695    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df4f9d7c-5c3d-4655-9d92-0d5a1e1aa5b4-maas-api-tls\") pod \"maas-api-e2e-explicit-bb8457-7b7fdf489c-w7ch8\" (UID: \"df4f9d7c-5c3d-4655-9d92-0d5a1e1aa5b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-bb8457-7b7fdf489c-w7ch8"
Sep 08 15:48:48.569150 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:48.569127    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgk95\" (UniqueName: \"kubernetes.io/projected/df4f9d7c-5c3d-4655-9d92-0d5a1e1aa5b4-kube-api-access-sgk95\") pod \"maas-api-e2e-explicit-bb8457-7b7fdf489c-w7ch8\" (UID: \"df4f9d7c-5c3d-4655-9d92-0d5a1e1aa5b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-bb8457-7b7fdf489c-w7ch8"
Sep 08 15:48:48.666119 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:48.666092    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-bb8457-7b7fdf489c-w7ch8"
Sep 08 15:48:48.946289 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:48.946265    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-bb8457-7b7fdf489c-w7ch8"]
Sep 08 15:48:48.948436 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:48:48.948410    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf4f9d7c_5c3d_4655_9d92_0d5a1e1aa5b4.slice/crio-c1484dca9b202dc9626f69a3b30d166a239bc284765ec4affcdf5a546971a32c WatchSource:0}: Error finding container c1484dca9b202dc9626f69a3b30d166a239bc284765ec4affcdf5a546971a32c: Status 404 returned error can't find the container with id c1484dca9b202dc9626f69a3b30d166a239bc284765ec4affcdf5a546971a32c
Sep 08 15:48:49.848058 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:49.848018    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-bb8457-7b7fdf489c-w7ch8" event={"ID":"df4f9d7c-5c3d-4655-9d92-0d5a1e1aa5b4","Type":"ContainerStarted","Data":"b79b8b1d21c57f99553f78b75968d12e5d1ed12678caf84fb541489559986160"}
Sep 08 15:48:49.848418 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:49.848071    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-bb8457-7b7fdf489c-w7ch8" event={"ID":"df4f9d7c-5c3d-4655-9d92-0d5a1e1aa5b4","Type":"ContainerStarted","Data":"c1484dca9b202dc9626f69a3b30d166a239bc284765ec4affcdf5a546971a32c"}
Sep 08 15:48:49.848418 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:49.848113    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-bb8457-7b7fdf489c-w7ch8"
Sep 08 15:48:49.869286 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:49.869230    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-bb8457-7b7fdf489c-w7ch8" podStartSLOduration=1.156754085 podStartE2EDuration="1.869214092s" totalImagesPullingTime="712.460007ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:48:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:48:49.868991742 +0000 UTC m=+967.958442831" watchObservedRunningTime="2026-09-08 15:48:49.869214092 +0000 UTC m=+967.958665180"
Sep 08 15:48:55.858165 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:48:55.858136    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-bb8457-7b7fdf489c-w7ch8"
Sep 08 15:49:27.924486 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:49:27.924452    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-bb8457/explicit-model-bb8457-kserve-586d668dbf-6nrfp"]
Sep 08 15:49:27.928292 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:49:27.928272    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-bb8457/explicit-model-bb8457-kserve-586d668dbf-6nrfp"
Sep 08 15:49:27.931020 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:49:27.930997    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-bb8457\"/\"kube-root-ca.crt\""
Sep 08 15:49:27.932278 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:49:27.932252    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-bb8457\"/\"default-dockercfg-np5ht\""
Sep 08 15:49:27.932390 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:49:27.932255    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-bb8457\"/\"openshift-service-ca.crt\""
Sep 08 15:49:27.932390 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:49:27.932322    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-bb8457\"/\"explicit-model-bb8457-kserve-self-signed-certs\""
Sep 08 15:49:27.935985 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:49:27.935964    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-bb8457/explicit-model-bb8457-kserve-586d668dbf-6nrfp"]
Sep 08 15:49:27.985677 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:49:27.985636    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4577a95e-391a-46ed-adb1-a33161933940-home\") pod \"explicit-model-bb8457-kserve-586d668dbf-6nrfp\" (UID: \"4577a95e-391a-46ed-adb1-a33161933940\") " pod="ai-tenant-e2e-explicit-bb8457/explicit-model-bb8457-kserve-586d668dbf-6nrfp"
Sep 08 15:49:27.985811 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:49:27.985695    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4577a95e-391a-46ed-adb1-a33161933940-dshm\") pod \"explicit-model-bb8457-kserve-586d668dbf-6nrfp\" (UID: \"4577a95e-391a-46ed-adb1-a33161933940\") " pod="ai-tenant-e2e-explicit-bb8457/explicit-model-bb8457-kserve-586d668dbf-6nrfp"
Sep 08 15:49:27.985811 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:49:27.985753    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4577a95e-391a-46ed-adb1-a33161933940-tmp-dir\") pod \"explicit-model-bb8457-kserve-586d668dbf-6nrfp\" (UID: \"4577a95e-391a-46ed-adb1-a33161933940\") " pod="ai-tenant-e2e-explicit-bb8457/explicit-model-bb8457-kserve-586d668dbf-6nrfp"
Sep 08 15:49:27.985811 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:49:27.985790    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4577a95e-391a-46ed-adb1-a33161933940-tls-certs\") pod \"explicit-model-bb8457-kserve-586d668dbf-6nrfp\" (UID: \"4577a95e-391a-46ed-adb1-a33161933940\") " pod="ai-tenant-e2e-explicit-bb8457/explicit-model-bb8457-kserve-586d668dbf-6nrfp"
Sep 08 15:49:27.985993 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:49:27.985846    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ml54l\" (UniqueName: \"kubernetes.io/projected/4577a95e-391a-46ed-adb1-a33161933940-kube-api-access-ml54l\") pod \"explicit-model-bb8457-kserve-586d668dbf-6nrfp\" (UID: \"4577a95e-391a-46ed-adb1-a33161933940\") " pod="ai-tenant-e2e-explicit-bb8457/explicit-model-bb8457-kserve-586d668dbf-6nrfp"
Sep 08 15:49:27.986049 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:49:27.985999    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4577a95e-391a-46ed-adb1-a33161933940-model-cache\") pod \"explicit-model-bb8457-kserve-586d668dbf-6nrfp\" (UID: \"4577a95e-391a-46ed-adb1-a33161933940\") " pod="ai-tenant-e2e-explicit-bb8457/explicit-model-bb8457-kserve-586d668dbf-6nrfp"
Sep 08 15:49:28.086967 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:49:28.086937    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4577a95e-391a-46ed-adb1-a33161933940-model-cache\") pod \"explicit-model-bb8457-kserve-586d668dbf-6nrfp\" (UID: \"4577a95e-391a-46ed-adb1-a33161933940\") " pod="ai-tenant-e2e-explicit-bb8457/explicit-model-bb8457-kserve-586d668dbf-6nrfp"
Sep 08 15:49:28.087100 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:49:28.087086    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4577a95e-391a-46ed-adb1-a33161933940-home\") pod \"explicit-model-bb8457-kserve-586d668dbf-6nrfp\" (UID: \"4577a95e-391a-46ed-adb1-a33161933940\") " pod="ai-tenant-e2e-explicit-bb8457/explicit-model-bb8457-kserve-586d668dbf-6nrfp"
Sep 08 15:49:28.087177 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:49:28.087123    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4577a95e-391a-46ed-adb1-a33161933940-dshm\") pod \"explicit-model-bb8457-kserve-586d668dbf-6nrfp\" (UID: \"4577a95e-391a-46ed-adb1-a33161933940\") " pod="ai-tenant-e2e-explicit-bb8457/explicit-model-bb8457-kserve-586d668dbf-6nrfp"
Sep 08 15:49:28.087177 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:49:28.087149    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4577a95e-391a-46ed-adb1-a33161933940-tmp-dir\") pod \"explicit-model-bb8457-kserve-586d668dbf-6nrfp\" (UID: \"4577a95e-391a-46ed-adb1-a33161933940\") " pod="ai-tenant-e2e-explicit-bb8457/explicit-model-bb8457-kserve-586d668dbf-6nrfp"
Sep 08 15:49:28.087177 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:49:28.087173    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4577a95e-391a-46ed-adb1-a33161933940-tls-certs\") pod \"explicit-model-bb8457-kserve-586d668dbf-6nrfp\" (UID: \"4577a95e-391a-46ed-adb1-a33161933940\") " pod="ai-tenant-e2e-explicit-bb8457/explicit-model-bb8457-kserve-586d668dbf-6nrfp"
Sep 08 15:49:28.087333 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:49:28.087206    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ml54l\" (UniqueName: \"kubernetes.io/projected/4577a95e-391a-46ed-adb1-a33161933940-kube-api-access-ml54l\") pod \"explicit-model-bb8457-kserve-586d668dbf-6nrfp\" (UID: \"4577a95e-391a-46ed-adb1-a33161933940\") " pod="ai-tenant-e2e-explicit-bb8457/explicit-model-bb8457-kserve-586d668dbf-6nrfp"
Sep 08 15:49:28.087333 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:49:28.087311    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4577a95e-391a-46ed-adb1-a33161933940-model-cache\") pod \"explicit-model-bb8457-kserve-586d668dbf-6nrfp\" (UID: \"4577a95e-391a-46ed-adb1-a33161933940\") " pod="ai-tenant-e2e-explicit-bb8457/explicit-model-bb8457-kserve-586d668dbf-6nrfp"
Sep 08 15:49:28.087453 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:49:28.087430    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4577a95e-391a-46ed-adb1-a33161933940-home\") pod \"explicit-model-bb8457-kserve-586d668dbf-6nrfp\" (UID: \"4577a95e-391a-46ed-adb1-a33161933940\") " pod="ai-tenant-e2e-explicit-bb8457/explicit-model-bb8457-kserve-586d668dbf-6nrfp"
Sep 08 15:49:28.087519 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:49:28.087486    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4577a95e-391a-46ed-adb1-a33161933940-tmp-dir\") pod \"explicit-model-bb8457-kserve-586d668dbf-6nrfp\" (UID: \"4577a95e-391a-46ed-adb1-a33161933940\") " pod="ai-tenant-e2e-explicit-bb8457/explicit-model-bb8457-kserve-586d668dbf-6nrfp"
Sep 08 15:49:28.089250 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:49:28.089230    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4577a95e-391a-46ed-adb1-a33161933940-dshm\") pod \"explicit-model-bb8457-kserve-586d668dbf-6nrfp\" (UID: \"4577a95e-391a-46ed-adb1-a33161933940\") " pod="ai-tenant-e2e-explicit-bb8457/explicit-model-bb8457-kserve-586d668dbf-6nrfp"
Sep 08 15:49:28.089557 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:49:28.089531    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4577a95e-391a-46ed-adb1-a33161933940-tls-certs\") pod \"explicit-model-bb8457-kserve-586d668dbf-6nrfp\" (UID: \"4577a95e-391a-46ed-adb1-a33161933940\") " pod="ai-tenant-e2e-explicit-bb8457/explicit-model-bb8457-kserve-586d668dbf-6nrfp"
Sep 08 15:49:28.095224 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:49:28.095202    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ml54l\" (UniqueName: \"kubernetes.io/projected/4577a95e-391a-46ed-adb1-a33161933940-kube-api-access-ml54l\") pod \"explicit-model-bb8457-kserve-586d668dbf-6nrfp\" (UID: \"4577a95e-391a-46ed-adb1-a33161933940\") " pod="ai-tenant-e2e-explicit-bb8457/explicit-model-bb8457-kserve-586d668dbf-6nrfp"
Sep 08 15:49:28.240457 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:49:28.240373    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-bb8457/explicit-model-bb8457-kserve-586d668dbf-6nrfp"
Sep 08 15:49:28.511574 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:49:28.511548    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-bb8457/explicit-model-bb8457-kserve-586d668dbf-6nrfp"]
Sep 08 15:49:28.513130 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:49:28.513102    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4577a95e_391a_46ed_adb1_a33161933940.slice/crio-33cc028ac7a1e3b2aba548b970202cceb4227e9b1ed73e1dbeae615e14deba87 WatchSource:0}: Error finding container 33cc028ac7a1e3b2aba548b970202cceb4227e9b1ed73e1dbeae615e14deba87: Status 404 returned error can't find the container with id 33cc028ac7a1e3b2aba548b970202cceb4227e9b1ed73e1dbeae615e14deba87
Sep 08 15:49:29.031281 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:49:29.031242    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-bb8457/explicit-model-bb8457-kserve-586d668dbf-6nrfp" event={"ID":"4577a95e-391a-46ed-adb1-a33161933940","Type":"ContainerStarted","Data":"ae003ea1b94d59d832d45222164e32e378a7628c32050b7127f76fd02b950bb2"}
Sep 08 15:49:29.031666 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:49:29.031290    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-bb8457/explicit-model-bb8457-kserve-586d668dbf-6nrfp" event={"ID":"4577a95e-391a-46ed-adb1-a33161933940","Type":"ContainerStarted","Data":"33cc028ac7a1e3b2aba548b970202cceb4227e9b1ed73e1dbeae615e14deba87"}
Sep 08 15:49:29.048131 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:49:29.048086    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-bb8457/explicit-model-bb8457-kserve-586d668dbf-6nrfp" podStartSLOduration=2.048070514 podStartE2EDuration="2.048070514s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:49:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:49:29.047135502 +0000 UTC m=+1007.136586592" watchObservedRunningTime="2026-09-08 15:49:29.048070514 +0000 UTC m=+1007.137521612"
Sep 08 15:49:30.135623 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:49:30.135588    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-bb8457/explicit-model-bb8457-kserve-586d668dbf-6nrfp"]
Sep 08 15:49:36.651305 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:49:36.651271    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-bbf4a143-7988bc7f7f-qc67s"]
Sep 08 15:49:36.654980 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:49:36.654956    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bbf4a143-7988bc7f7f-qc67s"
Sep 08 15:49:36.657466 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:49:36.657445    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-bbf4a143-serving-cert\""
Sep 08 15:49:36.667068 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:49:36.667047    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-bbf4a143-7988bc7f7f-qc67s"]
Sep 08 15:49:36.758521 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:49:36.758474    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a122e782-92e0-4607-9bf0-35e76ad317b7-maas-api-tls\") pod \"maas-api-e2e-mt-bbf4a143-7988bc7f7f-qc67s\" (UID: \"a122e782-92e0-4607-9bf0-35e76ad317b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bbf4a143-7988bc7f7f-qc67s"
Sep 08 15:49:36.758521 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:49:36.758521    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m75w8\" (UniqueName: \"kubernetes.io/projected/a122e782-92e0-4607-9bf0-35e76ad317b7-kube-api-access-m75w8\") pod \"maas-api-e2e-mt-bbf4a143-7988bc7f7f-qc67s\" (UID: \"a122e782-92e0-4607-9bf0-35e76ad317b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bbf4a143-7988bc7f7f-qc67s"
Sep 08 15:49:36.758721 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:49:36.758583    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a122e782-92e0-4607-9bf0-35e76ad317b7-openshift-service-ca\") pod \"maas-api-e2e-mt-bbf4a143-7988bc7f7f-qc67s\" (UID: \"a122e782-92e0-4607-9bf0-35e76ad317b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bbf4a143-7988bc7f7f-qc67s"
Sep 08 15:49:36.758721 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:49:36.758601    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a122e782-92e0-4607-9bf0-35e76ad317b7-metrics-certs\") pod \"maas-api-e2e-mt-bbf4a143-7988bc7f7f-qc67s\" (UID: \"a122e782-92e0-4607-9bf0-35e76ad317b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bbf4a143-7988bc7f7f-qc67s"
Sep 08 15:49:36.860000 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:49:36.859963    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a122e782-92e0-4607-9bf0-35e76ad317b7-maas-api-tls\") pod \"maas-api-e2e-mt-bbf4a143-7988bc7f7f-qc67s\" (UID: \"a122e782-92e0-4607-9bf0-35e76ad317b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bbf4a143-7988bc7f7f-qc67s"
Sep 08 15:49:36.860000 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:49:36.860004    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m75w8\" (UniqueName: \"kubernetes.io/projected/a122e782-92e0-4607-9bf0-35e76ad317b7-kube-api-access-m75w8\") pod \"maas-api-e2e-mt-bbf4a143-7988bc7f7f-qc67s\" (UID: \"a122e782-92e0-4607-9bf0-35e76ad317b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bbf4a143-7988bc7f7f-qc67s"
Sep 08 15:49:36.860210 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:49:36.860048    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a122e782-92e0-4607-9bf0-35e76ad317b7-openshift-service-ca\") pod \"maas-api-e2e-mt-bbf4a143-7988bc7f7f-qc67s\" (UID: \"a122e782-92e0-4607-9bf0-35e76ad317b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bbf4a143-7988bc7f7f-qc67s"
Sep 08 15:49:36.860210 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:49:36.860074    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a122e782-92e0-4607-9bf0-35e76ad317b7-metrics-certs\") pod \"maas-api-e2e-mt-bbf4a143-7988bc7f7f-qc67s\" (UID: \"a122e782-92e0-4607-9bf0-35e76ad317b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bbf4a143-7988bc7f7f-qc67s"
Sep 08 15:49:36.860210 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:49:36.860127    2549 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-bbf4a143-serving-cert: secret "maas-api-e2e-mt-bbf4a143-serving-cert" not found
Sep 08 15:49:36.860210 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:49:36.860203    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a122e782-92e0-4607-9bf0-35e76ad317b7-maas-api-tls podName:a122e782-92e0-4607-9bf0-35e76ad317b7 nodeName:}" failed. No retries permitted until 2026-09-08 15:49:37.360181782 +0000 UTC m=+1015.449632849 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a122e782-92e0-4607-9bf0-35e76ad317b7-maas-api-tls") pod "maas-api-e2e-mt-bbf4a143-7988bc7f7f-qc67s" (UID: "a122e782-92e0-4607-9bf0-35e76ad317b7") : secret "maas-api-e2e-mt-bbf4a143-serving-cert" not found
Sep 08 15:49:36.860804 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:49:36.860778    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a122e782-92e0-4607-9bf0-35e76ad317b7-openshift-service-ca\") pod \"maas-api-e2e-mt-bbf4a143-7988bc7f7f-qc67s\" (UID: \"a122e782-92e0-4607-9bf0-35e76ad317b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bbf4a143-7988bc7f7f-qc67s"
Sep 08 15:49:36.862489 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:49:36.862468    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a122e782-92e0-4607-9bf0-35e76ad317b7-metrics-certs\") pod \"maas-api-e2e-mt-bbf4a143-7988bc7f7f-qc67s\" (UID: \"a122e782-92e0-4607-9bf0-35e76ad317b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bbf4a143-7988bc7f7f-qc67s"
Sep 08 15:49:36.874647 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:49:36.874628    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m75w8\" (UniqueName: \"kubernetes.io/projected/a122e782-92e0-4607-9bf0-35e76ad317b7-kube-api-access-m75w8\") pod \"maas-api-e2e-mt-bbf4a143-7988bc7f7f-qc67s\" (UID: \"a122e782-92e0-4607-9bf0-35e76ad317b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bbf4a143-7988bc7f7f-qc67s"
Sep 08 15:49:37.365206 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:49:37.365171    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a122e782-92e0-4607-9bf0-35e76ad317b7-maas-api-tls\") pod \"maas-api-e2e-mt-bbf4a143-7988bc7f7f-qc67s\" (UID: \"a122e782-92e0-4607-9bf0-35e76ad317b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bbf4a143-7988bc7f7f-qc67s"
Sep 08 15:49:37.367618 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:49:37.367594    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a122e782-92e0-4607-9bf0-35e76ad317b7-maas-api-tls\") pod \"maas-api-e2e-mt-bbf4a143-7988bc7f7f-qc67s\" (UID: \"a122e782-92e0-4607-9bf0-35e76ad317b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bbf4a143-7988bc7f7f-qc67s"
Sep 08 15:49:37.567060 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:49:37.567027    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bbf4a143-7988bc7f7f-qc67s"
Sep 08 15:49:37.841022 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:49:37.840996    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-bbf4a143-7988bc7f7f-qc67s"]
Sep 08 15:49:37.843346 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:49:37.843315    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda122e782_92e0_4607_9bf0_35e76ad317b7.slice/crio-4e95c535505bf7f4f4dc057d35dff776ffe00a001f49eaef621e2c483a3ee3bc WatchSource:0}: Error finding container 4e95c535505bf7f4f4dc057d35dff776ffe00a001f49eaef621e2c483a3ee3bc: Status 404 returned error can't find the container with id 4e95c535505bf7f4f4dc057d35dff776ffe00a001f49eaef621e2c483a3ee3bc
Sep 08 15:49:38.075727 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:49:38.075694    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bbf4a143-7988bc7f7f-qc67s" event={"ID":"a122e782-92e0-4607-9bf0-35e76ad317b7","Type":"ContainerStarted","Data":"4e95c535505bf7f4f4dc057d35dff776ffe00a001f49eaef621e2c483a3ee3bc"}
Sep 08 15:49:38.240678 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:49:38.240649    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-bb8457/explicit-model-bb8457-kserve-586d668dbf-6nrfp"
Sep 08 15:49:40.086557 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:49:40.086523    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bbf4a143-7988bc7f7f-qc67s" event={"ID":"a122e782-92e0-4607-9bf0-35e76ad317b7","Type":"ContainerStarted","Data":"9e3f75c808e7812836748d51dfea9cf251c46520438dec062478bc7b4c1f7bf6"}
Sep 08 15:49:40.086939 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:49:40.086585    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bbf4a143-7988bc7f7f-qc67s"
Sep 08 15:49:40.102934 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:49:40.102879    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bbf4a143-7988bc7f7f-qc67s" podStartSLOduration=2.545607919 podStartE2EDuration="4.102847034s" totalImagesPullingTime="1.557239115s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:49:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:49:40.102350652 +0000 UTC m=+1018.191801743" watchObservedRunningTime="2026-09-08 15:49:40.102847034 +0000 UTC m=+1018.192298123"
Sep 08 15:49:40.977527 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:49:40.977488    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-bb8457-7b7fdf489c-w7ch8"]
Sep 08 15:49:40.977744 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:49:40.977719    2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-bb8457-7b7fdf489c-w7ch8" podUID="df4f9d7c-5c3d-4655-9d92-0d5a1e1aa5b4" containerName="maas-api" containerID="cri-o://b79b8b1d21c57f99553f78b75968d12e5d1ed12678caf84fb541489559986160" gracePeriod=30
Sep 08 15:49:41.234509 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:49:41.234452    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-bb8457-7b7fdf489c-w7ch8"
Sep 08 15:49:41.301435 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:49:41.301406    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df4f9d7c-5c3d-4655-9d92-0d5a1e1aa5b4-maas-api-tls\") pod \"df4f9d7c-5c3d-4655-9d92-0d5a1e1aa5b4\" (UID: \"df4f9d7c-5c3d-4655-9d92-0d5a1e1aa5b4\") "
Sep 08 15:49:41.301616 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:49:41.301598    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sgk95\" (UniqueName: \"kubernetes.io/projected/df4f9d7c-5c3d-4655-9d92-0d5a1e1aa5b4-kube-api-access-sgk95\") pod \"df4f9d7c-5c3d-4655-9d92-0d5a1e1aa5b4\" (UID: \"df4f9d7c-5c3d-4655-9d92-0d5a1e1aa5b4\") "
Sep 08 15:49:41.301660 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:49:41.301635    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df4f9d7c-5c3d-4655-9d92-0d5a1e1aa5b4-openshift-service-ca\") pod \"df4f9d7c-5c3d-4655-9d92-0d5a1e1aa5b4\" (UID: \"df4f9d7c-5c3d-4655-9d92-0d5a1e1aa5b4\") "
Sep 08 15:49:41.301704 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:49:41.301666    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df4f9d7c-5c3d-4655-9d92-0d5a1e1aa5b4-metrics-certs\") pod \"df4f9d7c-5c3d-4655-9d92-0d5a1e1aa5b4\" (UID: \"df4f9d7c-5c3d-4655-9d92-0d5a1e1aa5b4\") "
Sep 08 15:49:41.302001 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:49:41.301975    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df4f9d7c-5c3d-4655-9d92-0d5a1e1aa5b4-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "df4f9d7c-5c3d-4655-9d92-0d5a1e1aa5b4" (UID: "df4f9d7c-5c3d-4655-9d92-0d5a1e1aa5b4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 15:49:41.303547 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:49:41.303498    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df4f9d7c-5c3d-4655-9d92-0d5a1e1aa5b4-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "df4f9d7c-5c3d-4655-9d92-0d5a1e1aa5b4" (UID: "df4f9d7c-5c3d-4655-9d92-0d5a1e1aa5b4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 15:49:41.303654 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:49:41.303633    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df4f9d7c-5c3d-4655-9d92-0d5a1e1aa5b4-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "df4f9d7c-5c3d-4655-9d92-0d5a1e1aa5b4" (UID: "df4f9d7c-5c3d-4655-9d92-0d5a1e1aa5b4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 15:49:41.303712 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:49:41.303653    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df4f9d7c-5c3d-4655-9d92-0d5a1e1aa5b4-kube-api-access-sgk95" (OuterVolumeSpecName: "kube-api-access-sgk95") pod "df4f9d7c-5c3d-4655-9d92-0d5a1e1aa5b4" (UID: "df4f9d7c-5c3d-4655-9d92-0d5a1e1aa5b4"). InnerVolumeSpecName "kube-api-access-sgk95". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:49:41.402558 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:49:41.402525    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sgk95\" (UniqueName: \"kubernetes.io/projected/df4f9d7c-5c3d-4655-9d92-0d5a1e1aa5b4-kube-api-access-sgk95\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:49:41.402558 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:49:41.402555    2549 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df4f9d7c-5c3d-4655-9d92-0d5a1e1aa5b4-openshift-service-ca\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:49:41.402558 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:49:41.402566    2549 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df4f9d7c-5c3d-4655-9d92-0d5a1e1aa5b4-metrics-certs\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:49:41.402802 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:49:41.402574    2549 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df4f9d7c-5c3d-4655-9d92-0d5a1e1aa5b4-maas-api-tls\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:49:42.099630 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:49:42.099592    2549 generic.go:356] "Generic (PLEG): container finished" podID="df4f9d7c-5c3d-4655-9d92-0d5a1e1aa5b4" containerID="b79b8b1d21c57f99553f78b75968d12e5d1ed12678caf84fb541489559986160" exitCode=0
Sep 08 15:49:42.099832 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:49:42.099644    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-bb8457-7b7fdf489c-w7ch8"
Sep 08 15:49:42.099832 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:49:42.099689    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-bb8457-7b7fdf489c-w7ch8" event={"ID":"df4f9d7c-5c3d-4655-9d92-0d5a1e1aa5b4","Type":"ContainerDied","Data":"b79b8b1d21c57f99553f78b75968d12e5d1ed12678caf84fb541489559986160"}
Sep 08 15:49:42.099832 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:49:42.099753    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-bb8457-7b7fdf489c-w7ch8" event={"ID":"df4f9d7c-5c3d-4655-9d92-0d5a1e1aa5b4","Type":"ContainerDied","Data":"c1484dca9b202dc9626f69a3b30d166a239bc284765ec4affcdf5a546971a32c"}
Sep 08 15:49:42.099832 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:49:42.099783    2549 scope.go:117] "RemoveContainer" containerID="b79b8b1d21c57f99553f78b75968d12e5d1ed12678caf84fb541489559986160"
Sep 08 15:49:42.111586 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:49:42.111565    2549 scope.go:117] "RemoveContainer" containerID="b79b8b1d21c57f99553f78b75968d12e5d1ed12678caf84fb541489559986160"
Sep 08 15:49:42.112027 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:49:42.111995    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b79b8b1d21c57f99553f78b75968d12e5d1ed12678caf84fb541489559986160\": container with ID starting with b79b8b1d21c57f99553f78b75968d12e5d1ed12678caf84fb541489559986160 not found: ID does not exist" containerID="b79b8b1d21c57f99553f78b75968d12e5d1ed12678caf84fb541489559986160"
Sep 08 15:49:42.112139 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:49:42.112036    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b79b8b1d21c57f99553f78b75968d12e5d1ed12678caf84fb541489559986160"} err="failed to get container status \"b79b8b1d21c57f99553f78b75968d12e5d1ed12678caf84fb541489559986160\": rpc error: code = NotFound desc = could not find container \"b79b8b1d21c57f99553f78b75968d12e5d1ed12678caf84fb541489559986160\": container with ID starting with b79b8b1d21c57f99553f78b75968d12e5d1ed12678caf84fb541489559986160 not found: ID does not exist"
Sep 08 15:49:42.123653 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:49:42.123632    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-bb8457-7b7fdf489c-w7ch8"]
Sep 08 15:49:42.129538 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:49:42.129516    2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-bb8457-7b7fdf489c-w7ch8"]
Sep 08 15:49:42.407349 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:49:42.407316    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="df4f9d7c-5c3d-4655-9d92-0d5a1e1aa5b4" path="/var/lib/kubelet/pods/df4f9d7c-5c3d-4655-9d92-0d5a1e1aa5b4/volumes"
Sep 08 15:49:46.061972 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:49:46.061924    2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-bb8457/explicit-model-bb8457-kserve-586d668dbf-6nrfp" podUID="4577a95e-391a-46ed-adb1-a33161933940" containerName="main" containerID="cri-o://ae003ea1b94d59d832d45222164e32e378a7628c32050b7127f76fd02b950bb2" gracePeriod=45
Sep 08 15:49:46.098540 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:49:46.098514    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bbf4a143-7988bc7f7f-qc67s"
Sep 08 15:49:46.308287 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:49:46.308264    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-bb8457/explicit-model-bb8457-kserve-586d668dbf-6nrfp"
Sep 08 15:49:46.443758 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:49:46.443730    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4577a95e-391a-46ed-adb1-a33161933940-tls-certs\") pod \"4577a95e-391a-46ed-adb1-a33161933940\" (UID: \"4577a95e-391a-46ed-adb1-a33161933940\") "
Sep 08 15:49:46.443939 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:49:46.443770    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4577a95e-391a-46ed-adb1-a33161933940-model-cache\") pod \"4577a95e-391a-46ed-adb1-a33161933940\" (UID: \"4577a95e-391a-46ed-adb1-a33161933940\") "
Sep 08 15:49:46.443939 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:49:46.443805    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4577a95e-391a-46ed-adb1-a33161933940-home\") pod \"4577a95e-391a-46ed-adb1-a33161933940\" (UID: \"4577a95e-391a-46ed-adb1-a33161933940\") "
Sep 08 15:49:46.443939 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:49:46.443829    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4577a95e-391a-46ed-adb1-a33161933940-tmp-dir\") pod \"4577a95e-391a-46ed-adb1-a33161933940\" (UID: \"4577a95e-391a-46ed-adb1-a33161933940\") "
Sep 08 15:49:46.443939 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:49:46.443873    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ml54l\" (UniqueName: \"kubernetes.io/projected/4577a95e-391a-46ed-adb1-a33161933940-kube-api-access-ml54l\") pod \"4577a95e-391a-46ed-adb1-a33161933940\" (UID: \"4577a95e-391a-46ed-adb1-a33161933940\") "
Sep 08 15:49:46.443939 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:49:46.443892    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4577a95e-391a-46ed-adb1-a33161933940-dshm\") pod \"4577a95e-391a-46ed-adb1-a33161933940\" (UID: \"4577a95e-391a-46ed-adb1-a33161933940\") "
Sep 08 15:49:46.444269 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:49:46.444110    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4577a95e-391a-46ed-adb1-a33161933940-model-cache" (OuterVolumeSpecName: "model-cache") pod "4577a95e-391a-46ed-adb1-a33161933940" (UID: "4577a95e-391a-46ed-adb1-a33161933940"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:49:46.444269 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:49:46.444127    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4577a95e-391a-46ed-adb1-a33161933940-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4577a95e-391a-46ed-adb1-a33161933940" (UID: "4577a95e-391a-46ed-adb1-a33161933940"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:49:46.444269 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:49:46.444112    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4577a95e-391a-46ed-adb1-a33161933940-home" (OuterVolumeSpecName: "home") pod "4577a95e-391a-46ed-adb1-a33161933940" (UID: "4577a95e-391a-46ed-adb1-a33161933940"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:49:46.446050 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:49:46.446029    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4577a95e-391a-46ed-adb1-a33161933940-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4577a95e-391a-46ed-adb1-a33161933940" (UID: "4577a95e-391a-46ed-adb1-a33161933940"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 15:49:46.446446 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:49:46.446431    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4577a95e-391a-46ed-adb1-a33161933940-dshm" (OuterVolumeSpecName: "dshm") pod "4577a95e-391a-46ed-adb1-a33161933940" (UID: "4577a95e-391a-46ed-adb1-a33161933940"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:49:46.446513 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:49:46.446489    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4577a95e-391a-46ed-adb1-a33161933940-kube-api-access-ml54l" (OuterVolumeSpecName: "kube-api-access-ml54l") pod "4577a95e-391a-46ed-adb1-a33161933940" (UID: "4577a95e-391a-46ed-adb1-a33161933940"). InnerVolumeSpecName "kube-api-access-ml54l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:49:46.544503 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:49:46.544475    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ml54l\" (UniqueName: \"kubernetes.io/projected/4577a95e-391a-46ed-adb1-a33161933940-kube-api-access-ml54l\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:49:46.544503 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:49:46.544499    2549 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4577a95e-391a-46ed-adb1-a33161933940-dshm\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:49:46.544666 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:49:46.544514    2549 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4577a95e-391a-46ed-adb1-a33161933940-tls-certs\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:49:46.544666 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:49:46.544528    2549 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4577a95e-391a-46ed-adb1-a33161933940-model-cache\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:49:46.544666 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:49:46.544538    2549 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4577a95e-391a-46ed-adb1-a33161933940-home\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:49:46.544666 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:49:46.544548    2549 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4577a95e-391a-46ed-adb1-a33161933940-tmp-dir\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:49:47.125305 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:49:47.125268    2549 generic.go:356] "Generic (PLEG): container finished" podID="4577a95e-391a-46ed-adb1-a33161933940" containerID="ae003ea1b94d59d832d45222164e32e378a7628c32050b7127f76fd02b950bb2" exitCode=0
Sep 08 15:49:47.125767 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:49:47.125318    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-bb8457/explicit-model-bb8457-kserve-586d668dbf-6nrfp"
Sep 08 15:49:47.125767 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:49:47.125349    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-bb8457/explicit-model-bb8457-kserve-586d668dbf-6nrfp" event={"ID":"4577a95e-391a-46ed-adb1-a33161933940","Type":"ContainerDied","Data":"ae003ea1b94d59d832d45222164e32e378a7628c32050b7127f76fd02b950bb2"}
Sep 08 15:49:47.125767 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:49:47.125388    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-bb8457/explicit-model-bb8457-kserve-586d668dbf-6nrfp" event={"ID":"4577a95e-391a-46ed-adb1-a33161933940","Type":"ContainerDied","Data":"33cc028ac7a1e3b2aba548b970202cceb4227e9b1ed73e1dbeae615e14deba87"}
Sep 08 15:49:47.125767 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:49:47.125407    2549 scope.go:117] "RemoveContainer" containerID="ae003ea1b94d59d832d45222164e32e378a7628c32050b7127f76fd02b950bb2"
Sep 08 15:49:47.134520 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:49:47.134503    2549 scope.go:117] "RemoveContainer" containerID="ae003ea1b94d59d832d45222164e32e378a7628c32050b7127f76fd02b950bb2"
Sep 08 15:49:47.134818 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:49:47.134768    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae003ea1b94d59d832d45222164e32e378a7628c32050b7127f76fd02b950bb2\": container with ID starting with ae003ea1b94d59d832d45222164e32e378a7628c32050b7127f76fd02b950bb2 not found: ID does not exist" containerID="ae003ea1b94d59d832d45222164e32e378a7628c32050b7127f76fd02b950bb2"
Sep 08 15:49:47.134818 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:49:47.134803    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae003ea1b94d59d832d45222164e32e378a7628c32050b7127f76fd02b950bb2"} err="failed to get container status \"ae003ea1b94d59d832d45222164e32e378a7628c32050b7127f76fd02b950bb2\": rpc error: code = NotFound desc = could not find container \"ae003ea1b94d59d832d45222164e32e378a7628c32050b7127f76fd02b950bb2\": container with ID starting with ae003ea1b94d59d832d45222164e32e378a7628c32050b7127f76fd02b950bb2 not found: ID does not exist"
Sep 08 15:49:47.146523 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:49:47.146491    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-bb8457/explicit-model-bb8457-kserve-586d668dbf-6nrfp"]
Sep 08 15:49:47.148644 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:49:47.148623    2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-bb8457/explicit-model-bb8457-kserve-586d668dbf-6nrfp"]
Sep 08 15:49:48.400289 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:49:48.400244    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4577a95e-391a-46ed-adb1-a33161933940" path="/var/lib/kubelet/pods/4577a95e-391a-46ed-adb1-a33161933940/volumes"
Sep 08 15:50:09.817701 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:50:09.817665    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-69ef67/noten-model-69ef67-kserve-76579b859f-lbhp7"]
Sep 08 15:50:09.818281 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:50:09.818190    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4577a95e-391a-46ed-adb1-a33161933940" containerName="main"
Sep 08 15:50:09.818281 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:50:09.818211    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="4577a95e-391a-46ed-adb1-a33161933940" containerName="main"
Sep 08 15:50:09.818281 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:50:09.818232    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df4f9d7c-5c3d-4655-9d92-0d5a1e1aa5b4" containerName="maas-api"
Sep 08 15:50:09.818281 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:50:09.818242    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="df4f9d7c-5c3d-4655-9d92-0d5a1e1aa5b4" containerName="maas-api"
Sep 08 15:50:09.818494 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:50:09.818354    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="df4f9d7c-5c3d-4655-9d92-0d5a1e1aa5b4" containerName="maas-api"
Sep 08 15:50:09.818494 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:50:09.818366    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="4577a95e-391a-46ed-adb1-a33161933940" containerName="main"
Sep 08 15:50:09.822692 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:50:09.822671    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-69ef67/noten-model-69ef67-kserve-76579b859f-lbhp7"
Sep 08 15:50:09.825545 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:50:09.825523    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-69ef67\"/\"kube-root-ca.crt\""
Sep 08 15:50:09.826827 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:50:09.826775    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-69ef67\"/\"default-dockercfg-pjlzg\""
Sep 08 15:50:09.827056 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:50:09.826844    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-69ef67\"/\"openshift-service-ca.crt\""
Sep 08 15:50:09.827163 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:50:09.826899    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-69ef67\"/\"noten-model-69ef67-kserve-self-signed-certs\""
Sep 08 15:50:09.831485 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:50:09.831466    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-69ef67/noten-model-69ef67-kserve-76579b859f-lbhp7"]
Sep 08 15:50:09.944437 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:50:09.944412    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aede103b-5c4f-423d-b4f6-ffef53f7ebab-model-cache\") pod \"noten-model-69ef67-kserve-76579b859f-lbhp7\" (UID: \"aede103b-5c4f-423d-b4f6-ffef53f7ebab\") " pod="ai-tenant-e2e-noten-69ef67/noten-model-69ef67-kserve-76579b859f-lbhp7"
Sep 08 15:50:09.944569 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:50:09.944443    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aede103b-5c4f-423d-b4f6-ffef53f7ebab-tmp-dir\") pod \"noten-model-69ef67-kserve-76579b859f-lbhp7\" (UID: \"aede103b-5c4f-423d-b4f6-ffef53f7ebab\") " pod="ai-tenant-e2e-noten-69ef67/noten-model-69ef67-kserve-76579b859f-lbhp7"
Sep 08 15:50:09.944569 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:50:09.944507    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aede103b-5c4f-423d-b4f6-ffef53f7ebab-tls-certs\") pod \"noten-model-69ef67-kserve-76579b859f-lbhp7\" (UID: \"aede103b-5c4f-423d-b4f6-ffef53f7ebab\") " pod="ai-tenant-e2e-noten-69ef67/noten-model-69ef67-kserve-76579b859f-lbhp7"
Sep 08 15:50:09.944569 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:50:09.944525    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fp5x\" (UniqueName: \"kubernetes.io/projected/aede103b-5c4f-423d-b4f6-ffef53f7ebab-kube-api-access-9fp5x\") pod \"noten-model-69ef67-kserve-76579b859f-lbhp7\" (UID: \"aede103b-5c4f-423d-b4f6-ffef53f7ebab\") " pod="ai-tenant-e2e-noten-69ef67/noten-model-69ef67-kserve-76579b859f-lbhp7"
Sep 08 15:50:09.944569 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:50:09.944549    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aede103b-5c4f-423d-b4f6-ffef53f7ebab-home\") pod \"noten-model-69ef67-kserve-76579b859f-lbhp7\" (UID: \"aede103b-5c4f-423d-b4f6-ffef53f7ebab\") " pod="ai-tenant-e2e-noten-69ef67/noten-model-69ef67-kserve-76579b859f-lbhp7"
Sep 08 15:50:09.944712 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:50:09.944619    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aede103b-5c4f-423d-b4f6-ffef53f7ebab-dshm\") pod \"noten-model-69ef67-kserve-76579b859f-lbhp7\" (UID: \"aede103b-5c4f-423d-b4f6-ffef53f7ebab\") " pod="ai-tenant-e2e-noten-69ef67/noten-model-69ef67-kserve-76579b859f-lbhp7"
Sep 08 15:50:10.046108 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:50:10.046062    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aede103b-5c4f-423d-b4f6-ffef53f7ebab-tls-certs\") pod \"noten-model-69ef67-kserve-76579b859f-lbhp7\" (UID: \"aede103b-5c4f-423d-b4f6-ffef53f7ebab\") " pod="ai-tenant-e2e-noten-69ef67/noten-model-69ef67-kserve-76579b859f-lbhp7"
Sep 08 15:50:10.046108 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:50:10.046116    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fp5x\" (UniqueName: \"kubernetes.io/projected/aede103b-5c4f-423d-b4f6-ffef53f7ebab-kube-api-access-9fp5x\") pod \"noten-model-69ef67-kserve-76579b859f-lbhp7\" (UID: \"aede103b-5c4f-423d-b4f6-ffef53f7ebab\") " pod="ai-tenant-e2e-noten-69ef67/noten-model-69ef67-kserve-76579b859f-lbhp7"
Sep 08 15:50:10.046365 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:50:10.046338    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aede103b-5c4f-423d-b4f6-ffef53f7ebab-home\") pod \"noten-model-69ef67-kserve-76579b859f-lbhp7\" (UID: \"aede103b-5c4f-423d-b4f6-ffef53f7ebab\") " pod="ai-tenant-e2e-noten-69ef67/noten-model-69ef67-kserve-76579b859f-lbhp7"
Sep 08 15:50:10.046444 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:50:10.046429    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aede103b-5c4f-423d-b4f6-ffef53f7ebab-dshm\") pod \"noten-model-69ef67-kserve-76579b859f-lbhp7\" (UID: \"aede103b-5c4f-423d-b4f6-ffef53f7ebab\") " pod="ai-tenant-e2e-noten-69ef67/noten-model-69ef67-kserve-76579b859f-lbhp7"
Sep 08 15:50:10.046507 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:50:10.046490    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aede103b-5c4f-423d-b4f6-ffef53f7ebab-model-cache\") pod \"noten-model-69ef67-kserve-76579b859f-lbhp7\" (UID: \"aede103b-5c4f-423d-b4f6-ffef53f7ebab\") " pod="ai-tenant-e2e-noten-69ef67/noten-model-69ef67-kserve-76579b859f-lbhp7"
Sep 08 15:50:10.046548 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:50:10.046525    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aede103b-5c4f-423d-b4f6-ffef53f7ebab-tmp-dir\") pod \"noten-model-69ef67-kserve-76579b859f-lbhp7\" (UID: \"aede103b-5c4f-423d-b4f6-ffef53f7ebab\") " pod="ai-tenant-e2e-noten-69ef67/noten-model-69ef67-kserve-76579b859f-lbhp7"
Sep 08 15:50:10.046726 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:50:10.046696    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aede103b-5c4f-423d-b4f6-ffef53f7ebab-home\") pod \"noten-model-69ef67-kserve-76579b859f-lbhp7\" (UID: \"aede103b-5c4f-423d-b4f6-ffef53f7ebab\") " pod="ai-tenant-e2e-noten-69ef67/noten-model-69ef67-kserve-76579b859f-lbhp7"
Sep 08 15:50:10.046813 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:50:10.046787    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aede103b-5c4f-423d-b4f6-ffef53f7ebab-model-cache\") pod \"noten-model-69ef67-kserve-76579b859f-lbhp7\" (UID: \"aede103b-5c4f-423d-b4f6-ffef53f7ebab\") " pod="ai-tenant-e2e-noten-69ef67/noten-model-69ef67-kserve-76579b859f-lbhp7"
Sep 08 15:50:10.046900 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:50:10.046826    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aede103b-5c4f-423d-b4f6-ffef53f7ebab-tmp-dir\") pod \"noten-model-69ef67-kserve-76579b859f-lbhp7\" (UID: \"aede103b-5c4f-423d-b4f6-ffef53f7ebab\") " pod="ai-tenant-e2e-noten-69ef67/noten-model-69ef67-kserve-76579b859f-lbhp7"
Sep 08 15:50:10.048506 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:50:10.048482    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aede103b-5c4f-423d-b4f6-ffef53f7ebab-dshm\") pod \"noten-model-69ef67-kserve-76579b859f-lbhp7\" (UID: \"aede103b-5c4f-423d-b4f6-ffef53f7ebab\") " pod="ai-tenant-e2e-noten-69ef67/noten-model-69ef67-kserve-76579b859f-lbhp7"
Sep 08 15:50:10.048774 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:50:10.048757    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aede103b-5c4f-423d-b4f6-ffef53f7ebab-tls-certs\") pod \"noten-model-69ef67-kserve-76579b859f-lbhp7\" (UID: \"aede103b-5c4f-423d-b4f6-ffef53f7ebab\") " pod="ai-tenant-e2e-noten-69ef67/noten-model-69ef67-kserve-76579b859f-lbhp7"
Sep 08 15:50:10.054504 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:50:10.054481    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fp5x\" (UniqueName: \"kubernetes.io/projected/aede103b-5c4f-423d-b4f6-ffef53f7ebab-kube-api-access-9fp5x\") pod \"noten-model-69ef67-kserve-76579b859f-lbhp7\" (UID: \"aede103b-5c4f-423d-b4f6-ffef53f7ebab\") " pod="ai-tenant-e2e-noten-69ef67/noten-model-69ef67-kserve-76579b859f-lbhp7"
Sep 08 15:50:10.133686 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:50:10.133604    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-69ef67/noten-model-69ef67-kserve-76579b859f-lbhp7"
Sep 08 15:50:10.403294 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:50:10.403265    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-69ef67/noten-model-69ef67-kserve-76579b859f-lbhp7"]
Sep 08 15:50:10.405325 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:50:10.405301    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaede103b_5c4f_423d_b4f6_ffef53f7ebab.slice/crio-cb79252fcd8b1cf7a9360313440065b44b7abc70cfd750ca425717e69b823fe6 WatchSource:0}: Error finding container cb79252fcd8b1cf7a9360313440065b44b7abc70cfd750ca425717e69b823fe6: Status 404 returned error can't find the container with id cb79252fcd8b1cf7a9360313440065b44b7abc70cfd750ca425717e69b823fe6
Sep 08 15:50:11.241176 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:50:11.241140    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-69ef67/noten-model-69ef67-kserve-76579b859f-lbhp7" event={"ID":"aede103b-5c4f-423d-b4f6-ffef53f7ebab","Type":"ContainerStarted","Data":"59722e26848b6689518032122eb5923f2c3987df228bab66509adbcab7df0d09"}
Sep 08 15:50:11.241176 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:50:11.241180    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-69ef67/noten-model-69ef67-kserve-76579b859f-lbhp7" event={"ID":"aede103b-5c4f-423d-b4f6-ffef53f7ebab","Type":"ContainerStarted","Data":"cb79252fcd8b1cf7a9360313440065b44b7abc70cfd750ca425717e69b823fe6"}
Sep 08 15:50:11.259032 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:50:11.258989    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-noten-69ef67/noten-model-69ef67-kserve-76579b859f-lbhp7" podStartSLOduration=2.258974337 podStartE2EDuration="2.258974337s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:50:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:50:11.257781424 +0000 UTC m=+1049.347232513" watchObservedRunningTime="2026-09-08 15:50:11.258974337 +0000 UTC m=+1049.348425425"
Sep 08 15:50:12.152185 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:50:12.152149    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-28e012/test-model-28e012-kserve-78f4c6b865-rt9hj"]
Sep 08 15:50:18.189104 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:50:18.189059    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-bbf4a143-7988bc7f7f-qc67s"]
Sep 08 15:50:18.189582 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:50:18.189334    2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bbf4a143-7988bc7f7f-qc67s" podUID="a122e782-92e0-4607-9bf0-35e76ad317b7" containerName="maas-api" containerID="cri-o://9e3f75c808e7812836748d51dfea9cf251c46520438dec062478bc7b4c1f7bf6" gracePeriod=30
Sep 08 15:50:18.433402 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:50:18.433373    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bbf4a143-7988bc7f7f-qc67s"
Sep 08 15:50:18.519662 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:50:18.519588    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a122e782-92e0-4607-9bf0-35e76ad317b7-maas-api-tls\") pod \"a122e782-92e0-4607-9bf0-35e76ad317b7\" (UID: \"a122e782-92e0-4607-9bf0-35e76ad317b7\") "
Sep 08 15:50:18.519662 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:50:18.519646    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a122e782-92e0-4607-9bf0-35e76ad317b7-openshift-service-ca\") pod \"a122e782-92e0-4607-9bf0-35e76ad317b7\" (UID: \"a122e782-92e0-4607-9bf0-35e76ad317b7\") "
Sep 08 15:50:18.519841 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:50:18.519711    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m75w8\" (UniqueName: \"kubernetes.io/projected/a122e782-92e0-4607-9bf0-35e76ad317b7-kube-api-access-m75w8\") pod \"a122e782-92e0-4607-9bf0-35e76ad317b7\" (UID: \"a122e782-92e0-4607-9bf0-35e76ad317b7\") "
Sep 08 15:50:18.519841 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:50:18.519740    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a122e782-92e0-4607-9bf0-35e76ad317b7-metrics-certs\") pod \"a122e782-92e0-4607-9bf0-35e76ad317b7\" (UID: \"a122e782-92e0-4607-9bf0-35e76ad317b7\") "
Sep 08 15:50:18.520093 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:50:18.520061    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a122e782-92e0-4607-9bf0-35e76ad317b7-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a122e782-92e0-4607-9bf0-35e76ad317b7" (UID: "a122e782-92e0-4607-9bf0-35e76ad317b7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 15:50:18.521757 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:50:18.521730    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a122e782-92e0-4607-9bf0-35e76ad317b7-kube-api-access-m75w8" (OuterVolumeSpecName: "kube-api-access-m75w8") pod "a122e782-92e0-4607-9bf0-35e76ad317b7" (UID: "a122e782-92e0-4607-9bf0-35e76ad317b7"). InnerVolumeSpecName "kube-api-access-m75w8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:50:18.521955 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:50:18.521751    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a122e782-92e0-4607-9bf0-35e76ad317b7-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a122e782-92e0-4607-9bf0-35e76ad317b7" (UID: "a122e782-92e0-4607-9bf0-35e76ad317b7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 15:50:18.521955 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:50:18.521887    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a122e782-92e0-4607-9bf0-35e76ad317b7-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "a122e782-92e0-4607-9bf0-35e76ad317b7" (UID: "a122e782-92e0-4607-9bf0-35e76ad317b7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 15:50:18.620473 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:50:18.620422    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m75w8\" (UniqueName: \"kubernetes.io/projected/a122e782-92e0-4607-9bf0-35e76ad317b7-kube-api-access-m75w8\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:50:18.620473 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:50:18.620470    2549 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a122e782-92e0-4607-9bf0-35e76ad317b7-metrics-certs\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:50:18.620473 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:50:18.620480    2549 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a122e782-92e0-4607-9bf0-35e76ad317b7-maas-api-tls\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:50:18.620753 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:50:18.620488    2549 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a122e782-92e0-4607-9bf0-35e76ad317b7-openshift-service-ca\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:50:19.281002 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:50:19.280963    2549 generic.go:356] "Generic (PLEG): container finished" podID="a122e782-92e0-4607-9bf0-35e76ad317b7" containerID="9e3f75c808e7812836748d51dfea9cf251c46520438dec062478bc7b4c1f7bf6" exitCode=0
Sep 08 15:50:19.281386 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:50:19.281027    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bbf4a143-7988bc7f7f-qc67s"
Sep 08 15:50:19.281386 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:50:19.281053    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bbf4a143-7988bc7f7f-qc67s" event={"ID":"a122e782-92e0-4607-9bf0-35e76ad317b7","Type":"ContainerDied","Data":"9e3f75c808e7812836748d51dfea9cf251c46520438dec062478bc7b4c1f7bf6"}
Sep 08 15:50:19.281386 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:50:19.281129    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bbf4a143-7988bc7f7f-qc67s" event={"ID":"a122e782-92e0-4607-9bf0-35e76ad317b7","Type":"ContainerDied","Data":"4e95c535505bf7f4f4dc057d35dff776ffe00a001f49eaef621e2c483a3ee3bc"}
Sep 08 15:50:19.281386 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:50:19.281153    2549 scope.go:117] "RemoveContainer" containerID="9e3f75c808e7812836748d51dfea9cf251c46520438dec062478bc7b4c1f7bf6"
Sep 08 15:50:19.290703 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:50:19.290686    2549 scope.go:117] "RemoveContainer" containerID="9e3f75c808e7812836748d51dfea9cf251c46520438dec062478bc7b4c1f7bf6"
Sep 08 15:50:19.290977 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:50:19.290959    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e3f75c808e7812836748d51dfea9cf251c46520438dec062478bc7b4c1f7bf6\": container with ID starting with 9e3f75c808e7812836748d51dfea9cf251c46520438dec062478bc7b4c1f7bf6 not found: ID does not exist" containerID="9e3f75c808e7812836748d51dfea9cf251c46520438dec062478bc7b4c1f7bf6"
Sep 08 15:50:19.291020 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:50:19.290987    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e3f75c808e7812836748d51dfea9cf251c46520438dec062478bc7b4c1f7bf6"} err="failed to get container status \"9e3f75c808e7812836748d51dfea9cf251c46520438dec062478bc7b4c1f7bf6\": rpc error: code = NotFound desc = could not find container \"9e3f75c808e7812836748d51dfea9cf251c46520438dec062478bc7b4c1f7bf6\": container with ID starting with 9e3f75c808e7812836748d51dfea9cf251c46520438dec062478bc7b4c1f7bf6 not found: ID does not exist"
Sep 08 15:50:19.313892 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:50:19.313844    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-bbf4a143-7988bc7f7f-qc67s"]
Sep 08 15:50:19.351562 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:50:19.351536    2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-bbf4a143-7988bc7f7f-qc67s"]
Sep 08 15:50:20.134566 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:50:20.134532    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-69ef67/noten-model-69ef67-kserve-76579b859f-lbhp7"
Sep 08 15:50:20.134566 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:50:20.134572    2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-noten-69ef67/noten-model-69ef67-kserve-76579b859f-lbhp7"
Sep 08 15:50:20.146824 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:50:20.146802    2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-noten-69ef67/noten-model-69ef67-kserve-76579b859f-lbhp7"
Sep 08 15:50:20.297799 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:50:20.297770    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-noten-69ef67/noten-model-69ef67-kserve-76579b859f-lbhp7"
Sep 08 15:50:20.400681 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:50:20.400649    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a122e782-92e0-4607-9bf0-35e76ad317b7" path="/var/lib/kubelet/pods/a122e782-92e0-4607-9bf0-35e76ad317b7/volumes"
Sep 08 15:50:27.173407 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:50:27.173350    2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-28e012/test-model-28e012-kserve-78f4c6b865-rt9hj" podUID="ed1cf67d-6716-497c-9400-2f061528d436" containerName="main" containerID="cri-o://91164caed0c542f8d05ef4b20fbf860222f7189d93fae027039daa5d9c1b9822" gracePeriod=45
Sep 08 15:50:27.321043 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:50:27.321014    2549 generic.go:356] "Generic (PLEG): container finished" podID="ed1cf67d-6716-497c-9400-2f061528d436" containerID="91164caed0c542f8d05ef4b20fbf860222f7189d93fae027039daa5d9c1b9822" exitCode=0
Sep 08 15:50:27.321199 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:50:27.321090    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-28e012/test-model-28e012-kserve-78f4c6b865-rt9hj" event={"ID":"ed1cf67d-6716-497c-9400-2f061528d436","Type":"ContainerDied","Data":"91164caed0c542f8d05ef4b20fbf860222f7189d93fae027039daa5d9c1b9822"}
Sep 08 15:50:27.416541 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:50:27.416518    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-28e012/test-model-28e012-kserve-78f4c6b865-rt9hj"
Sep 08 15:50:27.489757 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:50:27.489684    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed1cf67d-6716-497c-9400-2f061528d436-tls-certs\") pod \"ed1cf67d-6716-497c-9400-2f061528d436\" (UID: \"ed1cf67d-6716-497c-9400-2f061528d436\") "
Sep 08 15:50:27.489757 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:50:27.489718    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pdl58\" (UniqueName: \"kubernetes.io/projected/ed1cf67d-6716-497c-9400-2f061528d436-kube-api-access-pdl58\") pod \"ed1cf67d-6716-497c-9400-2f061528d436\" (UID: \"ed1cf67d-6716-497c-9400-2f061528d436\") "
Sep 08 15:50:27.489757 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:50:27.489748    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed1cf67d-6716-497c-9400-2f061528d436-tmp-dir\") pod \"ed1cf67d-6716-497c-9400-2f061528d436\" (UID: \"ed1cf67d-6716-497c-9400-2f061528d436\") "
Sep 08 15:50:27.490069 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:50:27.489765    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed1cf67d-6716-497c-9400-2f061528d436-model-cache\") pod \"ed1cf67d-6716-497c-9400-2f061528d436\" (UID: \"ed1cf67d-6716-497c-9400-2f061528d436\") "
Sep 08 15:50:27.490069 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:50:27.489811    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed1cf67d-6716-497c-9400-2f061528d436-home\") pod \"ed1cf67d-6716-497c-9400-2f061528d436\" (UID: \"ed1cf67d-6716-497c-9400-2f061528d436\") "
Sep 08 15:50:27.490069 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:50:27.489830    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed1cf67d-6716-497c-9400-2f061528d436-dshm\") pod \"ed1cf67d-6716-497c-9400-2f061528d436\" (UID: \"ed1cf67d-6716-497c-9400-2f061528d436\") "
Sep 08 15:50:27.490069 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:50:27.490046    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed1cf67d-6716-497c-9400-2f061528d436-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ed1cf67d-6716-497c-9400-2f061528d436" (UID: "ed1cf67d-6716-497c-9400-2f061528d436"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:50:27.490280 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:50:27.490115    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed1cf67d-6716-497c-9400-2f061528d436-model-cache" (OuterVolumeSpecName: "model-cache") pod "ed1cf67d-6716-497c-9400-2f061528d436" (UID: "ed1cf67d-6716-497c-9400-2f061528d436"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:50:27.490280 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:50:27.490131    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed1cf67d-6716-497c-9400-2f061528d436-home" (OuterVolumeSpecName: "home") pod "ed1cf67d-6716-497c-9400-2f061528d436" (UID: "ed1cf67d-6716-497c-9400-2f061528d436"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:50:27.490524 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:50:27.490499    2549 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed1cf67d-6716-497c-9400-2f061528d436-tmp-dir\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:50:27.490524 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:50:27.490525    2549 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed1cf67d-6716-497c-9400-2f061528d436-model-cache\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:50:27.490734 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:50:27.490539    2549 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed1cf67d-6716-497c-9400-2f061528d436-home\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:50:27.492101 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:50:27.492069    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed1cf67d-6716-497c-9400-2f061528d436-dshm" (OuterVolumeSpecName: "dshm") pod "ed1cf67d-6716-497c-9400-2f061528d436" (UID: "ed1cf67d-6716-497c-9400-2f061528d436"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:50:27.492101 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:50:27.492098    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed1cf67d-6716-497c-9400-2f061528d436-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ed1cf67d-6716-497c-9400-2f061528d436" (UID: "ed1cf67d-6716-497c-9400-2f061528d436"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 15:50:27.492240 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:50:27.492078    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed1cf67d-6716-497c-9400-2f061528d436-kube-api-access-pdl58" (OuterVolumeSpecName: "kube-api-access-pdl58") pod "ed1cf67d-6716-497c-9400-2f061528d436" (UID: "ed1cf67d-6716-497c-9400-2f061528d436"). InnerVolumeSpecName "kube-api-access-pdl58". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:50:27.591265 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:50:27.591223    2549 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed1cf67d-6716-497c-9400-2f061528d436-tls-certs\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:50:27.591265 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:50:27.591258    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pdl58\" (UniqueName: \"kubernetes.io/projected/ed1cf67d-6716-497c-9400-2f061528d436-kube-api-access-pdl58\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:50:27.591265 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:50:27.591267    2549 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed1cf67d-6716-497c-9400-2f061528d436-dshm\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:50:28.327208 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:50:28.327175    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-28e012/test-model-28e012-kserve-78f4c6b865-rt9hj"
Sep 08 15:50:28.327208 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:50:28.327192    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-28e012/test-model-28e012-kserve-78f4c6b865-rt9hj" event={"ID":"ed1cf67d-6716-497c-9400-2f061528d436","Type":"ContainerDied","Data":"fb7565d0cf4dac24e490c8c2c8a9ae5acacacf4cc92baaeca25a5da004e3ffe8"}
Sep 08 15:50:28.327711 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:50:28.327246    2549 scope.go:117] "RemoveContainer" containerID="91164caed0c542f8d05ef4b20fbf860222f7189d93fae027039daa5d9c1b9822"
Sep 08 15:50:28.347619 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:50:28.347593    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-28e012/test-model-28e012-kserve-78f4c6b865-rt9hj"]
Sep 08 15:50:28.350916 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:50:28.350897    2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-28e012/test-model-28e012-kserve-78f4c6b865-rt9hj"]
Sep 08 15:50:28.400422 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:50:28.400394    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ed1cf67d-6716-497c-9400-2f061528d436" path="/var/lib/kubelet/pods/ed1cf67d-6716-497c-9400-2f061528d436/volumes"
Sep 08 15:50:38.504584 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:50:38.504510    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-69ef67/noten-model-69ef67-kserve-76579b859f-lbhp7"]
Sep 08 15:50:39.009835 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:50:39.009804    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-be2aba-openshift-default-865869f8bd-ft79s"]
Sep 08 15:50:39.010181 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:50:39.010167    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed1cf67d-6716-497c-9400-2f061528d436" containerName="main"
Sep 08 15:50:39.010231 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:50:39.010186    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed1cf67d-6716-497c-9400-2f061528d436" containerName="main"
Sep 08 15:50:39.010231 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:50:39.010199    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a122e782-92e0-4607-9bf0-35e76ad317b7" containerName="maas-api"
Sep 08 15:50:39.010231 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:50:39.010206    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="a122e782-92e0-4607-9bf0-35e76ad317b7" containerName="maas-api"
Sep 08 15:50:39.010323 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:50:39.010273    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="a122e782-92e0-4607-9bf0-35e76ad317b7" containerName="maas-api"
Sep 08 15:50:39.010323 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:50:39.010285    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed1cf67d-6716-497c-9400-2f061528d436" containerName="main"
Sep 08 15:50:39.014699 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:50:39.014676    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-be2aba-openshift-default-865869f8bd-ft79s"
Sep 08 15:50:39.018004 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:50:39.017982    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-be2aba-openshift-default-dockercfg-bvrpr\""
Sep 08 15:50:39.026273 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:50:39.026254    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-be2aba-openshift-default-865869f8bd-ft79s"]
Sep 08 15:50:39.093438 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:50:39.093409    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a33f03d3-323b-47eb-8680-b69489b21e0b-istio-podinfo\") pod \"e2e-shared-a-w2-be2aba-openshift-default-865869f8bd-ft79s\" (UID: \"a33f03d3-323b-47eb-8680-b69489b21e0b\") " pod="openshift-ingress/e2e-shared-a-w2-be2aba-openshift-default-865869f8bd-ft79s"
Sep 08 15:50:39.093438 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:50:39.093437    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsqdc\" (UniqueName: \"kubernetes.io/projected/a33f03d3-323b-47eb-8680-b69489b21e0b-kube-api-access-wsqdc\") pod \"e2e-shared-a-w2-be2aba-openshift-default-865869f8bd-ft79s\" (UID: \"a33f03d3-323b-47eb-8680-b69489b21e0b\") " pod="openshift-ingress/e2e-shared-a-w2-be2aba-openshift-default-865869f8bd-ft79s"
Sep 08 15:50:39.093619 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:50:39.093463    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a33f03d3-323b-47eb-8680-b69489b21e0b-istiod-ca-cert\") pod \"e2e-shared-a-w2-be2aba-openshift-default-865869f8bd-ft79s\" (UID: \"a33f03d3-323b-47eb-8680-b69489b21e0b\") " pod="openshift-ingress/e2e-shared-a-w2-be2aba-openshift-default-865869f8bd-ft79s"
Sep 08 15:50:39.093619 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:50:39.093483    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a33f03d3-323b-47eb-8680-b69489b21e0b-istio-envoy\") pod \"e2e-shared-a-w2-be2aba-openshift-default-865869f8bd-ft79s\" (UID: \"a33f03d3-323b-47eb-8680-b69489b21e0b\") " pod="openshift-ingress/e2e-shared-a-w2-be2aba-openshift-default-865869f8bd-ft79s"
Sep 08 15:50:39.093619 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:50:39.093582    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a33f03d3-323b-47eb-8680-b69489b21e0b-credential-socket\") pod \"e2e-shared-a-w2-be2aba-openshift-default-865869f8bd-ft79s\" (UID: \"a33f03d3-323b-47eb-8680-b69489b21e0b\") " pod="openshift-ingress/e2e-shared-a-w2-be2aba-openshift-default-865869f8bd-ft79s"
Sep 08 15:50:39.093727 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:50:39.093631    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a33f03d3-323b-47eb-8680-b69489b21e0b-workload-socket\") pod \"e2e-shared-a-w2-be2aba-openshift-default-865869f8bd-ft79s\" (UID: \"a33f03d3-323b-47eb-8680-b69489b21e0b\") " pod="openshift-ingress/e2e-shared-a-w2-be2aba-openshift-default-865869f8bd-ft79s"
Sep 08 15:50:39.093727 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:50:39.093667    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a33f03d3-323b-47eb-8680-b69489b21e0b-istio-data\") pod \"e2e-shared-a-w2-be2aba-openshift-default-865869f8bd-ft79s\" (UID: \"a33f03d3-323b-47eb-8680-b69489b21e0b\") " pod="openshift-ingress/e2e-shared-a-w2-be2aba-openshift-default-865869f8bd-ft79s"
Sep 08 15:50:39.093727 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:50:39.093712    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a33f03d3-323b-47eb-8680-b69489b21e0b-istio-token\") pod \"e2e-shared-a-w2-be2aba-openshift-default-865869f8bd-ft79s\" (UID: \"a33f03d3-323b-47eb-8680-b69489b21e0b\") " pod="openshift-ingress/e2e-shared-a-w2-be2aba-openshift-default-865869f8bd-ft79s"
Sep 08 15:50:39.093830 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:50:39.093744    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a33f03d3-323b-47eb-8680-b69489b21e0b-workload-certs\") pod \"e2e-shared-a-w2-be2aba-openshift-default-865869f8bd-ft79s\" (UID: \"a33f03d3-323b-47eb-8680-b69489b21e0b\") " pod="openshift-ingress/e2e-shared-a-w2-be2aba-openshift-default-865869f8bd-ft79s"
Sep 08 15:50:39.194527 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:50:39.194489    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a33f03d3-323b-47eb-8680-b69489b21e0b-istiod-ca-cert\") pod \"e2e-shared-a-w2-be2aba-openshift-default-865869f8bd-ft79s\" (UID: \"a33f03d3-323b-47eb-8680-b69489b21e0b\") " pod="openshift-ingress/e2e-shared-a-w2-be2aba-openshift-default-865869f8bd-ft79s"
Sep 08 15:50:39.194527 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:50:39.194531    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a33f03d3-323b-47eb-8680-b69489b21e0b-istio-envoy\") pod \"e2e-shared-a-w2-be2aba-openshift-default-865869f8bd-ft79s\" (UID: \"a33f03d3-323b-47eb-8680-b69489b21e0b\") " pod="openshift-ingress/e2e-shared-a-w2-be2aba-openshift-default-865869f8bd-ft79s"
Sep 08 15:50:39.194781 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:50:39.194566    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a33f03d3-323b-47eb-8680-b69489b21e0b-credential-socket\") pod \"e2e-shared-a-w2-be2aba-openshift-default-865869f8bd-ft79s\" (UID: \"a33f03d3-323b-47eb-8680-b69489b21e0b\") " pod="openshift-ingress/e2e-shared-a-w2-be2aba-openshift-default-865869f8bd-ft79s"
Sep 08 15:50:39.194781 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:50:39.194598    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a33f03d3-323b-47eb-8680-b69489b21e0b-workload-socket\") pod \"e2e-shared-a-w2-be2aba-openshift-default-865869f8bd-ft79s\" (UID: \"a33f03d3-323b-47eb-8680-b69489b21e0b\") " pod="openshift-ingress/e2e-shared-a-w2-be2aba-openshift-default-865869f8bd-ft79s"
Sep 08 15:50:39.194781 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:50:39.194620    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a33f03d3-323b-47eb-8680-b69489b21e0b-istio-data\") pod \"e2e-shared-a-w2-be2aba-openshift-default-865869f8bd-ft79s\" (UID: \"a33f03d3-323b-47eb-8680-b69489b21e0b\") " pod="openshift-ingress/e2e-shared-a-w2-be2aba-openshift-default-865869f8bd-ft79s"
Sep 08 15:50:39.194781 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:50:39.194670    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a33f03d3-323b-47eb-8680-b69489b21e0b-istio-token\") pod \"e2e-shared-a-w2-be2aba-openshift-default-865869f8bd-ft79s\" (UID: \"a33f03d3-323b-47eb-8680-b69489b21e0b\") " pod="openshift-ingress/e2e-shared-a-w2-be2aba-openshift-default-865869f8bd-ft79s"
Sep 08 15:50:39.194781 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:50:39.194714    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a33f03d3-323b-47eb-8680-b69489b21e0b-workload-certs\") pod \"e2e-shared-a-w2-be2aba-openshift-default-865869f8bd-ft79s\" (UID: \"a33f03d3-323b-47eb-8680-b69489b21e0b\") " pod="openshift-ingress/e2e-shared-a-w2-be2aba-openshift-default-865869f8bd-ft79s"
Sep 08 15:50:39.194781 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:50:39.194747    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a33f03d3-323b-47eb-8680-b69489b21e0b-istio-podinfo\") pod \"e2e-shared-a-w2-be2aba-openshift-default-865869f8bd-ft79s\" (UID: \"a33f03d3-323b-47eb-8680-b69489b21e0b\") " pod="openshift-ingress/e2e-shared-a-w2-be2aba-openshift-default-865869f8bd-ft79s"
Sep 08 15:50:39.194781 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:50:39.194769    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsqdc\" (UniqueName: \"kubernetes.io/projected/a33f03d3-323b-47eb-8680-b69489b21e0b-kube-api-access-wsqdc\") pod \"e2e-shared-a-w2-be2aba-openshift-default-865869f8bd-ft79s\" (UID: \"a33f03d3-323b-47eb-8680-b69489b21e0b\") " pod="openshift-ingress/e2e-shared-a-w2-be2aba-openshift-default-865869f8bd-ft79s"
Sep 08 15:50:39.195181 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:50:39.195056    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a33f03d3-323b-47eb-8680-b69489b21e0b-credential-socket\") pod \"e2e-shared-a-w2-be2aba-openshift-default-865869f8bd-ft79s\" (UID: \"a33f03d3-323b-47eb-8680-b69489b21e0b\") " pod="openshift-ingress/e2e-shared-a-w2-be2aba-openshift-default-865869f8bd-ft79s"
Sep 08 15:50:39.195247 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:50:39.195203    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a33f03d3-323b-47eb-8680-b69489b21e0b-workload-certs\") pod \"e2e-shared-a-w2-be2aba-openshift-default-865869f8bd-ft79s\" (UID: \"a33f03d3-323b-47eb-8680-b69489b21e0b\") " pod="openshift-ingress/e2e-shared-a-w2-be2aba-openshift-default-865869f8bd-ft79s"
Sep 08 15:50:39.195247 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:50:39.195206    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a33f03d3-323b-47eb-8680-b69489b21e0b-istio-data\") pod \"e2e-shared-a-w2-be2aba-openshift-default-865869f8bd-ft79s\" (UID: \"a33f03d3-323b-47eb-8680-b69489b21e0b\") " pod="openshift-ingress/e2e-shared-a-w2-be2aba-openshift-default-865869f8bd-ft79s"
Sep 08 15:50:39.195363 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:50:39.195255    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a33f03d3-323b-47eb-8680-b69489b21e0b-workload-socket\") pod \"e2e-shared-a-w2-be2aba-openshift-default-865869f8bd-ft79s\" (UID: \"a33f03d3-323b-47eb-8680-b69489b21e0b\") " pod="openshift-ingress/e2e-shared-a-w2-be2aba-openshift-default-865869f8bd-ft79s"
Sep 08 15:50:39.195417 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:50:39.195379    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a33f03d3-323b-47eb-8680-b69489b21e0b-istiod-ca-cert\") pod \"e2e-shared-a-w2-be2aba-openshift-default-865869f8bd-ft79s\" (UID: \"a33f03d3-323b-47eb-8680-b69489b21e0b\") " pod="openshift-ingress/e2e-shared-a-w2-be2aba-openshift-default-865869f8bd-ft79s"
Sep 08 15:50:39.197078 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:50:39.197059    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a33f03d3-323b-47eb-8680-b69489b21e0b-istio-envoy\") pod \"e2e-shared-a-w2-be2aba-openshift-default-865869f8bd-ft79s\" (UID: \"a33f03d3-323b-47eb-8680-b69489b21e0b\") " pod="openshift-ingress/e2e-shared-a-w2-be2aba-openshift-default-865869f8bd-ft79s"
Sep 08 15:50:39.197287 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:50:39.197269    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a33f03d3-323b-47eb-8680-b69489b21e0b-istio-podinfo\") pod \"e2e-shared-a-w2-be2aba-openshift-default-865869f8bd-ft79s\" (UID: \"a33f03d3-323b-47eb-8680-b69489b21e0b\") " pod="openshift-ingress/e2e-shared-a-w2-be2aba-openshift-default-865869f8bd-ft79s"
Sep 08 15:50:39.203462 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:50:39.203441    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsqdc\" (UniqueName: \"kubernetes.io/projected/a33f03d3-323b-47eb-8680-b69489b21e0b-kube-api-access-wsqdc\") pod \"e2e-shared-a-w2-be2aba-openshift-default-865869f8bd-ft79s\" (UID: \"a33f03d3-323b-47eb-8680-b69489b21e0b\") " pod="openshift-ingress/e2e-shared-a-w2-be2aba-openshift-default-865869f8bd-ft79s"
Sep 08 15:50:39.203614 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:50:39.203597    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a33f03d3-323b-47eb-8680-b69489b21e0b-istio-token\") pod \"e2e-shared-a-w2-be2aba-openshift-default-865869f8bd-ft79s\" (UID: \"a33f03d3-323b-47eb-8680-b69489b21e0b\") " pod="openshift-ingress/e2e-shared-a-w2-be2aba-openshift-default-865869f8bd-ft79s"
Sep 08 15:50:39.324720 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:50:39.324664    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-be2aba-openshift-default-865869f8bd-ft79s"
Sep 08 15:50:39.606564 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:50:39.606541    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-be2aba-openshift-default-865869f8bd-ft79s"]
Sep 08 15:50:39.608757 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:50:39.608729    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda33f03d3_323b_47eb_8680_b69489b21e0b.slice/crio-0562f4bbd843dd0ea9b0435ca78e341c669c0edde3d492f334d0e6d547dbebf0 WatchSource:0}: Error finding container 0562f4bbd843dd0ea9b0435ca78e341c669c0edde3d492f334d0e6d547dbebf0: Status 404 returned error can't find the container with id 0562f4bbd843dd0ea9b0435ca78e341c669c0edde3d492f334d0e6d547dbebf0
Sep 08 15:50:39.611048 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:50:39.611013    2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 08 15:50:39.611153 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:50:39.611078    2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 08 15:50:39.611153 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:50:39.611106    2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 08 15:50:40.388391 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:50:40.388360    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-be2aba-openshift-default-865869f8bd-ft79s" event={"ID":"a33f03d3-323b-47eb-8680-b69489b21e0b","Type":"ContainerStarted","Data":"999446d0e3799c5a9701374773c3faf79d7d475dcc51fd5401f2282c1a433b17"}
Sep 08 15:50:40.388596 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:50:40.388416    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-be2aba-openshift-default-865869f8bd-ft79s" event={"ID":"a33f03d3-323b-47eb-8680-b69489b21e0b","Type":"ContainerStarted","Data":"0562f4bbd843dd0ea9b0435ca78e341c669c0edde3d492f334d0e6d547dbebf0"}
Sep 08 15:50:40.412338 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:50:40.412291    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-be2aba-openshift-default-865869f8bd-ft79s" podStartSLOduration=2.412273862 podStartE2EDuration="2.412273862s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:50:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:50:40.411068388 +0000 UTC m=+1078.500519479" watchObservedRunningTime="2026-09-08 15:50:40.412273862 +0000 UTC m=+1078.501724950"
Sep 08 15:50:41.324819 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:50:41.324784    2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-be2aba-openshift-default-865869f8bd-ft79s"
Sep 08 15:50:41.329579 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:50:41.329555    2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-be2aba-openshift-default-865869f8bd-ft79s"
Sep 08 15:50:41.394290 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:50:41.394259    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-be2aba-openshift-default-865869f8bd-ft79s"
Sep 08 15:50:41.395073 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:50:41.395055    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-be2aba-openshift-default-865869f8bd-ft79s"
Sep 08 15:50:53.524629 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:50:53.524566    2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-noten-69ef67/noten-model-69ef67-kserve-76579b859f-lbhp7" podUID="aede103b-5c4f-423d-b4f6-ffef53f7ebab" containerName="main" containerID="cri-o://59722e26848b6689518032122eb5923f2c3987df228bab66509adbcab7df0d09" gracePeriod=45
Sep 08 15:50:53.813395 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:50:53.813370    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-69ef67/noten-model-69ef67-kserve-76579b859f-lbhp7"
Sep 08 15:50:53.927394 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:50:53.927360    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aede103b-5c4f-423d-b4f6-ffef53f7ebab-home\") pod \"aede103b-5c4f-423d-b4f6-ffef53f7ebab\" (UID: \"aede103b-5c4f-423d-b4f6-ffef53f7ebab\") "
Sep 08 15:50:53.927586 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:50:53.927430    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aede103b-5c4f-423d-b4f6-ffef53f7ebab-model-cache\") pod \"aede103b-5c4f-423d-b4f6-ffef53f7ebab\" (UID: \"aede103b-5c4f-423d-b4f6-ffef53f7ebab\") "
Sep 08 15:50:53.927586 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:50:53.927451    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aede103b-5c4f-423d-b4f6-ffef53f7ebab-tmp-dir\") pod \"aede103b-5c4f-423d-b4f6-ffef53f7ebab\" (UID: \"aede103b-5c4f-423d-b4f6-ffef53f7ebab\") "
Sep 08 15:50:53.927586 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:50:53.927478    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9fp5x\" (UniqueName: \"kubernetes.io/projected/aede103b-5c4f-423d-b4f6-ffef53f7ebab-kube-api-access-9fp5x\") pod \"aede103b-5c4f-423d-b4f6-ffef53f7ebab\" (UID: \"aede103b-5c4f-423d-b4f6-ffef53f7ebab\") "
Sep 08 15:50:53.927586 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:50:53.927520    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aede103b-5c4f-423d-b4f6-ffef53f7ebab-tls-certs\") pod \"aede103b-5c4f-423d-b4f6-ffef53f7ebab\" (UID: \"aede103b-5c4f-423d-b4f6-ffef53f7ebab\") "
Sep 08 15:50:53.927586 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:50:53.927570    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aede103b-5c4f-423d-b4f6-ffef53f7ebab-dshm\") pod \"aede103b-5c4f-423d-b4f6-ffef53f7ebab\" (UID: \"aede103b-5c4f-423d-b4f6-ffef53f7ebab\") "
Sep 08 15:50:53.927898 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:50:53.927670    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aede103b-5c4f-423d-b4f6-ffef53f7ebab-model-cache" (OuterVolumeSpecName: "model-cache") pod "aede103b-5c4f-423d-b4f6-ffef53f7ebab" (UID: "aede103b-5c4f-423d-b4f6-ffef53f7ebab"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:50:53.927898 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:50:53.927714    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aede103b-5c4f-423d-b4f6-ffef53f7ebab-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "aede103b-5c4f-423d-b4f6-ffef53f7ebab" (UID: "aede103b-5c4f-423d-b4f6-ffef53f7ebab"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:50:53.927898 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:50:53.927873    2549 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aede103b-5c4f-423d-b4f6-ffef53f7ebab-model-cache\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:50:53.927898 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:50:53.927887    2549 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aede103b-5c4f-423d-b4f6-ffef53f7ebab-tmp-dir\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:50:53.928177 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:50:53.928155    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aede103b-5c4f-423d-b4f6-ffef53f7ebab-home" (OuterVolumeSpecName: "home") pod "aede103b-5c4f-423d-b4f6-ffef53f7ebab" (UID: "aede103b-5c4f-423d-b4f6-ffef53f7ebab"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:50:53.929785 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:50:53.929764    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aede103b-5c4f-423d-b4f6-ffef53f7ebab-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "aede103b-5c4f-423d-b4f6-ffef53f7ebab" (UID: "aede103b-5c4f-423d-b4f6-ffef53f7ebab"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 15:50:53.929911 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:50:53.929891    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aede103b-5c4f-423d-b4f6-ffef53f7ebab-kube-api-access-9fp5x" (OuterVolumeSpecName: "kube-api-access-9fp5x") pod "aede103b-5c4f-423d-b4f6-ffef53f7ebab" (UID: "aede103b-5c4f-423d-b4f6-ffef53f7ebab"). InnerVolumeSpecName "kube-api-access-9fp5x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:50:53.931258 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:50:53.931239    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aede103b-5c4f-423d-b4f6-ffef53f7ebab-dshm" (OuterVolumeSpecName: "dshm") pod "aede103b-5c4f-423d-b4f6-ffef53f7ebab" (UID: "aede103b-5c4f-423d-b4f6-ffef53f7ebab"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:50:54.028445 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:50:54.028418    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9fp5x\" (UniqueName: \"kubernetes.io/projected/aede103b-5c4f-423d-b4f6-ffef53f7ebab-kube-api-access-9fp5x\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:50:54.028445 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:50:54.028442    2549 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aede103b-5c4f-423d-b4f6-ffef53f7ebab-tls-certs\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:50:54.028640 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:50:54.028455    2549 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aede103b-5c4f-423d-b4f6-ffef53f7ebab-dshm\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:50:54.028640 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:50:54.028463    2549 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aede103b-5c4f-423d-b4f6-ffef53f7ebab-home\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:50:54.471060 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:50:54.471023    2549 generic.go:356] "Generic (PLEG): container finished" podID="aede103b-5c4f-423d-b4f6-ffef53f7ebab" containerID="59722e26848b6689518032122eb5923f2c3987df228bab66509adbcab7df0d09" exitCode=0
Sep 08 15:50:54.471369 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:50:54.471093    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-69ef67/noten-model-69ef67-kserve-76579b859f-lbhp7"
Sep 08 15:50:54.471606 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:50:54.471120    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-69ef67/noten-model-69ef67-kserve-76579b859f-lbhp7" event={"ID":"aede103b-5c4f-423d-b4f6-ffef53f7ebab","Type":"ContainerDied","Data":"59722e26848b6689518032122eb5923f2c3987df228bab66509adbcab7df0d09"}
Sep 08 15:50:54.471778 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:50:54.471761    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-69ef67/noten-model-69ef67-kserve-76579b859f-lbhp7" event={"ID":"aede103b-5c4f-423d-b4f6-ffef53f7ebab","Type":"ContainerDied","Data":"cb79252fcd8b1cf7a9360313440065b44b7abc70cfd750ca425717e69b823fe6"}
Sep 08 15:50:54.471906 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:50:54.471893    2549 scope.go:117] "RemoveContainer" containerID="59722e26848b6689518032122eb5923f2c3987df228bab66509adbcab7df0d09"
Sep 08 15:50:54.481252 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:50:54.481235    2549 scope.go:117] "RemoveContainer" containerID="59722e26848b6689518032122eb5923f2c3987df228bab66509adbcab7df0d09"
Sep 08 15:50:54.481615 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:50:54.481594    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59722e26848b6689518032122eb5923f2c3987df228bab66509adbcab7df0d09\": container with ID starting with 59722e26848b6689518032122eb5923f2c3987df228bab66509adbcab7df0d09 not found: ID does not exist" containerID="59722e26848b6689518032122eb5923f2c3987df228bab66509adbcab7df0d09"
Sep 08 15:50:54.481658 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:50:54.481622    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59722e26848b6689518032122eb5923f2c3987df228bab66509adbcab7df0d09"} err="failed to get container status \"59722e26848b6689518032122eb5923f2c3987df228bab66509adbcab7df0d09\": rpc error: code = NotFound desc = could not find container \"59722e26848b6689518032122eb5923f2c3987df228bab66509adbcab7df0d09\": container with ID starting with 59722e26848b6689518032122eb5923f2c3987df228bab66509adbcab7df0d09 not found: ID does not exist"
Sep 08 15:50:54.489832 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:50:54.489810    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-69ef67/noten-model-69ef67-kserve-76579b859f-lbhp7"]
Sep 08 15:50:54.494419 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:50:54.494399    2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-69ef67/noten-model-69ef67-kserve-76579b859f-lbhp7"]
Sep 08 15:50:56.401870 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:50:56.401818    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aede103b-5c4f-423d-b4f6-ffef53f7ebab" path="/var/lib/kubelet/pods/aede103b-5c4f-423d-b4f6-ffef53f7ebab/volumes"
Sep 08 15:50:57.436017 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:50:57.435980    2549 patch_prober.go:28] interesting pod/e2e-shared-a-w2-be2aba-openshift-default-865869f8bd-ft79s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 08 15:50:57.436400 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:50:57.436040    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-be2aba-openshift-default-865869f8bd-ft79s" podUID="a33f03d3-323b-47eb-8680-b69489b21e0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 08 15:50:59.497222 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:50:59.497189    2549 generic.go:356] "Generic (PLEG): container finished" podID="a33f03d3-323b-47eb-8680-b69489b21e0b" containerID="999446d0e3799c5a9701374773c3faf79d7d475dcc51fd5401f2282c1a433b17" exitCode=0
Sep 08 15:50:59.497637 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:50:59.497263    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-be2aba-openshift-default-865869f8bd-ft79s" event={"ID":"a33f03d3-323b-47eb-8680-b69489b21e0b","Type":"ContainerDied","Data":"999446d0e3799c5a9701374773c3faf79d7d475dcc51fd5401f2282c1a433b17"}
Sep 08 15:50:59.497680 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:50:59.497639    2549 scope.go:117] "RemoveContainer" containerID="999446d0e3799c5a9701374773c3faf79d7d475dcc51fd5401f2282c1a433b17"
Sep 08 15:50:59.498690 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:50:59.498655    2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 08 15:50:59.498791 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:50:59.498730    2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 08 15:50:59.498791 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:50:59.498782    2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 08 15:51:00.503958 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:00.503927    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-be2aba-openshift-default-865869f8bd-ft79s" event={"ID":"a33f03d3-323b-47eb-8680-b69489b21e0b","Type":"ContainerStarted","Data":"7e07472ad3cfbabab9e72da2987b5a57753e726e884f1c16f85a76c2a62b9cf9"}
Sep 08 15:51:01.324991 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:01.324959    2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-be2aba-openshift-default-865869f8bd-ft79s"
Sep 08 15:51:01.325164 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:01.325134    2549 patch_prober.go:28] interesting pod/e2e-shared-a-w2-be2aba-openshift-default-865869f8bd-ft79s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body=
Sep 08 15:51:01.325211 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:01.325187    2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-be2aba-openshift-default-865869f8bd-ft79s" podUID="a33f03d3-323b-47eb-8680-b69489b21e0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused"
Sep 08 15:51:02.325193 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:02.325153    2549 patch_prober.go:28] interesting pod/e2e-shared-a-w2-be2aba-openshift-default-865869f8bd-ft79s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body=
Sep 08 15:51:02.325573 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:02.325209    2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-be2aba-openshift-default-865869f8bd-ft79s" podUID="a33f03d3-323b-47eb-8680-b69489b21e0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused"
Sep 08 15:51:03.325555 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:03.325520    2549 patch_prober.go:28] interesting pod/e2e-shared-a-w2-be2aba-openshift-default-865869f8bd-ft79s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body=
Sep 08 15:51:03.325951 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:03.325584    2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-be2aba-openshift-default-865869f8bd-ft79s" podUID="a33f03d3-323b-47eb-8680-b69489b21e0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused"
Sep 08 15:51:04.325630 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:04.325591    2549 patch_prober.go:28] interesting pod/e2e-shared-a-w2-be2aba-openshift-default-865869f8bd-ft79s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body=
Sep 08 15:51:04.326034 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:04.325649    2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-be2aba-openshift-default-865869f8bd-ft79s" podUID="a33f03d3-323b-47eb-8680-b69489b21e0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused"
Sep 08 15:51:05.325232 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:05.325194    2549 patch_prober.go:28] interesting pod/e2e-shared-a-w2-be2aba-openshift-default-865869f8bd-ft79s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body=
Sep 08 15:51:05.325419 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:05.325249    2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-be2aba-openshift-default-865869f8bd-ft79s" podUID="a33f03d3-323b-47eb-8680-b69489b21e0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused"
Sep 08 15:51:05.977138 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:05.977099    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-a057c5-78b7f5fbfd-kdgcz"]
Sep 08 15:51:05.977592 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:05.977332    2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a057c5-78b7f5fbfd-kdgcz" podUID="6c67f9b6-d044-475a-a032-c869b4a8c158" containerName="maas-api" containerID="cri-o://7359d0ca2f80d4ff3e01841cdc8466be566b583fc4c30ddc38f4c40e3f964630" gracePeriod=30
Sep 08 15:51:06.330253 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:06.330219    2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-be2aba-openshift-default-865869f8bd-ft79s"
Sep 08 15:51:06.330614 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:06.330594    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-be2aba-openshift-default-865869f8bd-ft79s"
Sep 08 15:51:06.331413 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:06.331393    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-be2aba-openshift-default-865869f8bd-ft79s"
Sep 08 15:51:06.443490 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:06.443460    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a057c5-78b7f5fbfd-kdgcz"
Sep 08 15:51:06.532849 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:06.532755    2549 generic.go:356] "Generic (PLEG): container finished" podID="6c67f9b6-d044-475a-a032-c869b4a8c158" containerID="7359d0ca2f80d4ff3e01841cdc8466be566b583fc4c30ddc38f4c40e3f964630" exitCode=0
Sep 08 15:51:06.532849 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:06.532801    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a057c5-78b7f5fbfd-kdgcz"
Sep 08 15:51:06.532849 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:06.532837    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a057c5-78b7f5fbfd-kdgcz" event={"ID":"6c67f9b6-d044-475a-a032-c869b4a8c158","Type":"ContainerDied","Data":"7359d0ca2f80d4ff3e01841cdc8466be566b583fc4c30ddc38f4c40e3f964630"}
Sep 08 15:51:06.533143 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:06.532899    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a057c5-78b7f5fbfd-kdgcz" event={"ID":"6c67f9b6-d044-475a-a032-c869b4a8c158","Type":"ContainerDied","Data":"6d5b4dad8892ec5a2a59f1e1c8b0673674f9282e07e0a29ebec635dd2caed906"}
Sep 08 15:51:06.533143 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:06.532923    2549 scope.go:117] "RemoveContainer" containerID="7359d0ca2f80d4ff3e01841cdc8466be566b583fc4c30ddc38f4c40e3f964630"
Sep 08 15:51:06.541244 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:06.541226    2549 scope.go:117] "RemoveContainer" containerID="7359d0ca2f80d4ff3e01841cdc8466be566b583fc4c30ddc38f4c40e3f964630"
Sep 08 15:51:06.541496 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:51:06.541478    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7359d0ca2f80d4ff3e01841cdc8466be566b583fc4c30ddc38f4c40e3f964630\": container with ID starting with 7359d0ca2f80d4ff3e01841cdc8466be566b583fc4c30ddc38f4c40e3f964630 not found: ID does not exist" containerID="7359d0ca2f80d4ff3e01841cdc8466be566b583fc4c30ddc38f4c40e3f964630"
Sep 08 15:51:06.541559 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:06.541503    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7359d0ca2f80d4ff3e01841cdc8466be566b583fc4c30ddc38f4c40e3f964630"} err="failed to get container status \"7359d0ca2f80d4ff3e01841cdc8466be566b583fc4c30ddc38f4c40e3f964630\": rpc error: code = NotFound desc = could not find container \"7359d0ca2f80d4ff3e01841cdc8466be566b583fc4c30ddc38f4c40e3f964630\": container with ID starting with 7359d0ca2f80d4ff3e01841cdc8466be566b583fc4c30ddc38f4c40e3f964630 not found: ID does not exist"
Sep 08 15:51:06.543769 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:06.543753    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c67f9b6-d044-475a-a032-c869b4a8c158-metrics-certs\") pod \"6c67f9b6-d044-475a-a032-c869b4a8c158\" (UID: \"6c67f9b6-d044-475a-a032-c869b4a8c158\") "
Sep 08 15:51:06.543830 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:06.543809    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c67f9b6-d044-475a-a032-c869b4a8c158-openshift-service-ca\") pod \"6c67f9b6-d044-475a-a032-c869b4a8c158\" (UID: \"6c67f9b6-d044-475a-a032-c869b4a8c158\") "
Sep 08 15:51:06.543830 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:06.543827    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-stpk2\" (UniqueName: \"kubernetes.io/projected/6c67f9b6-d044-475a-a032-c869b4a8c158-kube-api-access-stpk2\") pod \"6c67f9b6-d044-475a-a032-c869b4a8c158\" (UID: \"6c67f9b6-d044-475a-a032-c869b4a8c158\") "
Sep 08 15:51:06.543959 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:06.543850    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c67f9b6-d044-475a-a032-c869b4a8c158-maas-api-tls\") pod \"6c67f9b6-d044-475a-a032-c869b4a8c158\" (UID: \"6c67f9b6-d044-475a-a032-c869b4a8c158\") "
Sep 08 15:51:06.544265 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:06.544241    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c67f9b6-d044-475a-a032-c869b4a8c158-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6c67f9b6-d044-475a-a032-c869b4a8c158" (UID: "6c67f9b6-d044-475a-a032-c869b4a8c158"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 15:51:06.545886 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:06.545844    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c67f9b6-d044-475a-a032-c869b4a8c158-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6c67f9b6-d044-475a-a032-c869b4a8c158" (UID: "6c67f9b6-d044-475a-a032-c869b4a8c158"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 15:51:06.546314 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:06.546294    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c67f9b6-d044-475a-a032-c869b4a8c158-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "6c67f9b6-d044-475a-a032-c869b4a8c158" (UID: "6c67f9b6-d044-475a-a032-c869b4a8c158"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 15:51:06.546380 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:06.546310    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c67f9b6-d044-475a-a032-c869b4a8c158-kube-api-access-stpk2" (OuterVolumeSpecName: "kube-api-access-stpk2") pod "6c67f9b6-d044-475a-a032-c869b4a8c158" (UID: "6c67f9b6-d044-475a-a032-c869b4a8c158"). InnerVolumeSpecName "kube-api-access-stpk2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:51:06.645420 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:06.645385    2549 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c67f9b6-d044-475a-a032-c869b4a8c158-openshift-service-ca\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:51:06.645598 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:06.645429    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-stpk2\" (UniqueName: \"kubernetes.io/projected/6c67f9b6-d044-475a-a032-c869b4a8c158-kube-api-access-stpk2\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:51:06.645598 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:06.645458    2549 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c67f9b6-d044-475a-a032-c869b4a8c158-maas-api-tls\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:51:06.645598 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:06.645472    2549 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c67f9b6-d044-475a-a032-c869b4a8c158-metrics-certs\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:51:06.854560 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:06.854529    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-a057c5-78b7f5fbfd-kdgcz"]
Sep 08 15:51:06.859344 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:06.859317    2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-a057c5-78b7f5fbfd-kdgcz"]
Sep 08 15:51:08.400252 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:08.400219    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6c67f9b6-d044-475a-a032-c869b4a8c158" path="/var/lib/kubelet/pods/6c67f9b6-d044-475a-a032-c869b4a8c158/volumes"
Sep 08 15:51:11.944708 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:11.944677    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-0ac386-5b555b7776-xz5xw"]
Sep 08 15:51:11.945100 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:11.945039    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c67f9b6-d044-475a-a032-c869b4a8c158" containerName="maas-api"
Sep 08 15:51:11.945100 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:11.945051    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c67f9b6-d044-475a-a032-c869b4a8c158" containerName="maas-api"
Sep 08 15:51:11.945100 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:11.945064    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aede103b-5c4f-423d-b4f6-ffef53f7ebab" containerName="main"
Sep 08 15:51:11.945100 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:11.945069    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="aede103b-5c4f-423d-b4f6-ffef53f7ebab" containerName="main"
Sep 08 15:51:11.945234 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:11.945122    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="aede103b-5c4f-423d-b4f6-ffef53f7ebab" containerName="main"
Sep 08 15:51:11.945234 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:11.945130    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c67f9b6-d044-475a-a032-c869b4a8c158" containerName="maas-api"
Sep 08 15:51:11.948118 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:11.948099    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-0ac386-5b555b7776-xz5xw"
Sep 08 15:51:11.951136 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:11.951099    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-0ac386-serving-cert\""
Sep 08 15:51:11.951250 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:11.951158    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 08 15:51:11.951301 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:11.951245    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 08 15:51:11.952314 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:11.952294    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 08 15:51:11.952399 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:11.952294    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-tq4f4\""
Sep 08 15:51:11.958735 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:11.958701    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-0ac386-5b555b7776-xz5xw"]
Sep 08 15:51:12.094827 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:12.094555    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7518a6a7-c124-4aa6-beb2-8b5840c23e3c-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-0ac386-5b555b7776-xz5xw\" (UID: \"7518a6a7-c124-4aa6-beb2-8b5840c23e3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-0ac386-5b555b7776-xz5xw"
Sep 08 15:51:12.094827 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:12.094615    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7518a6a7-c124-4aa6-beb2-8b5840c23e3c-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-0ac386-5b555b7776-xz5xw\" (UID: \"7518a6a7-c124-4aa6-beb2-8b5840c23e3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-0ac386-5b555b7776-xz5xw"
Sep 08 15:51:12.094827 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:12.094681    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpbtd\" (UniqueName: \"kubernetes.io/projected/7518a6a7-c124-4aa6-beb2-8b5840c23e3c-kube-api-access-hpbtd\") pod \"maas-api-e2e-shared-b-w2-0ac386-5b555b7776-xz5xw\" (UID: \"7518a6a7-c124-4aa6-beb2-8b5840c23e3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-0ac386-5b555b7776-xz5xw"
Sep 08 15:51:12.094827 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:12.094733    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7518a6a7-c124-4aa6-beb2-8b5840c23e3c-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-0ac386-5b555b7776-xz5xw\" (UID: \"7518a6a7-c124-4aa6-beb2-8b5840c23e3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-0ac386-5b555b7776-xz5xw"
Sep 08 15:51:12.195830 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:12.195749    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7518a6a7-c124-4aa6-beb2-8b5840c23e3c-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-0ac386-5b555b7776-xz5xw\" (UID: \"7518a6a7-c124-4aa6-beb2-8b5840c23e3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-0ac386-5b555b7776-xz5xw"
Sep 08 15:51:12.195830 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:12.195788    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7518a6a7-c124-4aa6-beb2-8b5840c23e3c-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-0ac386-5b555b7776-xz5xw\" (UID: \"7518a6a7-c124-4aa6-beb2-8b5840c23e3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-0ac386-5b555b7776-xz5xw"
Sep 08 15:51:12.195830 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:12.195823    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpbtd\" (UniqueName: \"kubernetes.io/projected/7518a6a7-c124-4aa6-beb2-8b5840c23e3c-kube-api-access-hpbtd\") pod \"maas-api-e2e-shared-b-w2-0ac386-5b555b7776-xz5xw\" (UID: \"7518a6a7-c124-4aa6-beb2-8b5840c23e3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-0ac386-5b555b7776-xz5xw"
Sep 08 15:51:12.196102 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:12.195872    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7518a6a7-c124-4aa6-beb2-8b5840c23e3c-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-0ac386-5b555b7776-xz5xw\" (UID: \"7518a6a7-c124-4aa6-beb2-8b5840c23e3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-0ac386-5b555b7776-xz5xw"
Sep 08 15:51:12.196102 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:51:12.195988    2549 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-0ac386-serving-cert: secret "maas-api-e2e-shared-b-w2-0ac386-serving-cert" not found
Sep 08 15:51:12.196102 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:51:12.196042    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7518a6a7-c124-4aa6-beb2-8b5840c23e3c-maas-api-tls podName:7518a6a7-c124-4aa6-beb2-8b5840c23e3c nodeName:}" failed. No retries permitted until 2026-09-08 15:51:12.696027063 +0000 UTC m=+1110.785478130 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7518a6a7-c124-4aa6-beb2-8b5840c23e3c-maas-api-tls") pod "maas-api-e2e-shared-b-w2-0ac386-5b555b7776-xz5xw" (UID: "7518a6a7-c124-4aa6-beb2-8b5840c23e3c") : secret "maas-api-e2e-shared-b-w2-0ac386-serving-cert" not found
Sep 08 15:51:12.196472 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:12.196444    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7518a6a7-c124-4aa6-beb2-8b5840c23e3c-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-0ac386-5b555b7776-xz5xw\" (UID: \"7518a6a7-c124-4aa6-beb2-8b5840c23e3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-0ac386-5b555b7776-xz5xw"
Sep 08 15:51:12.198167 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:12.198149    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7518a6a7-c124-4aa6-beb2-8b5840c23e3c-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-0ac386-5b555b7776-xz5xw\" (UID: \"7518a6a7-c124-4aa6-beb2-8b5840c23e3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-0ac386-5b555b7776-xz5xw"
Sep 08 15:51:12.204907 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:12.204887    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpbtd\" (UniqueName: \"kubernetes.io/projected/7518a6a7-c124-4aa6-beb2-8b5840c23e3c-kube-api-access-hpbtd\") pod \"maas-api-e2e-shared-b-w2-0ac386-5b555b7776-xz5xw\" (UID: \"7518a6a7-c124-4aa6-beb2-8b5840c23e3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-0ac386-5b555b7776-xz5xw"
Sep 08 15:51:12.700327 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:12.699992    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7518a6a7-c124-4aa6-beb2-8b5840c23e3c-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-0ac386-5b555b7776-xz5xw\" (UID: \"7518a6a7-c124-4aa6-beb2-8b5840c23e3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-0ac386-5b555b7776-xz5xw"
Sep 08 15:51:12.703254 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:12.703214    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7518a6a7-c124-4aa6-beb2-8b5840c23e3c-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-0ac386-5b555b7776-xz5xw\" (UID: \"7518a6a7-c124-4aa6-beb2-8b5840c23e3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-0ac386-5b555b7776-xz5xw"
Sep 08 15:51:12.861660 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:12.861634    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-0ac386-5b555b7776-xz5xw"
Sep 08 15:51:13.141560 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:13.141535    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-0ac386-5b555b7776-xz5xw"]
Sep 08 15:51:13.143307 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:51:13.143280    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7518a6a7_c124_4aa6_beb2_8b5840c23e3c.slice/crio-aa9d319aab059e04121baffb7479aa35a8927fcb02886ac4585155ada0854827 WatchSource:0}: Error finding container aa9d319aab059e04121baffb7479aa35a8927fcb02886ac4585155ada0854827: Status 404 returned error can't find the container with id aa9d319aab059e04121baffb7479aa35a8927fcb02886ac4585155ada0854827
Sep 08 15:51:13.144638 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:13.144618    2549 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 08 15:51:13.571564 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:13.571529    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-0ac386-5b555b7776-xz5xw" event={"ID":"7518a6a7-c124-4aa6-beb2-8b5840c23e3c","Type":"ContainerStarted","Data":"aa9d319aab059e04121baffb7479aa35a8927fcb02886ac4585155ada0854827"}
Sep 08 15:51:13.773725 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:13.773692    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-a057c5-openshift-default-5f564fb8cd-2mflt"]
Sep 08 15:51:13.773922 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:13.773901    2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-a057c5-openshift-default-5f564fb8cd-2mflt" podUID="55c30a1d-0943-4733-9a53-ed49d3d4184d" containerName="istio-proxy" containerID="cri-o://2783d7dbbc01842411bd24980061e88bd7f0392b36ddd78aa81fb76724d495c3" gracePeriod=30
Sep 08 15:51:14.577663 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:14.577615    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-0ac386-5b555b7776-xz5xw" event={"ID":"7518a6a7-c124-4aa6-beb2-8b5840c23e3c","Type":"ContainerStarted","Data":"722a72e22980b8d15ea2f1e916d4132a404e76e4214f08ef6a16df182343bcc6"}
Sep 08 15:51:14.578114 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:14.577697    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-0ac386-5b555b7776-xz5xw"
Sep 08 15:51:14.603230 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:14.603180    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-0ac386-5b555b7776-xz5xw" podStartSLOduration=2.905099248 podStartE2EDuration="3.603164496s" totalImagesPullingTime="698.065248ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:51:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:51:14.601611987 +0000 UTC m=+1112.691063077" watchObservedRunningTime="2026-09-08 15:51:14.603164496 +0000 UTC m=+1112.692615720"
Sep 08 15:51:19.027137 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:19.027103    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-a057c5-openshift-default-5f564fb8cd-2mflt"
Sep 08 15:51:19.047977 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:19.047910    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55c30a1d-0943-4733-9a53-ed49d3d4184d-istiod-ca-cert\") pod \"55c30a1d-0943-4733-9a53-ed49d3d4184d\" (UID: \"55c30a1d-0943-4733-9a53-ed49d3d4184d\") "
Sep 08 15:51:19.047977 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:19.047951    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55c30a1d-0943-4733-9a53-ed49d3d4184d-istio-podinfo\") pod \"55c30a1d-0943-4733-9a53-ed49d3d4184d\" (UID: \"55c30a1d-0943-4733-9a53-ed49d3d4184d\") "
Sep 08 15:51:19.048164 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:19.047989    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55c30a1d-0943-4733-9a53-ed49d3d4184d-credential-socket\") pod \"55c30a1d-0943-4733-9a53-ed49d3d4184d\" (UID: \"55c30a1d-0943-4733-9a53-ed49d3d4184d\") "
Sep 08 15:51:19.048164 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:19.048087    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55c30a1d-0943-4733-9a53-ed49d3d4184d-istio-token\") pod \"55c30a1d-0943-4733-9a53-ed49d3d4184d\" (UID: \"55c30a1d-0943-4733-9a53-ed49d3d4184d\") "
Sep 08 15:51:19.048164 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:19.048139    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55c30a1d-0943-4733-9a53-ed49d3d4184d-istio-data\") pod \"55c30a1d-0943-4733-9a53-ed49d3d4184d\" (UID: \"55c30a1d-0943-4733-9a53-ed49d3d4184d\") "
Sep 08 15:51:19.048321 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:19.048168    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55c30a1d-0943-4733-9a53-ed49d3d4184d-workload-socket\") pod \"55c30a1d-0943-4733-9a53-ed49d3d4184d\" (UID: \"55c30a1d-0943-4733-9a53-ed49d3d4184d\") "
Sep 08 15:51:19.048321 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:19.048208    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n9hvl\" (UniqueName: \"kubernetes.io/projected/55c30a1d-0943-4733-9a53-ed49d3d4184d-kube-api-access-n9hvl\") pod \"55c30a1d-0943-4733-9a53-ed49d3d4184d\" (UID: \"55c30a1d-0943-4733-9a53-ed49d3d4184d\") "
Sep 08 15:51:19.048321 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:19.048226    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55c30a1d-0943-4733-9a53-ed49d3d4184d-istio-envoy\") pod \"55c30a1d-0943-4733-9a53-ed49d3d4184d\" (UID: \"55c30a1d-0943-4733-9a53-ed49d3d4184d\") "
Sep 08 15:51:19.048321 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:19.048242    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55c30a1d-0943-4733-9a53-ed49d3d4184d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "55c30a1d-0943-4733-9a53-ed49d3d4184d" (UID: "55c30a1d-0943-4733-9a53-ed49d3d4184d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:51:19.048321 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:19.048267    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55c30a1d-0943-4733-9a53-ed49d3d4184d-workload-certs\") pod \"55c30a1d-0943-4733-9a53-ed49d3d4184d\" (UID: \"55c30a1d-0943-4733-9a53-ed49d3d4184d\") "
Sep 08 15:51:19.048321 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:19.048291    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55c30a1d-0943-4733-9a53-ed49d3d4184d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "55c30a1d-0943-4733-9a53-ed49d3d4184d" (UID: "55c30a1d-0943-4733-9a53-ed49d3d4184d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 15:51:19.048631 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:19.048459    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55c30a1d-0943-4733-9a53-ed49d3d4184d-istio-data" (OuterVolumeSpecName: "istio-data") pod "55c30a1d-0943-4733-9a53-ed49d3d4184d" (UID: "55c30a1d-0943-4733-9a53-ed49d3d4184d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:51:19.048631 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:19.048548    2549 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55c30a1d-0943-4733-9a53-ed49d3d4184d-istiod-ca-cert\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:51:19.048631 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:19.048565    2549 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55c30a1d-0943-4733-9a53-ed49d3d4184d-credential-socket\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:51:19.048631 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:19.048579    2549 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55c30a1d-0943-4733-9a53-ed49d3d4184d-istio-data\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:51:19.048799 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:19.048666    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55c30a1d-0943-4733-9a53-ed49d3d4184d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "55c30a1d-0943-4733-9a53-ed49d3d4184d" (UID: "55c30a1d-0943-4733-9a53-ed49d3d4184d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:51:19.048799 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:19.048750    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55c30a1d-0943-4733-9a53-ed49d3d4184d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "55c30a1d-0943-4733-9a53-ed49d3d4184d" (UID: "55c30a1d-0943-4733-9a53-ed49d3d4184d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:51:19.050442 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:19.050416    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/55c30a1d-0943-4733-9a53-ed49d3d4184d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "55c30a1d-0943-4733-9a53-ed49d3d4184d" (UID: "55c30a1d-0943-4733-9a53-ed49d3d4184d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 08 15:51:19.050529 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:19.050465    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55c30a1d-0943-4733-9a53-ed49d3d4184d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "55c30a1d-0943-4733-9a53-ed49d3d4184d" (UID: "55c30a1d-0943-4733-9a53-ed49d3d4184d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:51:19.050650 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:19.050580    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55c30a1d-0943-4733-9a53-ed49d3d4184d-istio-token" (OuterVolumeSpecName: "istio-token") pod "55c30a1d-0943-4733-9a53-ed49d3d4184d" (UID: "55c30a1d-0943-4733-9a53-ed49d3d4184d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:51:19.050743 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:19.050667    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55c30a1d-0943-4733-9a53-ed49d3d4184d-kube-api-access-n9hvl" (OuterVolumeSpecName: "kube-api-access-n9hvl") pod "55c30a1d-0943-4733-9a53-ed49d3d4184d" (UID: "55c30a1d-0943-4733-9a53-ed49d3d4184d"). InnerVolumeSpecName "kube-api-access-n9hvl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:51:19.149642 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:19.149608    2549 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55c30a1d-0943-4733-9a53-ed49d3d4184d-workload-certs\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:51:19.149642 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:19.149635    2549 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55c30a1d-0943-4733-9a53-ed49d3d4184d-istio-podinfo\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:51:19.149642 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:19.149644    2549 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55c30a1d-0943-4733-9a53-ed49d3d4184d-istio-token\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:51:19.149877 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:19.149652    2549 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55c30a1d-0943-4733-9a53-ed49d3d4184d-workload-socket\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:51:19.149877 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:19.149660    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n9hvl\" (UniqueName: \"kubernetes.io/projected/55c30a1d-0943-4733-9a53-ed49d3d4184d-kube-api-access-n9hvl\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:51:19.149877 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:19.149669    2549 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55c30a1d-0943-4733-9a53-ed49d3d4184d-istio-envoy\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:51:19.602389 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:19.602355    2549 generic.go:356] "Generic (PLEG): container finished" podID="55c30a1d-0943-4733-9a53-ed49d3d4184d" containerID="2783d7dbbc01842411bd24980061e88bd7f0392b36ddd78aa81fb76724d495c3" exitCode=0
Sep 08 15:51:19.602544 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:19.602401    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-a057c5-openshift-default-5f564fb8cd-2mflt"
Sep 08 15:51:19.602544 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:19.602436    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a057c5-openshift-default-5f564fb8cd-2mflt" event={"ID":"55c30a1d-0943-4733-9a53-ed49d3d4184d","Type":"ContainerDied","Data":"2783d7dbbc01842411bd24980061e88bd7f0392b36ddd78aa81fb76724d495c3"}
Sep 08 15:51:19.602544 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:19.602488    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a057c5-openshift-default-5f564fb8cd-2mflt" event={"ID":"55c30a1d-0943-4733-9a53-ed49d3d4184d","Type":"ContainerDied","Data":"5a583b0657349218bfe3e99dfa19f4313c70be437013aa4f00d9aa227710ae87"}
Sep 08 15:51:19.602544 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:19.602511    2549 scope.go:117] "RemoveContainer" containerID="2783d7dbbc01842411bd24980061e88bd7f0392b36ddd78aa81fb76724d495c3"
Sep 08 15:51:19.613867 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:19.613073    2549 scope.go:117] "RemoveContainer" containerID="b6b9f975886b8fc8a2cf5c649c9d94b841ee530352f5b76a999fecfcd62d1073"
Sep 08 15:51:19.622129 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:19.622112    2549 scope.go:117] "RemoveContainer" containerID="2783d7dbbc01842411bd24980061e88bd7f0392b36ddd78aa81fb76724d495c3"
Sep 08 15:51:19.622385 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:51:19.622366    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2783d7dbbc01842411bd24980061e88bd7f0392b36ddd78aa81fb76724d495c3\": container with ID starting with 2783d7dbbc01842411bd24980061e88bd7f0392b36ddd78aa81fb76724d495c3 not found: ID does not exist" containerID="2783d7dbbc01842411bd24980061e88bd7f0392b36ddd78aa81fb76724d495c3"
Sep 08 15:51:19.622455 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:19.622391    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2783d7dbbc01842411bd24980061e88bd7f0392b36ddd78aa81fb76724d495c3"} err="failed to get container status \"2783d7dbbc01842411bd24980061e88bd7f0392b36ddd78aa81fb76724d495c3\": rpc error: code = NotFound desc = could not find container \"2783d7dbbc01842411bd24980061e88bd7f0392b36ddd78aa81fb76724d495c3\": container with ID starting with 2783d7dbbc01842411bd24980061e88bd7f0392b36ddd78aa81fb76724d495c3 not found: ID does not exist"
Sep 08 15:51:19.622455 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:19.622405    2549 scope.go:117] "RemoveContainer" containerID="b6b9f975886b8fc8a2cf5c649c9d94b841ee530352f5b76a999fecfcd62d1073"
Sep 08 15:51:19.622641 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:51:19.622624    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6b9f975886b8fc8a2cf5c649c9d94b841ee530352f5b76a999fecfcd62d1073\": container with ID starting with b6b9f975886b8fc8a2cf5c649c9d94b841ee530352f5b76a999fecfcd62d1073 not found: ID does not exist" containerID="b6b9f975886b8fc8a2cf5c649c9d94b841ee530352f5b76a999fecfcd62d1073"
Sep 08 15:51:19.622685 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:19.622647    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6b9f975886b8fc8a2cf5c649c9d94b841ee530352f5b76a999fecfcd62d1073"} err="failed to get container status \"b6b9f975886b8fc8a2cf5c649c9d94b841ee530352f5b76a999fecfcd62d1073\": rpc error: code = NotFound desc = could not find container \"b6b9f975886b8fc8a2cf5c649c9d94b841ee530352f5b76a999fecfcd62d1073\": container with ID starting with b6b9f975886b8fc8a2cf5c649c9d94b841ee530352f5b76a999fecfcd62d1073 not found: ID does not exist"
Sep 08 15:51:19.646307 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:19.646280    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-a057c5-openshift-default-5f564fb8cd-2mflt"]
Sep 08 15:51:19.658614 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:19.658585    2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-a057c5-openshift-default-5f564fb8cd-2mflt"]
Sep 08 15:51:20.401006 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:20.400972    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="55c30a1d-0943-4733-9a53-ed49d3d4184d" path="/var/lib/kubelet/pods/55c30a1d-0943-4733-9a53-ed49d3d4184d/volumes"
Sep 08 15:51:20.587844 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:20.587818    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-0ac386-5b555b7776-xz5xw"
Sep 08 15:51:36.451938 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:36.451904    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7778-139766118061808"]
Sep 08 15:51:36.452289 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:36.452228    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55c30a1d-0943-4733-9a53-ed49d3d4184d" containerName="istio-proxy"
Sep 08 15:51:36.452289 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:36.452240    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="55c30a1d-0943-4733-9a53-ed49d3d4184d" containerName="istio-proxy"
Sep 08 15:51:36.452368 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:36.452294    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="55c30a1d-0943-4733-9a53-ed49d3d4184d" containerName="istio-proxy"
Sep 08 15:51:36.456248 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:36.456230    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7778-139766118061808"
Sep 08 15:51:36.459135 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:36.459105    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-6hc5f\""
Sep 08 15:51:36.460604 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:36.460580    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7778-139766118061808"]
Sep 08 15:51:36.495447 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:36.495424    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vn2w\" (UniqueName: \"kubernetes.io/projected/54c71064-0560-4e4b-8b4e-4e830bf92b38-kube-api-access-8vn2w\") pod \"test-curl-7778-139766118061808\" (UID: \"54c71064-0560-4e4b-8b4e-4e830bf92b38\") " pod="opendatahub/test-curl-7778-139766118061808"
Sep 08 15:51:36.596331 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:36.596299    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vn2w\" (UniqueName: \"kubernetes.io/projected/54c71064-0560-4e4b-8b4e-4e830bf92b38-kube-api-access-8vn2w\") pod \"test-curl-7778-139766118061808\" (UID: \"54c71064-0560-4e4b-8b4e-4e830bf92b38\") " pod="opendatahub/test-curl-7778-139766118061808"
Sep 08 15:51:36.604492 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:36.604467    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vn2w\" (UniqueName: \"kubernetes.io/projected/54c71064-0560-4e4b-8b4e-4e830bf92b38-kube-api-access-8vn2w\") pod \"test-curl-7778-139766118061808\" (UID: \"54c71064-0560-4e4b-8b4e-4e830bf92b38\") " pod="opendatahub/test-curl-7778-139766118061808"
Sep 08 15:51:36.766628 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:36.766560    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7778-139766118061808"
Sep 08 15:51:37.040709 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:37.040680    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7778-139766118061808"]
Sep 08 15:51:37.043224 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:51:37.043194    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod54c71064_0560_4e4b_8b4e_4e830bf92b38.slice/crio-96e5aab06477a1f98d551e2ee53ae50fb80269ed68d547eb6122f5f108ec836a WatchSource:0}: Error finding container 96e5aab06477a1f98d551e2ee53ae50fb80269ed68d547eb6122f5f108ec836a: Status 404 returned error can't find the container with id 96e5aab06477a1f98d551e2ee53ae50fb80269ed68d547eb6122f5f108ec836a
Sep 08 15:51:37.685338 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:37.685302    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7778-139766118061808" event={"ID":"54c71064-0560-4e4b-8b4e-4e830bf92b38","Type":"ContainerStarted","Data":"96e5aab06477a1f98d551e2ee53ae50fb80269ed68d547eb6122f5f108ec836a"}
Sep 08 15:51:38.691160 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:38.691125    2549 generic.go:356] "Generic (PLEG): container finished" podID="54c71064-0560-4e4b-8b4e-4e830bf92b38" containerID="c59df72e2271f3c5f4b80a7823ed3a218ede66758ead075a15cc728076400f06" exitCode=0
Sep 08 15:51:38.691541 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:38.691175    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7778-139766118061808" event={"ID":"54c71064-0560-4e4b-8b4e-4e830bf92b38","Type":"ContainerDied","Data":"c59df72e2271f3c5f4b80a7823ed3a218ede66758ead075a15cc728076400f06"}
Sep 08 15:51:39.829771 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:39.829744    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7778-139766118061808"
Sep 08 15:51:39.866289 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:39.866264    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7778-139766118061808_54c71064-0560-4e4b-8b4e-4e830bf92b38/test-curl-7778-139766118061808/0.log"
Sep 08 15:51:39.886362 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:39.886337    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7778-139766118061808"]
Sep 08 15:51:39.889879 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:39.889843    2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7778-139766118061808"]
Sep 08 15:51:39.923015 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:39.922994    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8vn2w\" (UniqueName: \"kubernetes.io/projected/54c71064-0560-4e4b-8b4e-4e830bf92b38-kube-api-access-8vn2w\") pod \"54c71064-0560-4e4b-8b4e-4e830bf92b38\" (UID: \"54c71064-0560-4e4b-8b4e-4e830bf92b38\") "
Sep 08 15:51:39.925003 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:39.924975    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54c71064-0560-4e4b-8b4e-4e830bf92b38-kube-api-access-8vn2w" (OuterVolumeSpecName: "kube-api-access-8vn2w") pod "54c71064-0560-4e4b-8b4e-4e830bf92b38" (UID: "54c71064-0560-4e4b-8b4e-4e830bf92b38"). InnerVolumeSpecName "kube-api-access-8vn2w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:51:40.022035 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:40.021980    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7778-139766118061808"]
Sep 08 15:51:40.022317 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:40.022304    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55c30a1d-0943-4733-9a53-ed49d3d4184d" containerName="istio-proxy"
Sep 08 15:51:40.022362 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:40.022319    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="55c30a1d-0943-4733-9a53-ed49d3d4184d" containerName="istio-proxy"
Sep 08 15:51:40.022362 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:40.022328    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54c71064-0560-4e4b-8b4e-4e830bf92b38" containerName="test-curl-7778-139766118061808"
Sep 08 15:51:40.022362 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:40.022334    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="54c71064-0560-4e4b-8b4e-4e830bf92b38" containerName="test-curl-7778-139766118061808"
Sep 08 15:51:40.022454 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:40.022384    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="54c71064-0560-4e4b-8b4e-4e830bf92b38" containerName="test-curl-7778-139766118061808"
Sep 08 15:51:40.022454 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:40.022394    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="55c30a1d-0943-4733-9a53-ed49d3d4184d" containerName="istio-proxy"
Sep 08 15:51:40.023584 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:40.023567    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vn2w\" (UniqueName: \"kubernetes.io/projected/54c71064-0560-4e4b-8b4e-4e830bf92b38-kube-api-access-8vn2w\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:51:40.025178 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:40.025164    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7778-139766118061808"
Sep 08 15:51:40.030752 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:40.030729    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7778-139766118061808"]
Sep 08 15:51:40.124200 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:40.124167    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcn26\" (UniqueName: \"kubernetes.io/projected/cb2144b5-0399-4b1d-8456-ca40a9180789-kube-api-access-hcn26\") pod \"test-curl-7778-139766118061808\" (UID: \"cb2144b5-0399-4b1d-8456-ca40a9180789\") " pod="opendatahub/test-curl-7778-139766118061808"
Sep 08 15:51:40.224873 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:40.224829    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcn26\" (UniqueName: \"kubernetes.io/projected/cb2144b5-0399-4b1d-8456-ca40a9180789-kube-api-access-hcn26\") pod \"test-curl-7778-139766118061808\" (UID: \"cb2144b5-0399-4b1d-8456-ca40a9180789\") " pod="opendatahub/test-curl-7778-139766118061808"
Sep 08 15:51:40.233072 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:40.233053    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcn26\" (UniqueName: \"kubernetes.io/projected/cb2144b5-0399-4b1d-8456-ca40a9180789-kube-api-access-hcn26\") pod \"test-curl-7778-139766118061808\" (UID: \"cb2144b5-0399-4b1d-8456-ca40a9180789\") " pod="opendatahub/test-curl-7778-139766118061808"
Sep 08 15:51:40.334992 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:40.334932    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7778-139766118061808"
Sep 08 15:51:40.401199 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:40.401170    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="54c71064-0560-4e4b-8b4e-4e830bf92b38" path="/var/lib/kubelet/pods/54c71064-0560-4e4b-8b4e-4e830bf92b38/volumes"
Sep 08 15:51:40.604228 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:40.604180    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7778-139766118061808"]
Sep 08 15:51:40.606290 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:51:40.606264    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcb2144b5_0399_4b1d_8456_ca40a9180789.slice/crio-fc8fe7efa8911b153dabbffab654fb3c764787036a90d7dee9d1abb4c534de8d WatchSource:0}: Error finding container fc8fe7efa8911b153dabbffab654fb3c764787036a90d7dee9d1abb4c534de8d: Status 404 returned error can't find the container with id fc8fe7efa8911b153dabbffab654fb3c764787036a90d7dee9d1abb4c534de8d
Sep 08 15:51:40.701792 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:40.701768    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7778-139766118061808"
Sep 08 15:51:40.701792 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:40.701789    2549 scope.go:117] "RemoveContainer" containerID="c59df72e2271f3c5f4b80a7823ed3a218ede66758ead075a15cc728076400f06"
Sep 08 15:51:40.703237 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:40.703207    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7778-139766118061808" event={"ID":"cb2144b5-0399-4b1d-8456-ca40a9180789","Type":"ContainerStarted","Data":"fc8fe7efa8911b153dabbffab654fb3c764787036a90d7dee9d1abb4c534de8d"}
Sep 08 15:51:41.711042 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:41.711005    2549 generic.go:356] "Generic (PLEG): container finished" podID="cb2144b5-0399-4b1d-8456-ca40a9180789" containerID="52d3949d3815d362cb68932b0f8774098f8916669419902a54663a96903a8a08" exitCode=0
Sep 08 15:51:41.711465 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:41.711085    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7778-139766118061808" event={"ID":"cb2144b5-0399-4b1d-8456-ca40a9180789","Type":"ContainerDied","Data":"52d3949d3815d362cb68932b0f8774098f8916669419902a54663a96903a8a08"}
Sep 08 15:51:42.852143 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:42.852118    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7778-139766118061808"
Sep 08 15:51:42.887211 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:42.887186    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7778-139766118061808_cb2144b5-0399-4b1d-8456-ca40a9180789/test-curl-7778-139766118061808/0.log"
Sep 08 15:51:42.917876 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:42.917841    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7778-139766118061808"]
Sep 08 15:51:42.923709 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:42.923688    2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7778-139766118061808"]
Sep 08 15:51:42.946962 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:42.946941    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hcn26\" (UniqueName: \"kubernetes.io/projected/cb2144b5-0399-4b1d-8456-ca40a9180789-kube-api-access-hcn26\") pod \"cb2144b5-0399-4b1d-8456-ca40a9180789\" (UID: \"cb2144b5-0399-4b1d-8456-ca40a9180789\") "
Sep 08 15:51:42.948887 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:42.948843    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb2144b5-0399-4b1d-8456-ca40a9180789-kube-api-access-hcn26" (OuterVolumeSpecName: "kube-api-access-hcn26") pod "cb2144b5-0399-4b1d-8456-ca40a9180789" (UID: "cb2144b5-0399-4b1d-8456-ca40a9180789"). InnerVolumeSpecName "kube-api-access-hcn26". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:51:43.048223 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:43.048151    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hcn26\" (UniqueName: \"kubernetes.io/projected/cb2144b5-0399-4b1d-8456-ca40a9180789-kube-api-access-hcn26\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:51:43.056870 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:43.056830    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7778-139766118062384"]
Sep 08 15:51:43.057258 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:43.057240    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb2144b5-0399-4b1d-8456-ca40a9180789" containerName="test-curl-7778-139766118061808"
Sep 08 15:51:43.057350 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:43.057262    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb2144b5-0399-4b1d-8456-ca40a9180789" containerName="test-curl-7778-139766118061808"
Sep 08 15:51:43.057422 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:43.057359    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb2144b5-0399-4b1d-8456-ca40a9180789" containerName="test-curl-7778-139766118061808"
Sep 08 15:51:43.060350 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:43.060332    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7778-139766118062384"
Sep 08 15:51:43.064592 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:43.064570    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7778-139766118062384"]
Sep 08 15:51:43.148712 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:43.148681    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5b9g4\" (UniqueName: \"kubernetes.io/projected/84cd19d0-1b04-4c6b-a965-8f612b89e50b-kube-api-access-5b9g4\") pod \"test-curl-7778-139766118062384\" (UID: \"84cd19d0-1b04-4c6b-a965-8f612b89e50b\") " pod="opendatahub/test-curl-7778-139766118062384"
Sep 08 15:51:43.249448 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:43.249284    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5b9g4\" (UniqueName: \"kubernetes.io/projected/84cd19d0-1b04-4c6b-a965-8f612b89e50b-kube-api-access-5b9g4\") pod \"test-curl-7778-139766118062384\" (UID: \"84cd19d0-1b04-4c6b-a965-8f612b89e50b\") " pod="opendatahub/test-curl-7778-139766118062384"
Sep 08 15:51:43.257463 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:43.257436    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5b9g4\" (UniqueName: \"kubernetes.io/projected/84cd19d0-1b04-4c6b-a965-8f612b89e50b-kube-api-access-5b9g4\") pod \"test-curl-7778-139766118062384\" (UID: \"84cd19d0-1b04-4c6b-a965-8f612b89e50b\") " pod="opendatahub/test-curl-7778-139766118062384"
Sep 08 15:51:43.342467 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:43.342388    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-642d3b-openshift-default-596967c6f6-pxbln"]
Sep 08 15:51:43.345923 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:43.345901    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-642d3b-openshift-default-596967c6f6-pxbln"
Sep 08 15:51:43.349649 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:43.349627    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-642d3b-openshift-default-dockercfg-s8hsf\""
Sep 08 15:51:43.364897 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:43.364873    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-642d3b-openshift-default-596967c6f6-pxbln"]
Sep 08 15:51:43.370556 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:43.370536    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7778-139766118062384"
Sep 08 15:51:43.451553 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:43.451505    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b4f5c99-b0b3-4e45-82e8-01ba20a02907-istio-envoy\") pod \"e2e-shared-b-w3-642d3b-openshift-default-596967c6f6-pxbln\" (UID: \"1b4f5c99-b0b3-4e45-82e8-01ba20a02907\") " pod="openshift-ingress/e2e-shared-b-w3-642d3b-openshift-default-596967c6f6-pxbln"
Sep 08 15:51:43.451725 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:43.451567    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b4f5c99-b0b3-4e45-82e8-01ba20a02907-istio-data\") pod \"e2e-shared-b-w3-642d3b-openshift-default-596967c6f6-pxbln\" (UID: \"1b4f5c99-b0b3-4e45-82e8-01ba20a02907\") " pod="openshift-ingress/e2e-shared-b-w3-642d3b-openshift-default-596967c6f6-pxbln"
Sep 08 15:51:43.451725 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:43.451615    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b4f5c99-b0b3-4e45-82e8-01ba20a02907-workload-certs\") pod \"e2e-shared-b-w3-642d3b-openshift-default-596967c6f6-pxbln\" (UID: \"1b4f5c99-b0b3-4e45-82e8-01ba20a02907\") " pod="openshift-ingress/e2e-shared-b-w3-642d3b-openshift-default-596967c6f6-pxbln"
Sep 08 15:51:43.451725 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:43.451663    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b4f5c99-b0b3-4e45-82e8-01ba20a02907-istio-token\") pod \"e2e-shared-b-w3-642d3b-openshift-default-596967c6f6-pxbln\" (UID: \"1b4f5c99-b0b3-4e45-82e8-01ba20a02907\") " pod="openshift-ingress/e2e-shared-b-w3-642d3b-openshift-default-596967c6f6-pxbln"
Sep 08 15:51:43.451725 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:43.451696    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b4f5c99-b0b3-4e45-82e8-01ba20a02907-workload-socket\") pod \"e2e-shared-b-w3-642d3b-openshift-default-596967c6f6-pxbln\" (UID: \"1b4f5c99-b0b3-4e45-82e8-01ba20a02907\") " pod="openshift-ingress/e2e-shared-b-w3-642d3b-openshift-default-596967c6f6-pxbln"
Sep 08 15:51:43.451725 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:43.451722    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b4f5c99-b0b3-4e45-82e8-01ba20a02907-istio-podinfo\") pod \"e2e-shared-b-w3-642d3b-openshift-default-596967c6f6-pxbln\" (UID: \"1b4f5c99-b0b3-4e45-82e8-01ba20a02907\") " pod="openshift-ingress/e2e-shared-b-w3-642d3b-openshift-default-596967c6f6-pxbln"
Sep 08 15:51:43.452096 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:43.451747    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttgsq\" (UniqueName: \"kubernetes.io/projected/1b4f5c99-b0b3-4e45-82e8-01ba20a02907-kube-api-access-ttgsq\") pod \"e2e-shared-b-w3-642d3b-openshift-default-596967c6f6-pxbln\" (UID: \"1b4f5c99-b0b3-4e45-82e8-01ba20a02907\") " pod="openshift-ingress/e2e-shared-b-w3-642d3b-openshift-default-596967c6f6-pxbln"
Sep 08 15:51:43.452096 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:43.451811    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b4f5c99-b0b3-4e45-82e8-01ba20a02907-credential-socket\") pod \"e2e-shared-b-w3-642d3b-openshift-default-596967c6f6-pxbln\" (UID: \"1b4f5c99-b0b3-4e45-82e8-01ba20a02907\") " pod="openshift-ingress/e2e-shared-b-w3-642d3b-openshift-default-596967c6f6-pxbln"
Sep 08 15:51:43.452210 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:43.452172    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b4f5c99-b0b3-4e45-82e8-01ba20a02907-istiod-ca-cert\") pod \"e2e-shared-b-w3-642d3b-openshift-default-596967c6f6-pxbln\" (UID: \"1b4f5c99-b0b3-4e45-82e8-01ba20a02907\") " pod="openshift-ingress/e2e-shared-b-w3-642d3b-openshift-default-596967c6f6-pxbln"
Sep 08 15:51:43.553357 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:43.553319    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b4f5c99-b0b3-4e45-82e8-01ba20a02907-credential-socket\") pod \"e2e-shared-b-w3-642d3b-openshift-default-596967c6f6-pxbln\" (UID: \"1b4f5c99-b0b3-4e45-82e8-01ba20a02907\") " pod="openshift-ingress/e2e-shared-b-w3-642d3b-openshift-default-596967c6f6-pxbln"
Sep 08 15:51:43.553521 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:43.553367    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b4f5c99-b0b3-4e45-82e8-01ba20a02907-istiod-ca-cert\") pod \"e2e-shared-b-w3-642d3b-openshift-default-596967c6f6-pxbln\" (UID: \"1b4f5c99-b0b3-4e45-82e8-01ba20a02907\") " pod="openshift-ingress/e2e-shared-b-w3-642d3b-openshift-default-596967c6f6-pxbln"
Sep 08 15:51:43.553521 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:43.553495    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b4f5c99-b0b3-4e45-82e8-01ba20a02907-istio-envoy\") pod \"e2e-shared-b-w3-642d3b-openshift-default-596967c6f6-pxbln\" (UID: \"1b4f5c99-b0b3-4e45-82e8-01ba20a02907\") " pod="openshift-ingress/e2e-shared-b-w3-642d3b-openshift-default-596967c6f6-pxbln"
Sep 08 15:51:43.553620 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:43.553548    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b4f5c99-b0b3-4e45-82e8-01ba20a02907-istio-data\") pod \"e2e-shared-b-w3-642d3b-openshift-default-596967c6f6-pxbln\" (UID: \"1b4f5c99-b0b3-4e45-82e8-01ba20a02907\") " pod="openshift-ingress/e2e-shared-b-w3-642d3b-openshift-default-596967c6f6-pxbln"
Sep 08 15:51:43.553620 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:43.553585    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b4f5c99-b0b3-4e45-82e8-01ba20a02907-workload-certs\") pod \"e2e-shared-b-w3-642d3b-openshift-default-596967c6f6-pxbln\" (UID: \"1b4f5c99-b0b3-4e45-82e8-01ba20a02907\") " pod="openshift-ingress/e2e-shared-b-w3-642d3b-openshift-default-596967c6f6-pxbln"
Sep 08 15:51:43.553709 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:43.553652    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b4f5c99-b0b3-4e45-82e8-01ba20a02907-istio-token\") pod \"e2e-shared-b-w3-642d3b-openshift-default-596967c6f6-pxbln\" (UID: \"1b4f5c99-b0b3-4e45-82e8-01ba20a02907\") " pod="openshift-ingress/e2e-shared-b-w3-642d3b-openshift-default-596967c6f6-pxbln"
Sep 08 15:51:43.553818 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:43.553705    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b4f5c99-b0b3-4e45-82e8-01ba20a02907-workload-socket\") pod \"e2e-shared-b-w3-642d3b-openshift-default-596967c6f6-pxbln\" (UID: \"1b4f5c99-b0b3-4e45-82e8-01ba20a02907\") " pod="openshift-ingress/e2e-shared-b-w3-642d3b-openshift-default-596967c6f6-pxbln"
Sep 08 15:51:43.553818 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:43.553734    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b4f5c99-b0b3-4e45-82e8-01ba20a02907-istio-podinfo\") pod \"e2e-shared-b-w3-642d3b-openshift-default-596967c6f6-pxbln\" (UID: \"1b4f5c99-b0b3-4e45-82e8-01ba20a02907\") " pod="openshift-ingress/e2e-shared-b-w3-642d3b-openshift-default-596967c6f6-pxbln"
Sep 08 15:51:43.553818 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:43.553762    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttgsq\" (UniqueName: \"kubernetes.io/projected/1b4f5c99-b0b3-4e45-82e8-01ba20a02907-kube-api-access-ttgsq\") pod \"e2e-shared-b-w3-642d3b-openshift-default-596967c6f6-pxbln\" (UID: \"1b4f5c99-b0b3-4e45-82e8-01ba20a02907\") " pod="openshift-ingress/e2e-shared-b-w3-642d3b-openshift-default-596967c6f6-pxbln"
Sep 08 15:51:43.554128 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:43.554075    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b4f5c99-b0b3-4e45-82e8-01ba20a02907-workload-socket\") pod \"e2e-shared-b-w3-642d3b-openshift-default-596967c6f6-pxbln\" (UID: \"1b4f5c99-b0b3-4e45-82e8-01ba20a02907\") " pod="openshift-ingress/e2e-shared-b-w3-642d3b-openshift-default-596967c6f6-pxbln"
Sep 08 15:51:43.554278 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:43.554181    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b4f5c99-b0b3-4e45-82e8-01ba20a02907-credential-socket\") pod \"e2e-shared-b-w3-642d3b-openshift-default-596967c6f6-pxbln\" (UID: \"1b4f5c99-b0b3-4e45-82e8-01ba20a02907\") " pod="openshift-ingress/e2e-shared-b-w3-642d3b-openshift-default-596967c6f6-pxbln"
Sep 08 15:51:43.554278 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:43.554246    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b4f5c99-b0b3-4e45-82e8-01ba20a02907-istio-data\") pod \"e2e-shared-b-w3-642d3b-openshift-default-596967c6f6-pxbln\" (UID: \"1b4f5c99-b0b3-4e45-82e8-01ba20a02907\") " pod="openshift-ingress/e2e-shared-b-w3-642d3b-openshift-default-596967c6f6-pxbln"
Sep 08 15:51:43.554278 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:43.554260    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b4f5c99-b0b3-4e45-82e8-01ba20a02907-istiod-ca-cert\") pod \"e2e-shared-b-w3-642d3b-openshift-default-596967c6f6-pxbln\" (UID: \"1b4f5c99-b0b3-4e45-82e8-01ba20a02907\") " pod="openshift-ingress/e2e-shared-b-w3-642d3b-openshift-default-596967c6f6-pxbln"
Sep 08 15:51:43.554618 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:43.554593    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b4f5c99-b0b3-4e45-82e8-01ba20a02907-workload-certs\") pod \"e2e-shared-b-w3-642d3b-openshift-default-596967c6f6-pxbln\" (UID: \"1b4f5c99-b0b3-4e45-82e8-01ba20a02907\") " pod="openshift-ingress/e2e-shared-b-w3-642d3b-openshift-default-596967c6f6-pxbln"
Sep 08 15:51:43.555835 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:43.555811    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b4f5c99-b0b3-4e45-82e8-01ba20a02907-istio-envoy\") pod \"e2e-shared-b-w3-642d3b-openshift-default-596967c6f6-pxbln\" (UID: \"1b4f5c99-b0b3-4e45-82e8-01ba20a02907\") " pod="openshift-ingress/e2e-shared-b-w3-642d3b-openshift-default-596967c6f6-pxbln"
Sep 08 15:51:43.556233 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:43.556215    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b4f5c99-b0b3-4e45-82e8-01ba20a02907-istio-podinfo\") pod \"e2e-shared-b-w3-642d3b-openshift-default-596967c6f6-pxbln\" (UID: \"1b4f5c99-b0b3-4e45-82e8-01ba20a02907\") " pod="openshift-ingress/e2e-shared-b-w3-642d3b-openshift-default-596967c6f6-pxbln"
Sep 08 15:51:43.563240 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:43.563217    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b4f5c99-b0b3-4e45-82e8-01ba20a02907-istio-token\") pod \"e2e-shared-b-w3-642d3b-openshift-default-596967c6f6-pxbln\" (UID: \"1b4f5c99-b0b3-4e45-82e8-01ba20a02907\") " pod="openshift-ingress/e2e-shared-b-w3-642d3b-openshift-default-596967c6f6-pxbln"
Sep 08 15:51:43.563240 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:43.563236    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttgsq\" (UniqueName: \"kubernetes.io/projected/1b4f5c99-b0b3-4e45-82e8-01ba20a02907-kube-api-access-ttgsq\") pod \"e2e-shared-b-w3-642d3b-openshift-default-596967c6f6-pxbln\" (UID: \"1b4f5c99-b0b3-4e45-82e8-01ba20a02907\") " pod="openshift-ingress/e2e-shared-b-w3-642d3b-openshift-default-596967c6f6-pxbln"
Sep 08 15:51:43.644475 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:43.644446    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7778-139766118062384"]
Sep 08 15:51:43.645995 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:51:43.645969    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod84cd19d0_1b04_4c6b_a965_8f612b89e50b.slice/crio-3a8560da6d39d55309326cd3d200992ca77d4903fe8028c20fb89f1558c2d5a3 WatchSource:0}: Error finding container 3a8560da6d39d55309326cd3d200992ca77d4903fe8028c20fb89f1558c2d5a3: Status 404 returned error can't find the container with id 3a8560da6d39d55309326cd3d200992ca77d4903fe8028c20fb89f1558c2d5a3
Sep 08 15:51:43.657037 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:43.657017    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-642d3b-openshift-default-596967c6f6-pxbln"
Sep 08 15:51:43.723189 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:43.723168    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7778-139766118061808"
Sep 08 15:51:43.723322 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:43.723208    2549 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fc8fe7efa8911b153dabbffab654fb3c764787036a90d7dee9d1abb4c534de8d"
Sep 08 15:51:43.724683 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:43.724660    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7778-139766118062384" event={"ID":"84cd19d0-1b04-4c6b-a965-8f612b89e50b","Type":"ContainerStarted","Data":"3a8560da6d39d55309326cd3d200992ca77d4903fe8028c20fb89f1558c2d5a3"}
Sep 08 15:51:43.738301 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:51:43.738263    2549 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7778-139766118061808\" is forbidden: User \"system:node:ip-10-0-143-61.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-61.ec2.internal' and this object" podUID="cb2144b5-0399-4b1d-8456-ca40a9180789" pod="opendatahub/test-curl-7778-139766118061808"
Sep 08 15:51:43.939126 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:43.939093    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-642d3b-openshift-default-596967c6f6-pxbln"]
Sep 08 15:51:43.940555 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:51:43.940528    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b4f5c99_b0b3_4e45_82e8_01ba20a02907.slice/crio-de0904beb3919df912f71934b02011614b0a628092fb937247931cdbaca56772 WatchSource:0}: Error finding container de0904beb3919df912f71934b02011614b0a628092fb937247931cdbaca56772: Status 404 returned error can't find the container with id de0904beb3919df912f71934b02011614b0a628092fb937247931cdbaca56772
Sep 08 15:51:43.943130 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:43.943089    2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 08 15:51:43.943221 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:43.943179    2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 08 15:51:43.943291 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:43.943221    2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 08 15:51:44.402199 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:44.402167    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cb2144b5-0399-4b1d-8456-ca40a9180789" path="/var/lib/kubelet/pods/cb2144b5-0399-4b1d-8456-ca40a9180789/volumes"
Sep 08 15:51:44.731892 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:44.731785    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-642d3b-openshift-default-596967c6f6-pxbln" event={"ID":"1b4f5c99-b0b3-4e45-82e8-01ba20a02907","Type":"ContainerStarted","Data":"ad4d9b2b832b1b82773a45d8feb86ae8c664042c35d45d32d6a2c880e7a94814"}
Sep 08 15:51:44.732058 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:44.731909    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-642d3b-openshift-default-596967c6f6-pxbln" event={"ID":"1b4f5c99-b0b3-4e45-82e8-01ba20a02907","Type":"ContainerStarted","Data":"de0904beb3919df912f71934b02011614b0a628092fb937247931cdbaca56772"}
Sep 08 15:51:44.733348 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:44.733319    2549 generic.go:356] "Generic (PLEG): container finished" podID="84cd19d0-1b04-4c6b-a965-8f612b89e50b" containerID="8ec19537e2b6307584f896d87a87e9f2b7231bacbfe6f02fbf264ab7b282d09f" exitCode=0
Sep 08 15:51:44.733472 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:44.733376    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7778-139766118062384" event={"ID":"84cd19d0-1b04-4c6b-a965-8f612b89e50b","Type":"ContainerDied","Data":"8ec19537e2b6307584f896d87a87e9f2b7231bacbfe6f02fbf264ab7b282d09f"}
Sep 08 15:51:44.753942 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:44.753899    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-642d3b-openshift-default-596967c6f6-pxbln" podStartSLOduration=1.753882752 podStartE2EDuration="1.753882752s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:51:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:51:44.75131997 +0000 UTC m=+1142.840771060" watchObservedRunningTime="2026-09-08 15:51:44.753882752 +0000 UTC m=+1142.843333841"
Sep 08 15:51:45.657683 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:45.657646    2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-642d3b-openshift-default-596967c6f6-pxbln"
Sep 08 15:51:45.662610 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:45.662588    2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-642d3b-openshift-default-596967c6f6-pxbln"
Sep 08 15:51:45.738556 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:45.738527    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-642d3b-openshift-default-596967c6f6-pxbln"
Sep 08 15:51:45.739415 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:45.739394    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-642d3b-openshift-default-596967c6f6-pxbln"
Sep 08 15:51:45.881965 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:45.881940    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7778-139766118062384"
Sep 08 15:51:45.901169 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:45.901142    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7778-139766118062384_84cd19d0-1b04-4c6b-a965-8f612b89e50b/test-curl-7778-139766118062384/0.log"
Sep 08 15:51:45.922465 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:45.922403    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7778-139766118062384"]
Sep 08 15:51:45.925996 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:45.925955    2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7778-139766118062384"]
Sep 08 15:51:45.983221 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:45.983201    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5b9g4\" (UniqueName: \"kubernetes.io/projected/84cd19d0-1b04-4c6b-a965-8f612b89e50b-kube-api-access-5b9g4\") pod \"84cd19d0-1b04-4c6b-a965-8f612b89e50b\" (UID: \"84cd19d0-1b04-4c6b-a965-8f612b89e50b\") "
Sep 08 15:51:45.985210 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:45.985186    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84cd19d0-1b04-4c6b-a965-8f612b89e50b-kube-api-access-5b9g4" (OuterVolumeSpecName: "kube-api-access-5b9g4") pod "84cd19d0-1b04-4c6b-a965-8f612b89e50b" (UID: "84cd19d0-1b04-4c6b-a965-8f612b89e50b"). InnerVolumeSpecName "kube-api-access-5b9g4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:51:46.044899 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:46.044869    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7778-139766118062384"]
Sep 08 15:51:46.045242 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:46.045227    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84cd19d0-1b04-4c6b-a965-8f612b89e50b" containerName="test-curl-7778-139766118062384"
Sep 08 15:51:46.045292 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:46.045243    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="84cd19d0-1b04-4c6b-a965-8f612b89e50b" containerName="test-curl-7778-139766118062384"
Sep 08 15:51:46.045326 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:46.045315    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="84cd19d0-1b04-4c6b-a965-8f612b89e50b" containerName="test-curl-7778-139766118062384"
Sep 08 15:51:46.048145 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:46.048132    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7778-139766118062384"
Sep 08 15:51:46.051938 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:46.051918    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7778-139766118062384"]
Sep 08 15:51:46.084738 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:46.084713    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5b9g4\" (UniqueName: \"kubernetes.io/projected/84cd19d0-1b04-4c6b-a965-8f612b89e50b-kube-api-access-5b9g4\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:51:46.185258 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:46.185149    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qp8pl\" (UniqueName: \"kubernetes.io/projected/d02d52b3-90c8-4b2c-abf5-75c4582a2dce-kube-api-access-qp8pl\") pod \"test-curl-7778-139766118062384\" (UID: \"d02d52b3-90c8-4b2c-abf5-75c4582a2dce\") " pod="opendatahub/test-curl-7778-139766118062384"
Sep 08 15:51:46.286209 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:46.286180    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qp8pl\" (UniqueName: \"kubernetes.io/projected/d02d52b3-90c8-4b2c-abf5-75c4582a2dce-kube-api-access-qp8pl\") pod \"test-curl-7778-139766118062384\" (UID: \"d02d52b3-90c8-4b2c-abf5-75c4582a2dce\") " pod="opendatahub/test-curl-7778-139766118062384"
Sep 08 15:51:46.294505 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:46.294482    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qp8pl\" (UniqueName: \"kubernetes.io/projected/d02d52b3-90c8-4b2c-abf5-75c4582a2dce-kube-api-access-qp8pl\") pod \"test-curl-7778-139766118062384\" (UID: \"d02d52b3-90c8-4b2c-abf5-75c4582a2dce\") " pod="opendatahub/test-curl-7778-139766118062384"
Sep 08 15:51:46.359095 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:46.359071    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7778-139766118062384"
Sep 08 15:51:46.403047 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:46.403014    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="84cd19d0-1b04-4c6b-a965-8f612b89e50b" path="/var/lib/kubelet/pods/84cd19d0-1b04-4c6b-a965-8f612b89e50b/volumes"
Sep 08 15:51:46.631810 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:46.631728    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7778-139766118062384"]
Sep 08 15:51:46.635069 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:51:46.635041    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd02d52b3_90c8_4b2c_abf5_75c4582a2dce.slice/crio-362dceb994e3d6ac6a6f343d41e9a6c71cb459f321f253b32c92bf5ddf6ca2ea WatchSource:0}: Error finding container 362dceb994e3d6ac6a6f343d41e9a6c71cb459f321f253b32c92bf5ddf6ca2ea: Status 404 returned error can't find the container with id 362dceb994e3d6ac6a6f343d41e9a6c71cb459f321f253b32c92bf5ddf6ca2ea
Sep 08 15:51:46.743877 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:46.743822    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7778-139766118062384" event={"ID":"d02d52b3-90c8-4b2c-abf5-75c4582a2dce","Type":"ContainerStarted","Data":"362dceb994e3d6ac6a6f343d41e9a6c71cb459f321f253b32c92bf5ddf6ca2ea"}
Sep 08 15:51:46.745113 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:46.745099    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7778-139766118062384"
Sep 08 15:51:46.745188 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:46.745158    2549 scope.go:117] "RemoveContainer" containerID="8ec19537e2b6307584f896d87a87e9f2b7231bacbfe6f02fbf264ab7b282d09f"
Sep 08 15:51:47.751319 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:47.751285    2549 generic.go:356] "Generic (PLEG): container finished" podID="d02d52b3-90c8-4b2c-abf5-75c4582a2dce" containerID="ffefd5d453a60c250b1bfcf9242a782cc67bd1c02d7542c424aa7a29558562d7" exitCode=0
Sep 08 15:51:47.751727 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:47.751372    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7778-139766118062384" event={"ID":"d02d52b3-90c8-4b2c-abf5-75c4582a2dce","Type":"ContainerDied","Data":"ffefd5d453a60c250b1bfcf9242a782cc67bd1c02d7542c424aa7a29558562d7"}
Sep 08 15:51:48.883462 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:48.883437    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7778-139766118062384"
Sep 08 15:51:48.902815 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:48.902784    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7778-139766118062384_d02d52b3-90c8-4b2c-abf5-75c4582a2dce/test-curl-7778-139766118062384/0.log"
Sep 08 15:51:48.923565 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:48.923537    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7778-139766118062384"]
Sep 08 15:51:48.929249 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:48.929226    2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7778-139766118062384"]
Sep 08 15:51:49.010087 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:49.010028    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qp8pl\" (UniqueName: \"kubernetes.io/projected/d02d52b3-90c8-4b2c-abf5-75c4582a2dce-kube-api-access-qp8pl\") pod \"d02d52b3-90c8-4b2c-abf5-75c4582a2dce\" (UID: \"d02d52b3-90c8-4b2c-abf5-75c4582a2dce\") "
Sep 08 15:51:49.012157 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:49.012124    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d02d52b3-90c8-4b2c-abf5-75c4582a2dce-kube-api-access-qp8pl" (OuterVolumeSpecName: "kube-api-access-qp8pl") pod "d02d52b3-90c8-4b2c-abf5-75c4582a2dce" (UID: "d02d52b3-90c8-4b2c-abf5-75c4582a2dce"). InnerVolumeSpecName "kube-api-access-qp8pl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:51:49.110899 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:49.110851    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qp8pl\" (UniqueName: \"kubernetes.io/projected/d02d52b3-90c8-4b2c-abf5-75c4582a2dce-kube-api-access-qp8pl\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:51:49.764458 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:49.764421    2549 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="362dceb994e3d6ac6a6f343d41e9a6c71cb459f321f253b32c92bf5ddf6ca2ea"
Sep 08 15:51:49.764636 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:49.764471    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7778-139766118062384"
Sep 08 15:51:49.774623 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:51:49.774584    2549 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7778-139766118062384\" is forbidden: User \"system:node:ip-10-0-143-61.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-61.ec2.internal' and this object" podUID="d02d52b3-90c8-4b2c-abf5-75c4582a2dce" pod="opendatahub/test-curl-7778-139766118062384"
Sep 08 15:51:49.841173 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:49.841142    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-7eb8750f-openshift-default-76c7db7d88-cnfsh"]
Sep 08 15:51:49.841669 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:49.841647    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d02d52b3-90c8-4b2c-abf5-75c4582a2dce" containerName="test-curl-7778-139766118062384"
Sep 08 15:51:49.841669 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:49.841667    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="d02d52b3-90c8-4b2c-abf5-75c4582a2dce" containerName="test-curl-7778-139766118062384"
Sep 08 15:51:49.841833 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:49.841760    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="d02d52b3-90c8-4b2c-abf5-75c4582a2dce" containerName="test-curl-7778-139766118062384"
Sep 08 15:51:49.844734 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:49.844719    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7eb8750f-openshift-default-76c7db7d88-cnfsh"
Sep 08 15:51:49.847433 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:49.847414    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-7eb8750f-openshift-default-dockercfg-dbhr9\""
Sep 08 15:51:49.862889 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:49.862848    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-7eb8750f-openshift-default-76c7db7d88-cnfsh"]
Sep 08 15:51:49.881801 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:51:49.881771    2549 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7778-139766118062384\" is forbidden: User \"system:node:ip-10-0-143-61.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-61.ec2.internal' and this object" podUID="d02d52b3-90c8-4b2c-abf5-75c4582a2dce" pod="opendatahub/test-curl-7778-139766118062384"
Sep 08 15:51:49.918941 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:49.918913    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5f020da-9c3e-4f60-baf9-05d87fcee760-istio-token\") pod \"e2e-mt-7eb8750f-openshift-default-76c7db7d88-cnfsh\" (UID: \"c5f020da-9c3e-4f60-baf9-05d87fcee760\") " pod="openshift-ingress/e2e-mt-7eb8750f-openshift-default-76c7db7d88-cnfsh"
Sep 08 15:51:49.919286 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:49.918944    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5f020da-9c3e-4f60-baf9-05d87fcee760-istio-envoy\") pod \"e2e-mt-7eb8750f-openshift-default-76c7db7d88-cnfsh\" (UID: \"c5f020da-9c3e-4f60-baf9-05d87fcee760\") " pod="openshift-ingress/e2e-mt-7eb8750f-openshift-default-76c7db7d88-cnfsh"
Sep 08 15:51:49.919286 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:49.918966    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5f020da-9c3e-4f60-baf9-05d87fcee760-credential-socket\") pod \"e2e-mt-7eb8750f-openshift-default-76c7db7d88-cnfsh\" (UID: \"c5f020da-9c3e-4f60-baf9-05d87fcee760\") " pod="openshift-ingress/e2e-mt-7eb8750f-openshift-default-76c7db7d88-cnfsh"
Sep 08 15:51:49.919286 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:49.919014    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5f020da-9c3e-4f60-baf9-05d87fcee760-istiod-ca-cert\") pod \"e2e-mt-7eb8750f-openshift-default-76c7db7d88-cnfsh\" (UID: \"c5f020da-9c3e-4f60-baf9-05d87fcee760\") " pod="openshift-ingress/e2e-mt-7eb8750f-openshift-default-76c7db7d88-cnfsh"
Sep 08 15:51:49.919286 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:49.919056    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5f020da-9c3e-4f60-baf9-05d87fcee760-workload-certs\") pod \"e2e-mt-7eb8750f-openshift-default-76c7db7d88-cnfsh\" (UID: \"c5f020da-9c3e-4f60-baf9-05d87fcee760\") " pod="openshift-ingress/e2e-mt-7eb8750f-openshift-default-76c7db7d88-cnfsh"
Sep 08 15:51:49.919286 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:49.919083    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txxtt\" (UniqueName: \"kubernetes.io/projected/c5f020da-9c3e-4f60-baf9-05d87fcee760-kube-api-access-txxtt\") pod \"e2e-mt-7eb8750f-openshift-default-76c7db7d88-cnfsh\" (UID: \"c5f020da-9c3e-4f60-baf9-05d87fcee760\") " pod="openshift-ingress/e2e-mt-7eb8750f-openshift-default-76c7db7d88-cnfsh"
Sep 08 15:51:49.919286 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:49.919102    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5f020da-9c3e-4f60-baf9-05d87fcee760-workload-socket\") pod \"e2e-mt-7eb8750f-openshift-default-76c7db7d88-cnfsh\" (UID: \"c5f020da-9c3e-4f60-baf9-05d87fcee760\") " pod="openshift-ingress/e2e-mt-7eb8750f-openshift-default-76c7db7d88-cnfsh"
Sep 08 15:51:49.919286 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:49.919214    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5f020da-9c3e-4f60-baf9-05d87fcee760-istio-data\") pod \"e2e-mt-7eb8750f-openshift-default-76c7db7d88-cnfsh\" (UID: \"c5f020da-9c3e-4f60-baf9-05d87fcee760\") " pod="openshift-ingress/e2e-mt-7eb8750f-openshift-default-76c7db7d88-cnfsh"
Sep 08 15:51:49.919286 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:49.919253    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5f020da-9c3e-4f60-baf9-05d87fcee760-istio-podinfo\") pod \"e2e-mt-7eb8750f-openshift-default-76c7db7d88-cnfsh\" (UID: \"c5f020da-9c3e-4f60-baf9-05d87fcee760\") " pod="openshift-ingress/e2e-mt-7eb8750f-openshift-default-76c7db7d88-cnfsh"
Sep 08 15:51:50.020436 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:50.020337    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txxtt\" (UniqueName: \"kubernetes.io/projected/c5f020da-9c3e-4f60-baf9-05d87fcee760-kube-api-access-txxtt\") pod \"e2e-mt-7eb8750f-openshift-default-76c7db7d88-cnfsh\" (UID: \"c5f020da-9c3e-4f60-baf9-05d87fcee760\") " pod="openshift-ingress/e2e-mt-7eb8750f-openshift-default-76c7db7d88-cnfsh"
Sep 08 15:51:50.020436 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:50.020395    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5f020da-9c3e-4f60-baf9-05d87fcee760-workload-socket\") pod \"e2e-mt-7eb8750f-openshift-default-76c7db7d88-cnfsh\" (UID: \"c5f020da-9c3e-4f60-baf9-05d87fcee760\") " pod="openshift-ingress/e2e-mt-7eb8750f-openshift-default-76c7db7d88-cnfsh"
Sep 08 15:51:50.020670 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:50.020446    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5f020da-9c3e-4f60-baf9-05d87fcee760-istio-data\") pod \"e2e-mt-7eb8750f-openshift-default-76c7db7d88-cnfsh\" (UID: \"c5f020da-9c3e-4f60-baf9-05d87fcee760\") " pod="openshift-ingress/e2e-mt-7eb8750f-openshift-default-76c7db7d88-cnfsh"
Sep 08 15:51:50.020670 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:50.020501    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5f020da-9c3e-4f60-baf9-05d87fcee760-istio-podinfo\") pod \"e2e-mt-7eb8750f-openshift-default-76c7db7d88-cnfsh\" (UID: \"c5f020da-9c3e-4f60-baf9-05d87fcee760\") " pod="openshift-ingress/e2e-mt-7eb8750f-openshift-default-76c7db7d88-cnfsh"
Sep 08 15:51:50.020670 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:50.020559    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5f020da-9c3e-4f60-baf9-05d87fcee760-istio-token\") pod \"e2e-mt-7eb8750f-openshift-default-76c7db7d88-cnfsh\" (UID: \"c5f020da-9c3e-4f60-baf9-05d87fcee760\") " pod="openshift-ingress/e2e-mt-7eb8750f-openshift-default-76c7db7d88-cnfsh"
Sep 08 15:51:50.020814 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:50.020764    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5f020da-9c3e-4f60-baf9-05d87fcee760-istio-envoy\") pod \"e2e-mt-7eb8750f-openshift-default-76c7db7d88-cnfsh\" (UID: \"c5f020da-9c3e-4f60-baf9-05d87fcee760\") " pod="openshift-ingress/e2e-mt-7eb8750f-openshift-default-76c7db7d88-cnfsh"
Sep 08 15:51:50.020917 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:50.020831    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5f020da-9c3e-4f60-baf9-05d87fcee760-credential-socket\") pod \"e2e-mt-7eb8750f-openshift-default-76c7db7d88-cnfsh\" (UID: \"c5f020da-9c3e-4f60-baf9-05d87fcee760\") " pod="openshift-ingress/e2e-mt-7eb8750f-openshift-default-76c7db7d88-cnfsh"
Sep 08 15:51:50.020917 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:50.020887    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5f020da-9c3e-4f60-baf9-05d87fcee760-workload-socket\") pod \"e2e-mt-7eb8750f-openshift-default-76c7db7d88-cnfsh\" (UID: \"c5f020da-9c3e-4f60-baf9-05d87fcee760\") " pod="openshift-ingress/e2e-mt-7eb8750f-openshift-default-76c7db7d88-cnfsh"
Sep 08 15:51:50.020917 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:50.020902    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5f020da-9c3e-4f60-baf9-05d87fcee760-istiod-ca-cert\") pod \"e2e-mt-7eb8750f-openshift-default-76c7db7d88-cnfsh\" (UID: \"c5f020da-9c3e-4f60-baf9-05d87fcee760\") " pod="openshift-ingress/e2e-mt-7eb8750f-openshift-default-76c7db7d88-cnfsh"
Sep 08 15:51:50.021085 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:50.020913    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5f020da-9c3e-4f60-baf9-05d87fcee760-istio-data\") pod \"e2e-mt-7eb8750f-openshift-default-76c7db7d88-cnfsh\" (UID: \"c5f020da-9c3e-4f60-baf9-05d87fcee760\") " pod="openshift-ingress/e2e-mt-7eb8750f-openshift-default-76c7db7d88-cnfsh"
Sep 08 15:51:50.021085 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:50.020935    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5f020da-9c3e-4f60-baf9-05d87fcee760-workload-certs\") pod \"e2e-mt-7eb8750f-openshift-default-76c7db7d88-cnfsh\" (UID: \"c5f020da-9c3e-4f60-baf9-05d87fcee760\") " pod="openshift-ingress/e2e-mt-7eb8750f-openshift-default-76c7db7d88-cnfsh"
Sep 08 15:51:50.021213 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:50.021174    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5f020da-9c3e-4f60-baf9-05d87fcee760-credential-socket\") pod \"e2e-mt-7eb8750f-openshift-default-76c7db7d88-cnfsh\" (UID: \"c5f020da-9c3e-4f60-baf9-05d87fcee760\") " pod="openshift-ingress/e2e-mt-7eb8750f-openshift-default-76c7db7d88-cnfsh"
Sep 08 15:51:50.021264 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:50.021243    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5f020da-9c3e-4f60-baf9-05d87fcee760-workload-certs\") pod \"e2e-mt-7eb8750f-openshift-default-76c7db7d88-cnfsh\" (UID: \"c5f020da-9c3e-4f60-baf9-05d87fcee760\") " pod="openshift-ingress/e2e-mt-7eb8750f-openshift-default-76c7db7d88-cnfsh"
Sep 08 15:51:50.021514 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:50.021495    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5f020da-9c3e-4f60-baf9-05d87fcee760-istiod-ca-cert\") pod \"e2e-mt-7eb8750f-openshift-default-76c7db7d88-cnfsh\" (UID: \"c5f020da-9c3e-4f60-baf9-05d87fcee760\") " pod="openshift-ingress/e2e-mt-7eb8750f-openshift-default-76c7db7d88-cnfsh"
Sep 08 15:51:50.022980 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:50.022951    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5f020da-9c3e-4f60-baf9-05d87fcee760-istio-envoy\") pod \"e2e-mt-7eb8750f-openshift-default-76c7db7d88-cnfsh\" (UID: \"c5f020da-9c3e-4f60-baf9-05d87fcee760\") " pod="openshift-ingress/e2e-mt-7eb8750f-openshift-default-76c7db7d88-cnfsh"
Sep 08 15:51:50.023149 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:50.023128    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5f020da-9c3e-4f60-baf9-05d87fcee760-istio-podinfo\") pod \"e2e-mt-7eb8750f-openshift-default-76c7db7d88-cnfsh\" (UID: \"c5f020da-9c3e-4f60-baf9-05d87fcee760\") " pod="openshift-ingress/e2e-mt-7eb8750f-openshift-default-76c7db7d88-cnfsh"
Sep 08 15:51:50.028336 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:50.028309    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5f020da-9c3e-4f60-baf9-05d87fcee760-istio-token\") pod \"e2e-mt-7eb8750f-openshift-default-76c7db7d88-cnfsh\" (UID: \"c5f020da-9c3e-4f60-baf9-05d87fcee760\") " pod="openshift-ingress/e2e-mt-7eb8750f-openshift-default-76c7db7d88-cnfsh"
Sep 08 15:51:50.028582 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:50.028565    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-txxtt\" (UniqueName: \"kubernetes.io/projected/c5f020da-9c3e-4f60-baf9-05d87fcee760-kube-api-access-txxtt\") pod \"e2e-mt-7eb8750f-openshift-default-76c7db7d88-cnfsh\" (UID: \"c5f020da-9c3e-4f60-baf9-05d87fcee760\") " pod="openshift-ingress/e2e-mt-7eb8750f-openshift-default-76c7db7d88-cnfsh"
Sep 08 15:51:50.154227 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:50.154194    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7eb8750f-openshift-default-76c7db7d88-cnfsh"
Sep 08 15:51:50.400447 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:50.400417    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d02d52b3-90c8-4b2c-abf5-75c4582a2dce" path="/var/lib/kubelet/pods/d02d52b3-90c8-4b2c-abf5-75c4582a2dce/volumes"
Sep 08 15:51:50.430302 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:50.430278    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-7eb8750f-openshift-default-76c7db7d88-cnfsh"]
Sep 08 15:51:50.432781 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:51:50.432748    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5f020da_9c3e_4f60_baf9_05d87fcee760.slice/crio-91fc043bfe19d449eb656955eb610124a6a2f52c3352651d3105eb34fe67f6a3 WatchSource:0}: Error finding container 91fc043bfe19d449eb656955eb610124a6a2f52c3352651d3105eb34fe67f6a3: Status 404 returned error can't find the container with id 91fc043bfe19d449eb656955eb610124a6a2f52c3352651d3105eb34fe67f6a3
Sep 08 15:51:50.435103 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:50.435068    2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 08 15:51:50.435228 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:50.435143    2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 08 15:51:50.435228 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:50.435177    2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 08 15:51:50.770652 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:50.770569    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7eb8750f-openshift-default-76c7db7d88-cnfsh" event={"ID":"c5f020da-9c3e-4f60-baf9-05d87fcee760","Type":"ContainerStarted","Data":"c4fc9c1c9e6690e8e7e57043d148e2b6162e3e60af8c6520779649bfe0e5b28b"}
Sep 08 15:51:50.770652 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:50.770624    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7eb8750f-openshift-default-76c7db7d88-cnfsh" event={"ID":"c5f020da-9c3e-4f60-baf9-05d87fcee760","Type":"ContainerStarted","Data":"91fc043bfe19d449eb656955eb610124a6a2f52c3352651d3105eb34fe67f6a3"}
Sep 08 15:51:50.793370 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:50.793325    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-7eb8750f-openshift-default-76c7db7d88-cnfsh" podStartSLOduration=1.793310408 podStartE2EDuration="1.793310408s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:51:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:51:50.791725504 +0000 UTC m=+1148.881176588" watchObservedRunningTime="2026-09-08 15:51:50.793310408 +0000 UTC m=+1148.882761497"
Sep 08 15:51:51.154762 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:51.154733    2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-7eb8750f-openshift-default-76c7db7d88-cnfsh"
Sep 08 15:51:51.159704 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:51.159681    2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-7eb8750f-openshift-default-76c7db7d88-cnfsh"
Sep 08 15:51:51.775831 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:51.775794    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-7eb8750f-openshift-default-76c7db7d88-cnfsh"
Sep 08 15:51:51.776798 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:51.776773    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-7eb8750f-openshift-default-76c7db7d88-cnfsh"
Sep 08 15:51:53.923552 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:53.923521    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7eb8750f-569bd5896-xjnbd"]
Sep 08 15:51:53.927122 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:53.927088    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7eb8750f-569bd5896-xjnbd"
Sep 08 15:51:53.930037 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:53.930005    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-7eb8750f-serving-cert\""
Sep 08 15:51:53.936961 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:53.936925    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7eb8750f-569bd5896-xjnbd"]
Sep 08 15:51:54.063659 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:54.063618    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/654c65b6-d200-4301-894b-ed6257d97987-metrics-certs\") pod \"maas-api-e2e-mt-7eb8750f-569bd5896-xjnbd\" (UID: \"654c65b6-d200-4301-894b-ed6257d97987\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7eb8750f-569bd5896-xjnbd"
Sep 08 15:51:54.063832 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:54.063675    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/654c65b6-d200-4301-894b-ed6257d97987-openshift-service-ca\") pod \"maas-api-e2e-mt-7eb8750f-569bd5896-xjnbd\" (UID: \"654c65b6-d200-4301-894b-ed6257d97987\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7eb8750f-569bd5896-xjnbd"
Sep 08 15:51:54.063832 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:54.063708    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/654c65b6-d200-4301-894b-ed6257d97987-maas-api-tls\") pod \"maas-api-e2e-mt-7eb8750f-569bd5896-xjnbd\" (UID: \"654c65b6-d200-4301-894b-ed6257d97987\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7eb8750f-569bd5896-xjnbd"
Sep 08 15:51:54.063832 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:54.063724    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g264v\" (UniqueName: \"kubernetes.io/projected/654c65b6-d200-4301-894b-ed6257d97987-kube-api-access-g264v\") pod \"maas-api-e2e-mt-7eb8750f-569bd5896-xjnbd\" (UID: \"654c65b6-d200-4301-894b-ed6257d97987\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7eb8750f-569bd5896-xjnbd"
Sep 08 15:51:54.164954 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:54.164920    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/654c65b6-d200-4301-894b-ed6257d97987-metrics-certs\") pod \"maas-api-e2e-mt-7eb8750f-569bd5896-xjnbd\" (UID: \"654c65b6-d200-4301-894b-ed6257d97987\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7eb8750f-569bd5896-xjnbd"
Sep 08 15:51:54.165148 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:54.164972    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/654c65b6-d200-4301-894b-ed6257d97987-openshift-service-ca\") pod \"maas-api-e2e-mt-7eb8750f-569bd5896-xjnbd\" (UID: \"654c65b6-d200-4301-894b-ed6257d97987\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7eb8750f-569bd5896-xjnbd"
Sep 08 15:51:54.165148 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:54.164995    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/654c65b6-d200-4301-894b-ed6257d97987-maas-api-tls\") pod \"maas-api-e2e-mt-7eb8750f-569bd5896-xjnbd\" (UID: \"654c65b6-d200-4301-894b-ed6257d97987\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7eb8750f-569bd5896-xjnbd"
Sep 08 15:51:54.165148 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:51:54.165076    2549 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-7eb8750f-serving-cert: secret "maas-api-e2e-mt-7eb8750f-serving-cert" not found
Sep 08 15:51:54.165148 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:51:54.165125    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/654c65b6-d200-4301-894b-ed6257d97987-maas-api-tls podName:654c65b6-d200-4301-894b-ed6257d97987 nodeName:}" failed. No retries permitted until 2026-09-08 15:51:54.66511003 +0000 UTC m=+1152.754561096 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/654c65b6-d200-4301-894b-ed6257d97987-maas-api-tls") pod "maas-api-e2e-mt-7eb8750f-569bd5896-xjnbd" (UID: "654c65b6-d200-4301-894b-ed6257d97987") : secret "maas-api-e2e-mt-7eb8750f-serving-cert" not found
Sep 08 15:51:54.165411 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:54.165218    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g264v\" (UniqueName: \"kubernetes.io/projected/654c65b6-d200-4301-894b-ed6257d97987-kube-api-access-g264v\") pod \"maas-api-e2e-mt-7eb8750f-569bd5896-xjnbd\" (UID: \"654c65b6-d200-4301-894b-ed6257d97987\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7eb8750f-569bd5896-xjnbd"
Sep 08 15:51:54.165741 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:54.165722    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/654c65b6-d200-4301-894b-ed6257d97987-openshift-service-ca\") pod \"maas-api-e2e-mt-7eb8750f-569bd5896-xjnbd\" (UID: \"654c65b6-d200-4301-894b-ed6257d97987\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7eb8750f-569bd5896-xjnbd"
Sep 08 15:51:54.167548 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:54.167525    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/654c65b6-d200-4301-894b-ed6257d97987-metrics-certs\") pod \"maas-api-e2e-mt-7eb8750f-569bd5896-xjnbd\" (UID: \"654c65b6-d200-4301-894b-ed6257d97987\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7eb8750f-569bd5896-xjnbd"
Sep 08 15:51:54.178819 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:54.178762    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g264v\" (UniqueName: \"kubernetes.io/projected/654c65b6-d200-4301-894b-ed6257d97987-kube-api-access-g264v\") pod \"maas-api-e2e-mt-7eb8750f-569bd5896-xjnbd\" (UID: \"654c65b6-d200-4301-894b-ed6257d97987\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7eb8750f-569bd5896-xjnbd"
Sep 08 15:51:54.669721 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:54.669687    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/654c65b6-d200-4301-894b-ed6257d97987-maas-api-tls\") pod \"maas-api-e2e-mt-7eb8750f-569bd5896-xjnbd\" (UID: \"654c65b6-d200-4301-894b-ed6257d97987\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7eb8750f-569bd5896-xjnbd"
Sep 08 15:51:54.672080 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:54.672050    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/654c65b6-d200-4301-894b-ed6257d97987-maas-api-tls\") pod \"maas-api-e2e-mt-7eb8750f-569bd5896-xjnbd\" (UID: \"654c65b6-d200-4301-894b-ed6257d97987\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7eb8750f-569bd5896-xjnbd"
Sep 08 15:51:54.841051 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:54.841019    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7eb8750f-569bd5896-xjnbd"
Sep 08 15:51:55.113062 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:55.113038    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7eb8750f-569bd5896-xjnbd"]
Sep 08 15:51:55.114808 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:51:55.114779    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod654c65b6_d200_4301_894b_ed6257d97987.slice/crio-e8ec56c4692057883fb6a95d5700389e970d210fa1aa1b5aca3b327702f6342b WatchSource:0}: Error finding container e8ec56c4692057883fb6a95d5700389e970d210fa1aa1b5aca3b327702f6342b: Status 404 returned error can't find the container with id e8ec56c4692057883fb6a95d5700389e970d210fa1aa1b5aca3b327702f6342b
Sep 08 15:51:55.798661 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:55.798566    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7eb8750f-569bd5896-xjnbd" event={"ID":"654c65b6-d200-4301-894b-ed6257d97987","Type":"ContainerStarted","Data":"7b68e2a4b87c636e5efc146055d98107187ce17297acd58c10179bba5a8187c4"}
Sep 08 15:51:55.798661 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:55.798631    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7eb8750f-569bd5896-xjnbd" event={"ID":"654c65b6-d200-4301-894b-ed6257d97987","Type":"ContainerStarted","Data":"e8ec56c4692057883fb6a95d5700389e970d210fa1aa1b5aca3b327702f6342b"}
Sep 08 15:51:55.798881 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:51:55.798741    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7eb8750f-569bd5896-xjnbd"
Sep 08 15:52:01.809748 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:52:01.809716    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7eb8750f-569bd5896-xjnbd"
Sep 08 15:52:01.827264 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:52:01.827181    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7eb8750f-569bd5896-xjnbd" podStartSLOduration=8.445055827000001 podStartE2EDuration="8.827160915s" totalImagesPullingTime="382.105088ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:51:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:51:55.816545546 +0000 UTC m=+1153.905996636" watchObservedRunningTime="2026-09-08 15:52:01.827160915 +0000 UTC m=+1159.916612004"
Sep 08 15:52:04.845097 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:52:04.845005    2549 generic.go:356] "Generic (PLEG): container finished" podID="1b4f5c99-b0b3-4e45-82e8-01ba20a02907" containerID="ad4d9b2b832b1b82773a45d8feb86ae8c664042c35d45d32d6a2c880e7a94814" exitCode=0
Sep 08 15:52:04.845097 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:52:04.845064    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-642d3b-openshift-default-596967c6f6-pxbln" event={"ID":"1b4f5c99-b0b3-4e45-82e8-01ba20a02907","Type":"ContainerDied","Data":"ad4d9b2b832b1b82773a45d8feb86ae8c664042c35d45d32d6a2c880e7a94814"}
Sep 08 15:52:04.845668 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:52:04.845551    2549 scope.go:117] "RemoveContainer" containerID="ad4d9b2b832b1b82773a45d8feb86ae8c664042c35d45d32d6a2c880e7a94814"
Sep 08 15:52:04.846679 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:52:04.846646    2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 08 15:52:04.846733 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:52:04.846721    2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 08 15:52:04.846768 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:52:04.846753    2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 08 15:52:05.854971 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:52:05.854928    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-642d3b-openshift-default-596967c6f6-pxbln" event={"ID":"1b4f5c99-b0b3-4e45-82e8-01ba20a02907","Type":"ContainerStarted","Data":"73205b16d63fff9ba3223c370229309059a49a94ec20c1dbea18740ec3526359"}
Sep 08 15:52:06.658043 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:52:06.658004    2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-642d3b-openshift-default-596967c6f6-pxbln"
Sep 08 15:52:06.658262 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:52:06.658055    2549 patch_prober.go:28] interesting pod/e2e-shared-b-w3-642d3b-openshift-default-596967c6f6-pxbln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Sep 08 15:52:06.658262 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:52:06.658115    2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-642d3b-openshift-default-596967c6f6-pxbln" podUID="1b4f5c99-b0b3-4e45-82e8-01ba20a02907" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Sep 08 15:52:07.658492 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:52:07.658443    2549 patch_prober.go:28] interesting pod/e2e-shared-b-w3-642d3b-openshift-default-596967c6f6-pxbln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Sep 08 15:52:07.659010 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:52:07.658528    2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-642d3b-openshift-default-596967c6f6-pxbln" podUID="1b4f5c99-b0b3-4e45-82e8-01ba20a02907" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Sep 08 15:52:08.657789 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:52:08.657750    2549 patch_prober.go:28] interesting pod/e2e-shared-b-w3-642d3b-openshift-default-596967c6f6-pxbln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Sep 08 15:52:08.658103 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:52:08.658071    2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-642d3b-openshift-default-596967c6f6-pxbln" podUID="1b4f5c99-b0b3-4e45-82e8-01ba20a02907" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Sep 08 15:52:09.658218 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:52:09.658183    2549 patch_prober.go:28] interesting pod/e2e-shared-b-w3-642d3b-openshift-default-596967c6f6-pxbln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Sep 08 15:52:09.658784 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:52:09.658755    2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-642d3b-openshift-default-596967c6f6-pxbln" podUID="1b4f5c99-b0b3-4e45-82e8-01ba20a02907" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Sep 08 15:52:10.658225 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:52:10.658183    2549 patch_prober.go:28] interesting pod/e2e-shared-b-w3-642d3b-openshift-default-596967c6f6-pxbln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Sep 08 15:52:10.658798 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:52:10.658762    2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-642d3b-openshift-default-596967c6f6-pxbln" podUID="1b4f5c99-b0b3-4e45-82e8-01ba20a02907" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Sep 08 15:52:11.661664 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:52:11.661635    2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-642d3b-openshift-default-596967c6f6-pxbln"
Sep 08 15:52:11.662066 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:52:11.661923    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-642d3b-openshift-default-596967c6f6-pxbln"
Sep 08 15:52:11.662528 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:52:11.662510    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-642d3b-openshift-default-596967c6f6-pxbln"
Sep 08 15:52:13.907241 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:52:13.907205    2549 generic.go:356] "Generic (PLEG): container finished" podID="c5f020da-9c3e-4f60-baf9-05d87fcee760" containerID="c4fc9c1c9e6690e8e7e57043d148e2b6162e3e60af8c6520779649bfe0e5b28b" exitCode=0
Sep 08 15:52:13.907709 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:52:13.907282    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7eb8750f-openshift-default-76c7db7d88-cnfsh" event={"ID":"c5f020da-9c3e-4f60-baf9-05d87fcee760","Type":"ContainerDied","Data":"c4fc9c1c9e6690e8e7e57043d148e2b6162e3e60af8c6520779649bfe0e5b28b"}
Sep 08 15:52:13.907709 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:52:13.907681    2549 scope.go:117] "RemoveContainer" containerID="c4fc9c1c9e6690e8e7e57043d148e2b6162e3e60af8c6520779649bfe0e5b28b"
Sep 08 15:52:13.908875 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:52:13.908832    2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 08 15:52:13.908945 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:52:13.908924    2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 08 15:52:13.908981 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:52:13.908958    2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 08 15:52:14.914476 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:52:14.914440    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7eb8750f-openshift-default-76c7db7d88-cnfsh" event={"ID":"c5f020da-9c3e-4f60-baf9-05d87fcee760","Type":"ContainerStarted","Data":"c21513b255c56c288b2d645f3ec3ef7e6f615537295b281e9e5b3b40815b5a4d"}
Sep 08 15:52:15.155266 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:52:15.155228    2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-7eb8750f-openshift-default-76c7db7d88-cnfsh"
Sep 08 15:52:15.155644 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:52:15.155558    2549 patch_prober.go:28] interesting pod/e2e-mt-7eb8750f-openshift-default-76c7db7d88-cnfsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Sep 08 15:52:15.155825 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:52:15.155802    2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7eb8750f-openshift-default-76c7db7d88-cnfsh" podUID="c5f020da-9c3e-4f60-baf9-05d87fcee760" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Sep 08 15:52:16.154956 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:52:16.154916    2549 patch_prober.go:28] interesting pod/e2e-mt-7eb8750f-openshift-default-76c7db7d88-cnfsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Sep 08 15:52:16.155533 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:52:16.155501    2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7eb8750f-openshift-default-76c7db7d88-cnfsh" podUID="c5f020da-9c3e-4f60-baf9-05d87fcee760" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Sep 08 15:52:17.154617 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:52:17.154579    2549 patch_prober.go:28] interesting pod/e2e-mt-7eb8750f-openshift-default-76c7db7d88-cnfsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Sep 08 15:52:17.154923 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:52:17.154889    2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7eb8750f-openshift-default-76c7db7d88-cnfsh" podUID="c5f020da-9c3e-4f60-baf9-05d87fcee760" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Sep 08 15:52:18.154687 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:52:18.154648    2549 patch_prober.go:28] interesting pod/e2e-mt-7eb8750f-openshift-default-76c7db7d88-cnfsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Sep 08 15:52:18.155186 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:52:18.154717    2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7eb8750f-openshift-default-76c7db7d88-cnfsh" podUID="c5f020da-9c3e-4f60-baf9-05d87fcee760" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Sep 08 15:52:19.155048 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:52:19.155004    2549 patch_prober.go:28] interesting pod/e2e-mt-7eb8750f-openshift-default-76c7db7d88-cnfsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Sep 08 15:52:19.155550 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:52:19.155071    2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7eb8750f-openshift-default-76c7db7d88-cnfsh" podUID="c5f020da-9c3e-4f60-baf9-05d87fcee760" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Sep 08 15:52:20.160024 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:52:20.159986    2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-7eb8750f-openshift-default-76c7db7d88-cnfsh"
Sep 08 15:52:20.160477 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:52:20.160361    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-7eb8750f-openshift-default-76c7db7d88-cnfsh"
Sep 08 15:52:20.161062 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:52:20.161044    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-7eb8750f-openshift-default-76c7db7d88-cnfsh"
Sep 08 15:52:25.024490 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:52:25.024459    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-b30e99/auth-test-model-b30e99-kserve-7754746779-gzfkn"]
Sep 08 15:52:25.027513 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:52:25.027491    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-b30e99/auth-test-model-b30e99-kserve-7754746779-gzfkn"
Sep 08 15:52:25.030482 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:52:25.030461    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-b30e99\"/\"openshift-service-ca.crt\""
Sep 08 15:52:25.031831 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:52:25.031811    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-b30e99\"/\"auth-test-model-b30e99-kserve-self-signed-certs\""
Sep 08 15:52:25.031968 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:52:25.031837    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-b30e99\"/\"default-dockercfg-hsn87\""
Sep 08 15:52:25.031968 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:52:25.031903    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-b30e99\"/\"kube-root-ca.crt\""
Sep 08 15:52:25.036032 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:52:25.036006    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-b30e99/auth-test-model-b30e99-kserve-7754746779-gzfkn"]
Sep 08 15:52:25.160294 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:52:25.160256    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/566fc738-fff6-4ce2-bc29-16bfb9aee4b7-tmp-dir\") pod \"auth-test-model-b30e99-kserve-7754746779-gzfkn\" (UID: \"566fc738-fff6-4ce2-bc29-16bfb9aee4b7\") " pod="ai-tenant-e2e-shared-a-w3-b30e99/auth-test-model-b30e99-kserve-7754746779-gzfkn"
Sep 08 15:52:25.160454 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:52:25.160319    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kr6jm\" (UniqueName: \"kubernetes.io/projected/566fc738-fff6-4ce2-bc29-16bfb9aee4b7-kube-api-access-kr6jm\") pod \"auth-test-model-b30e99-kserve-7754746779-gzfkn\" (UID: \"566fc738-fff6-4ce2-bc29-16bfb9aee4b7\") " pod="ai-tenant-e2e-shared-a-w3-b30e99/auth-test-model-b30e99-kserve-7754746779-gzfkn"
Sep 08 15:52:25.160454 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:52:25.160372    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/566fc738-fff6-4ce2-bc29-16bfb9aee4b7-tls-certs\") pod \"auth-test-model-b30e99-kserve-7754746779-gzfkn\" (UID: \"566fc738-fff6-4ce2-bc29-16bfb9aee4b7\") " pod="ai-tenant-e2e-shared-a-w3-b30e99/auth-test-model-b30e99-kserve-7754746779-gzfkn"
Sep 08 15:52:25.160454 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:52:25.160395    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/566fc738-fff6-4ce2-bc29-16bfb9aee4b7-model-cache\") pod \"auth-test-model-b30e99-kserve-7754746779-gzfkn\" (UID: \"566fc738-fff6-4ce2-bc29-16bfb9aee4b7\") " pod="ai-tenant-e2e-shared-a-w3-b30e99/auth-test-model-b30e99-kserve-7754746779-gzfkn"
Sep 08 15:52:25.160454 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:52:25.160416    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/566fc738-fff6-4ce2-bc29-16bfb9aee4b7-dshm\") pod \"auth-test-model-b30e99-kserve-7754746779-gzfkn\" (UID: \"566fc738-fff6-4ce2-bc29-16bfb9aee4b7\") " pod="ai-tenant-e2e-shared-a-w3-b30e99/auth-test-model-b30e99-kserve-7754746779-gzfkn"
Sep 08 15:52:25.160597 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:52:25.160504    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/566fc738-fff6-4ce2-bc29-16bfb9aee4b7-home\") pod \"auth-test-model-b30e99-kserve-7754746779-gzfkn\" (UID: \"566fc738-fff6-4ce2-bc29-16bfb9aee4b7\") " pod="ai-tenant-e2e-shared-a-w3-b30e99/auth-test-model-b30e99-kserve-7754746779-gzfkn"
Sep 08 15:52:25.261766 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:52:25.261735    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kr6jm\" (UniqueName: \"kubernetes.io/projected/566fc738-fff6-4ce2-bc29-16bfb9aee4b7-kube-api-access-kr6jm\") pod \"auth-test-model-b30e99-kserve-7754746779-gzfkn\" (UID: \"566fc738-fff6-4ce2-bc29-16bfb9aee4b7\") " pod="ai-tenant-e2e-shared-a-w3-b30e99/auth-test-model-b30e99-kserve-7754746779-gzfkn"
Sep 08 15:52:25.261937 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:52:25.261777    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/566fc738-fff6-4ce2-bc29-16bfb9aee4b7-tls-certs\") pod \"auth-test-model-b30e99-kserve-7754746779-gzfkn\" (UID: \"566fc738-fff6-4ce2-bc29-16bfb9aee4b7\") " pod="ai-tenant-e2e-shared-a-w3-b30e99/auth-test-model-b30e99-kserve-7754746779-gzfkn"
Sep 08 15:52:25.261937 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:52:25.261809    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/566fc738-fff6-4ce2-bc29-16bfb9aee4b7-model-cache\") pod \"auth-test-model-b30e99-kserve-7754746779-gzfkn\" (UID: \"566fc738-fff6-4ce2-bc29-16bfb9aee4b7\") " pod="ai-tenant-e2e-shared-a-w3-b30e99/auth-test-model-b30e99-kserve-7754746779-gzfkn"
Sep 08 15:52:25.261937 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:52:25.261832    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/566fc738-fff6-4ce2-bc29-16bfb9aee4b7-dshm\") pod \"auth-test-model-b30e99-kserve-7754746779-gzfkn\" (UID: \"566fc738-fff6-4ce2-bc29-16bfb9aee4b7\") " pod="ai-tenant-e2e-shared-a-w3-b30e99/auth-test-model-b30e99-kserve-7754746779-gzfkn"
Sep 08 15:52:25.261937 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:52:25.261915    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/566fc738-fff6-4ce2-bc29-16bfb9aee4b7-home\") pod \"auth-test-model-b30e99-kserve-7754746779-gzfkn\" (UID: \"566fc738-fff6-4ce2-bc29-16bfb9aee4b7\") " pod="ai-tenant-e2e-shared-a-w3-b30e99/auth-test-model-b30e99-kserve-7754746779-gzfkn"
Sep 08 15:52:25.262162 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:52:25.261940    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/566fc738-fff6-4ce2-bc29-16bfb9aee4b7-tmp-dir\") pod \"auth-test-model-b30e99-kserve-7754746779-gzfkn\" (UID: \"566fc738-fff6-4ce2-bc29-16bfb9aee4b7\") " pod="ai-tenant-e2e-shared-a-w3-b30e99/auth-test-model-b30e99-kserve-7754746779-gzfkn"
Sep 08 15:52:25.262302 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:52:25.262281    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/566fc738-fff6-4ce2-bc29-16bfb9aee4b7-model-cache\") pod \"auth-test-model-b30e99-kserve-7754746779-gzfkn\" (UID: \"566fc738-fff6-4ce2-bc29-16bfb9aee4b7\") " pod="ai-tenant-e2e-shared-a-w3-b30e99/auth-test-model-b30e99-kserve-7754746779-gzfkn"
Sep 08 15:52:25.262370 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:52:25.262333    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/566fc738-fff6-4ce2-bc29-16bfb9aee4b7-tmp-dir\") pod \"auth-test-model-b30e99-kserve-7754746779-gzfkn\" (UID: \"566fc738-fff6-4ce2-bc29-16bfb9aee4b7\") " pod="ai-tenant-e2e-shared-a-w3-b30e99/auth-test-model-b30e99-kserve-7754746779-gzfkn"
Sep 08 15:52:25.262370 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:52:25.262359    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/566fc738-fff6-4ce2-bc29-16bfb9aee4b7-home\") pod \"auth-test-model-b30e99-kserve-7754746779-gzfkn\" (UID: \"566fc738-fff6-4ce2-bc29-16bfb9aee4b7\") " pod="ai-tenant-e2e-shared-a-w3-b30e99/auth-test-model-b30e99-kserve-7754746779-gzfkn"
Sep 08 15:52:25.264088 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:52:25.264063    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/566fc738-fff6-4ce2-bc29-16bfb9aee4b7-dshm\") pod \"auth-test-model-b30e99-kserve-7754746779-gzfkn\" (UID: \"566fc738-fff6-4ce2-bc29-16bfb9aee4b7\") " pod="ai-tenant-e2e-shared-a-w3-b30e99/auth-test-model-b30e99-kserve-7754746779-gzfkn"
Sep 08 15:52:25.264356 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:52:25.264338    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/566fc738-fff6-4ce2-bc29-16bfb9aee4b7-tls-certs\") pod \"auth-test-model-b30e99-kserve-7754746779-gzfkn\" (UID: \"566fc738-fff6-4ce2-bc29-16bfb9aee4b7\") " pod="ai-tenant-e2e-shared-a-w3-b30e99/auth-test-model-b30e99-kserve-7754746779-gzfkn"
Sep 08 15:52:25.269911 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:52:25.269889    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kr6jm\" (UniqueName: \"kubernetes.io/projected/566fc738-fff6-4ce2-bc29-16bfb9aee4b7-kube-api-access-kr6jm\") pod \"auth-test-model-b30e99-kserve-7754746779-gzfkn\" (UID: \"566fc738-fff6-4ce2-bc29-16bfb9aee4b7\") " pod="ai-tenant-e2e-shared-a-w3-b30e99/auth-test-model-b30e99-kserve-7754746779-gzfkn"
Sep 08 15:52:25.339044 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:52:25.338990    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-b30e99/auth-test-model-b30e99-kserve-7754746779-gzfkn"
Sep 08 15:52:25.617547 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:52:25.617520    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-b30e99/auth-test-model-b30e99-kserve-7754746779-gzfkn"]
Sep 08 15:52:25.619894 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:52:25.619845    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod566fc738_fff6_4ce2_bc29_16bfb9aee4b7.slice/crio-bdb6b53d32214046c2cf1e4b775bef688eb5632c8940ba83d3831de54335e59c WatchSource:0}: Error finding container bdb6b53d32214046c2cf1e4b775bef688eb5632c8940ba83d3831de54335e59c: Status 404 returned error can't find the container with id bdb6b53d32214046c2cf1e4b775bef688eb5632c8940ba83d3831de54335e59c
Sep 08 15:52:25.969392 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:52:25.969362    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-b30e99/auth-test-model-b30e99-kserve-7754746779-gzfkn" event={"ID":"566fc738-fff6-4ce2-bc29-16bfb9aee4b7","Type":"ContainerStarted","Data":"e3256a3bc4bf4e26e6bf1332dad79caf4700ecdd6692f9a1f58f8402e73b7c46"}
Sep 08 15:52:25.969561 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:52:25.969401    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-b30e99/auth-test-model-b30e99-kserve-7754746779-gzfkn" event={"ID":"566fc738-fff6-4ce2-bc29-16bfb9aee4b7","Type":"ContainerStarted","Data":"bdb6b53d32214046c2cf1e4b775bef688eb5632c8940ba83d3831de54335e59c"}
Sep 08 15:52:25.993708 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:52:25.993655    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-b30e99/auth-test-model-b30e99-kserve-7754746779-gzfkn" podStartSLOduration=0.993642034 podStartE2EDuration="993.642034ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:52:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:52:25.991848605 +0000 UTC m=+1184.081299695" watchObservedRunningTime="2026-09-08 15:52:25.993642034 +0000 UTC m=+1184.083093123"
Sep 08 15:52:34.311931 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:52:34.311895    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-7eb8750f/e2e-lifecycle-model-7eb8750f-kserve-787bfcdd59-pcwxf"]
Sep 08 15:52:34.314231 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:52:34.314206    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-7eb8750f/e2e-lifecycle-model-7eb8750f-kserve-787bfcdd59-pcwxf"
Sep 08 15:52:34.317379 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:52:34.317353    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-7eb8750f\"/\"openshift-service-ca.crt\""
Sep 08 15:52:34.318661 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:52:34.318634    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-7eb8750f\"/\"kube-root-ca.crt\""
Sep 08 15:52:34.318785 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:52:34.318643    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-7eb8750f\"/\"default-dockercfg-cbs9b\""
Sep 08 15:52:34.318785 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:52:34.318760    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-7eb8750f\"/\"e2e-lifecycle-model-7eb8750f-kserve-self-signed-certs\""
Sep 08 15:52:34.325445 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:52:34.325418    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-7eb8750f/e2e-lifecycle-model-7eb8750f-kserve-787bfcdd59-pcwxf"]
Sep 08 15:52:34.439693 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:52:34.439657    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc1375c9-af06-4eb8-9a0c-70be5a440705-home\") pod \"e2e-lifecycle-model-7eb8750f-kserve-787bfcdd59-pcwxf\" (UID: \"dc1375c9-af06-4eb8-9a0c-70be5a440705\") " pod="ai-tenant-e2e-mt-7eb8750f/e2e-lifecycle-model-7eb8750f-kserve-787bfcdd59-pcwxf"
Sep 08 15:52:34.439693 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:52:34.439700    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc1375c9-af06-4eb8-9a0c-70be5a440705-model-cache\") pod \"e2e-lifecycle-model-7eb8750f-kserve-787bfcdd59-pcwxf\" (UID: \"dc1375c9-af06-4eb8-9a0c-70be5a440705\") " pod="ai-tenant-e2e-mt-7eb8750f/e2e-lifecycle-model-7eb8750f-kserve-787bfcdd59-pcwxf"
Sep 08 15:52:34.439922 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:52:34.439734    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc1375c9-af06-4eb8-9a0c-70be5a440705-tmp-dir\") pod \"e2e-lifecycle-model-7eb8750f-kserve-787bfcdd59-pcwxf\" (UID: \"dc1375c9-af06-4eb8-9a0c-70be5a440705\") " pod="ai-tenant-e2e-mt-7eb8750f/e2e-lifecycle-model-7eb8750f-kserve-787bfcdd59-pcwxf"
Sep 08 15:52:34.439922 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:52:34.439758    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc1375c9-af06-4eb8-9a0c-70be5a440705-dshm\") pod \"e2e-lifecycle-model-7eb8750f-kserve-787bfcdd59-pcwxf\" (UID: \"dc1375c9-af06-4eb8-9a0c-70be5a440705\") " pod="ai-tenant-e2e-mt-7eb8750f/e2e-lifecycle-model-7eb8750f-kserve-787bfcdd59-pcwxf"
Sep 08 15:52:34.439922 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:52:34.439776    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d945d\" (UniqueName: \"kubernetes.io/projected/dc1375c9-af06-4eb8-9a0c-70be5a440705-kube-api-access-d945d\") pod \"e2e-lifecycle-model-7eb8750f-kserve-787bfcdd59-pcwxf\" (UID: \"dc1375c9-af06-4eb8-9a0c-70be5a440705\") " pod="ai-tenant-e2e-mt-7eb8750f/e2e-lifecycle-model-7eb8750f-kserve-787bfcdd59-pcwxf"
Sep 08 15:52:34.439922 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:52:34.439886    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc1375c9-af06-4eb8-9a0c-70be5a440705-tls-certs\") pod \"e2e-lifecycle-model-7eb8750f-kserve-787bfcdd59-pcwxf\" (UID: \"dc1375c9-af06-4eb8-9a0c-70be5a440705\") " pod="ai-tenant-e2e-mt-7eb8750f/e2e-lifecycle-model-7eb8750f-kserve-787bfcdd59-pcwxf"
Sep 08 15:52:34.540827 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:52:34.540791    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc1375c9-af06-4eb8-9a0c-70be5a440705-dshm\") pod \"e2e-lifecycle-model-7eb8750f-kserve-787bfcdd59-pcwxf\" (UID: \"dc1375c9-af06-4eb8-9a0c-70be5a440705\") " pod="ai-tenant-e2e-mt-7eb8750f/e2e-lifecycle-model-7eb8750f-kserve-787bfcdd59-pcwxf"
Sep 08 15:52:34.540827 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:52:34.540829    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d945d\" (UniqueName: \"kubernetes.io/projected/dc1375c9-af06-4eb8-9a0c-70be5a440705-kube-api-access-d945d\") pod \"e2e-lifecycle-model-7eb8750f-kserve-787bfcdd59-pcwxf\" (UID: \"dc1375c9-af06-4eb8-9a0c-70be5a440705\") " pod="ai-tenant-e2e-mt-7eb8750f/e2e-lifecycle-model-7eb8750f-kserve-787bfcdd59-pcwxf"
Sep 08 15:52:34.541060 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:52:34.540880    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc1375c9-af06-4eb8-9a0c-70be5a440705-tls-certs\") pod \"e2e-lifecycle-model-7eb8750f-kserve-787bfcdd59-pcwxf\" (UID: \"dc1375c9-af06-4eb8-9a0c-70be5a440705\") " pod="ai-tenant-e2e-mt-7eb8750f/e2e-lifecycle-model-7eb8750f-kserve-787bfcdd59-pcwxf"
Sep 08 15:52:34.541060 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:52:34.540948    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc1375c9-af06-4eb8-9a0c-70be5a440705-home\") pod \"e2e-lifecycle-model-7eb8750f-kserve-787bfcdd59-pcwxf\" (UID: \"dc1375c9-af06-4eb8-9a0c-70be5a440705\") " pod="ai-tenant-e2e-mt-7eb8750f/e2e-lifecycle-model-7eb8750f-kserve-787bfcdd59-pcwxf"
Sep 08 15:52:34.541060 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:52:34.540971    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc1375c9-af06-4eb8-9a0c-70be5a440705-model-cache\") pod \"e2e-lifecycle-model-7eb8750f-kserve-787bfcdd59-pcwxf\" (UID: \"dc1375c9-af06-4eb8-9a0c-70be5a440705\") " pod="ai-tenant-e2e-mt-7eb8750f/e2e-lifecycle-model-7eb8750f-kserve-787bfcdd59-pcwxf"
Sep 08 15:52:34.541060 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:52:34.541004    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc1375c9-af06-4eb8-9a0c-70be5a440705-tmp-dir\") pod \"e2e-lifecycle-model-7eb8750f-kserve-787bfcdd59-pcwxf\" (UID: \"dc1375c9-af06-4eb8-9a0c-70be5a440705\") " pod="ai-tenant-e2e-mt-7eb8750f/e2e-lifecycle-model-7eb8750f-kserve-787bfcdd59-pcwxf"
Sep 08 15:52:34.541364 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:52:34.541341    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc1375c9-af06-4eb8-9a0c-70be5a440705-tmp-dir\") pod \"e2e-lifecycle-model-7eb8750f-kserve-787bfcdd59-pcwxf\" (UID: \"dc1375c9-af06-4eb8-9a0c-70be5a440705\") " pod="ai-tenant-e2e-mt-7eb8750f/e2e-lifecycle-model-7eb8750f-kserve-787bfcdd59-pcwxf"
Sep 08 15:52:34.541446 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:52:34.541390    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc1375c9-af06-4eb8-9a0c-70be5a440705-model-cache\") pod \"e2e-lifecycle-model-7eb8750f-kserve-787bfcdd59-pcwxf\" (UID: \"dc1375c9-af06-4eb8-9a0c-70be5a440705\") " pod="ai-tenant-e2e-mt-7eb8750f/e2e-lifecycle-model-7eb8750f-kserve-787bfcdd59-pcwxf"
Sep 08 15:52:34.541446 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:52:34.541404    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc1375c9-af06-4eb8-9a0c-70be5a440705-home\") pod \"e2e-lifecycle-model-7eb8750f-kserve-787bfcdd59-pcwxf\" (UID: \"dc1375c9-af06-4eb8-9a0c-70be5a440705\") " pod="ai-tenant-e2e-mt-7eb8750f/e2e-lifecycle-model-7eb8750f-kserve-787bfcdd59-pcwxf"
Sep 08 15:52:34.543179 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:52:34.543156    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc1375c9-af06-4eb8-9a0c-70be5a440705-dshm\") pod \"e2e-lifecycle-model-7eb8750f-kserve-787bfcdd59-pcwxf\" (UID: \"dc1375c9-af06-4eb8-9a0c-70be5a440705\") " pod="ai-tenant-e2e-mt-7eb8750f/e2e-lifecycle-model-7eb8750f-kserve-787bfcdd59-pcwxf"
Sep 08 15:52:34.543336 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:52:34.543319    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc1375c9-af06-4eb8-9a0c-70be5a440705-tls-certs\") pod \"e2e-lifecycle-model-7eb8750f-kserve-787bfcdd59-pcwxf\" (UID: \"dc1375c9-af06-4eb8-9a0c-70be5a440705\") " pod="ai-tenant-e2e-mt-7eb8750f/e2e-lifecycle-model-7eb8750f-kserve-787bfcdd59-pcwxf"
Sep 08 15:52:34.548785 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:52:34.548765    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d945d\" (UniqueName: \"kubernetes.io/projected/dc1375c9-af06-4eb8-9a0c-70be5a440705-kube-api-access-d945d\") pod \"e2e-lifecycle-model-7eb8750f-kserve-787bfcdd59-pcwxf\" (UID: \"dc1375c9-af06-4eb8-9a0c-70be5a440705\") " pod="ai-tenant-e2e-mt-7eb8750f/e2e-lifecycle-model-7eb8750f-kserve-787bfcdd59-pcwxf"
Sep 08 15:52:34.630786 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:52:34.630717    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-7eb8750f/e2e-lifecycle-model-7eb8750f-kserve-787bfcdd59-pcwxf"
Sep 08 15:52:34.904777 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:52:34.904749    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-7eb8750f/e2e-lifecycle-model-7eb8750f-kserve-787bfcdd59-pcwxf"]
Sep 08 15:52:34.906806 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:52:34.906777    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddc1375c9_af06_4eb8_9a0c_70be5a440705.slice/crio-668a7330d8176e451079ab74b4a287b8a79553f35c4370a4072bb34decc27164 WatchSource:0}: Error finding container 668a7330d8176e451079ab74b4a287b8a79553f35c4370a4072bb34decc27164: Status 404 returned error can't find the container with id 668a7330d8176e451079ab74b4a287b8a79553f35c4370a4072bb34decc27164
Sep 08 15:52:35.015619 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:52:35.015588    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-7eb8750f/e2e-lifecycle-model-7eb8750f-kserve-787bfcdd59-pcwxf" event={"ID":"dc1375c9-af06-4eb8-9a0c-70be5a440705","Type":"ContainerStarted","Data":"d51b524ea9908d97c0d6cf62d248830c92c172045dd6da14357a660abbc1371a"}
Sep 08 15:52:35.015758 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:52:35.015631    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-7eb8750f/e2e-lifecycle-model-7eb8750f-kserve-787bfcdd59-pcwxf" event={"ID":"dc1375c9-af06-4eb8-9a0c-70be5a440705","Type":"ContainerStarted","Data":"668a7330d8176e451079ab74b4a287b8a79553f35c4370a4072bb34decc27164"}
Sep 08 15:52:35.339425 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:52:35.339328    2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-b30e99/auth-test-model-b30e99-kserve-7754746779-gzfkn"
Sep 08 15:52:35.339425 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:52:35.339391    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-b30e99/auth-test-model-b30e99-kserve-7754746779-gzfkn"
Sep 08 15:52:35.351587 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:52:35.351558    2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-b30e99/auth-test-model-b30e99-kserve-7754746779-gzfkn"
Sep 08 15:52:35.368549 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:52:35.368505    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-7eb8750f/e2e-lifecycle-model-7eb8750f-kserve-787bfcdd59-pcwxf" podStartSLOduration=1.368490828 podStartE2EDuration="1.368490828s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:52:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:52:35.034006588 +0000 UTC m=+1193.123457679" watchObservedRunningTime="2026-09-08 15:52:35.368490828 +0000 UTC m=+1193.457941917"
Sep 08 15:52:36.031791 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:52:36.031762    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-b30e99/auth-test-model-b30e99-kserve-7754746779-gzfkn"
Sep 08 15:52:42.390013 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:52:42.389985    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-wpwdq_5f7b0f68-a8e3-4058-b032-15f419682b2a/console-operator/2.log"
Sep 08 15:52:42.394219 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:52:42.394193    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-wpwdq_5f7b0f68-a8e3-4058-b032-15f419682b2a/console-operator/2.log"
Sep 08 15:52:42.397567 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:52:42.397548    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-f4vvt_195f1a74-19bc-49c9-aada-fa1c13488480/ovn-acl-logging/0.log"
Sep 08 15:52:42.401622 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:52:42.401605    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-f4vvt_195f1a74-19bc-49c9-aada-fa1c13488480/ovn-acl-logging/0.log"
Sep 08 15:52:44.631104 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:52:44.631072    2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-7eb8750f/e2e-lifecycle-model-7eb8750f-kserve-787bfcdd59-pcwxf"
Sep 08 15:52:44.631525 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:52:44.631137    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-7eb8750f/e2e-lifecycle-model-7eb8750f-kserve-787bfcdd59-pcwxf"
Sep 08 15:52:44.646327 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:52:44.646289    2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-7eb8750f/e2e-lifecycle-model-7eb8750f-kserve-787bfcdd59-pcwxf"
Sep 08 15:52:45.078499 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:52:45.078471    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-7eb8750f/e2e-lifecycle-model-7eb8750f-kserve-787bfcdd59-pcwxf"
Sep 08 15:53:18.166391 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:53:18.166357    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7eb8750f-569bd5896-xjnbd"]
Sep 08 15:53:18.166952 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:53:18.166639    2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7eb8750f-569bd5896-xjnbd" podUID="654c65b6-d200-4301-894b-ed6257d97987" containerName="maas-api" containerID="cri-o://7b68e2a4b87c636e5efc146055d98107187ce17297acd58c10179bba5a8187c4" gracePeriod=30
Sep 08 15:53:18.415387 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:53:18.415361    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7eb8750f-569bd5896-xjnbd"
Sep 08 15:53:18.512319 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:53:18.512236    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/654c65b6-d200-4301-894b-ed6257d97987-maas-api-tls\") pod \"654c65b6-d200-4301-894b-ed6257d97987\" (UID: \"654c65b6-d200-4301-894b-ed6257d97987\") "
Sep 08 15:53:18.512319 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:53:18.512299    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/654c65b6-d200-4301-894b-ed6257d97987-openshift-service-ca\") pod \"654c65b6-d200-4301-894b-ed6257d97987\" (UID: \"654c65b6-d200-4301-894b-ed6257d97987\") "
Sep 08 15:53:18.512556 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:53:18.512361    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g264v\" (UniqueName: \"kubernetes.io/projected/654c65b6-d200-4301-894b-ed6257d97987-kube-api-access-g264v\") pod \"654c65b6-d200-4301-894b-ed6257d97987\" (UID: \"654c65b6-d200-4301-894b-ed6257d97987\") "
Sep 08 15:53:18.512556 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:53:18.512390    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/654c65b6-d200-4301-894b-ed6257d97987-metrics-certs\") pod \"654c65b6-d200-4301-894b-ed6257d97987\" (UID: \"654c65b6-d200-4301-894b-ed6257d97987\") "
Sep 08 15:53:18.512696 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:53:18.512663    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/654c65b6-d200-4301-894b-ed6257d97987-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "654c65b6-d200-4301-894b-ed6257d97987" (UID: "654c65b6-d200-4301-894b-ed6257d97987"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 15:53:18.514668 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:53:18.514635    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/654c65b6-d200-4301-894b-ed6257d97987-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "654c65b6-d200-4301-894b-ed6257d97987" (UID: "654c65b6-d200-4301-894b-ed6257d97987"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 15:53:18.514944 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:53:18.514912    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/654c65b6-d200-4301-894b-ed6257d97987-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "654c65b6-d200-4301-894b-ed6257d97987" (UID: "654c65b6-d200-4301-894b-ed6257d97987"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 15:53:18.515034 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:53:18.514941    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/654c65b6-d200-4301-894b-ed6257d97987-kube-api-access-g264v" (OuterVolumeSpecName: "kube-api-access-g264v") pod "654c65b6-d200-4301-894b-ed6257d97987" (UID: "654c65b6-d200-4301-894b-ed6257d97987"). InnerVolumeSpecName "kube-api-access-g264v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:53:18.613909 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:53:18.613879    2549 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/654c65b6-d200-4301-894b-ed6257d97987-maas-api-tls\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:53:18.613909 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:53:18.613905    2549 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/654c65b6-d200-4301-894b-ed6257d97987-openshift-service-ca\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:53:18.614086 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:53:18.613916    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g264v\" (UniqueName: \"kubernetes.io/projected/654c65b6-d200-4301-894b-ed6257d97987-kube-api-access-g264v\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:53:18.614086 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:53:18.613927    2549 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/654c65b6-d200-4301-894b-ed6257d97987-metrics-certs\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:53:19.232209 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:53:19.232177    2549 generic.go:356] "Generic (PLEG): container finished" podID="654c65b6-d200-4301-894b-ed6257d97987" containerID="7b68e2a4b87c636e5efc146055d98107187ce17297acd58c10179bba5a8187c4" exitCode=0
Sep 08 15:53:19.232596 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:53:19.232225    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7eb8750f-569bd5896-xjnbd"
Sep 08 15:53:19.232596 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:53:19.232265    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7eb8750f-569bd5896-xjnbd" event={"ID":"654c65b6-d200-4301-894b-ed6257d97987","Type":"ContainerDied","Data":"7b68e2a4b87c636e5efc146055d98107187ce17297acd58c10179bba5a8187c4"}
Sep 08 15:53:19.232596 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:53:19.232320    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7eb8750f-569bd5896-xjnbd" event={"ID":"654c65b6-d200-4301-894b-ed6257d97987","Type":"ContainerDied","Data":"e8ec56c4692057883fb6a95d5700389e970d210fa1aa1b5aca3b327702f6342b"}
Sep 08 15:53:19.232596 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:53:19.232348    2549 scope.go:117] "RemoveContainer" containerID="7b68e2a4b87c636e5efc146055d98107187ce17297acd58c10179bba5a8187c4"
Sep 08 15:53:19.241962 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:53:19.241941    2549 scope.go:117] "RemoveContainer" containerID="7b68e2a4b87c636e5efc146055d98107187ce17297acd58c10179bba5a8187c4"
Sep 08 15:53:19.242239 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:53:19.242218    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b68e2a4b87c636e5efc146055d98107187ce17297acd58c10179bba5a8187c4\": container with ID starting with 7b68e2a4b87c636e5efc146055d98107187ce17297acd58c10179bba5a8187c4 not found: ID does not exist" containerID="7b68e2a4b87c636e5efc146055d98107187ce17297acd58c10179bba5a8187c4"
Sep 08 15:53:19.242326 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:53:19.242246    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b68e2a4b87c636e5efc146055d98107187ce17297acd58c10179bba5a8187c4"} err="failed to get container status \"7b68e2a4b87c636e5efc146055d98107187ce17297acd58c10179bba5a8187c4\": rpc error: code = NotFound desc = could not find container \"7b68e2a4b87c636e5efc146055d98107187ce17297acd58c10179bba5a8187c4\": container with ID starting with 7b68e2a4b87c636e5efc146055d98107187ce17297acd58c10179bba5a8187c4 not found: ID does not exist"
Sep 08 15:53:19.254480 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:53:19.254458    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7eb8750f-569bd5896-xjnbd"]
Sep 08 15:53:19.259673 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:53:19.259649    2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7eb8750f-569bd5896-xjnbd"]
Sep 08 15:53:20.400423 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:53:20.400389    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="654c65b6-d200-4301-894b-ed6257d97987" path="/var/lib/kubelet/pods/654c65b6-d200-4301-894b-ed6257d97987/volumes"
Sep 08 15:53:26.607158 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:53:26.607116    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-7eb8750f/e2e-lifecycle-model-7eb8750f-kserve-787bfcdd59-pcwxf"]
Sep 08 15:53:41.628089 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:53:41.627991    2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-7eb8750f/e2e-lifecycle-model-7eb8750f-kserve-787bfcdd59-pcwxf" podUID="dc1375c9-af06-4eb8-9a0c-70be5a440705" containerName="main" containerID="cri-o://d51b524ea9908d97c0d6cf62d248830c92c172045dd6da14357a660abbc1371a" gracePeriod=45
Sep 08 15:53:41.883633 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:53:41.883576    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-7eb8750f/e2e-lifecycle-model-7eb8750f-kserve-787bfcdd59-pcwxf"
Sep 08 15:53:42.006850 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:53:42.006817    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc1375c9-af06-4eb8-9a0c-70be5a440705-dshm\") pod \"dc1375c9-af06-4eb8-9a0c-70be5a440705\" (UID: \"dc1375c9-af06-4eb8-9a0c-70be5a440705\") "
Sep 08 15:53:42.007037 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:53:42.006919    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d945d\" (UniqueName: \"kubernetes.io/projected/dc1375c9-af06-4eb8-9a0c-70be5a440705-kube-api-access-d945d\") pod \"dc1375c9-af06-4eb8-9a0c-70be5a440705\" (UID: \"dc1375c9-af06-4eb8-9a0c-70be5a440705\") "
Sep 08 15:53:42.007037 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:53:42.006938    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc1375c9-af06-4eb8-9a0c-70be5a440705-home\") pod \"dc1375c9-af06-4eb8-9a0c-70be5a440705\" (UID: \"dc1375c9-af06-4eb8-9a0c-70be5a440705\") "
Sep 08 15:53:42.007037 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:53:42.006961    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc1375c9-af06-4eb8-9a0c-70be5a440705-model-cache\") pod \"dc1375c9-af06-4eb8-9a0c-70be5a440705\" (UID: \"dc1375c9-af06-4eb8-9a0c-70be5a440705\") "
Sep 08 15:53:42.007037 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:53:42.007031    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc1375c9-af06-4eb8-9a0c-70be5a440705-tmp-dir\") pod \"dc1375c9-af06-4eb8-9a0c-70be5a440705\" (UID: \"dc1375c9-af06-4eb8-9a0c-70be5a440705\") "
Sep 08 15:53:42.007256 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:53:42.007066    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc1375c9-af06-4eb8-9a0c-70be5a440705-tls-certs\") pod \"dc1375c9-af06-4eb8-9a0c-70be5a440705\" (UID: \"dc1375c9-af06-4eb8-9a0c-70be5a440705\") "
Sep 08 15:53:42.007256 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:53:42.007198    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc1375c9-af06-4eb8-9a0c-70be5a440705-home" (OuterVolumeSpecName: "home") pod "dc1375c9-af06-4eb8-9a0c-70be5a440705" (UID: "dc1375c9-af06-4eb8-9a0c-70be5a440705"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:53:42.007371 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:53:42.007262    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc1375c9-af06-4eb8-9a0c-70be5a440705-model-cache" (OuterVolumeSpecName: "model-cache") pod "dc1375c9-af06-4eb8-9a0c-70be5a440705" (UID: "dc1375c9-af06-4eb8-9a0c-70be5a440705"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:53:42.007371 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:53:42.007332    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc1375c9-af06-4eb8-9a0c-70be5a440705-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "dc1375c9-af06-4eb8-9a0c-70be5a440705" (UID: "dc1375c9-af06-4eb8-9a0c-70be5a440705"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:53:42.007488 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:53:42.007460    2549 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc1375c9-af06-4eb8-9a0c-70be5a440705-home\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:53:42.007488 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:53:42.007481    2549 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc1375c9-af06-4eb8-9a0c-70be5a440705-model-cache\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:53:42.007580 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:53:42.007497    2549 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc1375c9-af06-4eb8-9a0c-70be5a440705-tmp-dir\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:53:42.009118 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:53:42.009090    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc1375c9-af06-4eb8-9a0c-70be5a440705-dshm" (OuterVolumeSpecName: "dshm") pod "dc1375c9-af06-4eb8-9a0c-70be5a440705" (UID: "dc1375c9-af06-4eb8-9a0c-70be5a440705"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:53:42.009253 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:53:42.009159    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc1375c9-af06-4eb8-9a0c-70be5a440705-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "dc1375c9-af06-4eb8-9a0c-70be5a440705" (UID: "dc1375c9-af06-4eb8-9a0c-70be5a440705"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 15:53:42.009253 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:53:42.009188    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc1375c9-af06-4eb8-9a0c-70be5a440705-kube-api-access-d945d" (OuterVolumeSpecName: "kube-api-access-d945d") pod "dc1375c9-af06-4eb8-9a0c-70be5a440705" (UID: "dc1375c9-af06-4eb8-9a0c-70be5a440705"). InnerVolumeSpecName "kube-api-access-d945d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:53:42.108157 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:53:42.108118    2549 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc1375c9-af06-4eb8-9a0c-70be5a440705-tls-certs\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:53:42.108157 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:53:42.108144    2549 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc1375c9-af06-4eb8-9a0c-70be5a440705-dshm\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:53:42.108157 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:53:42.108155    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d945d\" (UniqueName: \"kubernetes.io/projected/dc1375c9-af06-4eb8-9a0c-70be5a440705-kube-api-access-d945d\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:53:42.358470 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:53:42.358438    2549 generic.go:356] "Generic (PLEG): container finished" podID="dc1375c9-af06-4eb8-9a0c-70be5a440705" containerID="d51b524ea9908d97c0d6cf62d248830c92c172045dd6da14357a660abbc1371a" exitCode=0
Sep 08 15:53:42.358635 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:53:42.358480    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-7eb8750f/e2e-lifecycle-model-7eb8750f-kserve-787bfcdd59-pcwxf"
Sep 08 15:53:42.358635 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:53:42.358514    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-7eb8750f/e2e-lifecycle-model-7eb8750f-kserve-787bfcdd59-pcwxf" event={"ID":"dc1375c9-af06-4eb8-9a0c-70be5a440705","Type":"ContainerDied","Data":"d51b524ea9908d97c0d6cf62d248830c92c172045dd6da14357a660abbc1371a"}
Sep 08 15:53:42.358635 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:53:42.358556    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-7eb8750f/e2e-lifecycle-model-7eb8750f-kserve-787bfcdd59-pcwxf" event={"ID":"dc1375c9-af06-4eb8-9a0c-70be5a440705","Type":"ContainerDied","Data":"668a7330d8176e451079ab74b4a287b8a79553f35c4370a4072bb34decc27164"}
Sep 08 15:53:42.358635 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:53:42.358574    2549 scope.go:117] "RemoveContainer" containerID="d51b524ea9908d97c0d6cf62d248830c92c172045dd6da14357a660abbc1371a"
Sep 08 15:53:42.370266 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:53:42.370243    2549 scope.go:117] "RemoveContainer" containerID="d51b524ea9908d97c0d6cf62d248830c92c172045dd6da14357a660abbc1371a"
Sep 08 15:53:42.370692 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:53:42.370653    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d51b524ea9908d97c0d6cf62d248830c92c172045dd6da14357a660abbc1371a\": container with ID starting with d51b524ea9908d97c0d6cf62d248830c92c172045dd6da14357a660abbc1371a not found: ID does not exist" containerID="d51b524ea9908d97c0d6cf62d248830c92c172045dd6da14357a660abbc1371a"
Sep 08 15:53:42.370777 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:53:42.370728    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d51b524ea9908d97c0d6cf62d248830c92c172045dd6da14357a660abbc1371a"} err="failed to get container status \"d51b524ea9908d97c0d6cf62d248830c92c172045dd6da14357a660abbc1371a\": rpc error: code = NotFound desc = could not find container \"d51b524ea9908d97c0d6cf62d248830c92c172045dd6da14357a660abbc1371a\": container with ID starting with d51b524ea9908d97c0d6cf62d248830c92c172045dd6da14357a660abbc1371a not found: ID does not exist"
Sep 08 15:53:42.386632 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:53:42.386611    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-7eb8750f/e2e-lifecycle-model-7eb8750f-kserve-787bfcdd59-pcwxf"]
Sep 08 15:53:42.390048 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:53:42.390028    2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-7eb8750f/e2e-lifecycle-model-7eb8750f-kserve-787bfcdd59-pcwxf"]
Sep 08 15:53:42.401786 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:53:42.401761    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dc1375c9-af06-4eb8-9a0c-70be5a440705" path="/var/lib/kubelet/pods/dc1375c9-af06-4eb8-9a0c-70be5a440705/volumes"
Sep 08 15:54:24.148117 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:54:24.148081    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-7eb8750f-openshift-default-76c7db7d88-cnfsh"]
Sep 08 15:54:24.148645 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:54:24.148303    2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-7eb8750f-openshift-default-76c7db7d88-cnfsh" podUID="c5f020da-9c3e-4f60-baf9-05d87fcee760" containerName="istio-proxy" containerID="cri-o://c21513b255c56c288b2d645f3ec3ef7e6f615537295b281e9e5b3b40815b5a4d" gracePeriod=30
Sep 08 15:54:29.394477 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:54:29.394452    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7eb8750f-openshift-default-76c7db7d88-cnfsh"
Sep 08 15:54:29.500391 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:54:29.500302    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5f020da-9c3e-4f60-baf9-05d87fcee760-istio-data\") pod \"c5f020da-9c3e-4f60-baf9-05d87fcee760\" (UID: \"c5f020da-9c3e-4f60-baf9-05d87fcee760\") "
Sep 08 15:54:29.500391 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:54:29.500348    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5f020da-9c3e-4f60-baf9-05d87fcee760-istiod-ca-cert\") pod \"c5f020da-9c3e-4f60-baf9-05d87fcee760\" (UID: \"c5f020da-9c3e-4f60-baf9-05d87fcee760\") "
Sep 08 15:54:29.500391 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:54:29.500374    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5f020da-9c3e-4f60-baf9-05d87fcee760-istio-envoy\") pod \"c5f020da-9c3e-4f60-baf9-05d87fcee760\" (UID: \"c5f020da-9c3e-4f60-baf9-05d87fcee760\") "
Sep 08 15:54:29.500688 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:54:29.500402    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5f020da-9c3e-4f60-baf9-05d87fcee760-istio-podinfo\") pod \"c5f020da-9c3e-4f60-baf9-05d87fcee760\" (UID: \"c5f020da-9c3e-4f60-baf9-05d87fcee760\") "
Sep 08 15:54:29.500688 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:54:29.500426    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5f020da-9c3e-4f60-baf9-05d87fcee760-istio-token\") pod \"c5f020da-9c3e-4f60-baf9-05d87fcee760\" (UID: \"c5f020da-9c3e-4f60-baf9-05d87fcee760\") "
Sep 08 15:54:29.500688 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:54:29.500483    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5f020da-9c3e-4f60-baf9-05d87fcee760-credential-socket\") pod \"c5f020da-9c3e-4f60-baf9-05d87fcee760\" (UID: \"c5f020da-9c3e-4f60-baf9-05d87fcee760\") "
Sep 08 15:54:29.500688 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:54:29.500517    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5f020da-9c3e-4f60-baf9-05d87fcee760-workload-socket\") pod \"c5f020da-9c3e-4f60-baf9-05d87fcee760\" (UID: \"c5f020da-9c3e-4f60-baf9-05d87fcee760\") "
Sep 08 15:54:29.500688 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:54:29.500545    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5f020da-9c3e-4f60-baf9-05d87fcee760-workload-certs\") pod \"c5f020da-9c3e-4f60-baf9-05d87fcee760\" (UID: \"c5f020da-9c3e-4f60-baf9-05d87fcee760\") "
Sep 08 15:54:29.500688 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:54:29.500574    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-txxtt\" (UniqueName: \"kubernetes.io/projected/c5f020da-9c3e-4f60-baf9-05d87fcee760-kube-api-access-txxtt\") pod \"c5f020da-9c3e-4f60-baf9-05d87fcee760\" (UID: \"c5f020da-9c3e-4f60-baf9-05d87fcee760\") "
Sep 08 15:54:29.500688 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:54:29.500575    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5f020da-9c3e-4f60-baf9-05d87fcee760-istio-data" (OuterVolumeSpecName: "istio-data") pod "c5f020da-9c3e-4f60-baf9-05d87fcee760" (UID: "c5f020da-9c3e-4f60-baf9-05d87fcee760"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:54:29.501094 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:54:29.500763    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5f020da-9c3e-4f60-baf9-05d87fcee760-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c5f020da-9c3e-4f60-baf9-05d87fcee760" (UID: "c5f020da-9c3e-4f60-baf9-05d87fcee760"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:54:29.501094 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:54:29.500826    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5f020da-9c3e-4f60-baf9-05d87fcee760-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c5f020da-9c3e-4f60-baf9-05d87fcee760" (UID: "c5f020da-9c3e-4f60-baf9-05d87fcee760"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 15:54:29.501094 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:54:29.500960    2549 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5f020da-9c3e-4f60-baf9-05d87fcee760-istio-data\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:54:29.501094 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:54:29.500980    2549 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5f020da-9c3e-4f60-baf9-05d87fcee760-istiod-ca-cert\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:54:29.501094 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:54:29.500994    2549 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5f020da-9c3e-4f60-baf9-05d87fcee760-workload-socket\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:54:29.501094 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:54:29.500990    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5f020da-9c3e-4f60-baf9-05d87fcee760-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c5f020da-9c3e-4f60-baf9-05d87fcee760" (UID: "c5f020da-9c3e-4f60-baf9-05d87fcee760"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:54:29.501094 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:54:29.501044    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5f020da-9c3e-4f60-baf9-05d87fcee760-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c5f020da-9c3e-4f60-baf9-05d87fcee760" (UID: "c5f020da-9c3e-4f60-baf9-05d87fcee760"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:54:29.502737 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:54:29.502706    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5f020da-9c3e-4f60-baf9-05d87fcee760-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c5f020da-9c3e-4f60-baf9-05d87fcee760" (UID: "c5f020da-9c3e-4f60-baf9-05d87fcee760"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:54:29.502926 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:54:29.502732    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c5f020da-9c3e-4f60-baf9-05d87fcee760-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c5f020da-9c3e-4f60-baf9-05d87fcee760" (UID: "c5f020da-9c3e-4f60-baf9-05d87fcee760"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 08 15:54:29.502926 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:54:29.502777    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5f020da-9c3e-4f60-baf9-05d87fcee760-kube-api-access-txxtt" (OuterVolumeSpecName: "kube-api-access-txxtt") pod "c5f020da-9c3e-4f60-baf9-05d87fcee760" (UID: "c5f020da-9c3e-4f60-baf9-05d87fcee760"). InnerVolumeSpecName "kube-api-access-txxtt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:54:29.502926 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:54:29.502821    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5f020da-9c3e-4f60-baf9-05d87fcee760-istio-token" (OuterVolumeSpecName: "istio-token") pod "c5f020da-9c3e-4f60-baf9-05d87fcee760" (UID: "c5f020da-9c3e-4f60-baf9-05d87fcee760"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:54:29.586649 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:54:29.586615    2549 generic.go:356] "Generic (PLEG): container finished" podID="c5f020da-9c3e-4f60-baf9-05d87fcee760" containerID="c21513b255c56c288b2d645f3ec3ef7e6f615537295b281e9e5b3b40815b5a4d" exitCode=0
Sep 08 15:54:29.586782 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:54:29.586661    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7eb8750f-openshift-default-76c7db7d88-cnfsh"
Sep 08 15:54:29.586782 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:54:29.586700    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7eb8750f-openshift-default-76c7db7d88-cnfsh" event={"ID":"c5f020da-9c3e-4f60-baf9-05d87fcee760","Type":"ContainerDied","Data":"c21513b255c56c288b2d645f3ec3ef7e6f615537295b281e9e5b3b40815b5a4d"}
Sep 08 15:54:29.586782 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:54:29.586757    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7eb8750f-openshift-default-76c7db7d88-cnfsh" event={"ID":"c5f020da-9c3e-4f60-baf9-05d87fcee760","Type":"ContainerDied","Data":"91fc043bfe19d449eb656955eb610124a6a2f52c3352651d3105eb34fe67f6a3"}
Sep 08 15:54:29.586929 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:54:29.586784    2549 scope.go:117] "RemoveContainer" containerID="c21513b255c56c288b2d645f3ec3ef7e6f615537295b281e9e5b3b40815b5a4d"
Sep 08 15:54:29.597904 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:54:29.596499    2549 scope.go:117] "RemoveContainer" containerID="c4fc9c1c9e6690e8e7e57043d148e2b6162e3e60af8c6520779649bfe0e5b28b"
Sep 08 15:54:29.601849 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:54:29.601828    2549 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5f020da-9c3e-4f60-baf9-05d87fcee760-istio-envoy\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:54:29.601849 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:54:29.601864    2549 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5f020da-9c3e-4f60-baf9-05d87fcee760-istio-podinfo\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:54:29.602004 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:54:29.601878    2549 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5f020da-9c3e-4f60-baf9-05d87fcee760-istio-token\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:54:29.602004 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:54:29.601890    2549 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5f020da-9c3e-4f60-baf9-05d87fcee760-credential-socket\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:54:29.602004 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:54:29.601903    2549 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5f020da-9c3e-4f60-baf9-05d87fcee760-workload-certs\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:54:29.602004 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:54:29.601917    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-txxtt\" (UniqueName: \"kubernetes.io/projected/c5f020da-9c3e-4f60-baf9-05d87fcee760-kube-api-access-txxtt\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:54:29.605267 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:54:29.605252    2549 scope.go:117] "RemoveContainer" containerID="c21513b255c56c288b2d645f3ec3ef7e6f615537295b281e9e5b3b40815b5a4d"
Sep 08 15:54:29.605527 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:54:29.605507    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c21513b255c56c288b2d645f3ec3ef7e6f615537295b281e9e5b3b40815b5a4d\": container with ID starting with c21513b255c56c288b2d645f3ec3ef7e6f615537295b281e9e5b3b40815b5a4d not found: ID does not exist" containerID="c21513b255c56c288b2d645f3ec3ef7e6f615537295b281e9e5b3b40815b5a4d"
Sep 08 15:54:29.605624 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:54:29.605536    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c21513b255c56c288b2d645f3ec3ef7e6f615537295b281e9e5b3b40815b5a4d"} err="failed to get container status \"c21513b255c56c288b2d645f3ec3ef7e6f615537295b281e9e5b3b40815b5a4d\": rpc error: code = NotFound desc = could not find container \"c21513b255c56c288b2d645f3ec3ef7e6f615537295b281e9e5b3b40815b5a4d\": container with ID starting with c21513b255c56c288b2d645f3ec3ef7e6f615537295b281e9e5b3b40815b5a4d not found: ID does not exist"
Sep 08 15:54:29.605624 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:54:29.605554    2549 scope.go:117] "RemoveContainer" containerID="c4fc9c1c9e6690e8e7e57043d148e2b6162e3e60af8c6520779649bfe0e5b28b"
Sep 08 15:54:29.605797 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:54:29.605778    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4fc9c1c9e6690e8e7e57043d148e2b6162e3e60af8c6520779649bfe0e5b28b\": container with ID starting with c4fc9c1c9e6690e8e7e57043d148e2b6162e3e60af8c6520779649bfe0e5b28b not found: ID does not exist" containerID="c4fc9c1c9e6690e8e7e57043d148e2b6162e3e60af8c6520779649bfe0e5b28b"
Sep 08 15:54:29.605889 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:54:29.605803    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4fc9c1c9e6690e8e7e57043d148e2b6162e3e60af8c6520779649bfe0e5b28b"} err="failed to get container status \"c4fc9c1c9e6690e8e7e57043d148e2b6162e3e60af8c6520779649bfe0e5b28b\": rpc error: code = NotFound desc = could not find container \"c4fc9c1c9e6690e8e7e57043d148e2b6162e3e60af8c6520779649bfe0e5b28b\": container with ID starting with c4fc9c1c9e6690e8e7e57043d148e2b6162e3e60af8c6520779649bfe0e5b28b not found: ID does not exist"
Sep 08 15:54:29.608681 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:54:29.608656    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-7eb8750f-openshift-default-76c7db7d88-cnfsh"]
Sep 08 15:54:29.612866 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:54:29.612836    2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-7eb8750f-openshift-default-76c7db7d88-cnfsh"]
Sep 08 15:54:30.400905 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:54:30.400851    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c5f020da-9c3e-4f60-baf9-05d87fcee760" path="/var/lib/kubelet/pods/c5f020da-9c3e-4f60-baf9-05d87fcee760/volumes"
Sep 08 15:55:01.038316 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:01.038272    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-b30e99/auth-test-model-b30e99-kserve-7754746779-gzfkn"]
Sep 08 15:55:11.944095 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:11.944057    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-b30e99/sub-test-model-b30e99-kserve-7b97bff7f5-jm5cg"]
Sep 08 15:55:11.944517 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:11.944395    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5f020da-9c3e-4f60-baf9-05d87fcee760" containerName="istio-proxy"
Sep 08 15:55:11.944517 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:11.944406    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5f020da-9c3e-4f60-baf9-05d87fcee760" containerName="istio-proxy"
Sep 08 15:55:11.944517 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:11.944420    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc1375c9-af06-4eb8-9a0c-70be5a440705" containerName="main"
Sep 08 15:55:11.944517 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:11.944428    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc1375c9-af06-4eb8-9a0c-70be5a440705" containerName="main"
Sep 08 15:55:11.944517 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:11.944440    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="654c65b6-d200-4301-894b-ed6257d97987" containerName="maas-api"
Sep 08 15:55:11.944517 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:11.944445    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="654c65b6-d200-4301-894b-ed6257d97987" containerName="maas-api"
Sep 08 15:55:11.944517 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:11.944500    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="c5f020da-9c3e-4f60-baf9-05d87fcee760" containerName="istio-proxy"
Sep 08 15:55:11.944517 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:11.944508    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="654c65b6-d200-4301-894b-ed6257d97987" containerName="maas-api"
Sep 08 15:55:11.944517 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:11.944515    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="c5f020da-9c3e-4f60-baf9-05d87fcee760" containerName="istio-proxy"
Sep 08 15:55:11.944517 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:11.944522    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc1375c9-af06-4eb8-9a0c-70be5a440705" containerName="main"
Sep 08 15:55:11.947408 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:11.947391    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-b30e99/sub-test-model-b30e99-kserve-7b97bff7f5-jm5cg"
Sep 08 15:55:11.950026 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:11.950005    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-b30e99\"/\"sub-test-model-b30e99-kserve-self-signed-certs\""
Sep 08 15:55:11.956544 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:11.956517    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-b30e99/sub-test-model-b30e99-kserve-7b97bff7f5-jm5cg"]
Sep 08 15:55:12.079641 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:12.079600    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/155f8e63-bfc2-476c-bbee-674645c242b3-tmp-dir\") pod \"sub-test-model-b30e99-kserve-7b97bff7f5-jm5cg\" (UID: \"155f8e63-bfc2-476c-bbee-674645c242b3\") " pod="ai-tenant-e2e-shared-a-w3-b30e99/sub-test-model-b30e99-kserve-7b97bff7f5-jm5cg"
Sep 08 15:55:12.079802 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:12.079676    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/155f8e63-bfc2-476c-bbee-674645c242b3-dshm\") pod \"sub-test-model-b30e99-kserve-7b97bff7f5-jm5cg\" (UID: \"155f8e63-bfc2-476c-bbee-674645c242b3\") " pod="ai-tenant-e2e-shared-a-w3-b30e99/sub-test-model-b30e99-kserve-7b97bff7f5-jm5cg"
Sep 08 15:55:12.079802 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:12.079744    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/155f8e63-bfc2-476c-bbee-674645c242b3-model-cache\") pod \"sub-test-model-b30e99-kserve-7b97bff7f5-jm5cg\" (UID: \"155f8e63-bfc2-476c-bbee-674645c242b3\") " pod="ai-tenant-e2e-shared-a-w3-b30e99/sub-test-model-b30e99-kserve-7b97bff7f5-jm5cg"
Sep 08 15:55:12.079802 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:12.079781    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/155f8e63-bfc2-476c-bbee-674645c242b3-tls-certs\") pod \"sub-test-model-b30e99-kserve-7b97bff7f5-jm5cg\" (UID: \"155f8e63-bfc2-476c-bbee-674645c242b3\") " pod="ai-tenant-e2e-shared-a-w3-b30e99/sub-test-model-b30e99-kserve-7b97bff7f5-jm5cg"
Sep 08 15:55:12.079802 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:12.079798    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/155f8e63-bfc2-476c-bbee-674645c242b3-home\") pod \"sub-test-model-b30e99-kserve-7b97bff7f5-jm5cg\" (UID: \"155f8e63-bfc2-476c-bbee-674645c242b3\") " pod="ai-tenant-e2e-shared-a-w3-b30e99/sub-test-model-b30e99-kserve-7b97bff7f5-jm5cg"
Sep 08 15:55:12.080037 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:12.079919    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzqcz\" (UniqueName: \"kubernetes.io/projected/155f8e63-bfc2-476c-bbee-674645c242b3-kube-api-access-vzqcz\") pod \"sub-test-model-b30e99-kserve-7b97bff7f5-jm5cg\" (UID: \"155f8e63-bfc2-476c-bbee-674645c242b3\") " pod="ai-tenant-e2e-shared-a-w3-b30e99/sub-test-model-b30e99-kserve-7b97bff7f5-jm5cg"
Sep 08 15:55:12.181351 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:12.181315    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/155f8e63-bfc2-476c-bbee-674645c242b3-dshm\") pod \"sub-test-model-b30e99-kserve-7b97bff7f5-jm5cg\" (UID: \"155f8e63-bfc2-476c-bbee-674645c242b3\") " pod="ai-tenant-e2e-shared-a-w3-b30e99/sub-test-model-b30e99-kserve-7b97bff7f5-jm5cg"
Sep 08 15:55:12.181537 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:12.181362    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/155f8e63-bfc2-476c-bbee-674645c242b3-model-cache\") pod \"sub-test-model-b30e99-kserve-7b97bff7f5-jm5cg\" (UID: \"155f8e63-bfc2-476c-bbee-674645c242b3\") " pod="ai-tenant-e2e-shared-a-w3-b30e99/sub-test-model-b30e99-kserve-7b97bff7f5-jm5cg"
Sep 08 15:55:12.181537 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:12.181402    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/155f8e63-bfc2-476c-bbee-674645c242b3-tls-certs\") pod \"sub-test-model-b30e99-kserve-7b97bff7f5-jm5cg\" (UID: \"155f8e63-bfc2-476c-bbee-674645c242b3\") " pod="ai-tenant-e2e-shared-a-w3-b30e99/sub-test-model-b30e99-kserve-7b97bff7f5-jm5cg"
Sep 08 15:55:12.181537 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:12.181420    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/155f8e63-bfc2-476c-bbee-674645c242b3-home\") pod \"sub-test-model-b30e99-kserve-7b97bff7f5-jm5cg\" (UID: \"155f8e63-bfc2-476c-bbee-674645c242b3\") " pod="ai-tenant-e2e-shared-a-w3-b30e99/sub-test-model-b30e99-kserve-7b97bff7f5-jm5cg"
Sep 08 15:55:12.181537 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:12.181450    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzqcz\" (UniqueName: \"kubernetes.io/projected/155f8e63-bfc2-476c-bbee-674645c242b3-kube-api-access-vzqcz\") pod \"sub-test-model-b30e99-kserve-7b97bff7f5-jm5cg\" (UID: \"155f8e63-bfc2-476c-bbee-674645c242b3\") " pod="ai-tenant-e2e-shared-a-w3-b30e99/sub-test-model-b30e99-kserve-7b97bff7f5-jm5cg"
Sep 08 15:55:12.181537 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:12.181486    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/155f8e63-bfc2-476c-bbee-674645c242b3-tmp-dir\") pod \"sub-test-model-b30e99-kserve-7b97bff7f5-jm5cg\" (UID: \"155f8e63-bfc2-476c-bbee-674645c242b3\") " pod="ai-tenant-e2e-shared-a-w3-b30e99/sub-test-model-b30e99-kserve-7b97bff7f5-jm5cg"
Sep 08 15:55:12.181842 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:12.181819    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/155f8e63-bfc2-476c-bbee-674645c242b3-model-cache\") pod \"sub-test-model-b30e99-kserve-7b97bff7f5-jm5cg\" (UID: \"155f8e63-bfc2-476c-bbee-674645c242b3\") " pod="ai-tenant-e2e-shared-a-w3-b30e99/sub-test-model-b30e99-kserve-7b97bff7f5-jm5cg"
Sep 08 15:55:12.181926 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:12.181878    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/155f8e63-bfc2-476c-bbee-674645c242b3-home\") pod \"sub-test-model-b30e99-kserve-7b97bff7f5-jm5cg\" (UID: \"155f8e63-bfc2-476c-bbee-674645c242b3\") " pod="ai-tenant-e2e-shared-a-w3-b30e99/sub-test-model-b30e99-kserve-7b97bff7f5-jm5cg"
Sep 08 15:55:12.181976 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:12.181952    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/155f8e63-bfc2-476c-bbee-674645c242b3-tmp-dir\") pod \"sub-test-model-b30e99-kserve-7b97bff7f5-jm5cg\" (UID: \"155f8e63-bfc2-476c-bbee-674645c242b3\") " pod="ai-tenant-e2e-shared-a-w3-b30e99/sub-test-model-b30e99-kserve-7b97bff7f5-jm5cg"
Sep 08 15:55:12.183588 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:12.183560    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/155f8e63-bfc2-476c-bbee-674645c242b3-dshm\") pod \"sub-test-model-b30e99-kserve-7b97bff7f5-jm5cg\" (UID: \"155f8e63-bfc2-476c-bbee-674645c242b3\") " pod="ai-tenant-e2e-shared-a-w3-b30e99/sub-test-model-b30e99-kserve-7b97bff7f5-jm5cg"
Sep 08 15:55:12.183869 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:12.183835    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/155f8e63-bfc2-476c-bbee-674645c242b3-tls-certs\") pod \"sub-test-model-b30e99-kserve-7b97bff7f5-jm5cg\" (UID: \"155f8e63-bfc2-476c-bbee-674645c242b3\") " pod="ai-tenant-e2e-shared-a-w3-b30e99/sub-test-model-b30e99-kserve-7b97bff7f5-jm5cg"
Sep 08 15:55:12.197743 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:12.197693    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzqcz\" (UniqueName: \"kubernetes.io/projected/155f8e63-bfc2-476c-bbee-674645c242b3-kube-api-access-vzqcz\") pod \"sub-test-model-b30e99-kserve-7b97bff7f5-jm5cg\" (UID: \"155f8e63-bfc2-476c-bbee-674645c242b3\") " pod="ai-tenant-e2e-shared-a-w3-b30e99/sub-test-model-b30e99-kserve-7b97bff7f5-jm5cg"
Sep 08 15:55:12.256996 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:12.256970    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-b30e99/sub-test-model-b30e99-kserve-7b97bff7f5-jm5cg"
Sep 08 15:55:12.529327 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:12.529301    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-b30e99/sub-test-model-b30e99-kserve-7b97bff7f5-jm5cg"]
Sep 08 15:55:12.531260 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:55:12.531237    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod155f8e63_bfc2_476c_bbee_674645c242b3.slice/crio-e0b7320cf08ccbf201f475ff10789eaecd92bf941abc8cd0a723844efd70fb4c WatchSource:0}: Error finding container e0b7320cf08ccbf201f475ff10789eaecd92bf941abc8cd0a723844efd70fb4c: Status 404 returned error can't find the container with id e0b7320cf08ccbf201f475ff10789eaecd92bf941abc8cd0a723844efd70fb4c
Sep 08 15:55:12.798319 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:12.798240    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-b30e99/sub-test-model-b30e99-kserve-7b97bff7f5-jm5cg" event={"ID":"155f8e63-bfc2-476c-bbee-674645c242b3","Type":"ContainerStarted","Data":"dbd01e43dc037c05406c2a29cf8c362801bdba26a36e5bebbce3767240cad50e"}
Sep 08 15:55:12.798319 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:12.798279    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-b30e99/sub-test-model-b30e99-kserve-7b97bff7f5-jm5cg" event={"ID":"155f8e63-bfc2-476c-bbee-674645c242b3","Type":"ContainerStarted","Data":"e0b7320cf08ccbf201f475ff10789eaecd92bf941abc8cd0a723844efd70fb4c"}
Sep 08 15:55:12.819190 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:12.819138    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-b30e99/sub-test-model-b30e99-kserve-7b97bff7f5-jm5cg" podStartSLOduration=1.819123003 podStartE2EDuration="1.819123003s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:55:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:55:12.816587893 +0000 UTC m=+1350.906038983" watchObservedRunningTime="2026-09-08 15:55:12.819123003 +0000 UTC m=+1350.908574092"
Sep 08 15:55:16.057380 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:16.057338    2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-b30e99/auth-test-model-b30e99-kserve-7754746779-gzfkn" podUID="566fc738-fff6-4ce2-bc29-16bfb9aee4b7" containerName="main" containerID="cri-o://e3256a3bc4bf4e26e6bf1332dad79caf4700ecdd6692f9a1f58f8402e73b7c46" gracePeriod=45
Sep 08 15:55:16.308891 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:16.308825    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-b30e99/auth-test-model-b30e99-kserve-7754746779-gzfkn"
Sep 08 15:55:16.416731 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:16.416700    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/566fc738-fff6-4ce2-bc29-16bfb9aee4b7-model-cache\") pod \"566fc738-fff6-4ce2-bc29-16bfb9aee4b7\" (UID: \"566fc738-fff6-4ce2-bc29-16bfb9aee4b7\") "
Sep 08 15:55:16.416929 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:16.416756    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/566fc738-fff6-4ce2-bc29-16bfb9aee4b7-tls-certs\") pod \"566fc738-fff6-4ce2-bc29-16bfb9aee4b7\" (UID: \"566fc738-fff6-4ce2-bc29-16bfb9aee4b7\") "
Sep 08 15:55:16.416929 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:16.416785    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/566fc738-fff6-4ce2-bc29-16bfb9aee4b7-tmp-dir\") pod \"566fc738-fff6-4ce2-bc29-16bfb9aee4b7\" (UID: \"566fc738-fff6-4ce2-bc29-16bfb9aee4b7\") "
Sep 08 15:55:16.416929 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:16.416874    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/566fc738-fff6-4ce2-bc29-16bfb9aee4b7-dshm\") pod \"566fc738-fff6-4ce2-bc29-16bfb9aee4b7\" (UID: \"566fc738-fff6-4ce2-bc29-16bfb9aee4b7\") "
Sep 08 15:55:16.416929 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:16.416901    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/566fc738-fff6-4ce2-bc29-16bfb9aee4b7-home\") pod \"566fc738-fff6-4ce2-bc29-16bfb9aee4b7\" (UID: \"566fc738-fff6-4ce2-bc29-16bfb9aee4b7\") "
Sep 08 15:55:16.417131 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:16.416957    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kr6jm\" (UniqueName: \"kubernetes.io/projected/566fc738-fff6-4ce2-bc29-16bfb9aee4b7-kube-api-access-kr6jm\") pod \"566fc738-fff6-4ce2-bc29-16bfb9aee4b7\" (UID: \"566fc738-fff6-4ce2-bc29-16bfb9aee4b7\") "
Sep 08 15:55:16.417131 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:16.416984    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/566fc738-fff6-4ce2-bc29-16bfb9aee4b7-model-cache" (OuterVolumeSpecName: "model-cache") pod "566fc738-fff6-4ce2-bc29-16bfb9aee4b7" (UID: "566fc738-fff6-4ce2-bc29-16bfb9aee4b7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:55:16.417131 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:16.417053    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/566fc738-fff6-4ce2-bc29-16bfb9aee4b7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "566fc738-fff6-4ce2-bc29-16bfb9aee4b7" (UID: "566fc738-fff6-4ce2-bc29-16bfb9aee4b7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:55:16.417275 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:16.417192    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/566fc738-fff6-4ce2-bc29-16bfb9aee4b7-home" (OuterVolumeSpecName: "home") pod "566fc738-fff6-4ce2-bc29-16bfb9aee4b7" (UID: "566fc738-fff6-4ce2-bc29-16bfb9aee4b7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:55:16.417275 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:16.417268    2549 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/566fc738-fff6-4ce2-bc29-16bfb9aee4b7-home\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:55:16.417368 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:16.417287    2549 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/566fc738-fff6-4ce2-bc29-16bfb9aee4b7-model-cache\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:55:16.417368 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:16.417300    2549 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/566fc738-fff6-4ce2-bc29-16bfb9aee4b7-tmp-dir\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:55:16.419057 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:16.419022    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/566fc738-fff6-4ce2-bc29-16bfb9aee4b7-dshm" (OuterVolumeSpecName: "dshm") pod "566fc738-fff6-4ce2-bc29-16bfb9aee4b7" (UID: "566fc738-fff6-4ce2-bc29-16bfb9aee4b7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:55:16.419173 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:16.419117    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/566fc738-fff6-4ce2-bc29-16bfb9aee4b7-kube-api-access-kr6jm" (OuterVolumeSpecName: "kube-api-access-kr6jm") pod "566fc738-fff6-4ce2-bc29-16bfb9aee4b7" (UID: "566fc738-fff6-4ce2-bc29-16bfb9aee4b7"). InnerVolumeSpecName "kube-api-access-kr6jm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:55:16.419173 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:16.419123    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/566fc738-fff6-4ce2-bc29-16bfb9aee4b7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "566fc738-fff6-4ce2-bc29-16bfb9aee4b7" (UID: "566fc738-fff6-4ce2-bc29-16bfb9aee4b7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 15:55:16.518024 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:16.517999    2549 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/566fc738-fff6-4ce2-bc29-16bfb9aee4b7-dshm\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:55:16.518169 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:16.518026    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kr6jm\" (UniqueName: \"kubernetes.io/projected/566fc738-fff6-4ce2-bc29-16bfb9aee4b7-kube-api-access-kr6jm\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:55:16.518169 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:16.518041    2549 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/566fc738-fff6-4ce2-bc29-16bfb9aee4b7-tls-certs\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:55:16.819599 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:16.819569    2549 generic.go:356] "Generic (PLEG): container finished" podID="566fc738-fff6-4ce2-bc29-16bfb9aee4b7" containerID="e3256a3bc4bf4e26e6bf1332dad79caf4700ecdd6692f9a1f58f8402e73b7c46" exitCode=0
Sep 08 15:55:16.819777 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:16.819618    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-b30e99/auth-test-model-b30e99-kserve-7754746779-gzfkn"
Sep 08 15:55:16.819777 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:16.819631    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-b30e99/auth-test-model-b30e99-kserve-7754746779-gzfkn" event={"ID":"566fc738-fff6-4ce2-bc29-16bfb9aee4b7","Type":"ContainerDied","Data":"e3256a3bc4bf4e26e6bf1332dad79caf4700ecdd6692f9a1f58f8402e73b7c46"}
Sep 08 15:55:16.819777 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:16.819662    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-b30e99/auth-test-model-b30e99-kserve-7754746779-gzfkn" event={"ID":"566fc738-fff6-4ce2-bc29-16bfb9aee4b7","Type":"ContainerDied","Data":"bdb6b53d32214046c2cf1e4b775bef688eb5632c8940ba83d3831de54335e59c"}
Sep 08 15:55:16.819777 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:16.819679    2549 scope.go:117] "RemoveContainer" containerID="e3256a3bc4bf4e26e6bf1332dad79caf4700ecdd6692f9a1f58f8402e73b7c46"
Sep 08 15:55:16.828795 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:16.828778    2549 scope.go:117] "RemoveContainer" containerID="e3256a3bc4bf4e26e6bf1332dad79caf4700ecdd6692f9a1f58f8402e73b7c46"
Sep 08 15:55:16.829072 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:55:16.829048    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3256a3bc4bf4e26e6bf1332dad79caf4700ecdd6692f9a1f58f8402e73b7c46\": container with ID starting with e3256a3bc4bf4e26e6bf1332dad79caf4700ecdd6692f9a1f58f8402e73b7c46 not found: ID does not exist" containerID="e3256a3bc4bf4e26e6bf1332dad79caf4700ecdd6692f9a1f58f8402e73b7c46"
Sep 08 15:55:16.829163 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:16.829081    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3256a3bc4bf4e26e6bf1332dad79caf4700ecdd6692f9a1f58f8402e73b7c46"} err="failed to get container status \"e3256a3bc4bf4e26e6bf1332dad79caf4700ecdd6692f9a1f58f8402e73b7c46\": rpc error: code = NotFound desc = could not find container \"e3256a3bc4bf4e26e6bf1332dad79caf4700ecdd6692f9a1f58f8402e73b7c46\": container with ID starting with e3256a3bc4bf4e26e6bf1332dad79caf4700ecdd6692f9a1f58f8402e73b7c46 not found: ID does not exist"
Sep 08 15:55:16.842778 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:16.842750    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-b30e99/auth-test-model-b30e99-kserve-7754746779-gzfkn"]
Sep 08 15:55:16.849791 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:16.849771    2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-b30e99/auth-test-model-b30e99-kserve-7754746779-gzfkn"]
Sep 08 15:55:18.400905 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:18.400847    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="566fc738-fff6-4ce2-bc29-16bfb9aee4b7" path="/var/lib/kubelet/pods/566fc738-fff6-4ce2-bc29-16bfb9aee4b7/volumes"
Sep 08 15:55:19.598045 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:19.598010    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-dfa395-openshift-default-799698ff7b-lkxcs"]
Sep 08 15:55:19.598396 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:19.598357    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5f020da-9c3e-4f60-baf9-05d87fcee760" containerName="istio-proxy"
Sep 08 15:55:19.598396 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:19.598369    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5f020da-9c3e-4f60-baf9-05d87fcee760" containerName="istio-proxy"
Sep 08 15:55:19.598396 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:19.598385    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="566fc738-fff6-4ce2-bc29-16bfb9aee4b7" containerName="main"
Sep 08 15:55:19.598396 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:19.598390    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="566fc738-fff6-4ce2-bc29-16bfb9aee4b7" containerName="main"
Sep 08 15:55:19.598529 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:19.598444    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="566fc738-fff6-4ce2-bc29-16bfb9aee4b7" containerName="main"
Sep 08 15:55:19.602639 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:19.602619    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-dfa395-openshift-default-799698ff7b-lkxcs"
Sep 08 15:55:19.605795 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:19.605764    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-dfa395-openshift-default-dockercfg-cqsct\""
Sep 08 15:55:19.625122 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:19.625100    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-dfa395-openshift-default-799698ff7b-lkxcs"]
Sep 08 15:55:19.745550 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:19.745513    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f73cff5c-48e7-4eae-816d-6c4fb8cc8e54-istio-data\") pod \"e2e-api-a-dfa395-openshift-default-799698ff7b-lkxcs\" (UID: \"f73cff5c-48e7-4eae-816d-6c4fb8cc8e54\") " pod="openshift-ingress/e2e-api-a-dfa395-openshift-default-799698ff7b-lkxcs"
Sep 08 15:55:19.745728 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:19.745557    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f73cff5c-48e7-4eae-816d-6c4fb8cc8e54-istio-envoy\") pod \"e2e-api-a-dfa395-openshift-default-799698ff7b-lkxcs\" (UID: \"f73cff5c-48e7-4eae-816d-6c4fb8cc8e54\") " pod="openshift-ingress/e2e-api-a-dfa395-openshift-default-799698ff7b-lkxcs"
Sep 08 15:55:19.745728 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:19.745636    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f73cff5c-48e7-4eae-816d-6c4fb8cc8e54-workload-certs\") pod \"e2e-api-a-dfa395-openshift-default-799698ff7b-lkxcs\" (UID: \"f73cff5c-48e7-4eae-816d-6c4fb8cc8e54\") " pod="openshift-ingress/e2e-api-a-dfa395-openshift-default-799698ff7b-lkxcs"
Sep 08 15:55:19.745728 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:19.745672    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f73cff5c-48e7-4eae-816d-6c4fb8cc8e54-istiod-ca-cert\") pod \"e2e-api-a-dfa395-openshift-default-799698ff7b-lkxcs\" (UID: \"f73cff5c-48e7-4eae-816d-6c4fb8cc8e54\") " pod="openshift-ingress/e2e-api-a-dfa395-openshift-default-799698ff7b-lkxcs"
Sep 08 15:55:19.745728 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:19.745709    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f73cff5c-48e7-4eae-816d-6c4fb8cc8e54-workload-socket\") pod \"e2e-api-a-dfa395-openshift-default-799698ff7b-lkxcs\" (UID: \"f73cff5c-48e7-4eae-816d-6c4fb8cc8e54\") " pod="openshift-ingress/e2e-api-a-dfa395-openshift-default-799698ff7b-lkxcs"
Sep 08 15:55:19.746000 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:19.745739    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f73cff5c-48e7-4eae-816d-6c4fb8cc8e54-istio-token\") pod \"e2e-api-a-dfa395-openshift-default-799698ff7b-lkxcs\" (UID: \"f73cff5c-48e7-4eae-816d-6c4fb8cc8e54\") " pod="openshift-ingress/e2e-api-a-dfa395-openshift-default-799698ff7b-lkxcs"
Sep 08 15:55:19.746000 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:19.745819    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f73cff5c-48e7-4eae-816d-6c4fb8cc8e54-credential-socket\") pod \"e2e-api-a-dfa395-openshift-default-799698ff7b-lkxcs\" (UID: \"f73cff5c-48e7-4eae-816d-6c4fb8cc8e54\") " pod="openshift-ingress/e2e-api-a-dfa395-openshift-default-799698ff7b-lkxcs"
Sep 08 15:55:19.746000 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:19.745888    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnlpx\" (UniqueName: \"kubernetes.io/projected/f73cff5c-48e7-4eae-816d-6c4fb8cc8e54-kube-api-access-fnlpx\") pod \"e2e-api-a-dfa395-openshift-default-799698ff7b-lkxcs\" (UID: \"f73cff5c-48e7-4eae-816d-6c4fb8cc8e54\") " pod="openshift-ingress/e2e-api-a-dfa395-openshift-default-799698ff7b-lkxcs"
Sep 08 15:55:19.746000 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:19.745935    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f73cff5c-48e7-4eae-816d-6c4fb8cc8e54-istio-podinfo\") pod \"e2e-api-a-dfa395-openshift-default-799698ff7b-lkxcs\" (UID: \"f73cff5c-48e7-4eae-816d-6c4fb8cc8e54\") " pod="openshift-ingress/e2e-api-a-dfa395-openshift-default-799698ff7b-lkxcs"
Sep 08 15:55:19.847287 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:19.847255    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f73cff5c-48e7-4eae-816d-6c4fb8cc8e54-istio-podinfo\") pod \"e2e-api-a-dfa395-openshift-default-799698ff7b-lkxcs\" (UID: \"f73cff5c-48e7-4eae-816d-6c4fb8cc8e54\") " pod="openshift-ingress/e2e-api-a-dfa395-openshift-default-799698ff7b-lkxcs"
Sep 08 15:55:19.847472 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:19.847301    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f73cff5c-48e7-4eae-816d-6c4fb8cc8e54-istio-data\") pod \"e2e-api-a-dfa395-openshift-default-799698ff7b-lkxcs\" (UID: \"f73cff5c-48e7-4eae-816d-6c4fb8cc8e54\") " pod="openshift-ingress/e2e-api-a-dfa395-openshift-default-799698ff7b-lkxcs"
Sep 08 15:55:19.847472 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:19.847340    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f73cff5c-48e7-4eae-816d-6c4fb8cc8e54-istio-envoy\") pod \"e2e-api-a-dfa395-openshift-default-799698ff7b-lkxcs\" (UID: \"f73cff5c-48e7-4eae-816d-6c4fb8cc8e54\") " pod="openshift-ingress/e2e-api-a-dfa395-openshift-default-799698ff7b-lkxcs"
Sep 08 15:55:19.847472 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:19.847390    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f73cff5c-48e7-4eae-816d-6c4fb8cc8e54-workload-certs\") pod \"e2e-api-a-dfa395-openshift-default-799698ff7b-lkxcs\" (UID: \"f73cff5c-48e7-4eae-816d-6c4fb8cc8e54\") " pod="openshift-ingress/e2e-api-a-dfa395-openshift-default-799698ff7b-lkxcs"
Sep 08 15:55:19.847880 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:19.847532    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f73cff5c-48e7-4eae-816d-6c4fb8cc8e54-istiod-ca-cert\") pod \"e2e-api-a-dfa395-openshift-default-799698ff7b-lkxcs\" (UID: \"f73cff5c-48e7-4eae-816d-6c4fb8cc8e54\") " pod="openshift-ingress/e2e-api-a-dfa395-openshift-default-799698ff7b-lkxcs"
Sep 08 15:55:19.847880 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:19.847579    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f73cff5c-48e7-4eae-816d-6c4fb8cc8e54-workload-socket\") pod \"e2e-api-a-dfa395-openshift-default-799698ff7b-lkxcs\" (UID: \"f73cff5c-48e7-4eae-816d-6c4fb8cc8e54\") " pod="openshift-ingress/e2e-api-a-dfa395-openshift-default-799698ff7b-lkxcs"
Sep 08 15:55:19.847880 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:19.847622    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f73cff5c-48e7-4eae-816d-6c4fb8cc8e54-istio-token\") pod \"e2e-api-a-dfa395-openshift-default-799698ff7b-lkxcs\" (UID: \"f73cff5c-48e7-4eae-816d-6c4fb8cc8e54\") " pod="openshift-ingress/e2e-api-a-dfa395-openshift-default-799698ff7b-lkxcs"
Sep 08 15:55:19.847880 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:19.847668    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f73cff5c-48e7-4eae-816d-6c4fb8cc8e54-credential-socket\") pod \"e2e-api-a-dfa395-openshift-default-799698ff7b-lkxcs\" (UID: \"f73cff5c-48e7-4eae-816d-6c4fb8cc8e54\") " pod="openshift-ingress/e2e-api-a-dfa395-openshift-default-799698ff7b-lkxcs"
Sep 08 15:55:19.847880 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:19.847713    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnlpx\" (UniqueName: \"kubernetes.io/projected/f73cff5c-48e7-4eae-816d-6c4fb8cc8e54-kube-api-access-fnlpx\") pod \"e2e-api-a-dfa395-openshift-default-799698ff7b-lkxcs\" (UID: \"f73cff5c-48e7-4eae-816d-6c4fb8cc8e54\") " pod="openshift-ingress/e2e-api-a-dfa395-openshift-default-799698ff7b-lkxcs"
Sep 08 15:55:19.847880 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:19.847811    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f73cff5c-48e7-4eae-816d-6c4fb8cc8e54-istio-data\") pod \"e2e-api-a-dfa395-openshift-default-799698ff7b-lkxcs\" (UID: \"f73cff5c-48e7-4eae-816d-6c4fb8cc8e54\") " pod="openshift-ingress/e2e-api-a-dfa395-openshift-default-799698ff7b-lkxcs"
Sep 08 15:55:19.848251 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:19.848001    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f73cff5c-48e7-4eae-816d-6c4fb8cc8e54-workload-socket\") pod \"e2e-api-a-dfa395-openshift-default-799698ff7b-lkxcs\" (UID: \"f73cff5c-48e7-4eae-816d-6c4fb8cc8e54\") " pod="openshift-ingress/e2e-api-a-dfa395-openshift-default-799698ff7b-lkxcs"
Sep 08 15:55:19.848251 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:19.848137    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f73cff5c-48e7-4eae-816d-6c4fb8cc8e54-credential-socket\") pod \"e2e-api-a-dfa395-openshift-default-799698ff7b-lkxcs\" (UID: \"f73cff5c-48e7-4eae-816d-6c4fb8cc8e54\") " pod="openshift-ingress/e2e-api-a-dfa395-openshift-default-799698ff7b-lkxcs"
Sep 08 15:55:19.848251 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:19.848231    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f73cff5c-48e7-4eae-816d-6c4fb8cc8e54-workload-certs\") pod \"e2e-api-a-dfa395-openshift-default-799698ff7b-lkxcs\" (UID: \"f73cff5c-48e7-4eae-816d-6c4fb8cc8e54\") " pod="openshift-ingress/e2e-api-a-dfa395-openshift-default-799698ff7b-lkxcs"
Sep 08 15:55:19.848360 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:19.848268    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f73cff5c-48e7-4eae-816d-6c4fb8cc8e54-istiod-ca-cert\") pod \"e2e-api-a-dfa395-openshift-default-799698ff7b-lkxcs\" (UID: \"f73cff5c-48e7-4eae-816d-6c4fb8cc8e54\") " pod="openshift-ingress/e2e-api-a-dfa395-openshift-default-799698ff7b-lkxcs"
Sep 08 15:55:19.849643 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:19.849618    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f73cff5c-48e7-4eae-816d-6c4fb8cc8e54-istio-envoy\") pod \"e2e-api-a-dfa395-openshift-default-799698ff7b-lkxcs\" (UID: \"f73cff5c-48e7-4eae-816d-6c4fb8cc8e54\") " pod="openshift-ingress/e2e-api-a-dfa395-openshift-default-799698ff7b-lkxcs"
Sep 08 15:55:19.849919 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:19.849902    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f73cff5c-48e7-4eae-816d-6c4fb8cc8e54-istio-podinfo\") pod \"e2e-api-a-dfa395-openshift-default-799698ff7b-lkxcs\" (UID: \"f73cff5c-48e7-4eae-816d-6c4fb8cc8e54\") " pod="openshift-ingress/e2e-api-a-dfa395-openshift-default-799698ff7b-lkxcs"
Sep 08 15:55:19.856705 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:19.856679    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f73cff5c-48e7-4eae-816d-6c4fb8cc8e54-istio-token\") pod \"e2e-api-a-dfa395-openshift-default-799698ff7b-lkxcs\" (UID: \"f73cff5c-48e7-4eae-816d-6c4fb8cc8e54\") " pod="openshift-ingress/e2e-api-a-dfa395-openshift-default-799698ff7b-lkxcs"
Sep 08 15:55:19.856836 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:19.856818    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnlpx\" (UniqueName: \"kubernetes.io/projected/f73cff5c-48e7-4eae-816d-6c4fb8cc8e54-kube-api-access-fnlpx\") pod \"e2e-api-a-dfa395-openshift-default-799698ff7b-lkxcs\" (UID: \"f73cff5c-48e7-4eae-816d-6c4fb8cc8e54\") " pod="openshift-ingress/e2e-api-a-dfa395-openshift-default-799698ff7b-lkxcs"
Sep 08 15:55:19.913405 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:19.913384    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-dfa395-openshift-default-799698ff7b-lkxcs"
Sep 08 15:55:20.189257 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:20.189231    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-dfa395-openshift-default-799698ff7b-lkxcs"]
Sep 08 15:55:20.191881 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:55:20.191841    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf73cff5c_48e7_4eae_816d_6c4fb8cc8e54.slice/crio-9a9d8a5b12f48d176617a27e7434f237115ce3ed1204d8d86d8c569c68207fbb WatchSource:0}: Error finding container 9a9d8a5b12f48d176617a27e7434f237115ce3ed1204d8d86d8c569c68207fbb: Status 404 returned error can't find the container with id 9a9d8a5b12f48d176617a27e7434f237115ce3ed1204d8d86d8c569c68207fbb
Sep 08 15:55:20.194502 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:20.194421    2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 08 15:55:20.194632 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:20.194561    2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 08 15:55:20.194632 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:20.194604    2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 08 15:55:20.843831 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:20.843745    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-dfa395-openshift-default-799698ff7b-lkxcs" event={"ID":"f73cff5c-48e7-4eae-816d-6c4fb8cc8e54","Type":"ContainerStarted","Data":"6c67ad5d7d6ac98213eef902291a31b2d996a772ec9d55e7b881519d384897b5"}
Sep 08 15:55:20.843831 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:20.843808    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-dfa395-openshift-default-799698ff7b-lkxcs" event={"ID":"f73cff5c-48e7-4eae-816d-6c4fb8cc8e54","Type":"ContainerStarted","Data":"9a9d8a5b12f48d176617a27e7434f237115ce3ed1204d8d86d8c569c68207fbb"}
Sep 08 15:55:20.868016 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:20.867963    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-dfa395-openshift-default-799698ff7b-lkxcs" podStartSLOduration=1.867941914 podStartE2EDuration="1.867941914s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:55:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:55:20.867237686 +0000 UTC m=+1358.956688776" watchObservedRunningTime="2026-09-08 15:55:20.867941914 +0000 UTC m=+1358.957393004"
Sep 08 15:55:20.914311 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:20.914283    2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-dfa395-openshift-default-799698ff7b-lkxcs"
Sep 08 15:55:21.920066 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:21.920033    2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-dfa395-openshift-default-799698ff7b-lkxcs"
Sep 08 15:55:22.258028 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:22.257925    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-b30e99/sub-test-model-b30e99-kserve-7b97bff7f5-jm5cg"
Sep 08 15:55:22.258028 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:22.257997    2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-b30e99/sub-test-model-b30e99-kserve-7b97bff7f5-jm5cg"
Sep 08 15:55:22.270649 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:22.270622    2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-b30e99/sub-test-model-b30e99-kserve-7b97bff7f5-jm5cg"
Sep 08 15:55:22.853549 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:22.853509    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-dfa395-openshift-default-799698ff7b-lkxcs"
Sep 08 15:55:22.854354 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:22.854335    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-dfa395-openshift-default-799698ff7b-lkxcs"
Sep 08 15:55:22.865466 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:22.865441    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-b30e99/sub-test-model-b30e99-kserve-7b97bff7f5-jm5cg"
Sep 08 15:55:24.079962 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:24.079929    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-dfa395-55797df967-xrc7t"]
Sep 08 15:55:24.083791 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:24.083772    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dfa395-55797df967-xrc7t"
Sep 08 15:55:24.087342 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:24.087314    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sj5j8\" (UniqueName: \"kubernetes.io/projected/d3b500bf-9cdf-4400-ac73-cb430b14798d-kube-api-access-sj5j8\") pod \"maas-api-e2e-api-a-dfa395-55797df967-xrc7t\" (UID: \"d3b500bf-9cdf-4400-ac73-cb430b14798d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dfa395-55797df967-xrc7t"
Sep 08 15:55:24.087454 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:24.087355    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3b500bf-9cdf-4400-ac73-cb430b14798d-metrics-certs\") pod \"maas-api-e2e-api-a-dfa395-55797df967-xrc7t\" (UID: \"d3b500bf-9cdf-4400-ac73-cb430b14798d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dfa395-55797df967-xrc7t"
Sep 08 15:55:24.087454 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:24.087424    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3b500bf-9cdf-4400-ac73-cb430b14798d-openshift-service-ca\") pod \"maas-api-e2e-api-a-dfa395-55797df967-xrc7t\" (UID: \"d3b500bf-9cdf-4400-ac73-cb430b14798d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dfa395-55797df967-xrc7t"
Sep 08 15:55:24.087561 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:24.087541    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3b500bf-9cdf-4400-ac73-cb430b14798d-maas-api-tls\") pod \"maas-api-e2e-api-a-dfa395-55797df967-xrc7t\" (UID: \"d3b500bf-9cdf-4400-ac73-cb430b14798d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dfa395-55797df967-xrc7t"
Sep 08 15:55:24.089020 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:24.089004    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-dfa395-serving-cert\""
Sep 08 15:55:24.101453 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:24.101433    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-dfa395-55797df967-xrc7t"]
Sep 08 15:55:24.188174 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:24.188141    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3b500bf-9cdf-4400-ac73-cb430b14798d-openshift-service-ca\") pod \"maas-api-e2e-api-a-dfa395-55797df967-xrc7t\" (UID: \"d3b500bf-9cdf-4400-ac73-cb430b14798d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dfa395-55797df967-xrc7t"
Sep 08 15:55:24.188321 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:24.188229    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3b500bf-9cdf-4400-ac73-cb430b14798d-maas-api-tls\") pod \"maas-api-e2e-api-a-dfa395-55797df967-xrc7t\" (UID: \"d3b500bf-9cdf-4400-ac73-cb430b14798d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dfa395-55797df967-xrc7t"
Sep 08 15:55:24.188321 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:24.188265    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sj5j8\" (UniqueName: \"kubernetes.io/projected/d3b500bf-9cdf-4400-ac73-cb430b14798d-kube-api-access-sj5j8\") pod \"maas-api-e2e-api-a-dfa395-55797df967-xrc7t\" (UID: \"d3b500bf-9cdf-4400-ac73-cb430b14798d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dfa395-55797df967-xrc7t"
Sep 08 15:55:24.188321 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:24.188282    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3b500bf-9cdf-4400-ac73-cb430b14798d-metrics-certs\") pod \"maas-api-e2e-api-a-dfa395-55797df967-xrc7t\" (UID: \"d3b500bf-9cdf-4400-ac73-cb430b14798d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dfa395-55797df967-xrc7t"
Sep 08 15:55:24.188485 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:55:24.188384    2549 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-dfa395-serving-cert: secret "maas-api-e2e-api-a-dfa395-serving-cert" not found
Sep 08 15:55:24.188485 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:55:24.188458    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3b500bf-9cdf-4400-ac73-cb430b14798d-maas-api-tls podName:d3b500bf-9cdf-4400-ac73-cb430b14798d nodeName:}" failed. No retries permitted until 2026-09-08 15:55:24.688435947 +0000 UTC m=+1362.777887029 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d3b500bf-9cdf-4400-ac73-cb430b14798d-maas-api-tls") pod "maas-api-e2e-api-a-dfa395-55797df967-xrc7t" (UID: "d3b500bf-9cdf-4400-ac73-cb430b14798d") : secret "maas-api-e2e-api-a-dfa395-serving-cert" not found
Sep 08 15:55:24.188707 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:24.188686    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3b500bf-9cdf-4400-ac73-cb430b14798d-openshift-service-ca\") pod \"maas-api-e2e-api-a-dfa395-55797df967-xrc7t\" (UID: \"d3b500bf-9cdf-4400-ac73-cb430b14798d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dfa395-55797df967-xrc7t"
Sep 08 15:55:24.190630 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:24.190610    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3b500bf-9cdf-4400-ac73-cb430b14798d-metrics-certs\") pod \"maas-api-e2e-api-a-dfa395-55797df967-xrc7t\" (UID: \"d3b500bf-9cdf-4400-ac73-cb430b14798d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dfa395-55797df967-xrc7t"
Sep 08 15:55:24.207751 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:24.205220    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sj5j8\" (UniqueName: \"kubernetes.io/projected/d3b500bf-9cdf-4400-ac73-cb430b14798d-kube-api-access-sj5j8\") pod \"maas-api-e2e-api-a-dfa395-55797df967-xrc7t\" (UID: \"d3b500bf-9cdf-4400-ac73-cb430b14798d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dfa395-55797df967-xrc7t"
Sep 08 15:55:24.692033 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:24.691996    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3b500bf-9cdf-4400-ac73-cb430b14798d-maas-api-tls\") pod \"maas-api-e2e-api-a-dfa395-55797df967-xrc7t\" (UID: \"d3b500bf-9cdf-4400-ac73-cb430b14798d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dfa395-55797df967-xrc7t"
Sep 08 15:55:24.694435 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:24.694404    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3b500bf-9cdf-4400-ac73-cb430b14798d-maas-api-tls\") pod \"maas-api-e2e-api-a-dfa395-55797df967-xrc7t\" (UID: \"d3b500bf-9cdf-4400-ac73-cb430b14798d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dfa395-55797df967-xrc7t"
Sep 08 15:55:24.694582 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:24.694532    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dfa395-55797df967-xrc7t"
Sep 08 15:55:24.973930 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:24.973906    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-dfa395-55797df967-xrc7t"]
Sep 08 15:55:24.975901 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:55:24.975873    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3b500bf_9cdf_4400_ac73_cb430b14798d.slice/crio-b30d66ca73588aaf8b22dbd028615f689e058b89116dd31744b51adb39b54051 WatchSource:0}: Error finding container b30d66ca73588aaf8b22dbd028615f689e058b89116dd31744b51adb39b54051: Status 404 returned error can't find the container with id b30d66ca73588aaf8b22dbd028615f689e058b89116dd31744b51adb39b54051
Sep 08 15:55:25.868436 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:25.868403    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dfa395-55797df967-xrc7t" event={"ID":"d3b500bf-9cdf-4400-ac73-cb430b14798d","Type":"ContainerStarted","Data":"b30d66ca73588aaf8b22dbd028615f689e058b89116dd31744b51adb39b54051"}
Sep 08 15:55:26.875216 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:26.875173    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dfa395-55797df967-xrc7t" event={"ID":"d3b500bf-9cdf-4400-ac73-cb430b14798d","Type":"ContainerStarted","Data":"33bcfd9b61c485a0ec1c34543d1cb67456cfb50f146e6fccc4c9af4de7b3de14"}
Sep 08 15:55:26.875602 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:26.875321    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dfa395-55797df967-xrc7t"
Sep 08 15:55:26.893208 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:26.893157    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dfa395-55797df967-xrc7t" podStartSLOduration=1.814216874 podStartE2EDuration="2.893136128s" totalImagesPullingTime="1.078919254s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:55:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:55:26.891493396 +0000 UTC m=+1364.980944500" watchObservedRunningTime="2026-09-08 15:55:26.893136128 +0000 UTC m=+1364.982587218"
Sep 08 15:55:32.885562 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:32.885529    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dfa395-55797df967-xrc7t"
Sep 08 15:55:43.960283 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:43.960250    2549 generic.go:356] "Generic (PLEG): container finished" podID="f73cff5c-48e7-4eae-816d-6c4fb8cc8e54" containerID="6c67ad5d7d6ac98213eef902291a31b2d996a772ec9d55e7b881519d384897b5" exitCode=0
Sep 08 15:55:43.960789 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:43.960307    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-dfa395-openshift-default-799698ff7b-lkxcs" event={"ID":"f73cff5c-48e7-4eae-816d-6c4fb8cc8e54","Type":"ContainerDied","Data":"6c67ad5d7d6ac98213eef902291a31b2d996a772ec9d55e7b881519d384897b5"}
Sep 08 15:55:43.960789 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:43.960778    2549 scope.go:117] "RemoveContainer" containerID="6c67ad5d7d6ac98213eef902291a31b2d996a772ec9d55e7b881519d384897b5"
Sep 08 15:55:43.961906 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:43.961876    2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 08 15:55:43.961996 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:43.961945    2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 08 15:55:43.962035 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:43.961978    2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 08 15:55:44.971254 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:44.971210    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-dfa395-openshift-default-799698ff7b-lkxcs" event={"ID":"f73cff5c-48e7-4eae-816d-6c4fb8cc8e54","Type":"ContainerStarted","Data":"028606c80d2592725bac7d696948326106a195b025d9f783feb34e90a5564d2b"}
Sep 08 15:55:45.538508 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:45.538472    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-642d3b/sub-test-model-642d3b-kserve-585c8f7f4f-k8mml"]
Sep 08 15:55:45.540804 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:45.540788    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-642d3b/sub-test-model-642d3b-kserve-585c8f7f4f-k8mml"
Sep 08 15:55:45.544644 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:45.544615    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-642d3b\"/\"openshift-service-ca.crt\""
Sep 08 15:55:45.544924 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:45.544903    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-642d3b\"/\"default-dockercfg-sp62x\""
Sep 08 15:55:45.545598 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:45.544942    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-642d3b\"/\"kube-root-ca.crt\""
Sep 08 15:55:45.545598 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:45.545043    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-642d3b\"/\"sub-test-model-642d3b-kserve-self-signed-certs\""
Sep 08 15:55:45.546022 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:45.546002    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-642d3b/sub-test-model-642d3b-kserve-585c8f7f4f-k8mml"]
Sep 08 15:55:45.681435 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:45.681401    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ec415c7-2031-4610-9e09-6eb7e37ef0d6-home\") pod \"sub-test-model-642d3b-kserve-585c8f7f4f-k8mml\" (UID: \"7ec415c7-2031-4610-9e09-6eb7e37ef0d6\") " pod="ai-tenant-e2e-shared-b-w3-642d3b/sub-test-model-642d3b-kserve-585c8f7f4f-k8mml"
Sep 08 15:55:45.681435 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:45.681438    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ft4m\" (UniqueName: \"kubernetes.io/projected/7ec415c7-2031-4610-9e09-6eb7e37ef0d6-kube-api-access-5ft4m\") pod \"sub-test-model-642d3b-kserve-585c8f7f4f-k8mml\" (UID: \"7ec415c7-2031-4610-9e09-6eb7e37ef0d6\") " pod="ai-tenant-e2e-shared-b-w3-642d3b/sub-test-model-642d3b-kserve-585c8f7f4f-k8mml"
Sep 08 15:55:45.681667 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:45.681474    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ec415c7-2031-4610-9e09-6eb7e37ef0d6-model-cache\") pod \"sub-test-model-642d3b-kserve-585c8f7f4f-k8mml\" (UID: \"7ec415c7-2031-4610-9e09-6eb7e37ef0d6\") " pod="ai-tenant-e2e-shared-b-w3-642d3b/sub-test-model-642d3b-kserve-585c8f7f4f-k8mml"
Sep 08 15:55:45.681667 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:45.681532    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ec415c7-2031-4610-9e09-6eb7e37ef0d6-tls-certs\") pod \"sub-test-model-642d3b-kserve-585c8f7f4f-k8mml\" (UID: \"7ec415c7-2031-4610-9e09-6eb7e37ef0d6\") " pod="ai-tenant-e2e-shared-b-w3-642d3b/sub-test-model-642d3b-kserve-585c8f7f4f-k8mml"
Sep 08 15:55:45.681667 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:45.681610    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ec415c7-2031-4610-9e09-6eb7e37ef0d6-tmp-dir\") pod \"sub-test-model-642d3b-kserve-585c8f7f4f-k8mml\" (UID: \"7ec415c7-2031-4610-9e09-6eb7e37ef0d6\") " pod="ai-tenant-e2e-shared-b-w3-642d3b/sub-test-model-642d3b-kserve-585c8f7f4f-k8mml"
Sep 08 15:55:45.681775 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:45.681663    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ec415c7-2031-4610-9e09-6eb7e37ef0d6-dshm\") pod \"sub-test-model-642d3b-kserve-585c8f7f4f-k8mml\" (UID: \"7ec415c7-2031-4610-9e09-6eb7e37ef0d6\") " pod="ai-tenant-e2e-shared-b-w3-642d3b/sub-test-model-642d3b-kserve-585c8f7f4f-k8mml"
Sep 08 15:55:45.782572 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:45.782535    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ec415c7-2031-4610-9e09-6eb7e37ef0d6-tls-certs\") pod \"sub-test-model-642d3b-kserve-585c8f7f4f-k8mml\" (UID: \"7ec415c7-2031-4610-9e09-6eb7e37ef0d6\") " pod="ai-tenant-e2e-shared-b-w3-642d3b/sub-test-model-642d3b-kserve-585c8f7f4f-k8mml"
Sep 08 15:55:45.782782 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:45.782623    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ec415c7-2031-4610-9e09-6eb7e37ef0d6-tmp-dir\") pod \"sub-test-model-642d3b-kserve-585c8f7f4f-k8mml\" (UID: \"7ec415c7-2031-4610-9e09-6eb7e37ef0d6\") " pod="ai-tenant-e2e-shared-b-w3-642d3b/sub-test-model-642d3b-kserve-585c8f7f4f-k8mml"
Sep 08 15:55:45.782782 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:45.782648    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ec415c7-2031-4610-9e09-6eb7e37ef0d6-dshm\") pod \"sub-test-model-642d3b-kserve-585c8f7f4f-k8mml\" (UID: \"7ec415c7-2031-4610-9e09-6eb7e37ef0d6\") " pod="ai-tenant-e2e-shared-b-w3-642d3b/sub-test-model-642d3b-kserve-585c8f7f4f-k8mml"
Sep 08 15:55:45.782782 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:45.782691    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ec415c7-2031-4610-9e09-6eb7e37ef0d6-home\") pod \"sub-test-model-642d3b-kserve-585c8f7f4f-k8mml\" (UID: \"7ec415c7-2031-4610-9e09-6eb7e37ef0d6\") " pod="ai-tenant-e2e-shared-b-w3-642d3b/sub-test-model-642d3b-kserve-585c8f7f4f-k8mml"
Sep 08 15:55:45.782782 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:45.782721    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ft4m\" (UniqueName: \"kubernetes.io/projected/7ec415c7-2031-4610-9e09-6eb7e37ef0d6-kube-api-access-5ft4m\") pod \"sub-test-model-642d3b-kserve-585c8f7f4f-k8mml\" (UID: \"7ec415c7-2031-4610-9e09-6eb7e37ef0d6\") " pod="ai-tenant-e2e-shared-b-w3-642d3b/sub-test-model-642d3b-kserve-585c8f7f4f-k8mml"
Sep 08 15:55:45.782782 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:45.782773    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ec415c7-2031-4610-9e09-6eb7e37ef0d6-model-cache\") pod \"sub-test-model-642d3b-kserve-585c8f7f4f-k8mml\" (UID: \"7ec415c7-2031-4610-9e09-6eb7e37ef0d6\") " pod="ai-tenant-e2e-shared-b-w3-642d3b/sub-test-model-642d3b-kserve-585c8f7f4f-k8mml"
Sep 08 15:55:45.783302 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:45.783275    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ec415c7-2031-4610-9e09-6eb7e37ef0d6-home\") pod \"sub-test-model-642d3b-kserve-585c8f7f4f-k8mml\" (UID: \"7ec415c7-2031-4610-9e09-6eb7e37ef0d6\") " pod="ai-tenant-e2e-shared-b-w3-642d3b/sub-test-model-642d3b-kserve-585c8f7f4f-k8mml"
Sep 08 15:55:45.783406 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:45.783309    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ec415c7-2031-4610-9e09-6eb7e37ef0d6-tmp-dir\") pod \"sub-test-model-642d3b-kserve-585c8f7f4f-k8mml\" (UID: \"7ec415c7-2031-4610-9e09-6eb7e37ef0d6\") " pod="ai-tenant-e2e-shared-b-w3-642d3b/sub-test-model-642d3b-kserve-585c8f7f4f-k8mml"
Sep 08 15:55:45.783561 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:45.783540    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ec415c7-2031-4610-9e09-6eb7e37ef0d6-model-cache\") pod \"sub-test-model-642d3b-kserve-585c8f7f4f-k8mml\" (UID: \"7ec415c7-2031-4610-9e09-6eb7e37ef0d6\") " pod="ai-tenant-e2e-shared-b-w3-642d3b/sub-test-model-642d3b-kserve-585c8f7f4f-k8mml"
Sep 08 15:55:45.785211 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:45.785187    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ec415c7-2031-4610-9e09-6eb7e37ef0d6-tls-certs\") pod \"sub-test-model-642d3b-kserve-585c8f7f4f-k8mml\" (UID: \"7ec415c7-2031-4610-9e09-6eb7e37ef0d6\") " pod="ai-tenant-e2e-shared-b-w3-642d3b/sub-test-model-642d3b-kserve-585c8f7f4f-k8mml"
Sep 08 15:55:45.785582 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:45.785559    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ec415c7-2031-4610-9e09-6eb7e37ef0d6-dshm\") pod \"sub-test-model-642d3b-kserve-585c8f7f4f-k8mml\" (UID: \"7ec415c7-2031-4610-9e09-6eb7e37ef0d6\") " pod="ai-tenant-e2e-shared-b-w3-642d3b/sub-test-model-642d3b-kserve-585c8f7f4f-k8mml"
Sep 08 15:55:45.791980 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:45.791930    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ft4m\" (UniqueName: \"kubernetes.io/projected/7ec415c7-2031-4610-9e09-6eb7e37ef0d6-kube-api-access-5ft4m\") pod \"sub-test-model-642d3b-kserve-585c8f7f4f-k8mml\" (UID: \"7ec415c7-2031-4610-9e09-6eb7e37ef0d6\") " pod="ai-tenant-e2e-shared-b-w3-642d3b/sub-test-model-642d3b-kserve-585c8f7f4f-k8mml"
Sep 08 15:55:45.876408 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:45.876375    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-642d3b/sub-test-model-642d3b-kserve-585c8f7f4f-k8mml"
Sep 08 15:55:45.914572 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:45.914534    2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-dfa395-openshift-default-799698ff7b-lkxcs"
Sep 08 15:55:45.914975 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:45.914947    2549 patch_prober.go:28] interesting pod/e2e-api-a-dfa395-openshift-default-799698ff7b-lkxcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Sep 08 15:55:45.915102 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:45.915003    2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-dfa395-openshift-default-799698ff7b-lkxcs" podUID="f73cff5c-48e7-4eae-816d-6c4fb8cc8e54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Sep 08 15:55:46.157398 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:46.157362    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-642d3b/sub-test-model-642d3b-kserve-585c8f7f4f-k8mml"]
Sep 08 15:55:46.914011 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:46.913973    2549 patch_prober.go:28] interesting pod/e2e-api-a-dfa395-openshift-default-799698ff7b-lkxcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Sep 08 15:55:46.914191 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:46.914029    2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-dfa395-openshift-default-799698ff7b-lkxcs" podUID="f73cff5c-48e7-4eae-816d-6c4fb8cc8e54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Sep 08 15:55:46.983600 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:46.983565    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-642d3b/sub-test-model-642d3b-kserve-585c8f7f4f-k8mml" event={"ID":"7ec415c7-2031-4610-9e09-6eb7e37ef0d6","Type":"ContainerStarted","Data":"d298d8ed401346de8226520876ba1a034afd0edbbe55ceeca86ee70063fd2e6c"}
Sep 08 15:55:46.983600 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:46.983605    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-642d3b/sub-test-model-642d3b-kserve-585c8f7f4f-k8mml" event={"ID":"7ec415c7-2031-4610-9e09-6eb7e37ef0d6","Type":"ContainerStarted","Data":"721a376da3fa837940d4ef422bcc0827c1c28d5996b6f4a273be05a135fb57fa"}
Sep 08 15:55:47.003879 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:47.003802    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-642d3b/sub-test-model-642d3b-kserve-585c8f7f4f-k8mml" podStartSLOduration=2.003787243 podStartE2EDuration="2.003787243s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:55:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:55:47.002039972 +0000 UTC m=+1385.091491073" watchObservedRunningTime="2026-09-08 15:55:47.003787243 +0000 UTC m=+1385.093238332"
Sep 08 15:55:47.914382 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:47.914345    2549 patch_prober.go:28] interesting pod/e2e-api-a-dfa395-openshift-default-799698ff7b-lkxcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Sep 08 15:55:47.914753 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:47.914401    2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-dfa395-openshift-default-799698ff7b-lkxcs" podUID="f73cff5c-48e7-4eae-816d-6c4fb8cc8e54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Sep 08 15:55:48.914530 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:48.914490    2549 patch_prober.go:28] interesting pod/e2e-api-a-dfa395-openshift-default-799698ff7b-lkxcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Sep 08 15:55:48.914924 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:48.914545    2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-dfa395-openshift-default-799698ff7b-lkxcs" podUID="f73cff5c-48e7-4eae-816d-6c4fb8cc8e54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Sep 08 15:55:49.914400 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:49.914364    2549 patch_prober.go:28] interesting pod/e2e-api-a-dfa395-openshift-default-799698ff7b-lkxcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Sep 08 15:55:49.914567 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:49.914414    2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-dfa395-openshift-default-799698ff7b-lkxcs" podUID="f73cff5c-48e7-4eae-816d-6c4fb8cc8e54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Sep 08 15:55:50.917952 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:50.917921    2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-dfa395-openshift-default-799698ff7b-lkxcs"
Sep 08 15:55:50.918353 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:50.918224    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-dfa395-openshift-default-799698ff7b-lkxcs"
Sep 08 15:55:50.918844 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:50.918828    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-dfa395-openshift-default-799698ff7b-lkxcs"
Sep 08 15:55:55.877180 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:55.877148    2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-642d3b/sub-test-model-642d3b-kserve-585c8f7f4f-k8mml"
Sep 08 15:55:55.877626 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:55.877190    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-642d3b/sub-test-model-642d3b-kserve-585c8f7f4f-k8mml"
Sep 08 15:55:55.889653 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:55.889629    2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-642d3b/sub-test-model-642d3b-kserve-585c8f7f4f-k8mml"
Sep 08 15:55:56.040989 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:55:56.040959    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-642d3b/sub-test-model-642d3b-kserve-585c8f7f4f-k8mml"
Sep 08 15:56:35.364868 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:56:35.364772    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-dfa395-55797df967-xrc7t"]
Sep 08 15:56:35.365418 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:56:35.365080    2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dfa395-55797df967-xrc7t" podUID="d3b500bf-9cdf-4400-ac73-cb430b14798d" containerName="maas-api" containerID="cri-o://33bcfd9b61c485a0ec1c34543d1cb67456cfb50f146e6fccc4c9af4de7b3de14" gracePeriod=30
Sep 08 15:56:35.602494 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:56:35.602468    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dfa395-55797df967-xrc7t"
Sep 08 15:56:35.706651 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:56:35.706622    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3b500bf-9cdf-4400-ac73-cb430b14798d-openshift-service-ca\") pod \"d3b500bf-9cdf-4400-ac73-cb430b14798d\" (UID: \"d3b500bf-9cdf-4400-ac73-cb430b14798d\") "
Sep 08 15:56:35.706804 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:56:35.706667    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3b500bf-9cdf-4400-ac73-cb430b14798d-metrics-certs\") pod \"d3b500bf-9cdf-4400-ac73-cb430b14798d\" (UID: \"d3b500bf-9cdf-4400-ac73-cb430b14798d\") "
Sep 08 15:56:35.706804 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:56:35.706716    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3b500bf-9cdf-4400-ac73-cb430b14798d-maas-api-tls\") pod \"d3b500bf-9cdf-4400-ac73-cb430b14798d\" (UID: \"d3b500bf-9cdf-4400-ac73-cb430b14798d\") "
Sep 08 15:56:35.706804 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:56:35.706759    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sj5j8\" (UniqueName: \"kubernetes.io/projected/d3b500bf-9cdf-4400-ac73-cb430b14798d-kube-api-access-sj5j8\") pod \"d3b500bf-9cdf-4400-ac73-cb430b14798d\" (UID: \"d3b500bf-9cdf-4400-ac73-cb430b14798d\") "
Sep 08 15:56:35.707064 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:56:35.707034    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3b500bf-9cdf-4400-ac73-cb430b14798d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d3b500bf-9cdf-4400-ac73-cb430b14798d" (UID: "d3b500bf-9cdf-4400-ac73-cb430b14798d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 15:56:35.708803 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:56:35.708777    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3b500bf-9cdf-4400-ac73-cb430b14798d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d3b500bf-9cdf-4400-ac73-cb430b14798d" (UID: "d3b500bf-9cdf-4400-ac73-cb430b14798d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 15:56:35.708926 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:56:35.708824    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3b500bf-9cdf-4400-ac73-cb430b14798d-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "d3b500bf-9cdf-4400-ac73-cb430b14798d" (UID: "d3b500bf-9cdf-4400-ac73-cb430b14798d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 15:56:35.708926 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:56:35.708841    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3b500bf-9cdf-4400-ac73-cb430b14798d-kube-api-access-sj5j8" (OuterVolumeSpecName: "kube-api-access-sj5j8") pod "d3b500bf-9cdf-4400-ac73-cb430b14798d" (UID: "d3b500bf-9cdf-4400-ac73-cb430b14798d"). InnerVolumeSpecName "kube-api-access-sj5j8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:56:35.807643 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:56:35.807613    2549 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3b500bf-9cdf-4400-ac73-cb430b14798d-openshift-service-ca\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:56:35.807643 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:56:35.807640    2549 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3b500bf-9cdf-4400-ac73-cb430b14798d-metrics-certs\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:56:35.807643 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:56:35.807650    2549 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3b500bf-9cdf-4400-ac73-cb430b14798d-maas-api-tls\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:56:35.807896 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:56:35.807660    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sj5j8\" (UniqueName: \"kubernetes.io/projected/d3b500bf-9cdf-4400-ac73-cb430b14798d-kube-api-access-sj5j8\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:56:36.242043 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:56:36.242011    2549 generic.go:356] "Generic (PLEG): container finished" podID="d3b500bf-9cdf-4400-ac73-cb430b14798d" containerID="33bcfd9b61c485a0ec1c34543d1cb67456cfb50f146e6fccc4c9af4de7b3de14" exitCode=0
Sep 08 15:56:36.242195 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:56:36.242053    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dfa395-55797df967-xrc7t"
Sep 08 15:56:36.242195 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:56:36.242079    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dfa395-55797df967-xrc7t" event={"ID":"d3b500bf-9cdf-4400-ac73-cb430b14798d","Type":"ContainerDied","Data":"33bcfd9b61c485a0ec1c34543d1cb67456cfb50f146e6fccc4c9af4de7b3de14"}
Sep 08 15:56:36.242195 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:56:36.242135    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dfa395-55797df967-xrc7t" event={"ID":"d3b500bf-9cdf-4400-ac73-cb430b14798d","Type":"ContainerDied","Data":"b30d66ca73588aaf8b22dbd028615f689e058b89116dd31744b51adb39b54051"}
Sep 08 15:56:36.242195 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:56:36.242158    2549 scope.go:117] "RemoveContainer" containerID="33bcfd9b61c485a0ec1c34543d1cb67456cfb50f146e6fccc4c9af4de7b3de14"
Sep 08 15:56:36.251787 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:56:36.251770    2549 scope.go:117] "RemoveContainer" containerID="33bcfd9b61c485a0ec1c34543d1cb67456cfb50f146e6fccc4c9af4de7b3de14"
Sep 08 15:56:36.252067 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:56:36.252046    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33bcfd9b61c485a0ec1c34543d1cb67456cfb50f146e6fccc4c9af4de7b3de14\": container with ID starting with 33bcfd9b61c485a0ec1c34543d1cb67456cfb50f146e6fccc4c9af4de7b3de14 not found: ID does not exist" containerID="33bcfd9b61c485a0ec1c34543d1cb67456cfb50f146e6fccc4c9af4de7b3de14"
Sep 08 15:56:36.252129 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:56:36.252074    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33bcfd9b61c485a0ec1c34543d1cb67456cfb50f146e6fccc4c9af4de7b3de14"} err="failed to get container status \"33bcfd9b61c485a0ec1c34543d1cb67456cfb50f146e6fccc4c9af4de7b3de14\": rpc error: code = NotFound desc = could not find container \"33bcfd9b61c485a0ec1c34543d1cb67456cfb50f146e6fccc4c9af4de7b3de14\": container with ID starting with 33bcfd9b61c485a0ec1c34543d1cb67456cfb50f146e6fccc4c9af4de7b3de14 not found: ID does not exist"
Sep 08 15:56:36.264063 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:56:36.263981    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-dfa395-55797df967-xrc7t"]
Sep 08 15:56:36.266040 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:56:36.266020    2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-dfa395-55797df967-xrc7t"]
Sep 08 15:56:36.400551 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:56:36.400527    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d3b500bf-9cdf-4400-ac73-cb430b14798d" path="/var/lib/kubelet/pods/d3b500bf-9cdf-4400-ac73-cb430b14798d/volumes"
Sep 08 15:56:42.337126 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:56:42.337080    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-dfa395-openshift-default-799698ff7b-lkxcs"]
Sep 08 15:56:42.337590 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:56:42.337291    2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-dfa395-openshift-default-799698ff7b-lkxcs" podUID="f73cff5c-48e7-4eae-816d-6c4fb8cc8e54" containerName="istio-proxy" containerID="cri-o://028606c80d2592725bac7d696948326106a195b025d9f783feb34e90a5564d2b" gracePeriod=30
Sep 08 15:56:43.037055 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:56:43.037023    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/mt-curl-7781-ce69cd"]
Sep 08 15:56:43.037405 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:56:43.037392    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3b500bf-9cdf-4400-ac73-cb430b14798d" containerName="maas-api"
Sep 08 15:56:43.037454 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:56:43.037407    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3b500bf-9cdf-4400-ac73-cb430b14798d" containerName="maas-api"
Sep 08 15:56:43.037489 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:56:43.037483    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3b500bf-9cdf-4400-ac73-cb430b14798d" containerName="maas-api"
Sep 08 15:56:43.039231 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:56:43.039213    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7781-ce69cd"
Sep 08 15:56:43.042506 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:56:43.042485    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-6hc5f\""
Sep 08 15:56:43.047847 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:56:43.047824    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/mt-curl-7781-ce69cd"]
Sep 08 15:56:43.074759 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:56:43.074737    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4d6k\" (UniqueName: \"kubernetes.io/projected/2b14067d-6206-4127-b430-4f579c2cfe8c-kube-api-access-g4d6k\") pod \"mt-curl-7781-ce69cd\" (UID: \"2b14067d-6206-4127-b430-4f579c2cfe8c\") " pod="opendatahub/mt-curl-7781-ce69cd"
Sep 08 15:56:43.175436 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:56:43.175391    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4d6k\" (UniqueName: \"kubernetes.io/projected/2b14067d-6206-4127-b430-4f579c2cfe8c-kube-api-access-g4d6k\") pod \"mt-curl-7781-ce69cd\" (UID: \"2b14067d-6206-4127-b430-4f579c2cfe8c\") " pod="opendatahub/mt-curl-7781-ce69cd"
Sep 08 15:56:43.188301 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:56:43.188268    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4d6k\" (UniqueName: \"kubernetes.io/projected/2b14067d-6206-4127-b430-4f579c2cfe8c-kube-api-access-g4d6k\") pod \"mt-curl-7781-ce69cd\" (UID: \"2b14067d-6206-4127-b430-4f579c2cfe8c\") " pod="opendatahub/mt-curl-7781-ce69cd"
Sep 08 15:56:43.349633 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:56:43.349555    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7781-ce69cd"
Sep 08 15:56:43.641891 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:56:43.641845    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/mt-curl-7781-ce69cd"]
Sep 08 15:56:43.642493 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:56:43.642463    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2b14067d_6206_4127_b430_4f579c2cfe8c.slice/crio-398abbb69ff740b34aff47266ba405c4672ef17aa4f0e3e2397d7b7423eb61fe WatchSource:0}: Error finding container 398abbb69ff740b34aff47266ba405c4672ef17aa4f0e3e2397d7b7423eb61fe: Status 404 returned error can't find the container with id 398abbb69ff740b34aff47266ba405c4672ef17aa4f0e3e2397d7b7423eb61fe
Sep 08 15:56:43.643930 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:56:43.643910    2549 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 08 15:56:44.284216 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:56:44.284180    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/mt-curl-7781-ce69cd" event={"ID":"2b14067d-6206-4127-b430-4f579c2cfe8c","Type":"ContainerStarted","Data":"398abbb69ff740b34aff47266ba405c4672ef17aa4f0e3e2397d7b7423eb61fe"}
Sep 08 15:56:45.290756 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:56:45.290719    2549 generic.go:356] "Generic (PLEG): container finished" podID="2b14067d-6206-4127-b430-4f579c2cfe8c" containerID="8ec126a3b72edb520580a77bde0a4c939eccf6645e77e86955b75d02eed41068" exitCode=0
Sep 08 15:56:45.291149 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:56:45.290798    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/mt-curl-7781-ce69cd" event={"ID":"2b14067d-6206-4127-b430-4f579c2cfe8c","Type":"ContainerDied","Data":"8ec126a3b72edb520580a77bde0a4c939eccf6645e77e86955b75d02eed41068"}
Sep 08 15:56:46.420351 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:56:46.420329    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7781-ce69cd"
Sep 08 15:56:46.454263 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:56:46.454240    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_mt-curl-7781-ce69cd_2b14067d-6206-4127-b430-4f579c2cfe8c/mt-curl-7781-ce69cd/0.log"
Sep 08 15:56:46.475557 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:56:46.475534    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/mt-curl-7781-ce69cd"]
Sep 08 15:56:46.481094 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:56:46.481072    2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/mt-curl-7781-ce69cd"]
Sep 08 15:56:46.504536 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:56:46.504514    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g4d6k\" (UniqueName: \"kubernetes.io/projected/2b14067d-6206-4127-b430-4f579c2cfe8c-kube-api-access-g4d6k\") pod \"2b14067d-6206-4127-b430-4f579c2cfe8c\" (UID: \"2b14067d-6206-4127-b430-4f579c2cfe8c\") "
Sep 08 15:56:46.506564 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:56:46.506537    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b14067d-6206-4127-b430-4f579c2cfe8c-kube-api-access-g4d6k" (OuterVolumeSpecName: "kube-api-access-g4d6k") pod "2b14067d-6206-4127-b430-4f579c2cfe8c" (UID: "2b14067d-6206-4127-b430-4f579c2cfe8c"). InnerVolumeSpecName "kube-api-access-g4d6k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:56:46.605395 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:56:46.605339    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4d6k\" (UniqueName: \"kubernetes.io/projected/2b14067d-6206-4127-b430-4f579c2cfe8c-kube-api-access-g4d6k\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:56:47.301068 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:56:47.301041    2549 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="398abbb69ff740b34aff47266ba405c4672ef17aa4f0e3e2397d7b7423eb61fe"
Sep 08 15:56:47.301068 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:56:47.301050    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7781-ce69cd"
Sep 08 15:56:47.310985 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:56:47.310959    2549 status_manager.go:1018] "Failed to get status for pod" err="pods \"mt-curl-7781-ce69cd\" is forbidden: User \"system:node:ip-10-0-143-61.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-61.ec2.internal' and this object" podUID="2b14067d-6206-4127-b430-4f579c2cfe8c" pod="opendatahub/mt-curl-7781-ce69cd"
Sep 08 15:56:47.587575 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:56:47.587553    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-dfa395-openshift-default-799698ff7b-lkxcs"
Sep 08 15:56:47.621868 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:56:47.621823    2549 status_manager.go:1018] "Failed to get status for pod" err="pods \"mt-curl-7781-ce69cd\" is forbidden: User \"system:node:ip-10-0-143-61.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-61.ec2.internal' and this object" podUID="2b14067d-6206-4127-b430-4f579c2cfe8c" pod="opendatahub/mt-curl-7781-ce69cd"
Sep 08 15:56:47.714100 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:56:47.714073    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f73cff5c-48e7-4eae-816d-6c4fb8cc8e54-workload-certs\") pod \"f73cff5c-48e7-4eae-816d-6c4fb8cc8e54\" (UID: \"f73cff5c-48e7-4eae-816d-6c4fb8cc8e54\") "
Sep 08 15:56:47.714232 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:56:47.714104    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f73cff5c-48e7-4eae-816d-6c4fb8cc8e54-credential-socket\") pod \"f73cff5c-48e7-4eae-816d-6c4fb8cc8e54\" (UID: \"f73cff5c-48e7-4eae-816d-6c4fb8cc8e54\") "
Sep 08 15:56:47.714232 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:56:47.714128    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fnlpx\" (UniqueName: \"kubernetes.io/projected/f73cff5c-48e7-4eae-816d-6c4fb8cc8e54-kube-api-access-fnlpx\") pod \"f73cff5c-48e7-4eae-816d-6c4fb8cc8e54\" (UID: \"f73cff5c-48e7-4eae-816d-6c4fb8cc8e54\") "
Sep 08 15:56:47.714232 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:56:47.714159    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f73cff5c-48e7-4eae-816d-6c4fb8cc8e54-istio-podinfo\") pod \"f73cff5c-48e7-4eae-816d-6c4fb8cc8e54\" (UID: \"f73cff5c-48e7-4eae-816d-6c4fb8cc8e54\") "
Sep 08 15:56:47.714232 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:56:47.714192    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f73cff5c-48e7-4eae-816d-6c4fb8cc8e54-istio-data\") pod \"f73cff5c-48e7-4eae-816d-6c4fb8cc8e54\" (UID: \"f73cff5c-48e7-4eae-816d-6c4fb8cc8e54\") "
Sep 08 15:56:47.714418 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:56:47.714245    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f73cff5c-48e7-4eae-816d-6c4fb8cc8e54-workload-socket\") pod \"f73cff5c-48e7-4eae-816d-6c4fb8cc8e54\" (UID: \"f73cff5c-48e7-4eae-816d-6c4fb8cc8e54\") "
Sep 08 15:56:47.714418 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:56:47.714289    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f73cff5c-48e7-4eae-816d-6c4fb8cc8e54-istio-envoy\") pod \"f73cff5c-48e7-4eae-816d-6c4fb8cc8e54\" (UID: \"f73cff5c-48e7-4eae-816d-6c4fb8cc8e54\") "
Sep 08 15:56:47.714418 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:56:47.714319    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f73cff5c-48e7-4eae-816d-6c4fb8cc8e54-istiod-ca-cert\") pod \"f73cff5c-48e7-4eae-816d-6c4fb8cc8e54\" (UID: \"f73cff5c-48e7-4eae-816d-6c4fb8cc8e54\") "
Sep 08 15:56:47.714418 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:56:47.714348    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f73cff5c-48e7-4eae-816d-6c4fb8cc8e54-istio-token\") pod \"f73cff5c-48e7-4eae-816d-6c4fb8cc8e54\" (UID: \"f73cff5c-48e7-4eae-816d-6c4fb8cc8e54\") "
Sep 08 15:56:47.714418 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:56:47.714358    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f73cff5c-48e7-4eae-816d-6c4fb8cc8e54-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f73cff5c-48e7-4eae-816d-6c4fb8cc8e54" (UID: "f73cff5c-48e7-4eae-816d-6c4fb8cc8e54"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:56:47.714675 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:56:47.714418    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f73cff5c-48e7-4eae-816d-6c4fb8cc8e54-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f73cff5c-48e7-4eae-816d-6c4fb8cc8e54" (UID: "f73cff5c-48e7-4eae-816d-6c4fb8cc8e54"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:56:47.714675 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:56:47.714560    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f73cff5c-48e7-4eae-816d-6c4fb8cc8e54-istio-data" (OuterVolumeSpecName: "istio-data") pod "f73cff5c-48e7-4eae-816d-6c4fb8cc8e54" (UID: "f73cff5c-48e7-4eae-816d-6c4fb8cc8e54"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:56:47.714784 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:56:47.714723    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f73cff5c-48e7-4eae-816d-6c4fb8cc8e54-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f73cff5c-48e7-4eae-816d-6c4fb8cc8e54" (UID: "f73cff5c-48e7-4eae-816d-6c4fb8cc8e54"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 15:56:47.714784 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:56:47.714747    2549 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f73cff5c-48e7-4eae-816d-6c4fb8cc8e54-workload-certs\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:56:47.714784 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:56:47.714767    2549 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f73cff5c-48e7-4eae-816d-6c4fb8cc8e54-credential-socket\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:56:47.714945 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:56:47.714782    2549 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f73cff5c-48e7-4eae-816d-6c4fb8cc8e54-istio-data\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:56:47.715005 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:56:47.714945    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f73cff5c-48e7-4eae-816d-6c4fb8cc8e54-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f73cff5c-48e7-4eae-816d-6c4fb8cc8e54" (UID: "f73cff5c-48e7-4eae-816d-6c4fb8cc8e54"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:56:47.716554 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:56:47.716523    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f73cff5c-48e7-4eae-816d-6c4fb8cc8e54-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f73cff5c-48e7-4eae-816d-6c4fb8cc8e54" (UID: "f73cff5c-48e7-4eae-816d-6c4fb8cc8e54"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:56:47.716554 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:56:47.716541    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f73cff5c-48e7-4eae-816d-6c4fb8cc8e54-istio-token" (OuterVolumeSpecName: "istio-token") pod "f73cff5c-48e7-4eae-816d-6c4fb8cc8e54" (UID: "f73cff5c-48e7-4eae-816d-6c4fb8cc8e54"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:56:47.716796 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:56:47.716776    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f73cff5c-48e7-4eae-816d-6c4fb8cc8e54-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f73cff5c-48e7-4eae-816d-6c4fb8cc8e54" (UID: "f73cff5c-48e7-4eae-816d-6c4fb8cc8e54"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 08 15:56:47.716850 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:56:47.716793    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f73cff5c-48e7-4eae-816d-6c4fb8cc8e54-kube-api-access-fnlpx" (OuterVolumeSpecName: "kube-api-access-fnlpx") pod "f73cff5c-48e7-4eae-816d-6c4fb8cc8e54" (UID: "f73cff5c-48e7-4eae-816d-6c4fb8cc8e54"). InnerVolumeSpecName "kube-api-access-fnlpx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:56:47.815787 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:56:47.815765    2549 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f73cff5c-48e7-4eae-816d-6c4fb8cc8e54-istio-envoy\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:56:47.815787 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:56:47.815784    2549 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f73cff5c-48e7-4eae-816d-6c4fb8cc8e54-istiod-ca-cert\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:56:47.815956 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:56:47.815792    2549 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f73cff5c-48e7-4eae-816d-6c4fb8cc8e54-istio-token\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:56:47.815956 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:56:47.815800    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fnlpx\" (UniqueName: \"kubernetes.io/projected/f73cff5c-48e7-4eae-816d-6c4fb8cc8e54-kube-api-access-fnlpx\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:56:47.815956 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:56:47.815808    2549 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f73cff5c-48e7-4eae-816d-6c4fb8cc8e54-istio-podinfo\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:56:47.815956 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:56:47.815816    2549 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f73cff5c-48e7-4eae-816d-6c4fb8cc8e54-workload-socket\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:56:48.307624 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:56:48.307591    2549 generic.go:356] "Generic (PLEG): container finished" podID="f73cff5c-48e7-4eae-816d-6c4fb8cc8e54" containerID="028606c80d2592725bac7d696948326106a195b025d9f783feb34e90a5564d2b" exitCode=0
Sep 08 15:56:48.307816 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:56:48.307636    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-dfa395-openshift-default-799698ff7b-lkxcs"
Sep 08 15:56:48.307816 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:56:48.307676    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-dfa395-openshift-default-799698ff7b-lkxcs" event={"ID":"f73cff5c-48e7-4eae-816d-6c4fb8cc8e54","Type":"ContainerDied","Data":"028606c80d2592725bac7d696948326106a195b025d9f783feb34e90a5564d2b"}
Sep 08 15:56:48.307816 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:56:48.307767    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-dfa395-openshift-default-799698ff7b-lkxcs" event={"ID":"f73cff5c-48e7-4eae-816d-6c4fb8cc8e54","Type":"ContainerDied","Data":"9a9d8a5b12f48d176617a27e7434f237115ce3ed1204d8d86d8c569c68207fbb"}
Sep 08 15:56:48.307816 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:56:48.307798    2549 scope.go:117] "RemoveContainer" containerID="028606c80d2592725bac7d696948326106a195b025d9f783feb34e90a5564d2b"
Sep 08 15:56:48.310255 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:56:48.310226    2549 status_manager.go:1018] "Failed to get status for pod" err="pods \"mt-curl-7781-ce69cd\" is forbidden: User \"system:node:ip-10-0-143-61.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-61.ec2.internal' and this object" podUID="2b14067d-6206-4127-b430-4f579c2cfe8c" pod="opendatahub/mt-curl-7781-ce69cd"
Sep 08 15:56:48.317416 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:56:48.317398    2549 scope.go:117] "RemoveContainer" containerID="6c67ad5d7d6ac98213eef902291a31b2d996a772ec9d55e7b881519d384897b5"
Sep 08 15:56:48.325390 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:56:48.325374    2549 scope.go:117] "RemoveContainer" containerID="028606c80d2592725bac7d696948326106a195b025d9f783feb34e90a5564d2b"
Sep 08 15:56:48.325630 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:56:48.325614    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"028606c80d2592725bac7d696948326106a195b025d9f783feb34e90a5564d2b\": container with ID starting with 028606c80d2592725bac7d696948326106a195b025d9f783feb34e90a5564d2b not found: ID does not exist" containerID="028606c80d2592725bac7d696948326106a195b025d9f783feb34e90a5564d2b"
Sep 08 15:56:48.325676 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:56:48.325637    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"028606c80d2592725bac7d696948326106a195b025d9f783feb34e90a5564d2b"} err="failed to get container status \"028606c80d2592725bac7d696948326106a195b025d9f783feb34e90a5564d2b\": rpc error: code = NotFound desc = could not find container \"028606c80d2592725bac7d696948326106a195b025d9f783feb34e90a5564d2b\": container with ID starting with 028606c80d2592725bac7d696948326106a195b025d9f783feb34e90a5564d2b not found: ID does not exist"
Sep 08 15:56:48.325676 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:56:48.325651    2549 scope.go:117] "RemoveContainer" containerID="6c67ad5d7d6ac98213eef902291a31b2d996a772ec9d55e7b881519d384897b5"
Sep 08 15:56:48.325896 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:56:48.325878    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c67ad5d7d6ac98213eef902291a31b2d996a772ec9d55e7b881519d384897b5\": container with ID starting with 6c67ad5d7d6ac98213eef902291a31b2d996a772ec9d55e7b881519d384897b5 not found: ID does not exist" containerID="6c67ad5d7d6ac98213eef902291a31b2d996a772ec9d55e7b881519d384897b5"
Sep 08 15:56:48.325950 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:56:48.325902    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c67ad5d7d6ac98213eef902291a31b2d996a772ec9d55e7b881519d384897b5"} err="failed to get container status \"6c67ad5d7d6ac98213eef902291a31b2d996a772ec9d55e7b881519d384897b5\": rpc error: code = NotFound desc = could not find container \"6c67ad5d7d6ac98213eef902291a31b2d996a772ec9d55e7b881519d384897b5\": container with ID starting with 6c67ad5d7d6ac98213eef902291a31b2d996a772ec9d55e7b881519d384897b5 not found: ID does not exist"
Sep 08 15:56:48.330522 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:56:48.330502    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-dfa395-openshift-default-799698ff7b-lkxcs"]
Sep 08 15:56:48.335734 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:56:48.335713    2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-dfa395-openshift-default-799698ff7b-lkxcs"]
Sep 08 15:56:48.335837 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:56:48.335815    2549 status_manager.go:1018] "Failed to get status for pod" err="pods \"mt-curl-7781-ce69cd\" is forbidden: User \"system:node:ip-10-0-143-61.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-61.ec2.internal' and this object" podUID="2b14067d-6206-4127-b430-4f579c2cfe8c" pod="opendatahub/mt-curl-7781-ce69cd"
Sep 08 15:56:48.399870 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:56:48.399830    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2b14067d-6206-4127-b430-4f579c2cfe8c" path="/var/lib/kubelet/pods/2b14067d-6206-4127-b430-4f579c2cfe8c/volumes"
Sep 08 15:56:48.400309 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:56:48.400289    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f73cff5c-48e7-4eae-816d-6c4fb8cc8e54" path="/var/lib/kubelet/pods/f73cff5c-48e7-4eae-816d-6c4fb8cc8e54/volumes"
Sep 08 15:56:51.467003 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:56:51.466967    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-b30e99/sub-test-model-b30e99-kserve-7b97bff7f5-jm5cg"]
Sep 08 15:56:52.421117 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:56:52.421082    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-642d3b/sub-test-model-642d3b-kserve-585c8f7f4f-k8mml"]
Sep 08 15:57:06.488160 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:06.488118    2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-b30e99/sub-test-model-b30e99-kserve-7b97bff7f5-jm5cg" podUID="155f8e63-bfc2-476c-bbee-674645c242b3" containerName="main" containerID="cri-o://dbd01e43dc037c05406c2a29cf8c362801bdba26a36e5bebbce3767240cad50e" gracePeriod=45
Sep 08 15:57:06.735013 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:06.734989    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-b30e99/sub-test-model-b30e99-kserve-7b97bff7f5-jm5cg"
Sep 08 15:57:06.877526 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:06.877455    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/155f8e63-bfc2-476c-bbee-674645c242b3-tmp-dir\") pod \"155f8e63-bfc2-476c-bbee-674645c242b3\" (UID: \"155f8e63-bfc2-476c-bbee-674645c242b3\") "
Sep 08 15:57:06.877526 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:06.877501    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/155f8e63-bfc2-476c-bbee-674645c242b3-model-cache\") pod \"155f8e63-bfc2-476c-bbee-674645c242b3\" (UID: \"155f8e63-bfc2-476c-bbee-674645c242b3\") "
Sep 08 15:57:06.877745 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:06.877533    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vzqcz\" (UniqueName: \"kubernetes.io/projected/155f8e63-bfc2-476c-bbee-674645c242b3-kube-api-access-vzqcz\") pod \"155f8e63-bfc2-476c-bbee-674645c242b3\" (UID: \"155f8e63-bfc2-476c-bbee-674645c242b3\") "
Sep 08 15:57:06.877745 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:06.877583    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/155f8e63-bfc2-476c-bbee-674645c242b3-tls-certs\") pod \"155f8e63-bfc2-476c-bbee-674645c242b3\" (UID: \"155f8e63-bfc2-476c-bbee-674645c242b3\") "
Sep 08 15:57:06.877745 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:06.877711    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/155f8e63-bfc2-476c-bbee-674645c242b3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "155f8e63-bfc2-476c-bbee-674645c242b3" (UID: "155f8e63-bfc2-476c-bbee-674645c242b3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:57:06.877897 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:06.877760    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/155f8e63-bfc2-476c-bbee-674645c242b3-home\") pod \"155f8e63-bfc2-476c-bbee-674645c242b3\" (UID: \"155f8e63-bfc2-476c-bbee-674645c242b3\") "
Sep 08 15:57:06.877897 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:06.877809    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/155f8e63-bfc2-476c-bbee-674645c242b3-dshm\") pod \"155f8e63-bfc2-476c-bbee-674645c242b3\" (UID: \"155f8e63-bfc2-476c-bbee-674645c242b3\") "
Sep 08 15:57:06.877897 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:06.877756    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/155f8e63-bfc2-476c-bbee-674645c242b3-model-cache" (OuterVolumeSpecName: "model-cache") pod "155f8e63-bfc2-476c-bbee-674645c242b3" (UID: "155f8e63-bfc2-476c-bbee-674645c242b3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:57:06.878020 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:06.877971    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/155f8e63-bfc2-476c-bbee-674645c242b3-home" (OuterVolumeSpecName: "home") pod "155f8e63-bfc2-476c-bbee-674645c242b3" (UID: "155f8e63-bfc2-476c-bbee-674645c242b3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:57:06.878147 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:06.878129    2549 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/155f8e63-bfc2-476c-bbee-674645c242b3-model-cache\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:57:06.878236 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:06.878149    2549 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/155f8e63-bfc2-476c-bbee-674645c242b3-home\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:57:06.878236 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:06.878161    2549 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/155f8e63-bfc2-476c-bbee-674645c242b3-tmp-dir\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:57:06.879722 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:06.879697    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/155f8e63-bfc2-476c-bbee-674645c242b3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "155f8e63-bfc2-476c-bbee-674645c242b3" (UID: "155f8e63-bfc2-476c-bbee-674645c242b3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 15:57:06.879823 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:06.879715    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/155f8e63-bfc2-476c-bbee-674645c242b3-dshm" (OuterVolumeSpecName: "dshm") pod "155f8e63-bfc2-476c-bbee-674645c242b3" (UID: "155f8e63-bfc2-476c-bbee-674645c242b3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:57:06.879823 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:06.879728    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/155f8e63-bfc2-476c-bbee-674645c242b3-kube-api-access-vzqcz" (OuterVolumeSpecName: "kube-api-access-vzqcz") pod "155f8e63-bfc2-476c-bbee-674645c242b3" (UID: "155f8e63-bfc2-476c-bbee-674645c242b3"). InnerVolumeSpecName "kube-api-access-vzqcz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:57:06.979063 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:06.979033    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vzqcz\" (UniqueName: \"kubernetes.io/projected/155f8e63-bfc2-476c-bbee-674645c242b3-kube-api-access-vzqcz\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:57:06.979063 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:06.979065    2549 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/155f8e63-bfc2-476c-bbee-674645c242b3-tls-certs\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:57:06.979221 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:06.979076    2549 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/155f8e63-bfc2-476c-bbee-674645c242b3-dshm\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:57:07.406242 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:07.406207    2549 generic.go:356] "Generic (PLEG): container finished" podID="155f8e63-bfc2-476c-bbee-674645c242b3" containerID="dbd01e43dc037c05406c2a29cf8c362801bdba26a36e5bebbce3767240cad50e" exitCode=0
Sep 08 15:57:07.406242 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:07.406240    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-b30e99/sub-test-model-b30e99-kserve-7b97bff7f5-jm5cg"
Sep 08 15:57:07.406438 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:07.406259    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-b30e99/sub-test-model-b30e99-kserve-7b97bff7f5-jm5cg" event={"ID":"155f8e63-bfc2-476c-bbee-674645c242b3","Type":"ContainerDied","Data":"dbd01e43dc037c05406c2a29cf8c362801bdba26a36e5bebbce3767240cad50e"}
Sep 08 15:57:07.406438 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:07.406293    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-b30e99/sub-test-model-b30e99-kserve-7b97bff7f5-jm5cg" event={"ID":"155f8e63-bfc2-476c-bbee-674645c242b3","Type":"ContainerDied","Data":"e0b7320cf08ccbf201f475ff10789eaecd92bf941abc8cd0a723844efd70fb4c"}
Sep 08 15:57:07.406438 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:07.406315    2549 scope.go:117] "RemoveContainer" containerID="dbd01e43dc037c05406c2a29cf8c362801bdba26a36e5bebbce3767240cad50e"
Sep 08 15:57:07.415217 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:07.415193    2549 scope.go:117] "RemoveContainer" containerID="dbd01e43dc037c05406c2a29cf8c362801bdba26a36e5bebbce3767240cad50e"
Sep 08 15:57:07.415488 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:57:07.415468    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dbd01e43dc037c05406c2a29cf8c362801bdba26a36e5bebbce3767240cad50e\": container with ID starting with dbd01e43dc037c05406c2a29cf8c362801bdba26a36e5bebbce3767240cad50e not found: ID does not exist" containerID="dbd01e43dc037c05406c2a29cf8c362801bdba26a36e5bebbce3767240cad50e"
Sep 08 15:57:07.415547 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:07.415501    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dbd01e43dc037c05406c2a29cf8c362801bdba26a36e5bebbce3767240cad50e"} err="failed to get container status \"dbd01e43dc037c05406c2a29cf8c362801bdba26a36e5bebbce3767240cad50e\": rpc error: code = NotFound desc = could not find container \"dbd01e43dc037c05406c2a29cf8c362801bdba26a36e5bebbce3767240cad50e\": container with ID starting with dbd01e43dc037c05406c2a29cf8c362801bdba26a36e5bebbce3767240cad50e not found: ID does not exist"
Sep 08 15:57:07.428048 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:07.428025    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-b30e99/sub-test-model-b30e99-kserve-7b97bff7f5-jm5cg"]
Sep 08 15:57:07.433169 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:07.433148    2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-b30e99/sub-test-model-b30e99-kserve-7b97bff7f5-jm5cg"]
Sep 08 15:57:07.441351 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:07.441324    2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-642d3b/sub-test-model-642d3b-kserve-585c8f7f4f-k8mml" podUID="7ec415c7-2031-4610-9e09-6eb7e37ef0d6" containerName="main" containerID="cri-o://d298d8ed401346de8226520876ba1a034afd0edbbe55ceeca86ee70063fd2e6c" gracePeriod=45
Sep 08 15:57:07.678039 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:07.678017    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-642d3b/sub-test-model-642d3b-kserve-585c8f7f4f-k8mml"
Sep 08 15:57:07.786025 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:07.785994    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ec415c7-2031-4610-9e09-6eb7e37ef0d6-tmp-dir\") pod \"7ec415c7-2031-4610-9e09-6eb7e37ef0d6\" (UID: \"7ec415c7-2031-4610-9e09-6eb7e37ef0d6\") "
Sep 08 15:57:07.786161 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:07.786031    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ec415c7-2031-4610-9e09-6eb7e37ef0d6-model-cache\") pod \"7ec415c7-2031-4610-9e09-6eb7e37ef0d6\" (UID: \"7ec415c7-2031-4610-9e09-6eb7e37ef0d6\") "
Sep 08 15:57:07.786161 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:07.786053    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ec415c7-2031-4610-9e09-6eb7e37ef0d6-tls-certs\") pod \"7ec415c7-2031-4610-9e09-6eb7e37ef0d6\" (UID: \"7ec415c7-2031-4610-9e09-6eb7e37ef0d6\") "
Sep 08 15:57:07.786161 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:07.786114    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5ft4m\" (UniqueName: \"kubernetes.io/projected/7ec415c7-2031-4610-9e09-6eb7e37ef0d6-kube-api-access-5ft4m\") pod \"7ec415c7-2031-4610-9e09-6eb7e37ef0d6\" (UID: \"7ec415c7-2031-4610-9e09-6eb7e37ef0d6\") "
Sep 08 15:57:07.786161 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:07.786130    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ec415c7-2031-4610-9e09-6eb7e37ef0d6-dshm\") pod \"7ec415c7-2031-4610-9e09-6eb7e37ef0d6\" (UID: \"7ec415c7-2031-4610-9e09-6eb7e37ef0d6\") "
Sep 08 15:57:07.786346 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:07.786200    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ec415c7-2031-4610-9e09-6eb7e37ef0d6-home\") pod \"7ec415c7-2031-4610-9e09-6eb7e37ef0d6\" (UID: \"7ec415c7-2031-4610-9e09-6eb7e37ef0d6\") "
Sep 08 15:57:07.786346 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:07.786262    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ec415c7-2031-4610-9e09-6eb7e37ef0d6-model-cache" (OuterVolumeSpecName: "model-cache") pod "7ec415c7-2031-4610-9e09-6eb7e37ef0d6" (UID: "7ec415c7-2031-4610-9e09-6eb7e37ef0d6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:57:07.786346 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:07.786271    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ec415c7-2031-4610-9e09-6eb7e37ef0d6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7ec415c7-2031-4610-9e09-6eb7e37ef0d6" (UID: "7ec415c7-2031-4610-9e09-6eb7e37ef0d6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:57:07.786513 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:07.786491    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ec415c7-2031-4610-9e09-6eb7e37ef0d6-home" (OuterVolumeSpecName: "home") pod "7ec415c7-2031-4610-9e09-6eb7e37ef0d6" (UID: "7ec415c7-2031-4610-9e09-6eb7e37ef0d6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:57:07.786558 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:07.786495    2549 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ec415c7-2031-4610-9e09-6eb7e37ef0d6-tmp-dir\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:57:07.786558 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:07.786535    2549 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ec415c7-2031-4610-9e09-6eb7e37ef0d6-model-cache\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:57:07.788283 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:07.788256    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ec415c7-2031-4610-9e09-6eb7e37ef0d6-dshm" (OuterVolumeSpecName: "dshm") pod "7ec415c7-2031-4610-9e09-6eb7e37ef0d6" (UID: "7ec415c7-2031-4610-9e09-6eb7e37ef0d6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:57:07.788386 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:07.788306    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ec415c7-2031-4610-9e09-6eb7e37ef0d6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7ec415c7-2031-4610-9e09-6eb7e37ef0d6" (UID: "7ec415c7-2031-4610-9e09-6eb7e37ef0d6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 15:57:07.788386 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:07.788321    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ec415c7-2031-4610-9e09-6eb7e37ef0d6-kube-api-access-5ft4m" (OuterVolumeSpecName: "kube-api-access-5ft4m") pod "7ec415c7-2031-4610-9e09-6eb7e37ef0d6" (UID: "7ec415c7-2031-4610-9e09-6eb7e37ef0d6"). InnerVolumeSpecName "kube-api-access-5ft4m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:57:07.887486 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:07.887455    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5ft4m\" (UniqueName: \"kubernetes.io/projected/7ec415c7-2031-4610-9e09-6eb7e37ef0d6-kube-api-access-5ft4m\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:57:07.887486 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:07.887481    2549 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ec415c7-2031-4610-9e09-6eb7e37ef0d6-dshm\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:57:07.887486 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:07.887491    2549 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ec415c7-2031-4610-9e09-6eb7e37ef0d6-home\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:57:07.887662 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:07.887499    2549 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ec415c7-2031-4610-9e09-6eb7e37ef0d6-tls-certs\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:57:08.400371 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:08.400328    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="155f8e63-bfc2-476c-bbee-674645c242b3" path="/var/lib/kubelet/pods/155f8e63-bfc2-476c-bbee-674645c242b3/volumes"
Sep 08 15:57:08.412603 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:08.412576    2549 generic.go:356] "Generic (PLEG): container finished" podID="7ec415c7-2031-4610-9e09-6eb7e37ef0d6" containerID="d298d8ed401346de8226520876ba1a034afd0edbbe55ceeca86ee70063fd2e6c" exitCode=0
Sep 08 15:57:08.412727 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:08.412611    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-642d3b/sub-test-model-642d3b-kserve-585c8f7f4f-k8mml"
Sep 08 15:57:08.412727 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:08.412619    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-642d3b/sub-test-model-642d3b-kserve-585c8f7f4f-k8mml" event={"ID":"7ec415c7-2031-4610-9e09-6eb7e37ef0d6","Type":"ContainerDied","Data":"d298d8ed401346de8226520876ba1a034afd0edbbe55ceeca86ee70063fd2e6c"}
Sep 08 15:57:08.412727 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:08.412641    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-642d3b/sub-test-model-642d3b-kserve-585c8f7f4f-k8mml" event={"ID":"7ec415c7-2031-4610-9e09-6eb7e37ef0d6","Type":"ContainerDied","Data":"721a376da3fa837940d4ef422bcc0827c1c28d5996b6f4a273be05a135fb57fa"}
Sep 08 15:57:08.412727 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:08.412656    2549 scope.go:117] "RemoveContainer" containerID="d298d8ed401346de8226520876ba1a034afd0edbbe55ceeca86ee70063fd2e6c"
Sep 08 15:57:08.421282 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:08.421265    2549 scope.go:117] "RemoveContainer" containerID="d298d8ed401346de8226520876ba1a034afd0edbbe55ceeca86ee70063fd2e6c"
Sep 08 15:57:08.421534 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:57:08.421515    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d298d8ed401346de8226520876ba1a034afd0edbbe55ceeca86ee70063fd2e6c\": container with ID starting with d298d8ed401346de8226520876ba1a034afd0edbbe55ceeca86ee70063fd2e6c not found: ID does not exist" containerID="d298d8ed401346de8226520876ba1a034afd0edbbe55ceeca86ee70063fd2e6c"
Sep 08 15:57:08.421593 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:08.421540    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d298d8ed401346de8226520876ba1a034afd0edbbe55ceeca86ee70063fd2e6c"} err="failed to get container status \"d298d8ed401346de8226520876ba1a034afd0edbbe55ceeca86ee70063fd2e6c\": rpc error: code = NotFound desc = could not find container \"d298d8ed401346de8226520876ba1a034afd0edbbe55ceeca86ee70063fd2e6c\": container with ID starting with d298d8ed401346de8226520876ba1a034afd0edbbe55ceeca86ee70063fd2e6c not found: ID does not exist"
Sep 08 15:57:08.428947 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:08.428925    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-642d3b/sub-test-model-642d3b-kserve-585c8f7f4f-k8mml"]
Sep 08 15:57:08.432761 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:08.432740    2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-642d3b/sub-test-model-642d3b-kserve-585c8f7f4f-k8mml"]
Sep 08 15:57:10.400932 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:10.400902    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7ec415c7-2031-4610-9e09-6eb7e37ef0d6" path="/var/lib/kubelet/pods/7ec415c7-2031-4610-9e09-6eb7e37ef0d6/volumes"
Sep 08 15:57:23.066139 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:23.066104    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-be2aba-openshift-default-865869f8bd-ft79s"]
Sep 08 15:57:23.066753 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:23.066363    2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-be2aba-openshift-default-865869f8bd-ft79s" podUID="a33f03d3-323b-47eb-8680-b69489b21e0b" containerName="istio-proxy" containerID="cri-o://7e07472ad3cfbabab9e72da2987b5a57753e726e884f1c16f85a76c2a62b9cf9" gracePeriod=30
Sep 08 15:57:28.308930 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:28.308906    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-be2aba-openshift-default-865869f8bd-ft79s"
Sep 08 15:57:28.342555 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:28.342494    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a33f03d3-323b-47eb-8680-b69489b21e0b-istiod-ca-cert\") pod \"a33f03d3-323b-47eb-8680-b69489b21e0b\" (UID: \"a33f03d3-323b-47eb-8680-b69489b21e0b\") "
Sep 08 15:57:28.342555 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:28.342531    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a33f03d3-323b-47eb-8680-b69489b21e0b-istio-token\") pod \"a33f03d3-323b-47eb-8680-b69489b21e0b\" (UID: \"a33f03d3-323b-47eb-8680-b69489b21e0b\") "
Sep 08 15:57:28.342555 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:28.342549    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a33f03d3-323b-47eb-8680-b69489b21e0b-istio-data\") pod \"a33f03d3-323b-47eb-8680-b69489b21e0b\" (UID: \"a33f03d3-323b-47eb-8680-b69489b21e0b\") "
Sep 08 15:57:28.342789 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:28.342567    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a33f03d3-323b-47eb-8680-b69489b21e0b-credential-socket\") pod \"a33f03d3-323b-47eb-8680-b69489b21e0b\" (UID: \"a33f03d3-323b-47eb-8680-b69489b21e0b\") "
Sep 08 15:57:28.342789 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:28.342588    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a33f03d3-323b-47eb-8680-b69489b21e0b-workload-certs\") pod \"a33f03d3-323b-47eb-8680-b69489b21e0b\" (UID: \"a33f03d3-323b-47eb-8680-b69489b21e0b\") "
Sep 08 15:57:28.342789 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:28.342617    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a33f03d3-323b-47eb-8680-b69489b21e0b-workload-socket\") pod \"a33f03d3-323b-47eb-8680-b69489b21e0b\" (UID: \"a33f03d3-323b-47eb-8680-b69489b21e0b\") "
Sep 08 15:57:28.342789 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:28.342646    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a33f03d3-323b-47eb-8680-b69489b21e0b-istio-envoy\") pod \"a33f03d3-323b-47eb-8680-b69489b21e0b\" (UID: \"a33f03d3-323b-47eb-8680-b69489b21e0b\") "
Sep 08 15:57:28.342789 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:28.342673    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a33f03d3-323b-47eb-8680-b69489b21e0b-istio-podinfo\") pod \"a33f03d3-323b-47eb-8680-b69489b21e0b\" (UID: \"a33f03d3-323b-47eb-8680-b69489b21e0b\") "
Sep 08 15:57:28.342789 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:28.342769    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wsqdc\" (UniqueName: \"kubernetes.io/projected/a33f03d3-323b-47eb-8680-b69489b21e0b-kube-api-access-wsqdc\") pod \"a33f03d3-323b-47eb-8680-b69489b21e0b\" (UID: \"a33f03d3-323b-47eb-8680-b69489b21e0b\") "
Sep 08 15:57:28.343123 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:28.342886    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a33f03d3-323b-47eb-8680-b69489b21e0b-istio-data" (OuterVolumeSpecName: "istio-data") pod "a33f03d3-323b-47eb-8680-b69489b21e0b" (UID: "a33f03d3-323b-47eb-8680-b69489b21e0b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:57:28.343123 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:28.342977    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a33f03d3-323b-47eb-8680-b69489b21e0b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a33f03d3-323b-47eb-8680-b69489b21e0b" (UID: "a33f03d3-323b-47eb-8680-b69489b21e0b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:57:28.343123 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:28.342995    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a33f03d3-323b-47eb-8680-b69489b21e0b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a33f03d3-323b-47eb-8680-b69489b21e0b" (UID: "a33f03d3-323b-47eb-8680-b69489b21e0b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:57:28.343123 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:28.343002    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a33f03d3-323b-47eb-8680-b69489b21e0b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a33f03d3-323b-47eb-8680-b69489b21e0b" (UID: "a33f03d3-323b-47eb-8680-b69489b21e0b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 15:57:28.343123 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:28.343059    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a33f03d3-323b-47eb-8680-b69489b21e0b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a33f03d3-323b-47eb-8680-b69489b21e0b" (UID: "a33f03d3-323b-47eb-8680-b69489b21e0b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:57:28.343369 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:28.343159    2549 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a33f03d3-323b-47eb-8680-b69489b21e0b-istiod-ca-cert\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:57:28.343369 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:28.343178    2549 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a33f03d3-323b-47eb-8680-b69489b21e0b-istio-data\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:57:28.343369 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:28.343191    2549 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a33f03d3-323b-47eb-8680-b69489b21e0b-credential-socket\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:57:28.343369 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:28.343214    2549 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a33f03d3-323b-47eb-8680-b69489b21e0b-workload-certs\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:57:28.343369 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:28.343228    2549 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a33f03d3-323b-47eb-8680-b69489b21e0b-workload-socket\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:57:28.344846 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:28.344817    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a33f03d3-323b-47eb-8680-b69489b21e0b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a33f03d3-323b-47eb-8680-b69489b21e0b" (UID: "a33f03d3-323b-47eb-8680-b69489b21e0b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 08 15:57:28.345168 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:28.345141    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a33f03d3-323b-47eb-8680-b69489b21e0b-istio-token" (OuterVolumeSpecName: "istio-token") pod "a33f03d3-323b-47eb-8680-b69489b21e0b" (UID: "a33f03d3-323b-47eb-8680-b69489b21e0b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:57:28.345250 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:28.345228    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a33f03d3-323b-47eb-8680-b69489b21e0b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a33f03d3-323b-47eb-8680-b69489b21e0b" (UID: "a33f03d3-323b-47eb-8680-b69489b21e0b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:57:28.345321 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:28.345305    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a33f03d3-323b-47eb-8680-b69489b21e0b-kube-api-access-wsqdc" (OuterVolumeSpecName: "kube-api-access-wsqdc") pod "a33f03d3-323b-47eb-8680-b69489b21e0b" (UID: "a33f03d3-323b-47eb-8680-b69489b21e0b"). InnerVolumeSpecName "kube-api-access-wsqdc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:57:28.443904 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:28.443879    2549 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a33f03d3-323b-47eb-8680-b69489b21e0b-istio-envoy\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:57:28.443904 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:28.443902    2549 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a33f03d3-323b-47eb-8680-b69489b21e0b-istio-podinfo\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:57:28.444041 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:28.443910    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wsqdc\" (UniqueName: \"kubernetes.io/projected/a33f03d3-323b-47eb-8680-b69489b21e0b-kube-api-access-wsqdc\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:57:28.444041 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:28.443919    2549 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a33f03d3-323b-47eb-8680-b69489b21e0b-istio-token\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:57:28.521659 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:28.521634    2549 generic.go:356] "Generic (PLEG): container finished" podID="a33f03d3-323b-47eb-8680-b69489b21e0b" containerID="7e07472ad3cfbabab9e72da2987b5a57753e726e884f1c16f85a76c2a62b9cf9" exitCode=0
Sep 08 15:57:28.521763 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:28.521682    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-be2aba-openshift-default-865869f8bd-ft79s" event={"ID":"a33f03d3-323b-47eb-8680-b69489b21e0b","Type":"ContainerDied","Data":"7e07472ad3cfbabab9e72da2987b5a57753e726e884f1c16f85a76c2a62b9cf9"}
Sep 08 15:57:28.521763 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:28.521684    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-be2aba-openshift-default-865869f8bd-ft79s"
Sep 08 15:57:28.521763 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:28.521721    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-be2aba-openshift-default-865869f8bd-ft79s" event={"ID":"a33f03d3-323b-47eb-8680-b69489b21e0b","Type":"ContainerDied","Data":"0562f4bbd843dd0ea9b0435ca78e341c669c0edde3d492f334d0e6d547dbebf0"}
Sep 08 15:57:28.521763 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:28.521743    2549 scope.go:117] "RemoveContainer" containerID="7e07472ad3cfbabab9e72da2987b5a57753e726e884f1c16f85a76c2a62b9cf9"
Sep 08 15:57:28.530476 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:28.530449    2549 scope.go:117] "RemoveContainer" containerID="999446d0e3799c5a9701374773c3faf79d7d475dcc51fd5401f2282c1a433b17"
Sep 08 15:57:28.538772 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:28.538745    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-be2aba-openshift-default-865869f8bd-ft79s"]
Sep 08 15:57:28.539285 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:28.539264    2549 scope.go:117] "RemoveContainer" containerID="7e07472ad3cfbabab9e72da2987b5a57753e726e884f1c16f85a76c2a62b9cf9"
Sep 08 15:57:28.539623 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:57:28.539591    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e07472ad3cfbabab9e72da2987b5a57753e726e884f1c16f85a76c2a62b9cf9\": container with ID starting with 7e07472ad3cfbabab9e72da2987b5a57753e726e884f1c16f85a76c2a62b9cf9 not found: ID does not exist" containerID="7e07472ad3cfbabab9e72da2987b5a57753e726e884f1c16f85a76c2a62b9cf9"
Sep 08 15:57:28.539775 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:28.539642    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e07472ad3cfbabab9e72da2987b5a57753e726e884f1c16f85a76c2a62b9cf9"} err="failed to get container status \"7e07472ad3cfbabab9e72da2987b5a57753e726e884f1c16f85a76c2a62b9cf9\": rpc error: code = NotFound desc = could not find container \"7e07472ad3cfbabab9e72da2987b5a57753e726e884f1c16f85a76c2a62b9cf9\": container with ID starting with 7e07472ad3cfbabab9e72da2987b5a57753e726e884f1c16f85a76c2a62b9cf9 not found: ID does not exist"
Sep 08 15:57:28.539775 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:28.539669    2549 scope.go:117] "RemoveContainer" containerID="999446d0e3799c5a9701374773c3faf79d7d475dcc51fd5401f2282c1a433b17"
Sep 08 15:57:28.540153 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:57:28.540125    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"999446d0e3799c5a9701374773c3faf79d7d475dcc51fd5401f2282c1a433b17\": container with ID starting with 999446d0e3799c5a9701374773c3faf79d7d475dcc51fd5401f2282c1a433b17 not found: ID does not exist" containerID="999446d0e3799c5a9701374773c3faf79d7d475dcc51fd5401f2282c1a433b17"
Sep 08 15:57:28.540536 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:28.540161    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"999446d0e3799c5a9701374773c3faf79d7d475dcc51fd5401f2282c1a433b17"} err="failed to get container status \"999446d0e3799c5a9701374773c3faf79d7d475dcc51fd5401f2282c1a433b17\": rpc error: code = NotFound desc = could not find container \"999446d0e3799c5a9701374773c3faf79d7d475dcc51fd5401f2282c1a433b17\": container with ID starting with 999446d0e3799c5a9701374773c3faf79d7d475dcc51fd5401f2282c1a433b17 not found: ID does not exist"
Sep 08 15:57:28.542258 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:28.542239    2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-be2aba-openshift-default-865869f8bd-ft79s"]
Sep 08 15:57:30.400438 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:30.400404    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a33f03d3-323b-47eb-8680-b69489b21e0b" path="/var/lib/kubelet/pods/a33f03d3-323b-47eb-8680-b69489b21e0b/volumes"
Sep 08 15:57:34.025444 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:34.025410    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-0ac386-5b555b7776-xz5xw"]
Sep 08 15:57:34.026041 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:34.025639    2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-0ac386-5b555b7776-xz5xw" podUID="7518a6a7-c124-4aa6-beb2-8b5840c23e3c" containerName="maas-api" containerID="cri-o://722a72e22980b8d15ea2f1e916d4132a404e76e4214f08ef6a16df182343bcc6" gracePeriod=30
Sep 08 15:57:34.269247 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:34.269221    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-0ac386-5b555b7776-xz5xw"
Sep 08 15:57:34.290361 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:34.290291    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7518a6a7-c124-4aa6-beb2-8b5840c23e3c-metrics-certs\") pod \"7518a6a7-c124-4aa6-beb2-8b5840c23e3c\" (UID: \"7518a6a7-c124-4aa6-beb2-8b5840c23e3c\") "
Sep 08 15:57:34.290361 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:34.290346    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hpbtd\" (UniqueName: \"kubernetes.io/projected/7518a6a7-c124-4aa6-beb2-8b5840c23e3c-kube-api-access-hpbtd\") pod \"7518a6a7-c124-4aa6-beb2-8b5840c23e3c\" (UID: \"7518a6a7-c124-4aa6-beb2-8b5840c23e3c\") "
Sep 08 15:57:34.290551 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:34.290417    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7518a6a7-c124-4aa6-beb2-8b5840c23e3c-maas-api-tls\") pod \"7518a6a7-c124-4aa6-beb2-8b5840c23e3c\" (UID: \"7518a6a7-c124-4aa6-beb2-8b5840c23e3c\") "
Sep 08 15:57:34.290551 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:34.290460    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7518a6a7-c124-4aa6-beb2-8b5840c23e3c-openshift-service-ca\") pod \"7518a6a7-c124-4aa6-beb2-8b5840c23e3c\" (UID: \"7518a6a7-c124-4aa6-beb2-8b5840c23e3c\") "
Sep 08 15:57:34.290919 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:34.290880    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7518a6a7-c124-4aa6-beb2-8b5840c23e3c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7518a6a7-c124-4aa6-beb2-8b5840c23e3c" (UID: "7518a6a7-c124-4aa6-beb2-8b5840c23e3c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 15:57:34.293226 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:34.293147    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7518a6a7-c124-4aa6-beb2-8b5840c23e3c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7518a6a7-c124-4aa6-beb2-8b5840c23e3c" (UID: "7518a6a7-c124-4aa6-beb2-8b5840c23e3c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 15:57:34.293334 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:34.293171    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7518a6a7-c124-4aa6-beb2-8b5840c23e3c-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "7518a6a7-c124-4aa6-beb2-8b5840c23e3c" (UID: "7518a6a7-c124-4aa6-beb2-8b5840c23e3c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 15:57:34.293842 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:34.293815    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7518a6a7-c124-4aa6-beb2-8b5840c23e3c-kube-api-access-hpbtd" (OuterVolumeSpecName: "kube-api-access-hpbtd") pod "7518a6a7-c124-4aa6-beb2-8b5840c23e3c" (UID: "7518a6a7-c124-4aa6-beb2-8b5840c23e3c"). InnerVolumeSpecName "kube-api-access-hpbtd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:57:34.392072 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:34.392045    2549 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7518a6a7-c124-4aa6-beb2-8b5840c23e3c-maas-api-tls\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:57:34.392072 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:34.392071    2549 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7518a6a7-c124-4aa6-beb2-8b5840c23e3c-openshift-service-ca\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:57:34.392220 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:34.392085    2549 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7518a6a7-c124-4aa6-beb2-8b5840c23e3c-metrics-certs\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:57:34.392220 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:34.392096    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hpbtd\" (UniqueName: \"kubernetes.io/projected/7518a6a7-c124-4aa6-beb2-8b5840c23e3c-kube-api-access-hpbtd\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:57:34.551619 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:34.551537    2549 generic.go:356] "Generic (PLEG): container finished" podID="7518a6a7-c124-4aa6-beb2-8b5840c23e3c" containerID="722a72e22980b8d15ea2f1e916d4132a404e76e4214f08ef6a16df182343bcc6" exitCode=0
Sep 08 15:57:34.551619 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:34.551585    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-0ac386-5b555b7776-xz5xw"
Sep 08 15:57:34.551619 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:34.551593    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-0ac386-5b555b7776-xz5xw" event={"ID":"7518a6a7-c124-4aa6-beb2-8b5840c23e3c","Type":"ContainerDied","Data":"722a72e22980b8d15ea2f1e916d4132a404e76e4214f08ef6a16df182343bcc6"}
Sep 08 15:57:34.551878 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:34.551634    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-0ac386-5b555b7776-xz5xw" event={"ID":"7518a6a7-c124-4aa6-beb2-8b5840c23e3c","Type":"ContainerDied","Data":"aa9d319aab059e04121baffb7479aa35a8927fcb02886ac4585155ada0854827"}
Sep 08 15:57:34.551878 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:34.551655    2549 scope.go:117] "RemoveContainer" containerID="722a72e22980b8d15ea2f1e916d4132a404e76e4214f08ef6a16df182343bcc6"
Sep 08 15:57:34.560603 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:34.560586    2549 scope.go:117] "RemoveContainer" containerID="722a72e22980b8d15ea2f1e916d4132a404e76e4214f08ef6a16df182343bcc6"
Sep 08 15:57:34.560889 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:57:34.560845    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"722a72e22980b8d15ea2f1e916d4132a404e76e4214f08ef6a16df182343bcc6\": container with ID starting with 722a72e22980b8d15ea2f1e916d4132a404e76e4214f08ef6a16df182343bcc6 not found: ID does not exist" containerID="722a72e22980b8d15ea2f1e916d4132a404e76e4214f08ef6a16df182343bcc6"
Sep 08 15:57:34.560992 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:34.560894    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"722a72e22980b8d15ea2f1e916d4132a404e76e4214f08ef6a16df182343bcc6"} err="failed to get container status \"722a72e22980b8d15ea2f1e916d4132a404e76e4214f08ef6a16df182343bcc6\": rpc error: code = NotFound desc = could not find container \"722a72e22980b8d15ea2f1e916d4132a404e76e4214f08ef6a16df182343bcc6\": container with ID starting with 722a72e22980b8d15ea2f1e916d4132a404e76e4214f08ef6a16df182343bcc6 not found: ID does not exist"
Sep 08 15:57:34.568695 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:34.568671    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-0ac386-5b555b7776-xz5xw"]
Sep 08 15:57:34.574710 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:34.574687    2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-0ac386-5b555b7776-xz5xw"]
Sep 08 15:57:36.400496 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:36.400460    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7518a6a7-c124-4aa6-beb2-8b5840c23e3c" path="/var/lib/kubelet/pods/7518a6a7-c124-4aa6-beb2-8b5840c23e3c/volumes"
Sep 08 15:57:42.034622 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:42.034583    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-642d3b/rate-test-model-642d3b-kserve-696597fd7-8xgfp"]
Sep 08 15:57:42.035026 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:42.034929    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ec415c7-2031-4610-9e09-6eb7e37ef0d6" containerName="main"
Sep 08 15:57:42.035026 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:42.034944    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ec415c7-2031-4610-9e09-6eb7e37ef0d6" containerName="main"
Sep 08 15:57:42.035026 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:42.034957    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b14067d-6206-4127-b430-4f579c2cfe8c" containerName="mt-curl-7781-ce69cd"
Sep 08 15:57:42.035026 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:42.034963    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b14067d-6206-4127-b430-4f579c2cfe8c" containerName="mt-curl-7781-ce69cd"
Sep 08 15:57:42.035026 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:42.034974    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f73cff5c-48e7-4eae-816d-6c4fb8cc8e54" containerName="istio-proxy"
Sep 08 15:57:42.035026 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:42.034980    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="f73cff5c-48e7-4eae-816d-6c4fb8cc8e54" containerName="istio-proxy"
Sep 08 15:57:42.035026 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:42.034992    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a33f03d3-323b-47eb-8680-b69489b21e0b" containerName="istio-proxy"
Sep 08 15:57:42.035026 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:42.034997    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="a33f03d3-323b-47eb-8680-b69489b21e0b" containerName="istio-proxy"
Sep 08 15:57:42.035026 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:42.035004    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7518a6a7-c124-4aa6-beb2-8b5840c23e3c" containerName="maas-api"
Sep 08 15:57:42.035026 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:42.035008    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="7518a6a7-c124-4aa6-beb2-8b5840c23e3c" containerName="maas-api"
Sep 08 15:57:42.035026 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:42.035030    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a33f03d3-323b-47eb-8680-b69489b21e0b" containerName="istio-proxy"
Sep 08 15:57:42.035026 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:42.035036    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="a33f03d3-323b-47eb-8680-b69489b21e0b" containerName="istio-proxy"
Sep 08 15:57:42.035461 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:42.035043    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="155f8e63-bfc2-476c-bbee-674645c242b3" containerName="main"
Sep 08 15:57:42.035461 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:42.035050    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="155f8e63-bfc2-476c-bbee-674645c242b3" containerName="main"
Sep 08 15:57:42.035461 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:42.035121    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="f73cff5c-48e7-4eae-816d-6c4fb8cc8e54" containerName="istio-proxy"
Sep 08 15:57:42.035461 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:42.035133    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="155f8e63-bfc2-476c-bbee-674645c242b3" containerName="main"
Sep 08 15:57:42.035461 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:42.035142    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b14067d-6206-4127-b430-4f579c2cfe8c" containerName="mt-curl-7781-ce69cd"
Sep 08 15:57:42.035461 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:42.035150    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="f73cff5c-48e7-4eae-816d-6c4fb8cc8e54" containerName="istio-proxy"
Sep 08 15:57:42.035461 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:42.035158    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ec415c7-2031-4610-9e09-6eb7e37ef0d6" containerName="main"
Sep 08 15:57:42.035461 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:42.035171    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="7518a6a7-c124-4aa6-beb2-8b5840c23e3c" containerName="maas-api"
Sep 08 15:57:42.035461 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:42.035177    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="a33f03d3-323b-47eb-8680-b69489b21e0b" containerName="istio-proxy"
Sep 08 15:57:42.039281 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:42.039265    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-642d3b/rate-test-model-642d3b-kserve-696597fd7-8xgfp"
Sep 08 15:57:42.042210 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:42.042187    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-642d3b\"/\"kube-root-ca.crt\""
Sep 08 15:57:42.043338 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:42.043321    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-642d3b\"/\"default-dockercfg-sp62x\""
Sep 08 15:57:42.043451 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:42.043352    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-642d3b\"/\"openshift-service-ca.crt\""
Sep 08 15:57:42.043451 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:42.043355    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-642d3b\"/\"rate-test-model-642d3b-kserve-self-signed-certs\""
Sep 08 15:57:42.048121 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:42.048099    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-642d3b/rate-test-model-642d3b-kserve-696597fd7-8xgfp"]
Sep 08 15:57:42.155832 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:42.155793    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d1ea8c0-df86-4917-9ded-766cd3100fce-tmp-dir\") pod \"rate-test-model-642d3b-kserve-696597fd7-8xgfp\" (UID: \"5d1ea8c0-df86-4917-9ded-766cd3100fce\") " pod="ai-tenant-e2e-shared-b-w3-642d3b/rate-test-model-642d3b-kserve-696597fd7-8xgfp"
Sep 08 15:57:42.155832 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:42.155834    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sg6m8\" (UniqueName: \"kubernetes.io/projected/5d1ea8c0-df86-4917-9ded-766cd3100fce-kube-api-access-sg6m8\") pod \"rate-test-model-642d3b-kserve-696597fd7-8xgfp\" (UID: \"5d1ea8c0-df86-4917-9ded-766cd3100fce\") " pod="ai-tenant-e2e-shared-b-w3-642d3b/rate-test-model-642d3b-kserve-696597fd7-8xgfp"
Sep 08 15:57:42.156046 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:42.155899    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d1ea8c0-df86-4917-9ded-766cd3100fce-home\") pod \"rate-test-model-642d3b-kserve-696597fd7-8xgfp\" (UID: \"5d1ea8c0-df86-4917-9ded-766cd3100fce\") " pod="ai-tenant-e2e-shared-b-w3-642d3b/rate-test-model-642d3b-kserve-696597fd7-8xgfp"
Sep 08 15:57:42.156046 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:42.155928    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d1ea8c0-df86-4917-9ded-766cd3100fce-tls-certs\") pod \"rate-test-model-642d3b-kserve-696597fd7-8xgfp\" (UID: \"5d1ea8c0-df86-4917-9ded-766cd3100fce\") " pod="ai-tenant-e2e-shared-b-w3-642d3b/rate-test-model-642d3b-kserve-696597fd7-8xgfp"
Sep 08 15:57:42.156046 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:42.155975    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d1ea8c0-df86-4917-9ded-766cd3100fce-dshm\") pod \"rate-test-model-642d3b-kserve-696597fd7-8xgfp\" (UID: \"5d1ea8c0-df86-4917-9ded-766cd3100fce\") " pod="ai-tenant-e2e-shared-b-w3-642d3b/rate-test-model-642d3b-kserve-696597fd7-8xgfp"
Sep 08 15:57:42.156046 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:42.156008    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d1ea8c0-df86-4917-9ded-766cd3100fce-model-cache\") pod \"rate-test-model-642d3b-kserve-696597fd7-8xgfp\" (UID: \"5d1ea8c0-df86-4917-9ded-766cd3100fce\") " pod="ai-tenant-e2e-shared-b-w3-642d3b/rate-test-model-642d3b-kserve-696597fd7-8xgfp"
Sep 08 15:57:42.257440 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:42.257405    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d1ea8c0-df86-4917-9ded-766cd3100fce-tmp-dir\") pod \"rate-test-model-642d3b-kserve-696597fd7-8xgfp\" (UID: \"5d1ea8c0-df86-4917-9ded-766cd3100fce\") " pod="ai-tenant-e2e-shared-b-w3-642d3b/rate-test-model-642d3b-kserve-696597fd7-8xgfp"
Sep 08 15:57:42.257440 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:42.257444    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sg6m8\" (UniqueName: \"kubernetes.io/projected/5d1ea8c0-df86-4917-9ded-766cd3100fce-kube-api-access-sg6m8\") pod \"rate-test-model-642d3b-kserve-696597fd7-8xgfp\" (UID: \"5d1ea8c0-df86-4917-9ded-766cd3100fce\") " pod="ai-tenant-e2e-shared-b-w3-642d3b/rate-test-model-642d3b-kserve-696597fd7-8xgfp"
Sep 08 15:57:42.257699 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:42.257471    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d1ea8c0-df86-4917-9ded-766cd3100fce-home\") pod \"rate-test-model-642d3b-kserve-696597fd7-8xgfp\" (UID: \"5d1ea8c0-df86-4917-9ded-766cd3100fce\") " pod="ai-tenant-e2e-shared-b-w3-642d3b/rate-test-model-642d3b-kserve-696597fd7-8xgfp"
Sep 08 15:57:42.257699 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:42.257495    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d1ea8c0-df86-4917-9ded-766cd3100fce-tls-certs\") pod \"rate-test-model-642d3b-kserve-696597fd7-8xgfp\" (UID: \"5d1ea8c0-df86-4917-9ded-766cd3100fce\") " pod="ai-tenant-e2e-shared-b-w3-642d3b/rate-test-model-642d3b-kserve-696597fd7-8xgfp"
Sep 08 15:57:42.257699 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:42.257511    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d1ea8c0-df86-4917-9ded-766cd3100fce-dshm\") pod \"rate-test-model-642d3b-kserve-696597fd7-8xgfp\" (UID: \"5d1ea8c0-df86-4917-9ded-766cd3100fce\") " pod="ai-tenant-e2e-shared-b-w3-642d3b/rate-test-model-642d3b-kserve-696597fd7-8xgfp"
Sep 08 15:57:42.257699 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:42.257539    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d1ea8c0-df86-4917-9ded-766cd3100fce-model-cache\") pod \"rate-test-model-642d3b-kserve-696597fd7-8xgfp\" (UID: \"5d1ea8c0-df86-4917-9ded-766cd3100fce\") " pod="ai-tenant-e2e-shared-b-w3-642d3b/rate-test-model-642d3b-kserve-696597fd7-8xgfp"
Sep 08 15:57:42.257896 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:42.257801    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d1ea8c0-df86-4917-9ded-766cd3100fce-tmp-dir\") pod \"rate-test-model-642d3b-kserve-696597fd7-8xgfp\" (UID: \"5d1ea8c0-df86-4917-9ded-766cd3100fce\") " pod="ai-tenant-e2e-shared-b-w3-642d3b/rate-test-model-642d3b-kserve-696597fd7-8xgfp"
Sep 08 15:57:42.257896 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:42.257877    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d1ea8c0-df86-4917-9ded-766cd3100fce-home\") pod \"rate-test-model-642d3b-kserve-696597fd7-8xgfp\" (UID: \"5d1ea8c0-df86-4917-9ded-766cd3100fce\") " pod="ai-tenant-e2e-shared-b-w3-642d3b/rate-test-model-642d3b-kserve-696597fd7-8xgfp"
Sep 08 15:57:42.258000 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:42.257982    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d1ea8c0-df86-4917-9ded-766cd3100fce-model-cache\") pod \"rate-test-model-642d3b-kserve-696597fd7-8xgfp\" (UID: \"5d1ea8c0-df86-4917-9ded-766cd3100fce\") " pod="ai-tenant-e2e-shared-b-w3-642d3b/rate-test-model-642d3b-kserve-696597fd7-8xgfp"
Sep 08 15:57:42.260016 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:42.259988    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d1ea8c0-df86-4917-9ded-766cd3100fce-dshm\") pod \"rate-test-model-642d3b-kserve-696597fd7-8xgfp\" (UID: \"5d1ea8c0-df86-4917-9ded-766cd3100fce\") " pod="ai-tenant-e2e-shared-b-w3-642d3b/rate-test-model-642d3b-kserve-696597fd7-8xgfp"
Sep 08 15:57:42.260016 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:42.260003    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d1ea8c0-df86-4917-9ded-766cd3100fce-tls-certs\") pod \"rate-test-model-642d3b-kserve-696597fd7-8xgfp\" (UID: \"5d1ea8c0-df86-4917-9ded-766cd3100fce\") " pod="ai-tenant-e2e-shared-b-w3-642d3b/rate-test-model-642d3b-kserve-696597fd7-8xgfp"
Sep 08 15:57:42.265650 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:42.265626    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sg6m8\" (UniqueName: \"kubernetes.io/projected/5d1ea8c0-df86-4917-9ded-766cd3100fce-kube-api-access-sg6m8\") pod \"rate-test-model-642d3b-kserve-696597fd7-8xgfp\" (UID: \"5d1ea8c0-df86-4917-9ded-766cd3100fce\") " pod="ai-tenant-e2e-shared-b-w3-642d3b/rate-test-model-642d3b-kserve-696597fd7-8xgfp"
Sep 08 15:57:42.351576 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:42.351516    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-642d3b\"/\"default-dockercfg-sp62x\""
Sep 08 15:57:42.360041 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:42.360021    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-642d3b/rate-test-model-642d3b-kserve-696597fd7-8xgfp"
Sep 08 15:57:42.435190 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:42.435162    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-wpwdq_5f7b0f68-a8e3-4058-b032-15f419682b2a/console-operator/2.log"
Sep 08 15:57:42.436393 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:42.436369    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-wpwdq_5f7b0f68-a8e3-4058-b032-15f419682b2a/console-operator/2.log"
Sep 08 15:57:42.445114 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:42.445093    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-f4vvt_195f1a74-19bc-49c9-aada-fa1c13488480/ovn-acl-logging/0.log"
Sep 08 15:57:42.445696 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:42.445677    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-f4vvt_195f1a74-19bc-49c9-aada-fa1c13488480/ovn-acl-logging/0.log"
Sep 08 15:57:42.550434 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:42.550409    2549 scope.go:117] "RemoveContainer" containerID="52d3949d3815d362cb68932b0f8774098f8916669419902a54663a96903a8a08"
Sep 08 15:57:42.634046 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:42.634023    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-642d3b/rate-test-model-642d3b-kserve-696597fd7-8xgfp"]
Sep 08 15:57:42.636472 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:57:42.636444    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5d1ea8c0_df86_4917_9ded_766cd3100fce.slice/crio-a26ef2346a23f283626714d94054ae200d471dbd061f575179b22c5eafef71c3 WatchSource:0}: Error finding container a26ef2346a23f283626714d94054ae200d471dbd061f575179b22c5eafef71c3: Status 404 returned error can't find the container with id a26ef2346a23f283626714d94054ae200d471dbd061f575179b22c5eafef71c3
Sep 08 15:57:43.596383 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:43.596350    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-642d3b/rate-test-model-642d3b-kserve-696597fd7-8xgfp" event={"ID":"5d1ea8c0-df86-4917-9ded-766cd3100fce","Type":"ContainerStarted","Data":"4d6c56697af3d5c183b555ea2cbaab568d9952efd6f1d4cbfb7d71110a834a0e"}
Sep 08 15:57:43.596383 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:43.596387    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-642d3b/rate-test-model-642d3b-kserve-696597fd7-8xgfp" event={"ID":"5d1ea8c0-df86-4917-9ded-766cd3100fce","Type":"ContainerStarted","Data":"a26ef2346a23f283626714d94054ae200d471dbd061f575179b22c5eafef71c3"}
Sep 08 15:57:43.613842 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:43.613793    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-642d3b/rate-test-model-642d3b-kserve-696597fd7-8xgfp" podStartSLOduration=1.613778275 podStartE2EDuration="1.613778275s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:57:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:57:43.612276507 +0000 UTC m=+1501.701727595" watchObservedRunningTime="2026-09-08 15:57:43.613778275 +0000 UTC m=+1501.703229365"
Sep 08 15:57:52.360851 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:52.360815    2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-642d3b/rate-test-model-642d3b-kserve-696597fd7-8xgfp"
Sep 08 15:57:52.360851 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:52.360879    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-642d3b/rate-test-model-642d3b-kserve-696597fd7-8xgfp"
Sep 08 15:57:52.373411 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:52.373381    2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-642d3b/rate-test-model-642d3b-kserve-696597fd7-8xgfp"
Sep 08 15:57:52.647891 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:57:52.647866    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-642d3b/rate-test-model-642d3b-kserve-696597fd7-8xgfp"
Sep 08 15:58:20.221415 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:20.221381    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-642d3b/rate-test-model-642d3b-kserve-696597fd7-8xgfp"]
Sep 08 15:58:24.985948 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:24.985912    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-e0da5a-8f59c46fb-8c7sm"]
Sep 08 15:58:24.986380 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:24.986271    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f73cff5c-48e7-4eae-816d-6c4fb8cc8e54" containerName="istio-proxy"
Sep 08 15:58:24.986380 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:24.986284    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="f73cff5c-48e7-4eae-816d-6c4fb8cc8e54" containerName="istio-proxy"
Sep 08 15:58:24.986380 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:24.986342    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="a33f03d3-323b-47eb-8680-b69489b21e0b" containerName="istio-proxy"
Sep 08 15:58:24.989087 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:24.989064    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e0da5a-8f59c46fb-8c7sm"
Sep 08 15:58:24.992159 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:24.992132    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-e0da5a-serving-cert\""
Sep 08 15:58:24.992447 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:24.992428    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 08 15:58:24.993636 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:24.993618    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-tq4f4\""
Sep 08 15:58:24.993741 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:24.993648    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 08 15:58:24.993741 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:24.993658    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 08 15:58:25.003510 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:25.003486    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-e0da5a-8f59c46fb-8c7sm"]
Sep 08 15:58:25.021931 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:25.021904    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9dbeaa09-4695-4eb4-80b6-08da3893d90f-maas-api-tls\") pod \"maas-api-e2e-ipp-a-e0da5a-8f59c46fb-8c7sm\" (UID: \"9dbeaa09-4695-4eb4-80b6-08da3893d90f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e0da5a-8f59c46fb-8c7sm"
Sep 08 15:58:25.022036 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:25.021993    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9dbeaa09-4695-4eb4-80b6-08da3893d90f-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-e0da5a-8f59c46fb-8c7sm\" (UID: \"9dbeaa09-4695-4eb4-80b6-08da3893d90f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e0da5a-8f59c46fb-8c7sm"
Sep 08 15:58:25.022036 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:25.022030    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9dbeaa09-4695-4eb4-80b6-08da3893d90f-metrics-certs\") pod \"maas-api-e2e-ipp-a-e0da5a-8f59c46fb-8c7sm\" (UID: \"9dbeaa09-4695-4eb4-80b6-08da3893d90f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e0da5a-8f59c46fb-8c7sm"
Sep 08 15:58:25.022156 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:25.022089    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qt82l\" (UniqueName: \"kubernetes.io/projected/9dbeaa09-4695-4eb4-80b6-08da3893d90f-kube-api-access-qt82l\") pod \"maas-api-e2e-ipp-a-e0da5a-8f59c46fb-8c7sm\" (UID: \"9dbeaa09-4695-4eb4-80b6-08da3893d90f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e0da5a-8f59c46fb-8c7sm"
Sep 08 15:58:25.123104 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:25.123071    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9dbeaa09-4695-4eb4-80b6-08da3893d90f-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-e0da5a-8f59c46fb-8c7sm\" (UID: \"9dbeaa09-4695-4eb4-80b6-08da3893d90f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e0da5a-8f59c46fb-8c7sm"
Sep 08 15:58:25.123239 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:25.123107    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9dbeaa09-4695-4eb4-80b6-08da3893d90f-metrics-certs\") pod \"maas-api-e2e-ipp-a-e0da5a-8f59c46fb-8c7sm\" (UID: \"9dbeaa09-4695-4eb4-80b6-08da3893d90f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e0da5a-8f59c46fb-8c7sm"
Sep 08 15:58:25.123239 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:25.123136    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qt82l\" (UniqueName: \"kubernetes.io/projected/9dbeaa09-4695-4eb4-80b6-08da3893d90f-kube-api-access-qt82l\") pod \"maas-api-e2e-ipp-a-e0da5a-8f59c46fb-8c7sm\" (UID: \"9dbeaa09-4695-4eb4-80b6-08da3893d90f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e0da5a-8f59c46fb-8c7sm"
Sep 08 15:58:25.123239 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:25.123174    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9dbeaa09-4695-4eb4-80b6-08da3893d90f-maas-api-tls\") pod \"maas-api-e2e-ipp-a-e0da5a-8f59c46fb-8c7sm\" (UID: \"9dbeaa09-4695-4eb4-80b6-08da3893d90f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e0da5a-8f59c46fb-8c7sm"
Sep 08 15:58:25.123890 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:25.123843    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9dbeaa09-4695-4eb4-80b6-08da3893d90f-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-e0da5a-8f59c46fb-8c7sm\" (UID: \"9dbeaa09-4695-4eb4-80b6-08da3893d90f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e0da5a-8f59c46fb-8c7sm"
Sep 08 15:58:25.125646 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:25.125619    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9dbeaa09-4695-4eb4-80b6-08da3893d90f-metrics-certs\") pod \"maas-api-e2e-ipp-a-e0da5a-8f59c46fb-8c7sm\" (UID: \"9dbeaa09-4695-4eb4-80b6-08da3893d90f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e0da5a-8f59c46fb-8c7sm"
Sep 08 15:58:25.125780 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:25.125759    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9dbeaa09-4695-4eb4-80b6-08da3893d90f-maas-api-tls\") pod \"maas-api-e2e-ipp-a-e0da5a-8f59c46fb-8c7sm\" (UID: \"9dbeaa09-4695-4eb4-80b6-08da3893d90f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e0da5a-8f59c46fb-8c7sm"
Sep 08 15:58:25.134088 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:25.134061    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qt82l\" (UniqueName: \"kubernetes.io/projected/9dbeaa09-4695-4eb4-80b6-08da3893d90f-kube-api-access-qt82l\") pod \"maas-api-e2e-ipp-a-e0da5a-8f59c46fb-8c7sm\" (UID: \"9dbeaa09-4695-4eb4-80b6-08da3893d90f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e0da5a-8f59c46fb-8c7sm"
Sep 08 15:58:25.300277 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:25.300192    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e0da5a-8f59c46fb-8c7sm"
Sep 08 15:58:25.579715 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:25.579661    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-e0da5a-8f59c46fb-8c7sm"]
Sep 08 15:58:25.789708 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:25.789662    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e0da5a-8f59c46fb-8c7sm" event={"ID":"9dbeaa09-4695-4eb4-80b6-08da3893d90f","Type":"ContainerStarted","Data":"e23c752edba199009875376988d82110d622902865c0bb7a801835272093c7dc"}
Sep 08 15:58:26.796193 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:26.796158    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e0da5a-8f59c46fb-8c7sm" event={"ID":"9dbeaa09-4695-4eb4-80b6-08da3893d90f","Type":"ContainerStarted","Data":"ba525a946b181ed231a21a5bd74bf3d4a4969f0ce73e664993d998a5f3273327"}
Sep 08 15:58:26.796571 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:26.796291    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e0da5a-8f59c46fb-8c7sm"
Sep 08 15:58:26.814648 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:26.814601    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e0da5a-8f59c46fb-8c7sm" podStartSLOduration=2.424866706 podStartE2EDuration="2.814585916s" totalImagesPullingTime="389.71921ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:58:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:58:26.814370628 +0000 UTC m=+1544.903821717" watchObservedRunningTime="2026-09-08 15:58:26.814585916 +0000 UTC m=+1544.904037004"
Sep 08 15:58:27.144830 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:27.144803    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7v8687"]
Sep 08 15:58:27.148513 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:27.148495    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7v8687"
Sep 08 15:58:27.151286 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:27.151265    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 08 15:58:27.151392 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:27.151298    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 08 15:58:27.151570 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:27.151552    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-qc6rj\""
Sep 08 15:58:27.157710 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:27.157686    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7v8687"]
Sep 08 15:58:27.239157 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:27.239129    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69vhgw"]
Sep 08 15:58:27.239414 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:27.239393    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75llj\" (UniqueName: \"kubernetes.io/projected/23b78a32-811f-4864-83fb-8da7130eddb0-kube-api-access-75llj\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7v8687\" (UID: \"23b78a32-811f-4864-83fb-8da7130eddb0\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7v8687"
Sep 08 15:58:27.239469 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:27.239436    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/23b78a32-811f-4864-83fb-8da7130eddb0-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7v8687\" (UID: \"23b78a32-811f-4864-83fb-8da7130eddb0\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7v8687"
Sep 08 15:58:27.239575 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:27.239555    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/23b78a32-811f-4864-83fb-8da7130eddb0-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7v8687\" (UID: \"23b78a32-811f-4864-83fb-8da7130eddb0\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7v8687"
Sep 08 15:58:27.242977 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:27.242960    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69vhgw"
Sep 08 15:58:27.252193 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:27.252170    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69vhgw"]
Sep 08 15:58:27.340392 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:27.340355    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpsxs\" (UniqueName: \"kubernetes.io/projected/ee187f09-5a2b-42f7-a8d8-d720996c0a19-kube-api-access-rpsxs\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69vhgw\" (UID: \"ee187f09-5a2b-42f7-a8d8-d720996c0a19\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69vhgw"
Sep 08 15:58:27.340572 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:27.340408    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ee187f09-5a2b-42f7-a8d8-d720996c0a19-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69vhgw\" (UID: \"ee187f09-5a2b-42f7-a8d8-d720996c0a19\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69vhgw"
Sep 08 15:58:27.340572 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:27.340446    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75llj\" (UniqueName: \"kubernetes.io/projected/23b78a32-811f-4864-83fb-8da7130eddb0-kube-api-access-75llj\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7v8687\" (UID: \"23b78a32-811f-4864-83fb-8da7130eddb0\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7v8687"
Sep 08 15:58:27.340572 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:27.340495    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/23b78a32-811f-4864-83fb-8da7130eddb0-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7v8687\" (UID: \"23b78a32-811f-4864-83fb-8da7130eddb0\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7v8687"
Sep 08 15:58:27.340784 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:27.340610    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ee187f09-5a2b-42f7-a8d8-d720996c0a19-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69vhgw\" (UID: \"ee187f09-5a2b-42f7-a8d8-d720996c0a19\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69vhgw"
Sep 08 15:58:27.340784 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:27.340652    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/23b78a32-811f-4864-83fb-8da7130eddb0-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7v8687\" (UID: \"23b78a32-811f-4864-83fb-8da7130eddb0\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7v8687"
Sep 08 15:58:27.340969 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:27.340943    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/23b78a32-811f-4864-83fb-8da7130eddb0-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7v8687\" (UID: \"23b78a32-811f-4864-83fb-8da7130eddb0\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7v8687"
Sep 08 15:58:27.341087 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:27.341064    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/23b78a32-811f-4864-83fb-8da7130eddb0-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7v8687\" (UID: \"23b78a32-811f-4864-83fb-8da7130eddb0\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7v8687"
Sep 08 15:58:27.351836 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:27.351813    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atwdg5"]
Sep 08 15:58:27.353302 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:27.353277    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-75llj\" (UniqueName: \"kubernetes.io/projected/23b78a32-811f-4864-83fb-8da7130eddb0-kube-api-access-75llj\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7v8687\" (UID: \"23b78a32-811f-4864-83fb-8da7130eddb0\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7v8687"
Sep 08 15:58:27.355481 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:27.355465    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atwdg5"
Sep 08 15:58:27.368525 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:27.368502    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atwdg5"]
Sep 08 15:58:27.441652 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:27.441591    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ee187f09-5a2b-42f7-a8d8-d720996c0a19-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69vhgw\" (UID: \"ee187f09-5a2b-42f7-a8d8-d720996c0a19\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69vhgw"
Sep 08 15:58:27.441652 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:27.441642    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpsxs\" (UniqueName: \"kubernetes.io/projected/ee187f09-5a2b-42f7-a8d8-d720996c0a19-kube-api-access-rpsxs\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69vhgw\" (UID: \"ee187f09-5a2b-42f7-a8d8-d720996c0a19\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69vhgw"
Sep 08 15:58:27.441844 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:27.441667    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ee187f09-5a2b-42f7-a8d8-d720996c0a19-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69vhgw\" (UID: \"ee187f09-5a2b-42f7-a8d8-d720996c0a19\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69vhgw"
Sep 08 15:58:27.441844 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:27.441723    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7e5a3717-1782-4fb9-89c9-4ede7b591461-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atwdg5\" (UID: \"7e5a3717-1782-4fb9-89c9-4ede7b591461\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atwdg5"
Sep 08 15:58:27.441844 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:27.441754    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7e5a3717-1782-4fb9-89c9-4ede7b591461-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atwdg5\" (UID: \"7e5a3717-1782-4fb9-89c9-4ede7b591461\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atwdg5"
Sep 08 15:58:27.441844 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:27.441779    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbpxc\" (UniqueName: \"kubernetes.io/projected/7e5a3717-1782-4fb9-89c9-4ede7b591461-kube-api-access-xbpxc\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atwdg5\" (UID: \"7e5a3717-1782-4fb9-89c9-4ede7b591461\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atwdg5"
Sep 08 15:58:27.442047 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:27.441991    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ee187f09-5a2b-42f7-a8d8-d720996c0a19-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69vhgw\" (UID: \"ee187f09-5a2b-42f7-a8d8-d720996c0a19\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69vhgw"
Sep 08 15:58:27.442119 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:27.442099    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ee187f09-5a2b-42f7-a8d8-d720996c0a19-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69vhgw\" (UID: \"ee187f09-5a2b-42f7-a8d8-d720996c0a19\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69vhgw"
Sep 08 15:58:27.448610 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:27.448589    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwc6qz"]
Sep 08 15:58:27.452547 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:27.452531    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwc6qz"
Sep 08 15:58:27.454332 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:27.454313    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpsxs\" (UniqueName: \"kubernetes.io/projected/ee187f09-5a2b-42f7-a8d8-d720996c0a19-kube-api-access-rpsxs\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69vhgw\" (UID: \"ee187f09-5a2b-42f7-a8d8-d720996c0a19\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69vhgw"
Sep 08 15:58:27.458498 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:27.458480    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7v8687"
Sep 08 15:58:27.462700 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:27.462658    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwc6qz"]
Sep 08 15:58:27.543197 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:27.543169    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcz6z\" (UniqueName: \"kubernetes.io/projected/1afd9af4-dee3-4c13-9133-dc9c89572d51-kube-api-access-zcz6z\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwc6qz\" (UID: \"1afd9af4-dee3-4c13-9133-dc9c89572d51\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwc6qz"
Sep 08 15:58:27.543346 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:27.543226    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1afd9af4-dee3-4c13-9133-dc9c89572d51-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwc6qz\" (UID: \"1afd9af4-dee3-4c13-9133-dc9c89572d51\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwc6qz"
Sep 08 15:58:27.543346 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:27.543304    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1afd9af4-dee3-4c13-9133-dc9c89572d51-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwc6qz\" (UID: \"1afd9af4-dee3-4c13-9133-dc9c89572d51\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwc6qz"
Sep 08 15:58:27.543456 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:27.543370    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7e5a3717-1782-4fb9-89c9-4ede7b591461-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atwdg5\" (UID: \"7e5a3717-1782-4fb9-89c9-4ede7b591461\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atwdg5"
Sep 08 15:58:27.543456 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:27.543403    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7e5a3717-1782-4fb9-89c9-4ede7b591461-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atwdg5\" (UID: \"7e5a3717-1782-4fb9-89c9-4ede7b591461\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atwdg5"
Sep 08 15:58:27.543456 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:27.543422    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbpxc\" (UniqueName: \"kubernetes.io/projected/7e5a3717-1782-4fb9-89c9-4ede7b591461-kube-api-access-xbpxc\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atwdg5\" (UID: \"7e5a3717-1782-4fb9-89c9-4ede7b591461\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atwdg5"
Sep 08 15:58:27.543735 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:27.543715    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7e5a3717-1782-4fb9-89c9-4ede7b591461-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atwdg5\" (UID: \"7e5a3717-1782-4fb9-89c9-4ede7b591461\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atwdg5"
Sep 08 15:58:27.543804 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:27.543744    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7e5a3717-1782-4fb9-89c9-4ede7b591461-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atwdg5\" (UID: \"7e5a3717-1782-4fb9-89c9-4ede7b591461\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atwdg5"
Sep 08 15:58:27.552953 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:27.552931    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69vhgw"
Sep 08 15:58:27.557159 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:27.557137    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbpxc\" (UniqueName: \"kubernetes.io/projected/7e5a3717-1782-4fb9-89c9-4ede7b591461-kube-api-access-xbpxc\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atwdg5\" (UID: \"7e5a3717-1782-4fb9-89c9-4ede7b591461\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atwdg5"
Sep 08 15:58:27.644232 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:27.644203    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcz6z\" (UniqueName: \"kubernetes.io/projected/1afd9af4-dee3-4c13-9133-dc9c89572d51-kube-api-access-zcz6z\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwc6qz\" (UID: \"1afd9af4-dee3-4c13-9133-dc9c89572d51\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwc6qz"
Sep 08 15:58:27.644371 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:27.644247    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1afd9af4-dee3-4c13-9133-dc9c89572d51-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwc6qz\" (UID: \"1afd9af4-dee3-4c13-9133-dc9c89572d51\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwc6qz"
Sep 08 15:58:27.644371 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:27.644288    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1afd9af4-dee3-4c13-9133-dc9c89572d51-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwc6qz\" (UID: \"1afd9af4-dee3-4c13-9133-dc9c89572d51\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwc6qz"
Sep 08 15:58:27.644714 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:27.644689    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1afd9af4-dee3-4c13-9133-dc9c89572d51-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwc6qz\" (UID: \"1afd9af4-dee3-4c13-9133-dc9c89572d51\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwc6qz"
Sep 08 15:58:27.644811 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:27.644724    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1afd9af4-dee3-4c13-9133-dc9c89572d51-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwc6qz\" (UID: \"1afd9af4-dee3-4c13-9133-dc9c89572d51\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwc6qz"
Sep 08 15:58:27.653628 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:27.653603    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcz6z\" (UniqueName: \"kubernetes.io/projected/1afd9af4-dee3-4c13-9133-dc9c89572d51-kube-api-access-zcz6z\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwc6qz\" (UID: \"1afd9af4-dee3-4c13-9133-dc9c89572d51\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwc6qz"
Sep 08 15:58:27.668980 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:27.668961    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atwdg5"
Sep 08 15:58:27.734212 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:27.734180    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7v8687"]
Sep 08 15:58:27.737140 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:58:27.737109    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23b78a32_811f_4864_83fb_8da7130eddb0.slice/crio-168efecc24a099a98e244a021592c4493d2be5dbaa5bd7bf6cbf1832c55b100d WatchSource:0}: Error finding container 168efecc24a099a98e244a021592c4493d2be5dbaa5bd7bf6cbf1832c55b100d: Status 404 returned error can't find the container with id 168efecc24a099a98e244a021592c4493d2be5dbaa5bd7bf6cbf1832c55b100d
Sep 08 15:58:27.784140 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:27.784119    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwc6qz"
Sep 08 15:58:27.811788 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:27.811755    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7v8687" event={"ID":"23b78a32-811f-4864-83fb-8da7130eddb0","Type":"ContainerStarted","Data":"168efecc24a099a98e244a021592c4493d2be5dbaa5bd7bf6cbf1832c55b100d"}
Sep 08 15:58:27.841368 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:27.841337    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69vhgw"]
Sep 08 15:58:27.846176 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:58:27.845683    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee187f09_5a2b_42f7_a8d8_d720996c0a19.slice/crio-29af0a58a0ba53335d9165f1075d4d438a176db18e0a4d190e5a7394ead8ce26 WatchSource:0}: Error finding container 29af0a58a0ba53335d9165f1075d4d438a176db18e0a4d190e5a7394ead8ce26: Status 404 returned error can't find the container with id 29af0a58a0ba53335d9165f1075d4d438a176db18e0a4d190e5a7394ead8ce26
Sep 08 15:58:27.952817 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:27.952716    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atwdg5"]
Sep 08 15:58:27.953769 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:58:27.953738    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e5a3717_1782_4fb9_89c9_4ede7b591461.slice/crio-5181a9912a94bd67e87add2809552ab2b598fe61fe1a18069072136467d74e68 WatchSource:0}: Error finding container 5181a9912a94bd67e87add2809552ab2b598fe61fe1a18069072136467d74e68: Status 404 returned error can't find the container with id 5181a9912a94bd67e87add2809552ab2b598fe61fe1a18069072136467d74e68
Sep 08 15:58:28.063645 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:28.063622    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwc6qz"]
Sep 08 15:58:28.086708 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:58:28.086677    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1afd9af4_dee3_4c13_9133_dc9c89572d51.slice/crio-c828a819e80d3c4850fa4975a6773ed05c9c79f9c71988e1df310b8fbd297239 WatchSource:0}: Error finding container c828a819e80d3c4850fa4975a6773ed05c9c79f9c71988e1df310b8fbd297239: Status 404 returned error can't find the container with id c828a819e80d3c4850fa4975a6773ed05c9c79f9c71988e1df310b8fbd297239
Sep 08 15:58:28.819802 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:28.819769    2549 generic.go:356] "Generic (PLEG): container finished" podID="ee187f09-5a2b-42f7-a8d8-d720996c0a19" containerID="dcbbdf8cf00886d43e51a729ceaab510a0916f3775071de1c9d674eec2b41dbc" exitCode=0
Sep 08 15:58:28.820298 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:28.819833    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69vhgw" event={"ID":"ee187f09-5a2b-42f7-a8d8-d720996c0a19","Type":"ContainerDied","Data":"dcbbdf8cf00886d43e51a729ceaab510a0916f3775071de1c9d674eec2b41dbc"}
Sep 08 15:58:28.820298 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:28.819910    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69vhgw" event={"ID":"ee187f09-5a2b-42f7-a8d8-d720996c0a19","Type":"ContainerStarted","Data":"29af0a58a0ba53335d9165f1075d4d438a176db18e0a4d190e5a7394ead8ce26"}
Sep 08 15:58:28.821550 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:28.821448    2549 generic.go:356] "Generic (PLEG): container finished" podID="23b78a32-811f-4864-83fb-8da7130eddb0" containerID="c719263d19b50864117a3061fb3041daae02a9469433bcbdb138fa4fc9cbb738" exitCode=0
Sep 08 15:58:28.821550 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:28.821515    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7v8687" event={"ID":"23b78a32-811f-4864-83fb-8da7130eddb0","Type":"ContainerDied","Data":"c719263d19b50864117a3061fb3041daae02a9469433bcbdb138fa4fc9cbb738"}
Sep 08 15:58:28.823168 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:28.823142    2549 generic.go:356] "Generic (PLEG): container finished" podID="7e5a3717-1782-4fb9-89c9-4ede7b591461" containerID="d22016a4cc3984f0c4f1f4957428d2a264885b368a055f8dc7884758d9448415" exitCode=0
Sep 08 15:58:28.823267 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:28.823228    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atwdg5" event={"ID":"7e5a3717-1782-4fb9-89c9-4ede7b591461","Type":"ContainerDied","Data":"d22016a4cc3984f0c4f1f4957428d2a264885b368a055f8dc7884758d9448415"}
Sep 08 15:58:28.823343 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:28.823275    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atwdg5" event={"ID":"7e5a3717-1782-4fb9-89c9-4ede7b591461","Type":"ContainerStarted","Data":"5181a9912a94bd67e87add2809552ab2b598fe61fe1a18069072136467d74e68"}
Sep 08 15:58:28.824985 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:28.824965    2549 generic.go:356] "Generic (PLEG): container finished" podID="1afd9af4-dee3-4c13-9133-dc9c89572d51" containerID="d0428573dc9c89579568fc5a706de5b33700ad4ace2c6ad48c309b00cf463811" exitCode=0
Sep 08 15:58:28.825090 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:28.825035    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwc6qz" event={"ID":"1afd9af4-dee3-4c13-9133-dc9c89572d51","Type":"ContainerDied","Data":"d0428573dc9c89579568fc5a706de5b33700ad4ace2c6ad48c309b00cf463811"}
Sep 08 15:58:28.825090 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:28.825076    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwc6qz" event={"ID":"1afd9af4-dee3-4c13-9133-dc9c89572d51","Type":"ContainerStarted","Data":"c828a819e80d3c4850fa4975a6773ed05c9c79f9c71988e1df310b8fbd297239"}
Sep 08 15:58:29.833183 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:29.833099    2549 generic.go:356] "Generic (PLEG): container finished" podID="23b78a32-811f-4864-83fb-8da7130eddb0" containerID="5f065b88e092d9db6c44bf1dd913c1e3e9c06a2d0ff950f63a124f20abc08dae" exitCode=0
Sep 08 15:58:29.833530 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:29.833182    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7v8687" event={"ID":"23b78a32-811f-4864-83fb-8da7130eddb0","Type":"ContainerDied","Data":"5f065b88e092d9db6c44bf1dd913c1e3e9c06a2d0ff950f63a124f20abc08dae"}
Sep 08 15:58:29.835291 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:29.835270    2549 generic.go:356] "Generic (PLEG): container finished" podID="1afd9af4-dee3-4c13-9133-dc9c89572d51" containerID="1755fa79fcadfd51cc2c036d0856fa773bdcf87f3625cf37a1a4b3e3d1521f8f" exitCode=0
Sep 08 15:58:29.835380 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:29.835302    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwc6qz" event={"ID":"1afd9af4-dee3-4c13-9133-dc9c89572d51","Type":"ContainerDied","Data":"1755fa79fcadfd51cc2c036d0856fa773bdcf87f3625cf37a1a4b3e3d1521f8f"}
Sep 08 15:58:30.842509 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:30.842436    2549 generic.go:356] "Generic (PLEG): container finished" podID="1afd9af4-dee3-4c13-9133-dc9c89572d51" containerID="eb2c947f8963074a7dc6836da41b9aaef932aa493fd3407f8d0dd32c73b4239b" exitCode=0
Sep 08 15:58:30.842906 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:30.842585    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwc6qz" event={"ID":"1afd9af4-dee3-4c13-9133-dc9c89572d51","Type":"ContainerDied","Data":"eb2c947f8963074a7dc6836da41b9aaef932aa493fd3407f8d0dd32c73b4239b"}
Sep 08 15:58:30.845314 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:30.845282    2549 generic.go:356] "Generic (PLEG): container finished" podID="23b78a32-811f-4864-83fb-8da7130eddb0" containerID="d537bd1b68386843c717e108a39869a8bed3bf9ef5b4d368c3a3a51f6e73a4ec" exitCode=0
Sep 08 15:58:30.845428 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:30.845318    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7v8687" event={"ID":"23b78a32-811f-4864-83fb-8da7130eddb0","Type":"ContainerDied","Data":"d537bd1b68386843c717e108a39869a8bed3bf9ef5b4d368c3a3a51f6e73a4ec"}
Sep 08 15:58:31.858702 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:31.858665    2549 generic.go:356] "Generic (PLEG): container finished" podID="7e5a3717-1782-4fb9-89c9-4ede7b591461" containerID="9b994aba32c67929aa3cf812d6d204630c66e4681d319003c045aea32f78eb64" exitCode=0
Sep 08 15:58:31.859096 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:31.858746    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atwdg5" event={"ID":"7e5a3717-1782-4fb9-89c9-4ede7b591461","Type":"ContainerDied","Data":"9b994aba32c67929aa3cf812d6d204630c66e4681d319003c045aea32f78eb64"}
Sep 08 15:58:32.026626 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:32.026597    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwc6qz"
Sep 08 15:58:32.030046 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:32.030028    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7v8687"
Sep 08 15:58:32.083538 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:32.083511    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1afd9af4-dee3-4c13-9133-dc9c89572d51-bundle\") pod \"1afd9af4-dee3-4c13-9133-dc9c89572d51\" (UID: \"1afd9af4-dee3-4c13-9133-dc9c89572d51\") "
Sep 08 15:58:32.083660 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:32.083572    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1afd9af4-dee3-4c13-9133-dc9c89572d51-util\") pod \"1afd9af4-dee3-4c13-9133-dc9c89572d51\" (UID: \"1afd9af4-dee3-4c13-9133-dc9c89572d51\") "
Sep 08 15:58:32.083660 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:32.083618    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/23b78a32-811f-4864-83fb-8da7130eddb0-bundle\") pod \"23b78a32-811f-4864-83fb-8da7130eddb0\" (UID: \"23b78a32-811f-4864-83fb-8da7130eddb0\") "
Sep 08 15:58:32.083775 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:32.083663    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/23b78a32-811f-4864-83fb-8da7130eddb0-util\") pod \"23b78a32-811f-4864-83fb-8da7130eddb0\" (UID: \"23b78a32-811f-4864-83fb-8da7130eddb0\") "
Sep 08 15:58:32.083775 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:32.083719    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zcz6z\" (UniqueName: \"kubernetes.io/projected/1afd9af4-dee3-4c13-9133-dc9c89572d51-kube-api-access-zcz6z\") pod \"1afd9af4-dee3-4c13-9133-dc9c89572d51\" (UID: \"1afd9af4-dee3-4c13-9133-dc9c89572d51\") "
Sep 08 15:58:32.083775 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:32.083751    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-75llj\" (UniqueName: \"kubernetes.io/projected/23b78a32-811f-4864-83fb-8da7130eddb0-kube-api-access-75llj\") pod \"23b78a32-811f-4864-83fb-8da7130eddb0\" (UID: \"23b78a32-811f-4864-83fb-8da7130eddb0\") "
Sep 08 15:58:32.084604 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:32.084215    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23b78a32-811f-4864-83fb-8da7130eddb0-bundle" (OuterVolumeSpecName: "bundle") pod "23b78a32-811f-4864-83fb-8da7130eddb0" (UID: "23b78a32-811f-4864-83fb-8da7130eddb0"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:58:32.085416 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:32.085231    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1afd9af4-dee3-4c13-9133-dc9c89572d51-bundle" (OuterVolumeSpecName: "bundle") pod "1afd9af4-dee3-4c13-9133-dc9c89572d51" (UID: "1afd9af4-dee3-4c13-9133-dc9c89572d51"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:58:32.086901 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:32.086831    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23b78a32-811f-4864-83fb-8da7130eddb0-kube-api-access-75llj" (OuterVolumeSpecName: "kube-api-access-75llj") pod "23b78a32-811f-4864-83fb-8da7130eddb0" (UID: "23b78a32-811f-4864-83fb-8da7130eddb0"). InnerVolumeSpecName "kube-api-access-75llj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:58:32.089902 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:32.089775    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1afd9af4-dee3-4c13-9133-dc9c89572d51-kube-api-access-zcz6z" (OuterVolumeSpecName: "kube-api-access-zcz6z") pod "1afd9af4-dee3-4c13-9133-dc9c89572d51" (UID: "1afd9af4-dee3-4c13-9133-dc9c89572d51"). InnerVolumeSpecName "kube-api-access-zcz6z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:58:32.092501 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:32.092478    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1afd9af4-dee3-4c13-9133-dc9c89572d51-util" (OuterVolumeSpecName: "util") pod "1afd9af4-dee3-4c13-9133-dc9c89572d51" (UID: "1afd9af4-dee3-4c13-9133-dc9c89572d51"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:58:32.092672 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:32.092651    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23b78a32-811f-4864-83fb-8da7130eddb0-util" (OuterVolumeSpecName: "util") pod "23b78a32-811f-4864-83fb-8da7130eddb0" (UID: "23b78a32-811f-4864-83fb-8da7130eddb0"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:58:32.185145 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:32.185123    2549 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1afd9af4-dee3-4c13-9133-dc9c89572d51-util\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:58:32.185145 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:32.185144    2549 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/23b78a32-811f-4864-83fb-8da7130eddb0-bundle\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:58:32.185277 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:32.185152    2549 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/23b78a32-811f-4864-83fb-8da7130eddb0-util\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:58:32.185277 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:32.185160    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zcz6z\" (UniqueName: \"kubernetes.io/projected/1afd9af4-dee3-4c13-9133-dc9c89572d51-kube-api-access-zcz6z\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:58:32.185277 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:32.185171    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75llj\" (UniqueName: \"kubernetes.io/projected/23b78a32-811f-4864-83fb-8da7130eddb0-kube-api-access-75llj\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:58:32.185277 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:32.185180    2549 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1afd9af4-dee3-4c13-9133-dc9c89572d51-bundle\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:58:32.817196 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:32.817169    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e0da5a-8f59c46fb-8c7sm"
Sep 08 15:58:32.871630 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:32.871604    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7v8687"
Sep 08 15:58:32.871630 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:32.871617    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7v8687" event={"ID":"23b78a32-811f-4864-83fb-8da7130eddb0","Type":"ContainerDied","Data":"168efecc24a099a98e244a021592c4493d2be5dbaa5bd7bf6cbf1832c55b100d"}
Sep 08 15:58:32.872172 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:32.871664    2549 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="168efecc24a099a98e244a021592c4493d2be5dbaa5bd7bf6cbf1832c55b100d"
Sep 08 15:58:32.874357 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:32.874330    2549 generic.go:356] "Generic (PLEG): container finished" podID="7e5a3717-1782-4fb9-89c9-4ede7b591461" containerID="89722a1c2c5f7e5a77825862da3a0b327a7dcfbc44a8b5f7d6564604ee6d8201" exitCode=0
Sep 08 15:58:32.874492 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:32.874423    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atwdg5" event={"ID":"7e5a3717-1782-4fb9-89c9-4ede7b591461","Type":"ContainerDied","Data":"89722a1c2c5f7e5a77825862da3a0b327a7dcfbc44a8b5f7d6564604ee6d8201"}
Sep 08 15:58:32.876637 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:32.876618    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwc6qz"
Sep 08 15:58:32.876718 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:32.876623    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwc6qz" event={"ID":"1afd9af4-dee3-4c13-9133-dc9c89572d51","Type":"ContainerDied","Data":"c828a819e80d3c4850fa4975a6773ed05c9c79f9c71988e1df310b8fbd297239"}
Sep 08 15:58:32.876774 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:32.876737    2549 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c828a819e80d3c4850fa4975a6773ed05c9c79f9c71988e1df310b8fbd297239"
Sep 08 15:58:34.016503 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:34.016480    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atwdg5"
Sep 08 15:58:34.099577 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:34.099549    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7e5a3717-1782-4fb9-89c9-4ede7b591461-util\") pod \"7e5a3717-1782-4fb9-89c9-4ede7b591461\" (UID: \"7e5a3717-1782-4fb9-89c9-4ede7b591461\") "
Sep 08 15:58:34.099751 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:34.099606    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7e5a3717-1782-4fb9-89c9-4ede7b591461-bundle\") pod \"7e5a3717-1782-4fb9-89c9-4ede7b591461\" (UID: \"7e5a3717-1782-4fb9-89c9-4ede7b591461\") "
Sep 08 15:58:34.099751 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:34.099627    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xbpxc\" (UniqueName: \"kubernetes.io/projected/7e5a3717-1782-4fb9-89c9-4ede7b591461-kube-api-access-xbpxc\") pod \"7e5a3717-1782-4fb9-89c9-4ede7b591461\" (UID: \"7e5a3717-1782-4fb9-89c9-4ede7b591461\") "
Sep 08 15:58:34.100392 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:34.100365    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e5a3717-1782-4fb9-89c9-4ede7b591461-bundle" (OuterVolumeSpecName: "bundle") pod "7e5a3717-1782-4fb9-89c9-4ede7b591461" (UID: "7e5a3717-1782-4fb9-89c9-4ede7b591461"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:58:34.101809 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:34.101788    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e5a3717-1782-4fb9-89c9-4ede7b591461-kube-api-access-xbpxc" (OuterVolumeSpecName: "kube-api-access-xbpxc") pod "7e5a3717-1782-4fb9-89c9-4ede7b591461" (UID: "7e5a3717-1782-4fb9-89c9-4ede7b591461"). InnerVolumeSpecName "kube-api-access-xbpxc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:58:34.104094 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:34.104056    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e5a3717-1782-4fb9-89c9-4ede7b591461-util" (OuterVolumeSpecName: "util") pod "7e5a3717-1782-4fb9-89c9-4ede7b591461" (UID: "7e5a3717-1782-4fb9-89c9-4ede7b591461"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:58:34.200364 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:34.200336    2549 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7e5a3717-1782-4fb9-89c9-4ede7b591461-util\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:58:34.200364 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:34.200362    2549 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7e5a3717-1782-4fb9-89c9-4ede7b591461-bundle\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:58:34.200502 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:34.200377    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xbpxc\" (UniqueName: \"kubernetes.io/projected/7e5a3717-1782-4fb9-89c9-4ede7b591461-kube-api-access-xbpxc\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:58:34.890120 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:34.890077    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atwdg5" event={"ID":"7e5a3717-1782-4fb9-89c9-4ede7b591461","Type":"ContainerDied","Data":"5181a9912a94bd67e87add2809552ab2b598fe61fe1a18069072136467d74e68"}
Sep 08 15:58:34.890120 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:34.890105    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atwdg5"
Sep 08 15:58:34.890344 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:34.890154    2549 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5181a9912a94bd67e87add2809552ab2b598fe61fe1a18069072136467d74e68"
Sep 08 15:58:35.240596 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:35.240556    2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-642d3b/rate-test-model-642d3b-kserve-696597fd7-8xgfp" podUID="5d1ea8c0-df86-4917-9ded-766cd3100fce" containerName="main" containerID="cri-o://4d6c56697af3d5c183b555ea2cbaab568d9952efd6f1d4cbfb7d71110a834a0e" gracePeriod=45
Sep 08 15:58:35.485626 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:35.485604    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-642d3b/rate-test-model-642d3b-kserve-696597fd7-8xgfp"
Sep 08 15:58:35.612122 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:35.612040    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d1ea8c0-df86-4917-9ded-766cd3100fce-dshm\") pod \"5d1ea8c0-df86-4917-9ded-766cd3100fce\" (UID: \"5d1ea8c0-df86-4917-9ded-766cd3100fce\") "
Sep 08 15:58:35.612122 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:35.612102    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sg6m8\" (UniqueName: \"kubernetes.io/projected/5d1ea8c0-df86-4917-9ded-766cd3100fce-kube-api-access-sg6m8\") pod \"5d1ea8c0-df86-4917-9ded-766cd3100fce\" (UID: \"5d1ea8c0-df86-4917-9ded-766cd3100fce\") "
Sep 08 15:58:35.612346 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:35.612146    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d1ea8c0-df86-4917-9ded-766cd3100fce-model-cache\") pod \"5d1ea8c0-df86-4917-9ded-766cd3100fce\" (UID: \"5d1ea8c0-df86-4917-9ded-766cd3100fce\") "
Sep 08 15:58:35.612346 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:35.612172    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d1ea8c0-df86-4917-9ded-766cd3100fce-tmp-dir\") pod \"5d1ea8c0-df86-4917-9ded-766cd3100fce\" (UID: \"5d1ea8c0-df86-4917-9ded-766cd3100fce\") "
Sep 08 15:58:35.612346 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:35.612198    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d1ea8c0-df86-4917-9ded-766cd3100fce-tls-certs\") pod \"5d1ea8c0-df86-4917-9ded-766cd3100fce\" (UID: \"5d1ea8c0-df86-4917-9ded-766cd3100fce\") "
Sep 08 15:58:35.612346 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:35.612259    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d1ea8c0-df86-4917-9ded-766cd3100fce-home\") pod \"5d1ea8c0-df86-4917-9ded-766cd3100fce\" (UID: \"5d1ea8c0-df86-4917-9ded-766cd3100fce\") "
Sep 08 15:58:35.612527 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:35.612378    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d1ea8c0-df86-4917-9ded-766cd3100fce-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5d1ea8c0-df86-4917-9ded-766cd3100fce" (UID: "5d1ea8c0-df86-4917-9ded-766cd3100fce"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:58:35.612527 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:35.612404    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d1ea8c0-df86-4917-9ded-766cd3100fce-model-cache" (OuterVolumeSpecName: "model-cache") pod "5d1ea8c0-df86-4917-9ded-766cd3100fce" (UID: "5d1ea8c0-df86-4917-9ded-766cd3100fce"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:58:35.612615 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:35.612579    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d1ea8c0-df86-4917-9ded-766cd3100fce-home" (OuterVolumeSpecName: "home") pod "5d1ea8c0-df86-4917-9ded-766cd3100fce" (UID: "5d1ea8c0-df86-4917-9ded-766cd3100fce"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:58:35.612676 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:35.612623    2549 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d1ea8c0-df86-4917-9ded-766cd3100fce-model-cache\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:58:35.612676 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:35.612642    2549 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d1ea8c0-df86-4917-9ded-766cd3100fce-tmp-dir\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:58:35.614105 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:35.614077    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d1ea8c0-df86-4917-9ded-766cd3100fce-dshm" (OuterVolumeSpecName: "dshm") pod "5d1ea8c0-df86-4917-9ded-766cd3100fce" (UID: "5d1ea8c0-df86-4917-9ded-766cd3100fce"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:58:35.614241 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:35.614217    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d1ea8c0-df86-4917-9ded-766cd3100fce-kube-api-access-sg6m8" (OuterVolumeSpecName: "kube-api-access-sg6m8") pod "5d1ea8c0-df86-4917-9ded-766cd3100fce" (UID: "5d1ea8c0-df86-4917-9ded-766cd3100fce"). InnerVolumeSpecName "kube-api-access-sg6m8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:58:35.614291 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:35.614225    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d1ea8c0-df86-4917-9ded-766cd3100fce-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5d1ea8c0-df86-4917-9ded-766cd3100fce" (UID: "5d1ea8c0-df86-4917-9ded-766cd3100fce"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 15:58:35.713611 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:35.713573    2549 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d1ea8c0-df86-4917-9ded-766cd3100fce-tls-certs\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:58:35.713611 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:35.713604    2549 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d1ea8c0-df86-4917-9ded-766cd3100fce-home\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:58:35.713611 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:35.713613    2549 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d1ea8c0-df86-4917-9ded-766cd3100fce-dshm\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:58:35.713821 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:35.713623    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sg6m8\" (UniqueName: \"kubernetes.io/projected/5d1ea8c0-df86-4917-9ded-766cd3100fce-kube-api-access-sg6m8\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:58:35.895952 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:35.895919    2549 generic.go:356] "Generic (PLEG): container finished" podID="5d1ea8c0-df86-4917-9ded-766cd3100fce" containerID="4d6c56697af3d5c183b555ea2cbaab568d9952efd6f1d4cbfb7d71110a834a0e" exitCode=0
Sep 08 15:58:35.896123 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:35.895966    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-642d3b/rate-test-model-642d3b-kserve-696597fd7-8xgfp"
Sep 08 15:58:35.896123 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:35.896016    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-642d3b/rate-test-model-642d3b-kserve-696597fd7-8xgfp" event={"ID":"5d1ea8c0-df86-4917-9ded-766cd3100fce","Type":"ContainerDied","Data":"4d6c56697af3d5c183b555ea2cbaab568d9952efd6f1d4cbfb7d71110a834a0e"}
Sep 08 15:58:35.896123 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:35.896069    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-642d3b/rate-test-model-642d3b-kserve-696597fd7-8xgfp" event={"ID":"5d1ea8c0-df86-4917-9ded-766cd3100fce","Type":"ContainerDied","Data":"a26ef2346a23f283626714d94054ae200d471dbd061f575179b22c5eafef71c3"}
Sep 08 15:58:35.896123 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:35.896090    2549 scope.go:117] "RemoveContainer" containerID="4d6c56697af3d5c183b555ea2cbaab568d9952efd6f1d4cbfb7d71110a834a0e"
Sep 08 15:58:35.905435 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:35.905414    2549 scope.go:117] "RemoveContainer" containerID="4d6c56697af3d5c183b555ea2cbaab568d9952efd6f1d4cbfb7d71110a834a0e"
Sep 08 15:58:35.905682 ip-10-0-143-61 kubenswrapper[2549]: E0908 15:58:35.905663    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d6c56697af3d5c183b555ea2cbaab568d9952efd6f1d4cbfb7d71110a834a0e\": container with ID starting with 4d6c56697af3d5c183b555ea2cbaab568d9952efd6f1d4cbfb7d71110a834a0e not found: ID does not exist" containerID="4d6c56697af3d5c183b555ea2cbaab568d9952efd6f1d4cbfb7d71110a834a0e"
Sep 08 15:58:35.905758 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:35.905691    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d6c56697af3d5c183b555ea2cbaab568d9952efd6f1d4cbfb7d71110a834a0e"} err="failed to get container status \"4d6c56697af3d5c183b555ea2cbaab568d9952efd6f1d4cbfb7d71110a834a0e\": rpc error: code = NotFound desc = could not find container \"4d6c56697af3d5c183b555ea2cbaab568d9952efd6f1d4cbfb7d71110a834a0e\": container with ID starting with 4d6c56697af3d5c183b555ea2cbaab568d9952efd6f1d4cbfb7d71110a834a0e not found: ID does not exist"
Sep 08 15:58:35.923786 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:35.923764    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-642d3b/rate-test-model-642d3b-kserve-696597fd7-8xgfp"]
Sep 08 15:58:35.926099 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:35.926078    2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-642d3b/rate-test-model-642d3b-kserve-696597fd7-8xgfp"]
Sep 08 15:58:36.400544 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:36.400510    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5d1ea8c0-df86-4917-9ded-766cd3100fce" path="/var/lib/kubelet/pods/5d1ea8c0-df86-4917-9ded-766cd3100fce/volumes"
Sep 08 15:58:42.589594 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:42.589557    2549 scope.go:117] "RemoveContainer" containerID="ffefd5d453a60c250b1bfcf9242a782cc67bd1c02d7542c424aa7a29558562d7"
Sep 08 15:58:47.957322 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:47.957285    2549 generic.go:356] "Generic (PLEG): container finished" podID="ee187f09-5a2b-42f7-a8d8-d720996c0a19" containerID="d1ff7592384548f11d05d2fce184b6bfb7d578a0991161bc43a1af83230905cc" exitCode=0
Sep 08 15:58:47.957769 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:47.957381    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69vhgw" event={"ID":"ee187f09-5a2b-42f7-a8d8-d720996c0a19","Type":"ContainerDied","Data":"d1ff7592384548f11d05d2fce184b6bfb7d578a0991161bc43a1af83230905cc"}
Sep 08 15:58:48.964642 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:48.964608    2549 generic.go:356] "Generic (PLEG): container finished" podID="ee187f09-5a2b-42f7-a8d8-d720996c0a19" containerID="f7361a7f02fc49dadfaa16d9c42bafeda842ecd93ab321f566deb90bf94046df" exitCode=0
Sep 08 15:58:48.965001 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:48.964679    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69vhgw" event={"ID":"ee187f09-5a2b-42f7-a8d8-d720996c0a19","Type":"ContainerDied","Data":"f7361a7f02fc49dadfaa16d9c42bafeda842ecd93ab321f566deb90bf94046df"}
Sep 08 15:58:49.788385 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:49.788347    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-a3a348-openshift-default-888d95576-zz4tm"]
Sep 08 15:58:49.788716 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:49.788701    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e5a3717-1782-4fb9-89c9-4ede7b591461" containerName="pull"
Sep 08 15:58:49.788716 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:49.788717    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e5a3717-1782-4fb9-89c9-4ede7b591461" containerName="pull"
Sep 08 15:58:49.788843 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:49.788727    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23b78a32-811f-4864-83fb-8da7130eddb0" containerName="pull"
Sep 08 15:58:49.788843 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:49.788734    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="23b78a32-811f-4864-83fb-8da7130eddb0" containerName="pull"
Sep 08 15:58:49.788843 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:49.788742    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23b78a32-811f-4864-83fb-8da7130eddb0" containerName="extract"
Sep 08 15:58:49.788843 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:49.788748    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="23b78a32-811f-4864-83fb-8da7130eddb0" containerName="extract"
Sep 08 15:58:49.788843 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:49.788762    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1afd9af4-dee3-4c13-9133-dc9c89572d51" containerName="util"
Sep 08 15:58:49.788843 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:49.788770    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="1afd9af4-dee3-4c13-9133-dc9c89572d51" containerName="util"
Sep 08 15:58:49.788843 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:49.788783    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e5a3717-1782-4fb9-89c9-4ede7b591461" containerName="util"
Sep 08 15:58:49.788843 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:49.788788    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e5a3717-1782-4fb9-89c9-4ede7b591461" containerName="util"
Sep 08 15:58:49.788843 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:49.788795    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1afd9af4-dee3-4c13-9133-dc9c89572d51" containerName="pull"
Sep 08 15:58:49.788843 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:49.788800    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="1afd9af4-dee3-4c13-9133-dc9c89572d51" containerName="pull"
Sep 08 15:58:49.788843 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:49.788805    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e5a3717-1782-4fb9-89c9-4ede7b591461" containerName="extract"
Sep 08 15:58:49.788843 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:49.788809    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e5a3717-1782-4fb9-89c9-4ede7b591461" containerName="extract"
Sep 08 15:58:49.788843 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:49.788816    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d1ea8c0-df86-4917-9ded-766cd3100fce" containerName="main"
Sep 08 15:58:49.788843 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:49.788820    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d1ea8c0-df86-4917-9ded-766cd3100fce" containerName="main"
Sep 08 15:58:49.788843 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:49.788826    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23b78a32-811f-4864-83fb-8da7130eddb0" containerName="util"
Sep 08 15:58:49.788843 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:49.788831    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="23b78a32-811f-4864-83fb-8da7130eddb0" containerName="util"
Sep 08 15:58:49.788843 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:49.788843    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1afd9af4-dee3-4c13-9133-dc9c89572d51" containerName="extract"
Sep 08 15:58:49.788843 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:49.788850    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="1afd9af4-dee3-4c13-9133-dc9c89572d51" containerName="extract"
Sep 08 15:58:49.789780 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:49.788928    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="1afd9af4-dee3-4c13-9133-dc9c89572d51" containerName="extract"
Sep 08 15:58:49.789780 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:49.788937    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="23b78a32-811f-4864-83fb-8da7130eddb0" containerName="extract"
Sep 08 15:58:49.789780 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:49.788944    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e5a3717-1782-4fb9-89c9-4ede7b591461" containerName="extract"
Sep 08 15:58:49.789780 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:49.788951    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d1ea8c0-df86-4917-9ded-766cd3100fce" containerName="main"
Sep 08 15:58:49.792065 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:49.792043    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-a3a348-openshift-default-888d95576-zz4tm"
Sep 08 15:58:49.794765 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:49.794741    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-a3a348-openshift-default-dockercfg-t72bg\""
Sep 08 15:58:49.805687 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:49.805663    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-a3a348-openshift-default-888d95576-zz4tm"]
Sep 08 15:58:49.931050 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:49.931021    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f989139-19c6-4ee7-bc23-a118c031e801-istio-envoy\") pod \"e2e-ipp-b-a3a348-openshift-default-888d95576-zz4tm\" (UID: \"8f989139-19c6-4ee7-bc23-a118c031e801\") " pod="openshift-ingress/e2e-ipp-b-a3a348-openshift-default-888d95576-zz4tm"
Sep 08 15:58:49.931220 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:49.931065    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f989139-19c6-4ee7-bc23-a118c031e801-istiod-ca-cert\") pod \"e2e-ipp-b-a3a348-openshift-default-888d95576-zz4tm\" (UID: \"8f989139-19c6-4ee7-bc23-a118c031e801\") " pod="openshift-ingress/e2e-ipp-b-a3a348-openshift-default-888d95576-zz4tm"
Sep 08 15:58:49.931220 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:49.931098    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdvkq\" (UniqueName: \"kubernetes.io/projected/8f989139-19c6-4ee7-bc23-a118c031e801-kube-api-access-bdvkq\") pod \"e2e-ipp-b-a3a348-openshift-default-888d95576-zz4tm\" (UID: \"8f989139-19c6-4ee7-bc23-a118c031e801\") " pod="openshift-ingress/e2e-ipp-b-a3a348-openshift-default-888d95576-zz4tm"
Sep 08 15:58:49.931220 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:49.931114    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f989139-19c6-4ee7-bc23-a118c031e801-istio-data\") pod \"e2e-ipp-b-a3a348-openshift-default-888d95576-zz4tm\" (UID: \"8f989139-19c6-4ee7-bc23-a118c031e801\") " pod="openshift-ingress/e2e-ipp-b-a3a348-openshift-default-888d95576-zz4tm"
Sep 08 15:58:49.931220 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:49.931137    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f989139-19c6-4ee7-bc23-a118c031e801-workload-certs\") pod \"e2e-ipp-b-a3a348-openshift-default-888d95576-zz4tm\" (UID: \"8f989139-19c6-4ee7-bc23-a118c031e801\") " pod="openshift-ingress/e2e-ipp-b-a3a348-openshift-default-888d95576-zz4tm"
Sep 08 15:58:49.931385 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:49.931241    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f989139-19c6-4ee7-bc23-a118c031e801-istio-token\") pod \"e2e-ipp-b-a3a348-openshift-default-888d95576-zz4tm\" (UID: \"8f989139-19c6-4ee7-bc23-a118c031e801\") " pod="openshift-ingress/e2e-ipp-b-a3a348-openshift-default-888d95576-zz4tm"
Sep 08 15:58:49.931385 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:49.931281    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f989139-19c6-4ee7-bc23-a118c031e801-workload-socket\") pod \"e2e-ipp-b-a3a348-openshift-default-888d95576-zz4tm\" (UID: \"8f989139-19c6-4ee7-bc23-a118c031e801\") " pod="openshift-ingress/e2e-ipp-b-a3a348-openshift-default-888d95576-zz4tm"
Sep 08 15:58:49.931385 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:49.931301    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f989139-19c6-4ee7-bc23-a118c031e801-istio-podinfo\") pod \"e2e-ipp-b-a3a348-openshift-default-888d95576-zz4tm\" (UID: \"8f989139-19c6-4ee7-bc23-a118c031e801\") " pod="openshift-ingress/e2e-ipp-b-a3a348-openshift-default-888d95576-zz4tm"
Sep 08 15:58:49.931385 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:49.931319    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f989139-19c6-4ee7-bc23-a118c031e801-credential-socket\") pod \"e2e-ipp-b-a3a348-openshift-default-888d95576-zz4tm\" (UID: \"8f989139-19c6-4ee7-bc23-a118c031e801\") " pod="openshift-ingress/e2e-ipp-b-a3a348-openshift-default-888d95576-zz4tm"
Sep 08 15:58:50.032487 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:50.032455    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f989139-19c6-4ee7-bc23-a118c031e801-istio-token\") pod \"e2e-ipp-b-a3a348-openshift-default-888d95576-zz4tm\" (UID: \"8f989139-19c6-4ee7-bc23-a118c031e801\") " pod="openshift-ingress/e2e-ipp-b-a3a348-openshift-default-888d95576-zz4tm"
Sep 08 15:58:50.032915 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:50.032504    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f989139-19c6-4ee7-bc23-a118c031e801-workload-socket\") pod \"e2e-ipp-b-a3a348-openshift-default-888d95576-zz4tm\" (UID: \"8f989139-19c6-4ee7-bc23-a118c031e801\") " pod="openshift-ingress/e2e-ipp-b-a3a348-openshift-default-888d95576-zz4tm"
Sep 08 15:58:50.032915 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:50.032614    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f989139-19c6-4ee7-bc23-a118c031e801-istio-podinfo\") pod \"e2e-ipp-b-a3a348-openshift-default-888d95576-zz4tm\" (UID: \"8f989139-19c6-4ee7-bc23-a118c031e801\") " pod="openshift-ingress/e2e-ipp-b-a3a348-openshift-default-888d95576-zz4tm"
Sep 08 15:58:50.032915 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:50.032666    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f989139-19c6-4ee7-bc23-a118c031e801-credential-socket\") pod \"e2e-ipp-b-a3a348-openshift-default-888d95576-zz4tm\" (UID: \"8f989139-19c6-4ee7-bc23-a118c031e801\") " pod="openshift-ingress/e2e-ipp-b-a3a348-openshift-default-888d95576-zz4tm"
Sep 08 15:58:50.032915 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:50.032711    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f989139-19c6-4ee7-bc23-a118c031e801-istio-envoy\") pod \"e2e-ipp-b-a3a348-openshift-default-888d95576-zz4tm\" (UID: \"8f989139-19c6-4ee7-bc23-a118c031e801\") " pod="openshift-ingress/e2e-ipp-b-a3a348-openshift-default-888d95576-zz4tm"
Sep 08 15:58:50.032915 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:50.032770    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f989139-19c6-4ee7-bc23-a118c031e801-istiod-ca-cert\") pod \"e2e-ipp-b-a3a348-openshift-default-888d95576-zz4tm\" (UID: \"8f989139-19c6-4ee7-bc23-a118c031e801\") " pod="openshift-ingress/e2e-ipp-b-a3a348-openshift-default-888d95576-zz4tm"
Sep 08 15:58:50.032915 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:50.032810    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdvkq\" (UniqueName: \"kubernetes.io/projected/8f989139-19c6-4ee7-bc23-a118c031e801-kube-api-access-bdvkq\") pod \"e2e-ipp-b-a3a348-openshift-default-888d95576-zz4tm\" (UID: \"8f989139-19c6-4ee7-bc23-a118c031e801\") " pod="openshift-ingress/e2e-ipp-b-a3a348-openshift-default-888d95576-zz4tm"
Sep 08 15:58:50.032915 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:50.032835    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f989139-19c6-4ee7-bc23-a118c031e801-istio-data\") pod \"e2e-ipp-b-a3a348-openshift-default-888d95576-zz4tm\" (UID: \"8f989139-19c6-4ee7-bc23-a118c031e801\") " pod="openshift-ingress/e2e-ipp-b-a3a348-openshift-default-888d95576-zz4tm"
Sep 08 15:58:50.032915 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:50.032907    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f989139-19c6-4ee7-bc23-a118c031e801-workload-certs\") pod \"e2e-ipp-b-a3a348-openshift-default-888d95576-zz4tm\" (UID: \"8f989139-19c6-4ee7-bc23-a118c031e801\") " pod="openshift-ingress/e2e-ipp-b-a3a348-openshift-default-888d95576-zz4tm"
Sep 08 15:58:50.033599 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:50.032917    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f989139-19c6-4ee7-bc23-a118c031e801-workload-socket\") pod \"e2e-ipp-b-a3a348-openshift-default-888d95576-zz4tm\" (UID: \"8f989139-19c6-4ee7-bc23-a118c031e801\") " pod="openshift-ingress/e2e-ipp-b-a3a348-openshift-default-888d95576-zz4tm"
Sep 08 15:58:50.033599 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:50.033042    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f989139-19c6-4ee7-bc23-a118c031e801-credential-socket\") pod \"e2e-ipp-b-a3a348-openshift-default-888d95576-zz4tm\" (UID: \"8f989139-19c6-4ee7-bc23-a118c031e801\") " pod="openshift-ingress/e2e-ipp-b-a3a348-openshift-default-888d95576-zz4tm"
Sep 08 15:58:50.033599 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:50.033297    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f989139-19c6-4ee7-bc23-a118c031e801-istio-data\") pod \"e2e-ipp-b-a3a348-openshift-default-888d95576-zz4tm\" (UID: \"8f989139-19c6-4ee7-bc23-a118c031e801\") " pod="openshift-ingress/e2e-ipp-b-a3a348-openshift-default-888d95576-zz4tm"
Sep 08 15:58:50.033599 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:50.033480    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f989139-19c6-4ee7-bc23-a118c031e801-workload-certs\") pod \"e2e-ipp-b-a3a348-openshift-default-888d95576-zz4tm\" (UID: \"8f989139-19c6-4ee7-bc23-a118c031e801\") " pod="openshift-ingress/e2e-ipp-b-a3a348-openshift-default-888d95576-zz4tm"
Sep 08 15:58:50.034058 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:50.034010    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f989139-19c6-4ee7-bc23-a118c031e801-istiod-ca-cert\") pod \"e2e-ipp-b-a3a348-openshift-default-888d95576-zz4tm\" (UID: \"8f989139-19c6-4ee7-bc23-a118c031e801\") " pod="openshift-ingress/e2e-ipp-b-a3a348-openshift-default-888d95576-zz4tm"
Sep 08 15:58:50.036978 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:50.036954    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f989139-19c6-4ee7-bc23-a118c031e801-istio-podinfo\") pod \"e2e-ipp-b-a3a348-openshift-default-888d95576-zz4tm\" (UID: \"8f989139-19c6-4ee7-bc23-a118c031e801\") " pod="openshift-ingress/e2e-ipp-b-a3a348-openshift-default-888d95576-zz4tm"
Sep 08 15:58:50.037071 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:50.037028    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f989139-19c6-4ee7-bc23-a118c031e801-istio-envoy\") pod \"e2e-ipp-b-a3a348-openshift-default-888d95576-zz4tm\" (UID: \"8f989139-19c6-4ee7-bc23-a118c031e801\") " pod="openshift-ingress/e2e-ipp-b-a3a348-openshift-default-888d95576-zz4tm"
Sep 08 15:58:50.041653 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:50.041578    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdvkq\" (UniqueName: \"kubernetes.io/projected/8f989139-19c6-4ee7-bc23-a118c031e801-kube-api-access-bdvkq\") pod \"e2e-ipp-b-a3a348-openshift-default-888d95576-zz4tm\" (UID: \"8f989139-19c6-4ee7-bc23-a118c031e801\") " pod="openshift-ingress/e2e-ipp-b-a3a348-openshift-default-888d95576-zz4tm"
Sep 08 15:58:50.042869 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:50.042836    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f989139-19c6-4ee7-bc23-a118c031e801-istio-token\") pod \"e2e-ipp-b-a3a348-openshift-default-888d95576-zz4tm\" (UID: \"8f989139-19c6-4ee7-bc23-a118c031e801\") " pod="openshift-ingress/e2e-ipp-b-a3a348-openshift-default-888d95576-zz4tm"
Sep 08 15:58:50.104903 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:50.104880    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69vhgw"
Sep 08 15:58:50.105010 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:50.104893    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-a3a348-openshift-default-888d95576-zz4tm"
Sep 08 15:58:50.234674 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:50.234648    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ee187f09-5a2b-42f7-a8d8-d720996c0a19-util\") pod \"ee187f09-5a2b-42f7-a8d8-d720996c0a19\" (UID: \"ee187f09-5a2b-42f7-a8d8-d720996c0a19\") "
Sep 08 15:58:50.234841 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:50.234684    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ee187f09-5a2b-42f7-a8d8-d720996c0a19-bundle\") pod \"ee187f09-5a2b-42f7-a8d8-d720996c0a19\" (UID: \"ee187f09-5a2b-42f7-a8d8-d720996c0a19\") "
Sep 08 15:58:50.234841 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:50.234774    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rpsxs\" (UniqueName: \"kubernetes.io/projected/ee187f09-5a2b-42f7-a8d8-d720996c0a19-kube-api-access-rpsxs\") pod \"ee187f09-5a2b-42f7-a8d8-d720996c0a19\" (UID: \"ee187f09-5a2b-42f7-a8d8-d720996c0a19\") "
Sep 08 15:58:50.235237 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:50.235198    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee187f09-5a2b-42f7-a8d8-d720996c0a19-bundle" (OuterVolumeSpecName: "bundle") pod "ee187f09-5a2b-42f7-a8d8-d720996c0a19" (UID: "ee187f09-5a2b-42f7-a8d8-d720996c0a19"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:58:50.236743 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:50.236721    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee187f09-5a2b-42f7-a8d8-d720996c0a19-kube-api-access-rpsxs" (OuterVolumeSpecName: "kube-api-access-rpsxs") pod "ee187f09-5a2b-42f7-a8d8-d720996c0a19" (UID: "ee187f09-5a2b-42f7-a8d8-d720996c0a19"). InnerVolumeSpecName "kube-api-access-rpsxs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:58:50.239590 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:50.239570    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee187f09-5a2b-42f7-a8d8-d720996c0a19-util" (OuterVolumeSpecName: "util") pod "ee187f09-5a2b-42f7-a8d8-d720996c0a19" (UID: "ee187f09-5a2b-42f7-a8d8-d720996c0a19"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:58:50.335509 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:50.335445    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rpsxs\" (UniqueName: \"kubernetes.io/projected/ee187f09-5a2b-42f7-a8d8-d720996c0a19-kube-api-access-rpsxs\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:58:50.335509 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:50.335485    2549 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ee187f09-5a2b-42f7-a8d8-d720996c0a19-util\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:58:50.335509 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:50.335496    2549 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ee187f09-5a2b-42f7-a8d8-d720996c0a19-bundle\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 15:58:50.380992 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:50.380969    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-a3a348-openshift-default-888d95576-zz4tm"]
Sep 08 15:58:50.382680 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:58:50.382651    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f989139_19c6_4ee7_bc23_a118c031e801.slice/crio-041a686e69cc4ff6aa6b9258e7f77df78dcf4ca503bf958e39b389a2efbfa458 WatchSource:0}: Error finding container 041a686e69cc4ff6aa6b9258e7f77df78dcf4ca503bf958e39b389a2efbfa458: Status 404 returned error can't find the container with id 041a686e69cc4ff6aa6b9258e7f77df78dcf4ca503bf958e39b389a2efbfa458
Sep 08 15:58:50.384830 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:50.384801    2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 08 15:58:50.384918 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:50.384876    2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 08 15:58:50.384918 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:50.384907    2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 08 15:58:50.977435 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:50.977391    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a3a348-openshift-default-888d95576-zz4tm" event={"ID":"8f989139-19c6-4ee7-bc23-a118c031e801","Type":"ContainerStarted","Data":"5ae7b77b9339058d3be203ed3aca7f80706b79d3b1bf50b71e7453cba5d00a10"}
Sep 08 15:58:50.977614 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:50.977455    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a3a348-openshift-default-888d95576-zz4tm" event={"ID":"8f989139-19c6-4ee7-bc23-a118c031e801","Type":"ContainerStarted","Data":"041a686e69cc4ff6aa6b9258e7f77df78dcf4ca503bf958e39b389a2efbfa458"}
Sep 08 15:58:50.979335 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:50.979310    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69vhgw" event={"ID":"ee187f09-5a2b-42f7-a8d8-d720996c0a19","Type":"ContainerDied","Data":"29af0a58a0ba53335d9165f1075d4d438a176db18e0a4d190e5a7394ead8ce26"}
Sep 08 15:58:50.979443 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:50.979346    2549 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="29af0a58a0ba53335d9165f1075d4d438a176db18e0a4d190e5a7394ead8ce26"
Sep 08 15:58:50.979443 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:50.979353    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69vhgw"
Sep 08 15:58:51.000971 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:51.000919    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-a3a348-openshift-default-888d95576-zz4tm" podStartSLOduration=2.000900729 podStartE2EDuration="2.000900729s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:58:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:58:51.000509411 +0000 UTC m=+1569.089960551" watchObservedRunningTime="2026-09-08 15:58:51.000900729 +0000 UTC m=+1569.090351831"
Sep 08 15:58:51.105775 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:51.105750    2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-a3a348-openshift-default-888d95576-zz4tm"
Sep 08 15:58:51.110739 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:51.110721    2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-a3a348-openshift-default-888d95576-zz4tm"
Sep 08 15:58:51.985442 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:51.985404    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-a3a348-openshift-default-888d95576-zz4tm"
Sep 08 15:58:51.986450 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:51.986428    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-a3a348-openshift-default-888d95576-zz4tm"
Sep 08 15:58:54.165905 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:54.165849    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a3a348-6fb49499f7-m4q4c"]
Sep 08 15:58:54.166292 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:54.166268    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee187f09-5a2b-42f7-a8d8-d720996c0a19" containerName="util"
Sep 08 15:58:54.166292 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:54.166280    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee187f09-5a2b-42f7-a8d8-d720996c0a19" containerName="util"
Sep 08 15:58:54.166292 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:54.166289    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee187f09-5a2b-42f7-a8d8-d720996c0a19" containerName="extract"
Sep 08 15:58:54.166394 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:54.166298    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee187f09-5a2b-42f7-a8d8-d720996c0a19" containerName="extract"
Sep 08 15:58:54.166394 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:54.166311    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee187f09-5a2b-42f7-a8d8-d720996c0a19" containerName="pull"
Sep 08 15:58:54.166394 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:54.166317    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee187f09-5a2b-42f7-a8d8-d720996c0a19" containerName="pull"
Sep 08 15:58:54.166482 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:54.166399    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee187f09-5a2b-42f7-a8d8-d720996c0a19" containerName="extract"
Sep 08 15:58:54.169632 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:54.169607    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a3a348-6fb49499f7-m4q4c"
Sep 08 15:58:54.173285 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:54.173261    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-a3a348-serving-cert\""
Sep 08 15:58:54.182050 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:54.182030    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a3a348-6fb49499f7-m4q4c"]
Sep 08 15:58:54.270945 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:54.270915    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bl5kz\" (UniqueName: \"kubernetes.io/projected/dc333166-7aa5-4c4b-9906-9994c344e4d4-kube-api-access-bl5kz\") pod \"maas-api-e2e-ipp-b-a3a348-6fb49499f7-m4q4c\" (UID: \"dc333166-7aa5-4c4b-9906-9994c344e4d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a3a348-6fb49499f7-m4q4c"
Sep 08 15:58:54.271091 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:54.270966    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc333166-7aa5-4c4b-9906-9994c344e4d4-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-a3a348-6fb49499f7-m4q4c\" (UID: \"dc333166-7aa5-4c4b-9906-9994c344e4d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a3a348-6fb49499f7-m4q4c"
Sep 08 15:58:54.271091 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:54.271064    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc333166-7aa5-4c4b-9906-9994c344e4d4-maas-api-tls\") pod \"maas-api-e2e-ipp-b-a3a348-6fb49499f7-m4q4c\" (UID: \"dc333166-7aa5-4c4b-9906-9994c344e4d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a3a348-6fb49499f7-m4q4c"
Sep 08 15:58:54.271174 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:54.271110    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc333166-7aa5-4c4b-9906-9994c344e4d4-metrics-certs\") pod \"maas-api-e2e-ipp-b-a3a348-6fb49499f7-m4q4c\" (UID: \"dc333166-7aa5-4c4b-9906-9994c344e4d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a3a348-6fb49499f7-m4q4c"
Sep 08 15:58:54.371689 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:54.371647    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bl5kz\" (UniqueName: \"kubernetes.io/projected/dc333166-7aa5-4c4b-9906-9994c344e4d4-kube-api-access-bl5kz\") pod \"maas-api-e2e-ipp-b-a3a348-6fb49499f7-m4q4c\" (UID: \"dc333166-7aa5-4c4b-9906-9994c344e4d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a3a348-6fb49499f7-m4q4c"
Sep 08 15:58:54.371896 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:54.371720    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc333166-7aa5-4c4b-9906-9994c344e4d4-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-a3a348-6fb49499f7-m4q4c\" (UID: \"dc333166-7aa5-4c4b-9906-9994c344e4d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a3a348-6fb49499f7-m4q4c"
Sep 08 15:58:54.371896 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:54.371784    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc333166-7aa5-4c4b-9906-9994c344e4d4-maas-api-tls\") pod \"maas-api-e2e-ipp-b-a3a348-6fb49499f7-m4q4c\" (UID: \"dc333166-7aa5-4c4b-9906-9994c344e4d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a3a348-6fb49499f7-m4q4c"
Sep 08 15:58:54.371896 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:54.371827    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc333166-7aa5-4c4b-9906-9994c344e4d4-metrics-certs\") pod \"maas-api-e2e-ipp-b-a3a348-6fb49499f7-m4q4c\" (UID: \"dc333166-7aa5-4c4b-9906-9994c344e4d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a3a348-6fb49499f7-m4q4c"
Sep 08 15:58:54.373388 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:54.373354    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc333166-7aa5-4c4b-9906-9994c344e4d4-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-a3a348-6fb49499f7-m4q4c\" (UID: \"dc333166-7aa5-4c4b-9906-9994c344e4d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a3a348-6fb49499f7-m4q4c"
Sep 08 15:58:54.375406 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:54.375382    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc333166-7aa5-4c4b-9906-9994c344e4d4-maas-api-tls\") pod \"maas-api-e2e-ipp-b-a3a348-6fb49499f7-m4q4c\" (UID: \"dc333166-7aa5-4c4b-9906-9994c344e4d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a3a348-6fb49499f7-m4q4c"
Sep 08 15:58:54.375530 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:54.375405    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc333166-7aa5-4c4b-9906-9994c344e4d4-metrics-certs\") pod \"maas-api-e2e-ipp-b-a3a348-6fb49499f7-m4q4c\" (UID: \"dc333166-7aa5-4c4b-9906-9994c344e4d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a3a348-6fb49499f7-m4q4c"
Sep 08 15:58:54.382143 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:54.382117    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bl5kz\" (UniqueName: \"kubernetes.io/projected/dc333166-7aa5-4c4b-9906-9994c344e4d4-kube-api-access-bl5kz\") pod \"maas-api-e2e-ipp-b-a3a348-6fb49499f7-m4q4c\" (UID: \"dc333166-7aa5-4c4b-9906-9994c344e4d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a3a348-6fb49499f7-m4q4c"
Sep 08 15:58:54.481783 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:54.481693    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a3a348-6fb49499f7-m4q4c"
Sep 08 15:58:54.761589 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:54.761560    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a3a348-6fb49499f7-m4q4c"]
Sep 08 15:58:54.761954 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:58:54.761928    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc333166_7aa5_4c4b_9906_9994c344e4d4.slice/crio-788003c7f852b5563c8c4e4f65a7265425dd43c77806b871e44c43cc679940bd WatchSource:0}: Error finding container 788003c7f852b5563c8c4e4f65a7265425dd43c77806b871e44c43cc679940bd: Status 404 returned error can't find the container with id 788003c7f852b5563c8c4e4f65a7265425dd43c77806b871e44c43cc679940bd
Sep 08 15:58:55.003583 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:55.003548    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a3a348-6fb49499f7-m4q4c" event={"ID":"dc333166-7aa5-4c4b-9906-9994c344e4d4","Type":"ContainerStarted","Data":"788003c7f852b5563c8c4e4f65a7265425dd43c77806b871e44c43cc679940bd"}
Sep 08 15:58:56.010620 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:56.010582    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a3a348-6fb49499f7-m4q4c" event={"ID":"dc333166-7aa5-4c4b-9906-9994c344e4d4","Type":"ContainerStarted","Data":"fc3c76a46b31f1ae4a5ad602e20ac1987c9134033d554a1ec2c4c3acd24b935b"}
Sep 08 15:58:56.011119 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:56.010679    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a3a348-6fb49499f7-m4q4c"
Sep 08 15:58:56.050126 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:56.050075    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a3a348-6fb49499f7-m4q4c" podStartSLOduration=1.746846688 podStartE2EDuration="2.050060711s" totalImagesPullingTime="303.214023ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:58:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:58:56.047379592 +0000 UTC m=+1574.136830680" watchObservedRunningTime="2026-09-08 15:58:56.050060711 +0000 UTC m=+1574.139511800"
Sep 08 15:58:59.045158 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:59.045118    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-p7lvk"]
Sep 08 15:58:59.049091 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:59.049067    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-p7lvk"
Sep 08 15:58:59.052010 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:59.051988    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 08 15:58:59.052112 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:59.052041    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-sjv2h\""
Sep 08 15:58:59.052176 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:59.052115    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 08 15:58:59.058618 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:59.058595    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-p7lvk"]
Sep 08 15:58:59.215215 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:59.215181    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8t4mz\" (UniqueName: \"kubernetes.io/projected/eb4fccc1-c0ed-423b-86dc-1226e8f7a666-kube-api-access-8t4mz\") pod \"authorino-operator-76b478bd78-p7lvk\" (UID: \"eb4fccc1-c0ed-423b-86dc-1226e8f7a666\") " pod="kuadrant-system/authorino-operator-76b478bd78-p7lvk"
Sep 08 15:58:59.316572 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:59.316501    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8t4mz\" (UniqueName: \"kubernetes.io/projected/eb4fccc1-c0ed-423b-86dc-1226e8f7a666-kube-api-access-8t4mz\") pod \"authorino-operator-76b478bd78-p7lvk\" (UID: \"eb4fccc1-c0ed-423b-86dc-1226e8f7a666\") " pod="kuadrant-system/authorino-operator-76b478bd78-p7lvk"
Sep 08 15:58:59.325761 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:59.325731    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8t4mz\" (UniqueName: \"kubernetes.io/projected/eb4fccc1-c0ed-423b-86dc-1226e8f7a666-kube-api-access-8t4mz\") pod \"authorino-operator-76b478bd78-p7lvk\" (UID: \"eb4fccc1-c0ed-423b-86dc-1226e8f7a666\") " pod="kuadrant-system/authorino-operator-76b478bd78-p7lvk"
Sep 08 15:58:59.359677 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:59.359653    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-p7lvk"
Sep 08 15:58:59.642375 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:58:59.642298    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-p7lvk"]
Sep 08 15:58:59.645014 ip-10-0-143-61 kubenswrapper[2549]: W0908 15:58:59.644990    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb4fccc1_c0ed_423b_86dc_1226e8f7a666.slice/crio-a131c5c2833d603c01263bec548449b51a20c875b3a490cba05ede88811fcdb7 WatchSource:0}: Error finding container a131c5c2833d603c01263bec548449b51a20c875b3a490cba05ede88811fcdb7: Status 404 returned error can't find the container with id a131c5c2833d603c01263bec548449b51a20c875b3a490cba05ede88811fcdb7
Sep 08 15:59:00.033219 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:59:00.033187    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-p7lvk" event={"ID":"eb4fccc1-c0ed-423b-86dc-1226e8f7a666","Type":"ContainerStarted","Data":"a131c5c2833d603c01263bec548449b51a20c875b3a490cba05ede88811fcdb7"}
Sep 08 15:59:02.020526 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:59:02.020498    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a3a348-6fb49499f7-m4q4c"
Sep 08 15:59:02.045075 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:59:02.045037    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-p7lvk" event={"ID":"eb4fccc1-c0ed-423b-86dc-1226e8f7a666","Type":"ContainerStarted","Data":"62133793c86378fb8117571e6cb20f31362101e1163ed3577ba94522e91f5481"}
Sep 08 15:59:02.045220 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:59:02.045103    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-76b478bd78-p7lvk"
Sep 08 15:59:02.098289 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:59:02.098234    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-76b478bd78-p7lvk" podStartSLOduration=1.666262829 podStartE2EDuration="3.098219505s" totalImagesPullingTime="1.431956676s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:58:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:59:02.095977054 +0000 UTC m=+1580.185428170" watchObservedRunningTime="2026-09-08 15:59:02.098219505 +0000 UTC m=+1580.187670593"
Sep 08 15:59:11.107015 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:59:11.106931    2549 generic.go:356] "Generic (PLEG): container finished" podID="8f989139-19c6-4ee7-bc23-a118c031e801" containerID="5ae7b77b9339058d3be203ed3aca7f80706b79d3b1bf50b71e7453cba5d00a10" exitCode=0
Sep 08 15:59:11.107015 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:59:11.106992    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a3a348-openshift-default-888d95576-zz4tm" event={"ID":"8f989139-19c6-4ee7-bc23-a118c031e801","Type":"ContainerDied","Data":"5ae7b77b9339058d3be203ed3aca7f80706b79d3b1bf50b71e7453cba5d00a10"}
Sep 08 15:59:11.107523 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:59:11.107341    2549 scope.go:117] "RemoveContainer" containerID="5ae7b77b9339058d3be203ed3aca7f80706b79d3b1bf50b71e7453cba5d00a10"
Sep 08 15:59:11.108428 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:59:11.108389    2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 08 15:59:11.108519 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:59:11.108461    2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 08 15:59:11.108519 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:59:11.108490    2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 08 15:59:12.113740 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:59:12.113706    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a3a348-openshift-default-888d95576-zz4tm" event={"ID":"8f989139-19c6-4ee7-bc23-a118c031e801","Type":"ContainerStarted","Data":"d00d7d9024c7f8422e9223064d54486458ebdd8433d54556be7e2d92f87a772a"}
Sep 08 15:59:13.056505 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:59:13.056464    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-76b478bd78-p7lvk"
Sep 08 15:59:13.105424 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:59:13.105385    2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-a3a348-openshift-default-888d95576-zz4tm"
Sep 08 15:59:13.105600 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:59:13.105558    2549 patch_prober.go:28] interesting pod/e2e-ipp-b-a3a348-openshift-default-888d95576-zz4tm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Sep 08 15:59:13.105675 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:59:13.105613    2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a3a348-openshift-default-888d95576-zz4tm" podUID="8f989139-19c6-4ee7-bc23-a118c031e801" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Sep 08 15:59:14.105325 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:59:14.105286    2549 patch_prober.go:28] interesting pod/e2e-ipp-b-a3a348-openshift-default-888d95576-zz4tm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Sep 08 15:59:14.105704 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:59:14.105341    2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a3a348-openshift-default-888d95576-zz4tm" podUID="8f989139-19c6-4ee7-bc23-a118c031e801" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Sep 08 15:59:15.106225 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:59:15.106187    2549 patch_prober.go:28] interesting pod/e2e-ipp-b-a3a348-openshift-default-888d95576-zz4tm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Sep 08 15:59:15.106680 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:59:15.106249    2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a3a348-openshift-default-888d95576-zz4tm" podUID="8f989139-19c6-4ee7-bc23-a118c031e801" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Sep 08 15:59:16.106020 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:59:16.105971    2549 patch_prober.go:28] interesting pod/e2e-ipp-b-a3a348-openshift-default-888d95576-zz4tm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Sep 08 15:59:16.106232 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:59:16.106066    2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a3a348-openshift-default-888d95576-zz4tm" podUID="8f989139-19c6-4ee7-bc23-a118c031e801" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Sep 08 15:59:17.379587 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:59:17.379553    2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-a3a348-openshift-default-888d95576-zz4tm"
Sep 08 15:59:17.380563 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:59:17.380531    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-a3a348-openshift-default-888d95576-zz4tm"
Sep 08 15:59:17.381710 ip-10-0-143-61 kubenswrapper[2549]: I0908 15:59:17.381686    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-a3a348-openshift-default-888d95576-zz4tm"
Sep 08 16:00:00.139615 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:00:00.139535    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-642d3b-29814720-4m9qt"]
Sep 08 16:00:00.143042 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:00:00.143025    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-642d3b-29814720-4m9qt"
Sep 08 16:00:00.145595 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:00:00.145579    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vcp9j\""
Sep 08 16:00:00.154241 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:00:00.154216    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-642d3b-29814720-4m9qt"
Sep 08 16:00:00.156444 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:00:00.156419    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-642d3b-29814720-4m9qt"]
Sep 08 16:00:00.432929 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:00:00.432902    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-642d3b-29814720-4m9qt"]
Sep 08 16:00:00.435125 ip-10-0-143-61 kubenswrapper[2549]: W0908 16:00:00.435094    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4db3b95c_b863_4a09_a9a1_e0dd0dc282c0.slice/crio-82e9090d7f04107801ea9180d4d9932536a7f9310347c251e0730e5ce0c363a6 WatchSource:0}: Error finding container 82e9090d7f04107801ea9180d4d9932536a7f9310347c251e0730e5ce0c363a6: Status 404 returned error can't find the container with id 82e9090d7f04107801ea9180d4d9932536a7f9310347c251e0730e5ce0c363a6
Sep 08 16:00:01.385200 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:00:01.385167    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-642d3b-29814720-4m9qt" event={"ID":"4db3b95c-b863-4a09-a9a1-e0dd0dc282c0","Type":"ContainerStarted","Data":"82e9090d7f04107801ea9180d4d9932536a7f9310347c251e0730e5ce0c363a6"}
Sep 08 16:00:03.397642 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:00:03.397611    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-642d3b-29814720-4m9qt" event={"ID":"4db3b95c-b863-4a09-a9a1-e0dd0dc282c0","Type":"ContainerStarted","Data":"965563f3ea75f67dc257cca888ce2827c3d20c91bb1a2d4d4ebb4e7478e48b2c"}
Sep 08 16:00:03.413279 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:00:03.413233    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-642d3b-29814720-4m9qt" podStartSLOduration=0.893046421 podStartE2EDuration="3.413218959s" totalImagesPullingTime="2.520172538s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:00:03.411832893 +0000 UTC m=+1641.501283979" watchObservedRunningTime="2026-09-08 16:00:03.413218959 +0000 UTC m=+1641.502670100"
Sep 08 16:00:04.403880 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:00:04.403836    2549 generic.go:356] "Generic (PLEG): container finished" podID="4db3b95c-b863-4a09-a9a1-e0dd0dc282c0" containerID="965563f3ea75f67dc257cca888ce2827c3d20c91bb1a2d4d4ebb4e7478e48b2c" exitCode=0
Sep 08 16:00:04.404250 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:00:04.403891    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-642d3b-29814720-4m9qt" event={"ID":"4db3b95c-b863-4a09-a9a1-e0dd0dc282c0","Type":"ContainerDied","Data":"965563f3ea75f67dc257cca888ce2827c3d20c91bb1a2d4d4ebb4e7478e48b2c"}
Sep 08 16:00:05.537642 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:00:05.537614    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-642d3b-29814720-4m9qt"
Sep 08 16:00:06.415968 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:00:06.415943    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-642d3b-29814720-4m9qt"
Sep 08 16:00:06.416099 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:00:06.415945    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-642d3b-29814720-4m9qt" event={"ID":"4db3b95c-b863-4a09-a9a1-e0dd0dc282c0","Type":"ContainerDied","Data":"82e9090d7f04107801ea9180d4d9932536a7f9310347c251e0730e5ce0c363a6"}
Sep 08 16:00:06.416099 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:00:06.416064    2549 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="82e9090d7f04107801ea9180d4d9932536a7f9310347c251e0730e5ce0c363a6"
Sep 08 16:00:58.019751 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:00:58.019717    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-e0da5a-8f59c46fb-8c7sm"]
Sep 08 16:00:58.020308 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:00:58.019958    2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e0da5a-8f59c46fb-8c7sm" podUID="9dbeaa09-4695-4eb4-80b6-08da3893d90f" containerName="maas-api" containerID="cri-o://ba525a946b181ed231a21a5bd74bf3d4a4969f0ce73e664993d998a5f3273327" gracePeriod=30
Sep 08 16:00:58.266960 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:00:58.266936    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e0da5a-8f59c46fb-8c7sm"
Sep 08 16:00:58.377273 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:00:58.377185    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9dbeaa09-4695-4eb4-80b6-08da3893d90f-maas-api-tls\") pod \"9dbeaa09-4695-4eb4-80b6-08da3893d90f\" (UID: \"9dbeaa09-4695-4eb4-80b6-08da3893d90f\") "
Sep 08 16:00:58.377438 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:00:58.377273    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qt82l\" (UniqueName: \"kubernetes.io/projected/9dbeaa09-4695-4eb4-80b6-08da3893d90f-kube-api-access-qt82l\") pod \"9dbeaa09-4695-4eb4-80b6-08da3893d90f\" (UID: \"9dbeaa09-4695-4eb4-80b6-08da3893d90f\") "
Sep 08 16:00:58.377438 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:00:58.377325    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9dbeaa09-4695-4eb4-80b6-08da3893d90f-metrics-certs\") pod \"9dbeaa09-4695-4eb4-80b6-08da3893d90f\" (UID: \"9dbeaa09-4695-4eb4-80b6-08da3893d90f\") "
Sep 08 16:00:58.377438 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:00:58.377371    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9dbeaa09-4695-4eb4-80b6-08da3893d90f-openshift-service-ca\") pod \"9dbeaa09-4695-4eb4-80b6-08da3893d90f\" (UID: \"9dbeaa09-4695-4eb4-80b6-08da3893d90f\") "
Sep 08 16:00:58.377973 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:00:58.377940    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9dbeaa09-4695-4eb4-80b6-08da3893d90f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9dbeaa09-4695-4eb4-80b6-08da3893d90f" (UID: "9dbeaa09-4695-4eb4-80b6-08da3893d90f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:00:58.379552 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:00:58.379520    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9dbeaa09-4695-4eb4-80b6-08da3893d90f-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "9dbeaa09-4695-4eb4-80b6-08da3893d90f" (UID: "9dbeaa09-4695-4eb4-80b6-08da3893d90f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 16:00:58.379552 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:00:58.379542    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9dbeaa09-4695-4eb4-80b6-08da3893d90f-kube-api-access-qt82l" (OuterVolumeSpecName: "kube-api-access-qt82l") pod "9dbeaa09-4695-4eb4-80b6-08da3893d90f" (UID: "9dbeaa09-4695-4eb4-80b6-08da3893d90f"). InnerVolumeSpecName "kube-api-access-qt82l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:00:58.379784 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:00:58.379760    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9dbeaa09-4695-4eb4-80b6-08da3893d90f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9dbeaa09-4695-4eb4-80b6-08da3893d90f" (UID: "9dbeaa09-4695-4eb4-80b6-08da3893d90f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 16:00:58.478498 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:00:58.478469    2549 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9dbeaa09-4695-4eb4-80b6-08da3893d90f-openshift-service-ca\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 16:00:58.478712 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:00:58.478594    2549 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9dbeaa09-4695-4eb4-80b6-08da3893d90f-maas-api-tls\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 16:00:58.478712 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:00:58.478703    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qt82l\" (UniqueName: \"kubernetes.io/projected/9dbeaa09-4695-4eb4-80b6-08da3893d90f-kube-api-access-qt82l\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 16:00:58.478802 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:00:58.478720    2549 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9dbeaa09-4695-4eb4-80b6-08da3893d90f-metrics-certs\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 16:00:58.672564 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:00:58.672530    2549 generic.go:356] "Generic (PLEG): container finished" podID="9dbeaa09-4695-4eb4-80b6-08da3893d90f" containerID="ba525a946b181ed231a21a5bd74bf3d4a4969f0ce73e664993d998a5f3273327" exitCode=0
Sep 08 16:00:58.672717 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:00:58.672570    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e0da5a-8f59c46fb-8c7sm"
Sep 08 16:00:58.672717 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:00:58.672620    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e0da5a-8f59c46fb-8c7sm" event={"ID":"9dbeaa09-4695-4eb4-80b6-08da3893d90f","Type":"ContainerDied","Data":"ba525a946b181ed231a21a5bd74bf3d4a4969f0ce73e664993d998a5f3273327"}
Sep 08 16:00:58.672717 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:00:58.672676    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e0da5a-8f59c46fb-8c7sm" event={"ID":"9dbeaa09-4695-4eb4-80b6-08da3893d90f","Type":"ContainerDied","Data":"e23c752edba199009875376988d82110d622902865c0bb7a801835272093c7dc"}
Sep 08 16:00:58.672717 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:00:58.672703    2549 scope.go:117] "RemoveContainer" containerID="ba525a946b181ed231a21a5bd74bf3d4a4969f0ce73e664993d998a5f3273327"
Sep 08 16:00:58.681677 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:00:58.681660    2549 scope.go:117] "RemoveContainer" containerID="ba525a946b181ed231a21a5bd74bf3d4a4969f0ce73e664993d998a5f3273327"
Sep 08 16:00:58.681961 ip-10-0-143-61 kubenswrapper[2549]: E0908 16:00:58.681942    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba525a946b181ed231a21a5bd74bf3d4a4969f0ce73e664993d998a5f3273327\": container with ID starting with ba525a946b181ed231a21a5bd74bf3d4a4969f0ce73e664993d998a5f3273327 not found: ID does not exist" containerID="ba525a946b181ed231a21a5bd74bf3d4a4969f0ce73e664993d998a5f3273327"
Sep 08 16:00:58.682017 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:00:58.681969    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba525a946b181ed231a21a5bd74bf3d4a4969f0ce73e664993d998a5f3273327"} err="failed to get container status \"ba525a946b181ed231a21a5bd74bf3d4a4969f0ce73e664993d998a5f3273327\": rpc error: code = NotFound desc = could not find container \"ba525a946b181ed231a21a5bd74bf3d4a4969f0ce73e664993d998a5f3273327\": container with ID starting with ba525a946b181ed231a21a5bd74bf3d4a4969f0ce73e664993d998a5f3273327 not found: ID does not exist"
Sep 08 16:00:58.688742 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:00:58.688722    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-e0da5a-8f59c46fb-8c7sm"]
Sep 08 16:00:58.694704 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:00:58.694674    2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-e0da5a-8f59c46fb-8c7sm"]
Sep 08 16:01:00.400492 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:01:00.400460    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9dbeaa09-4695-4eb4-80b6-08da3893d90f" path="/var/lib/kubelet/pods/9dbeaa09-4695-4eb4-80b6-08da3893d90f/volumes"
Sep 08 16:01:57.447429 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:01:57.447389    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a3a348-6fb49499f7-m4q4c"]
Sep 08 16:01:57.448024 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:01:57.447599    2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a3a348-6fb49499f7-m4q4c" podUID="dc333166-7aa5-4c4b-9906-9994c344e4d4" containerName="maas-api" containerID="cri-o://fc3c76a46b31f1ae4a5ad602e20ac1987c9134033d554a1ec2c4c3acd24b935b" gracePeriod=30
Sep 08 16:01:57.464573 ip-10-0-143-61 kubenswrapper[2549]: E0908 16:01:57.464388    2549 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc333166_7aa5_4c4b_9906_9994c344e4d4.slice/crio-fc3c76a46b31f1ae4a5ad602e20ac1987c9134033d554a1ec2c4c3acd24b935b.scope\": RecentStats: unable to find data in memory cache]"
Sep 08 16:01:57.688125 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:01:57.688101    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a3a348-6fb49499f7-m4q4c"
Sep 08 16:01:57.766807 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:01:57.766743    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc333166-7aa5-4c4b-9906-9994c344e4d4-maas-api-tls\") pod \"dc333166-7aa5-4c4b-9906-9994c344e4d4\" (UID: \"dc333166-7aa5-4c4b-9906-9994c344e4d4\") "
Sep 08 16:01:57.766951 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:01:57.766831    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc333166-7aa5-4c4b-9906-9994c344e4d4-metrics-certs\") pod \"dc333166-7aa5-4c4b-9906-9994c344e4d4\" (UID: \"dc333166-7aa5-4c4b-9906-9994c344e4d4\") "
Sep 08 16:01:57.766951 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:01:57.766868    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bl5kz\" (UniqueName: \"kubernetes.io/projected/dc333166-7aa5-4c4b-9906-9994c344e4d4-kube-api-access-bl5kz\") pod \"dc333166-7aa5-4c4b-9906-9994c344e4d4\" (UID: \"dc333166-7aa5-4c4b-9906-9994c344e4d4\") "
Sep 08 16:01:57.766951 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:01:57.766947    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc333166-7aa5-4c4b-9906-9994c344e4d4-openshift-service-ca\") pod \"dc333166-7aa5-4c4b-9906-9994c344e4d4\" (UID: \"dc333166-7aa5-4c4b-9906-9994c344e4d4\") "
Sep 08 16:01:57.767378 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:01:57.767353    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc333166-7aa5-4c4b-9906-9994c344e4d4-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "dc333166-7aa5-4c4b-9906-9994c344e4d4" (UID: "dc333166-7aa5-4c4b-9906-9994c344e4d4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:01:57.768904 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:01:57.768879    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc333166-7aa5-4c4b-9906-9994c344e4d4-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "dc333166-7aa5-4c4b-9906-9994c344e4d4" (UID: "dc333166-7aa5-4c4b-9906-9994c344e4d4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 16:01:57.768904 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:01:57.768881    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc333166-7aa5-4c4b-9906-9994c344e4d4-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "dc333166-7aa5-4c4b-9906-9994c344e4d4" (UID: "dc333166-7aa5-4c4b-9906-9994c344e4d4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 16:01:57.769031 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:01:57.768929    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc333166-7aa5-4c4b-9906-9994c344e4d4-kube-api-access-bl5kz" (OuterVolumeSpecName: "kube-api-access-bl5kz") pod "dc333166-7aa5-4c4b-9906-9994c344e4d4" (UID: "dc333166-7aa5-4c4b-9906-9994c344e4d4"). InnerVolumeSpecName "kube-api-access-bl5kz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:01:57.868375 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:01:57.868337    2549 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc333166-7aa5-4c4b-9906-9994c344e4d4-metrics-certs\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 16:01:57.868375 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:01:57.868368    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bl5kz\" (UniqueName: \"kubernetes.io/projected/dc333166-7aa5-4c4b-9906-9994c344e4d4-kube-api-access-bl5kz\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 16:01:57.868375 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:01:57.868379    2549 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc333166-7aa5-4c4b-9906-9994c344e4d4-openshift-service-ca\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 16:01:57.868633 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:01:57.868389    2549 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc333166-7aa5-4c4b-9906-9994c344e4d4-maas-api-tls\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 16:01:57.981538 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:01:57.981502    2549 generic.go:356] "Generic (PLEG): container finished" podID="dc333166-7aa5-4c4b-9906-9994c344e4d4" containerID="fc3c76a46b31f1ae4a5ad602e20ac1987c9134033d554a1ec2c4c3acd24b935b" exitCode=0
Sep 08 16:01:57.981741 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:01:57.981549    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a3a348-6fb49499f7-m4q4c"
Sep 08 16:01:57.981741 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:01:57.981550    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a3a348-6fb49499f7-m4q4c" event={"ID":"dc333166-7aa5-4c4b-9906-9994c344e4d4","Type":"ContainerDied","Data":"fc3c76a46b31f1ae4a5ad602e20ac1987c9134033d554a1ec2c4c3acd24b935b"}
Sep 08 16:01:57.981741 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:01:57.981617    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a3a348-6fb49499f7-m4q4c" event={"ID":"dc333166-7aa5-4c4b-9906-9994c344e4d4","Type":"ContainerDied","Data":"788003c7f852b5563c8c4e4f65a7265425dd43c77806b871e44c43cc679940bd"}
Sep 08 16:01:57.981741 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:01:57.981648    2549 scope.go:117] "RemoveContainer" containerID="fc3c76a46b31f1ae4a5ad602e20ac1987c9134033d554a1ec2c4c3acd24b935b"
Sep 08 16:01:57.990904 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:01:57.990884    2549 scope.go:117] "RemoveContainer" containerID="fc3c76a46b31f1ae4a5ad602e20ac1987c9134033d554a1ec2c4c3acd24b935b"
Sep 08 16:01:57.991210 ip-10-0-143-61 kubenswrapper[2549]: E0908 16:01:57.991181    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc3c76a46b31f1ae4a5ad602e20ac1987c9134033d554a1ec2c4c3acd24b935b\": container with ID starting with fc3c76a46b31f1ae4a5ad602e20ac1987c9134033d554a1ec2c4c3acd24b935b not found: ID does not exist" containerID="fc3c76a46b31f1ae4a5ad602e20ac1987c9134033d554a1ec2c4c3acd24b935b"
Sep 08 16:01:57.991269 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:01:57.991222    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc3c76a46b31f1ae4a5ad602e20ac1987c9134033d554a1ec2c4c3acd24b935b"} err="failed to get container status \"fc3c76a46b31f1ae4a5ad602e20ac1987c9134033d554a1ec2c4c3acd24b935b\": rpc error: code = NotFound desc = could not find container \"fc3c76a46b31f1ae4a5ad602e20ac1987c9134033d554a1ec2c4c3acd24b935b\": container with ID starting with fc3c76a46b31f1ae4a5ad602e20ac1987c9134033d554a1ec2c4c3acd24b935b not found: ID does not exist"
Sep 08 16:01:58.015296 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:01:58.015271    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a3a348-6fb49499f7-m4q4c"]
Sep 08 16:01:58.030157 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:01:58.030104    2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a3a348-6fb49499f7-m4q4c"]
Sep 08 16:01:58.400008 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:01:58.399975    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dc333166-7aa5-4c4b-9906-9994c344e4d4" path="/var/lib/kubelet/pods/dc333166-7aa5-4c4b-9906-9994c344e4d4/volumes"
Sep 08 16:02:04.138595 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:02:04.138557    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-a3a348-openshift-default-888d95576-zz4tm"]
Sep 08 16:02:04.139048 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:02:04.138794    2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-a3a348-openshift-default-888d95576-zz4tm" podUID="8f989139-19c6-4ee7-bc23-a118c031e801" containerName="istio-proxy" containerID="cri-o://d00d7d9024c7f8422e9223064d54486458ebdd8433d54556be7e2d92f87a772a" gracePeriod=30
Sep 08 16:02:04.503314 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:02:04.503279    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-b30e99-zqwl7"]
Sep 08 16:02:04.503831 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:02:04.503810    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9dbeaa09-4695-4eb4-80b6-08da3893d90f" containerName="maas-api"
Sep 08 16:02:04.503906 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:02:04.503838    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="9dbeaa09-4695-4eb4-80b6-08da3893d90f" containerName="maas-api"
Sep 08 16:02:04.503906 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:02:04.503885    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc333166-7aa5-4c4b-9906-9994c344e4d4" containerName="maas-api"
Sep 08 16:02:04.503906 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:02:04.503896    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc333166-7aa5-4c4b-9906-9994c344e4d4" containerName="maas-api"
Sep 08 16:02:04.504022 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:02:04.503907    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4db3b95c-b863-4a09-a9a1-e0dd0dc282c0" containerName="cleanup"
Sep 08 16:02:04.504022 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:02:04.503916    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="4db3b95c-b863-4a09-a9a1-e0dd0dc282c0" containerName="cleanup"
Sep 08 16:02:04.504022 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:02:04.504005    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="4db3b95c-b863-4a09-a9a1-e0dd0dc282c0" containerName="cleanup"
Sep 08 16:02:04.504022 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:02:04.504019    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="9dbeaa09-4695-4eb4-80b6-08da3893d90f" containerName="maas-api"
Sep 08 16:02:04.504161 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:02:04.504029    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc333166-7aa5-4c4b-9906-9994c344e4d4" containerName="maas-api"
Sep 08 16:02:04.508377 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:02:04.508360    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-b30e99-zqwl7"
Sep 08 16:02:04.511706 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:02:04.511680    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vcp9j\""
Sep 08 16:02:04.511826 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:02:04.511680    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 08 16:02:04.514564 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:02:04.514538    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-b30e99-zqwl7"]
Sep 08 16:02:04.623210 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:02:04.623177    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba920924-a704-4d87-958a-4736b62d84c4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-b30e99-zqwl7\" (UID: \"ba920924-a704-4d87-958a-4736b62d84c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-b30e99-zqwl7"
Sep 08 16:02:04.724238 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:02:04.724207    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba920924-a704-4d87-958a-4736b62d84c4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-b30e99-zqwl7\" (UID: \"ba920924-a704-4d87-958a-4736b62d84c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-b30e99-zqwl7"
Sep 08 16:02:04.724793 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:02:04.724772    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba920924-a704-4d87-958a-4736b62d84c4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-b30e99-zqwl7\" (UID: \"ba920924-a704-4d87-958a-4736b62d84c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-b30e99-zqwl7"
Sep 08 16:02:04.819608 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:02:04.819533    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-b30e99-zqwl7"
Sep 08 16:02:05.090555 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:02:05.090520    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-b30e99-zqwl7"]
Sep 08 16:02:05.092222 ip-10-0-143-61 kubenswrapper[2549]: W0908 16:02:05.092196    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba920924_a704_4d87_958a_4736b62d84c4.slice/crio-8df556044cf4577ef2e048986a6b26d71d95ec704d7df5a4320b2b12eb847be8 WatchSource:0}: Error finding container 8df556044cf4577ef2e048986a6b26d71d95ec704d7df5a4320b2b12eb847be8: Status 404 returned error can't find the container with id 8df556044cf4577ef2e048986a6b26d71d95ec704d7df5a4320b2b12eb847be8
Sep 08 16:02:06.027325 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:02:06.027235    2549 generic.go:356] "Generic (PLEG): container finished" podID="ba920924-a704-4d87-958a-4736b62d84c4" containerID="41fb839138f6db517d3af8f26a57e690eff8cbb6a68dcebe38874f7efe226d7b" exitCode=0
Sep 08 16:02:06.027325 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:02:06.027276    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-b30e99-zqwl7" event={"ID":"ba920924-a704-4d87-958a-4736b62d84c4","Type":"ContainerDied","Data":"41fb839138f6db517d3af8f26a57e690eff8cbb6a68dcebe38874f7efe226d7b"}
Sep 08 16:02:06.027713 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:02:06.027337    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-b30e99-zqwl7" event={"ID":"ba920924-a704-4d87-958a-4736b62d84c4","Type":"ContainerStarted","Data":"8df556044cf4577ef2e048986a6b26d71d95ec704d7df5a4320b2b12eb847be8"}
Sep 08 16:02:07.156749 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:02:07.156726    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-b30e99-zqwl7"
Sep 08 16:02:07.245386 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:02:07.245355    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba920924-a704-4d87-958a-4736b62d84c4-maas-api-service-ca\") pod \"ba920924-a704-4d87-958a-4736b62d84c4\" (UID: \"ba920924-a704-4d87-958a-4736b62d84c4\") "
Sep 08 16:02:07.245716 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:02:07.245694    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba920924-a704-4d87-958a-4736b62d84c4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ba920924-a704-4d87-958a-4736b62d84c4" (UID: "ba920924-a704-4d87-958a-4736b62d84c4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:02:07.346913 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:02:07.346836    2549 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba920924-a704-4d87-958a-4736b62d84c4-maas-api-service-ca\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 16:02:08.038757 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:02:08.038732    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-b30e99-zqwl7"
Sep 08 16:02:08.038943 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:02:08.038769    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-b30e99-zqwl7" event={"ID":"ba920924-a704-4d87-958a-4736b62d84c4","Type":"ContainerDied","Data":"8df556044cf4577ef2e048986a6b26d71d95ec704d7df5a4320b2b12eb847be8"}
Sep 08 16:02:08.038943 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:02:08.038810    2549 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8df556044cf4577ef2e048986a6b26d71d95ec704d7df5a4320b2b12eb847be8"
Sep 08 16:02:09.379063 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:02:09.379041    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-a3a348-openshift-default-888d95576-zz4tm"
Sep 08 16:02:09.464409 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:02:09.464345    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f989139-19c6-4ee7-bc23-a118c031e801-istio-data\") pod \"8f989139-19c6-4ee7-bc23-a118c031e801\" (UID: \"8f989139-19c6-4ee7-bc23-a118c031e801\") "
Sep 08 16:02:09.464409 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:02:09.464387    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f989139-19c6-4ee7-bc23-a118c031e801-istio-envoy\") pod \"8f989139-19c6-4ee7-bc23-a118c031e801\" (UID: \"8f989139-19c6-4ee7-bc23-a118c031e801\") "
Sep 08 16:02:09.464628 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:02:09.464407    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bdvkq\" (UniqueName: \"kubernetes.io/projected/8f989139-19c6-4ee7-bc23-a118c031e801-kube-api-access-bdvkq\") pod \"8f989139-19c6-4ee7-bc23-a118c031e801\" (UID: \"8f989139-19c6-4ee7-bc23-a118c031e801\") "
Sep 08 16:02:09.464628 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:02:09.464437    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f989139-19c6-4ee7-bc23-a118c031e801-istio-podinfo\") pod \"8f989139-19c6-4ee7-bc23-a118c031e801\" (UID: \"8f989139-19c6-4ee7-bc23-a118c031e801\") "
Sep 08 16:02:09.464628 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:02:09.464461    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f989139-19c6-4ee7-bc23-a118c031e801-credential-socket\") pod \"8f989139-19c6-4ee7-bc23-a118c031e801\" (UID: \"8f989139-19c6-4ee7-bc23-a118c031e801\") "
Sep 08 16:02:09.464628 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:02:09.464485    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f989139-19c6-4ee7-bc23-a118c031e801-workload-certs\") pod \"8f989139-19c6-4ee7-bc23-a118c031e801\" (UID: \"8f989139-19c6-4ee7-bc23-a118c031e801\") "
Sep 08 16:02:09.464628 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:02:09.464508    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f989139-19c6-4ee7-bc23-a118c031e801-istiod-ca-cert\") pod \"8f989139-19c6-4ee7-bc23-a118c031e801\" (UID: \"8f989139-19c6-4ee7-bc23-a118c031e801\") "
Sep 08 16:02:09.464628 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:02:09.464559    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f989139-19c6-4ee7-bc23-a118c031e801-workload-socket\") pod \"8f989139-19c6-4ee7-bc23-a118c031e801\" (UID: \"8f989139-19c6-4ee7-bc23-a118c031e801\") "
Sep 08 16:02:09.464628 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:02:09.464610    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f989139-19c6-4ee7-bc23-a118c031e801-istio-data" (OuterVolumeSpecName: "istio-data") pod "8f989139-19c6-4ee7-bc23-a118c031e801" (UID: "8f989139-19c6-4ee7-bc23-a118c031e801"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:02:09.464628 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:02:09.464627    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f989139-19c6-4ee7-bc23-a118c031e801-istio-token\") pod \"8f989139-19c6-4ee7-bc23-a118c031e801\" (UID: \"8f989139-19c6-4ee7-bc23-a118c031e801\") "
Sep 08 16:02:09.465083 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:02:09.464800    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f989139-19c6-4ee7-bc23-a118c031e801-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8f989139-19c6-4ee7-bc23-a118c031e801" (UID: "8f989139-19c6-4ee7-bc23-a118c031e801"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:02:09.465083 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:02:09.464813    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f989139-19c6-4ee7-bc23-a118c031e801-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8f989139-19c6-4ee7-bc23-a118c031e801" (UID: "8f989139-19c6-4ee7-bc23-a118c031e801"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:02:09.465083 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:02:09.465040    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f989139-19c6-4ee7-bc23-a118c031e801-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8f989139-19c6-4ee7-bc23-a118c031e801" (UID: "8f989139-19c6-4ee7-bc23-a118c031e801"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:02:09.465240 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:02:09.465111    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f989139-19c6-4ee7-bc23-a118c031e801-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8f989139-19c6-4ee7-bc23-a118c031e801" (UID: "8f989139-19c6-4ee7-bc23-a118c031e801"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:02:09.465304 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:02:09.465280    2549 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f989139-19c6-4ee7-bc23-a118c031e801-workload-socket\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 16:02:09.465304 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:02:09.465298    2549 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f989139-19c6-4ee7-bc23-a118c031e801-istio-data\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 16:02:09.465408 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:02:09.465311    2549 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f989139-19c6-4ee7-bc23-a118c031e801-credential-socket\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 16:02:09.465408 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:02:09.465321    2549 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f989139-19c6-4ee7-bc23-a118c031e801-workload-certs\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 16:02:09.465408 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:02:09.465329    2549 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f989139-19c6-4ee7-bc23-a118c031e801-istiod-ca-cert\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 16:02:09.466716 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:02:09.466687    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f989139-19c6-4ee7-bc23-a118c031e801-kube-api-access-bdvkq" (OuterVolumeSpecName: "kube-api-access-bdvkq") pod "8f989139-19c6-4ee7-bc23-a118c031e801" (UID: "8f989139-19c6-4ee7-bc23-a118c031e801"). InnerVolumeSpecName "kube-api-access-bdvkq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:02:09.466824 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:02:09.466717    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8f989139-19c6-4ee7-bc23-a118c031e801-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8f989139-19c6-4ee7-bc23-a118c031e801" (UID: "8f989139-19c6-4ee7-bc23-a118c031e801"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 08 16:02:09.466961 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:02:09.466920    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f989139-19c6-4ee7-bc23-a118c031e801-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8f989139-19c6-4ee7-bc23-a118c031e801" (UID: "8f989139-19c6-4ee7-bc23-a118c031e801"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:02:09.467016 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:02:09.466963    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f989139-19c6-4ee7-bc23-a118c031e801-istio-token" (OuterVolumeSpecName: "istio-token") pod "8f989139-19c6-4ee7-bc23-a118c031e801" (UID: "8f989139-19c6-4ee7-bc23-a118c031e801"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:02:09.565734 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:02:09.565703    2549 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f989139-19c6-4ee7-bc23-a118c031e801-istio-envoy\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 16:02:09.565734 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:02:09.565727    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bdvkq\" (UniqueName: \"kubernetes.io/projected/8f989139-19c6-4ee7-bc23-a118c031e801-kube-api-access-bdvkq\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 16:02:09.565734 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:02:09.565736    2549 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f989139-19c6-4ee7-bc23-a118c031e801-istio-podinfo\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 16:02:09.565990 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:02:09.565745    2549 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f989139-19c6-4ee7-bc23-a118c031e801-istio-token\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 16:02:10.052301 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:02:10.052264    2549 generic.go:356] "Generic (PLEG): container finished" podID="8f989139-19c6-4ee7-bc23-a118c031e801" containerID="d00d7d9024c7f8422e9223064d54486458ebdd8433d54556be7e2d92f87a772a" exitCode=0
Sep 08 16:02:10.052472 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:02:10.052310    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-a3a348-openshift-default-888d95576-zz4tm"
Sep 08 16:02:10.052472 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:02:10.052314    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a3a348-openshift-default-888d95576-zz4tm" event={"ID":"8f989139-19c6-4ee7-bc23-a118c031e801","Type":"ContainerDied","Data":"d00d7d9024c7f8422e9223064d54486458ebdd8433d54556be7e2d92f87a772a"}
Sep 08 16:02:10.052472 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:02:10.052417    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a3a348-openshift-default-888d95576-zz4tm" event={"ID":"8f989139-19c6-4ee7-bc23-a118c031e801","Type":"ContainerDied","Data":"041a686e69cc4ff6aa6b9258e7f77df78dcf4ca503bf958e39b389a2efbfa458"}
Sep 08 16:02:10.052472 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:02:10.052438    2549 scope.go:117] "RemoveContainer" containerID="d00d7d9024c7f8422e9223064d54486458ebdd8433d54556be7e2d92f87a772a"
Sep 08 16:02:10.061996 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:02:10.061977    2549 scope.go:117] "RemoveContainer" containerID="5ae7b77b9339058d3be203ed3aca7f80706b79d3b1bf50b71e7453cba5d00a10"
Sep 08 16:02:10.069513 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:02:10.069490    2549 scope.go:117] "RemoveContainer" containerID="d00d7d9024c7f8422e9223064d54486458ebdd8433d54556be7e2d92f87a772a"
Sep 08 16:02:10.069769 ip-10-0-143-61 kubenswrapper[2549]: E0908 16:02:10.069739    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d00d7d9024c7f8422e9223064d54486458ebdd8433d54556be7e2d92f87a772a\": container with ID starting with d00d7d9024c7f8422e9223064d54486458ebdd8433d54556be7e2d92f87a772a not found: ID does not exist" containerID="d00d7d9024c7f8422e9223064d54486458ebdd8433d54556be7e2d92f87a772a"
Sep 08 16:02:10.069809 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:02:10.069779    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d00d7d9024c7f8422e9223064d54486458ebdd8433d54556be7e2d92f87a772a"} err="failed to get container status \"d00d7d9024c7f8422e9223064d54486458ebdd8433d54556be7e2d92f87a772a\": rpc error: code = NotFound desc = could not find container \"d00d7d9024c7f8422e9223064d54486458ebdd8433d54556be7e2d92f87a772a\": container with ID starting with d00d7d9024c7f8422e9223064d54486458ebdd8433d54556be7e2d92f87a772a not found: ID does not exist"
Sep 08 16:02:10.069809 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:02:10.069797    2549 scope.go:117] "RemoveContainer" containerID="5ae7b77b9339058d3be203ed3aca7f80706b79d3b1bf50b71e7453cba5d00a10"
Sep 08 16:02:10.070051 ip-10-0-143-61 kubenswrapper[2549]: E0908 16:02:10.070037    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ae7b77b9339058d3be203ed3aca7f80706b79d3b1bf50b71e7453cba5d00a10\": container with ID starting with 5ae7b77b9339058d3be203ed3aca7f80706b79d3b1bf50b71e7453cba5d00a10 not found: ID does not exist" containerID="5ae7b77b9339058d3be203ed3aca7f80706b79d3b1bf50b71e7453cba5d00a10"
Sep 08 16:02:10.070108 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:02:10.070054    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ae7b77b9339058d3be203ed3aca7f80706b79d3b1bf50b71e7453cba5d00a10"} err="failed to get container status \"5ae7b77b9339058d3be203ed3aca7f80706b79d3b1bf50b71e7453cba5d00a10\": rpc error: code = NotFound desc = could not find container \"5ae7b77b9339058d3be203ed3aca7f80706b79d3b1bf50b71e7453cba5d00a10\": container with ID starting with 5ae7b77b9339058d3be203ed3aca7f80706b79d3b1bf50b71e7453cba5d00a10 not found: ID does not exist"
Sep 08 16:02:10.075357 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:02:10.075336    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-a3a348-openshift-default-888d95576-zz4tm"]
Sep 08 16:02:10.079090 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:02:10.079071    2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-a3a348-openshift-default-888d95576-zz4tm"]
Sep 08 16:02:10.400835 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:02:10.400801    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8f989139-19c6-4ee7-bc23-a118c031e801" path="/var/lib/kubelet/pods/8f989139-19c6-4ee7-bc23-a118c031e801/volumes"
Sep 08 16:02:15.476655 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:02:15.476619    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-b30e99-zqwl7"]
Sep 08 16:02:15.480035 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:02:15.480009    2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-b30e99-zqwl7"]
Sep 08 16:02:16.401067 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:02:16.401034    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ba920924-a704-4d87-958a-4736b62d84c4" path="/var/lib/kubelet/pods/ba920924-a704-4d87-958a-4736b62d84c4/volumes"
Sep 08 16:02:22.322243 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:02:22.322209    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-642d3b-9wgv2"]
Sep 08 16:02:22.322596 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:02:22.322550    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba920924-a704-4d87-958a-4736b62d84c4" containerName="revoke-keys"
Sep 08 16:02:22.322596 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:02:22.322561    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba920924-a704-4d87-958a-4736b62d84c4" containerName="revoke-keys"
Sep 08 16:02:22.322596 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:02:22.322569    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f989139-19c6-4ee7-bc23-a118c031e801" containerName="istio-proxy"
Sep 08 16:02:22.322596 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:02:22.322575    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f989139-19c6-4ee7-bc23-a118c031e801" containerName="istio-proxy"
Sep 08 16:02:22.322596 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:02:22.322586    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f989139-19c6-4ee7-bc23-a118c031e801" containerName="istio-proxy"
Sep 08 16:02:22.322596 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:02:22.322591    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f989139-19c6-4ee7-bc23-a118c031e801" containerName="istio-proxy"
Sep 08 16:02:22.322800 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:02:22.322649    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f989139-19c6-4ee7-bc23-a118c031e801" containerName="istio-proxy"
Sep 08 16:02:22.322800 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:02:22.322656    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f989139-19c6-4ee7-bc23-a118c031e801" containerName="istio-proxy"
Sep 08 16:02:22.322800 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:02:22.322662    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba920924-a704-4d87-958a-4736b62d84c4" containerName="revoke-keys"
Sep 08 16:02:22.325538 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:02:22.325521    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-642d3b-9wgv2"
Sep 08 16:02:22.328251 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:02:22.328226    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vcp9j\""
Sep 08 16:02:22.328387 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:02:22.328276    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 08 16:02:22.332567 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:02:22.332544    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-642d3b-9wgv2"]
Sep 08 16:02:22.475269 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:02:22.475234    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b819926-bbcf-4c06-ba74-670452cff153-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-642d3b-9wgv2\" (UID: \"0b819926-bbcf-4c06-ba74-670452cff153\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-642d3b-9wgv2"
Sep 08 16:02:22.576769 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:02:22.576683    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b819926-bbcf-4c06-ba74-670452cff153-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-642d3b-9wgv2\" (UID: \"0b819926-bbcf-4c06-ba74-670452cff153\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-642d3b-9wgv2"
Sep 08 16:02:22.577296 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:02:22.577275    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b819926-bbcf-4c06-ba74-670452cff153-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-642d3b-9wgv2\" (UID: \"0b819926-bbcf-4c06-ba74-670452cff153\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-642d3b-9wgv2"
Sep 08 16:02:22.636693 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:02:22.636665    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-642d3b-9wgv2"
Sep 08 16:02:22.906796 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:02:22.906766    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-642d3b-9wgv2"]
Sep 08 16:02:22.908755 ip-10-0-143-61 kubenswrapper[2549]: W0908 16:02:22.908727    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b819926_bbcf_4c06_ba74_670452cff153.slice/crio-17287ecb0e779bec7c2164c6d557447da848262e10ad98d969f28e06a3309219 WatchSource:0}: Error finding container 17287ecb0e779bec7c2164c6d557447da848262e10ad98d969f28e06a3309219: Status 404 returned error can't find the container with id 17287ecb0e779bec7c2164c6d557447da848262e10ad98d969f28e06a3309219
Sep 08 16:02:23.117929 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:02:23.117898    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-642d3b-9wgv2" event={"ID":"0b819926-bbcf-4c06-ba74-670452cff153","Type":"ContainerStarted","Data":"17287ecb0e779bec7c2164c6d557447da848262e10ad98d969f28e06a3309219"}
Sep 08 16:02:24.125102 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:02:24.125066    2549 generic.go:356] "Generic (PLEG): container finished" podID="0b819926-bbcf-4c06-ba74-670452cff153" containerID="0c34e3fa1599c8c1c3fb04d9fffce351738427a893725c1e4b2ce4c22c173e0b" exitCode=0
Sep 08 16:02:24.125489 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:02:24.125153    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-642d3b-9wgv2" event={"ID":"0b819926-bbcf-4c06-ba74-670452cff153","Type":"ContainerDied","Data":"0c34e3fa1599c8c1c3fb04d9fffce351738427a893725c1e4b2ce4c22c173e0b"}
Sep 08 16:02:25.254918 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:02:25.254894    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-642d3b-9wgv2"
Sep 08 16:02:25.399893 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:02:25.399842    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b819926-bbcf-4c06-ba74-670452cff153-maas-api-service-ca\") pod \"0b819926-bbcf-4c06-ba74-670452cff153\" (UID: \"0b819926-bbcf-4c06-ba74-670452cff153\") "
Sep 08 16:02:25.400111 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:02:25.400091    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b819926-bbcf-4c06-ba74-670452cff153-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0b819926-bbcf-4c06-ba74-670452cff153" (UID: "0b819926-bbcf-4c06-ba74-670452cff153"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:02:25.501195 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:02:25.501149    2549 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b819926-bbcf-4c06-ba74-670452cff153-maas-api-service-ca\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 16:02:26.137877 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:02:26.137825    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-642d3b-9wgv2"
Sep 08 16:02:26.138051 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:02:26.137880    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-642d3b-9wgv2" event={"ID":"0b819926-bbcf-4c06-ba74-670452cff153","Type":"ContainerDied","Data":"17287ecb0e779bec7c2164c6d557447da848262e10ad98d969f28e06a3309219"}
Sep 08 16:02:26.138051 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:02:26.137950    2549 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="17287ecb0e779bec7c2164c6d557447da848262e10ad98d969f28e06a3309219"
Sep 08 16:02:32.818930 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:02:32.818898    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-642d3b-29814720-4m9qt"]
Sep 08 16:02:32.822934 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:02:32.822907    2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-642d3b-29814720-4m9qt"]
Sep 08 16:02:33.010605 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:02:33.010576    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-642d3b-9wgv2"]
Sep 08 16:02:33.012378 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:02:33.012355    2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-642d3b-9wgv2"]
Sep 08 16:02:34.400080 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:02:34.400044    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0b819926-bbcf-4c06-ba74-670452cff153" path="/var/lib/kubelet/pods/0b819926-bbcf-4c06-ba74-670452cff153/volumes"
Sep 08 16:02:34.400568 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:02:34.400455    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4db3b95c-b863-4a09-a9a1-e0dd0dc282c0" path="/var/lib/kubelet/pods/4db3b95c-b863-4a09-a9a1-e0dd0dc282c0/volumes"
Sep 08 16:02:39.702393 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:02:39.702351    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-642d3b-openshift-default-596967c6f6-pxbln"]
Sep 08 16:02:39.702804 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:02:39.702611    2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-642d3b-openshift-default-596967c6f6-pxbln" podUID="1b4f5c99-b0b3-4e45-82e8-01ba20a02907" containerName="istio-proxy" containerID="cri-o://73205b16d63fff9ba3223c370229309059a49a94ec20c1dbea18740ec3526359" gracePeriod=30
Sep 08 16:02:41.664943 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:02:41.664897    2549 patch_prober.go:28] interesting pod/e2e-shared-b-w3-642d3b-openshift-default-596967c6f6-pxbln container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 08 16:02:41.665325 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:02:41.664960    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-642d3b-openshift-default-596967c6f6-pxbln" podUID="1b4f5c99-b0b3-4e45-82e8-01ba20a02907" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 08 16:02:42.474490 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:02:42.474464    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-wpwdq_5f7b0f68-a8e3-4058-b032-15f419682b2a/console-operator/2.log"
Sep 08 16:02:42.475968 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:02:42.475943    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-wpwdq_5f7b0f68-a8e3-4058-b032-15f419682b2a/console-operator/2.log"
Sep 08 16:02:42.480882 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:02:42.480844    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-f4vvt_195f1a74-19bc-49c9-aada-fa1c13488480/ovn-acl-logging/0.log"
Sep 08 16:02:42.482279 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:02:42.482259    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-f4vvt_195f1a74-19bc-49c9-aada-fa1c13488480/ovn-acl-logging/0.log"
Sep 08 16:02:44.937983 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:02:44.937958    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-642d3b-openshift-default-596967c6f6-pxbln"
Sep 08 16:02:44.953940 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:02:44.953920    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b4f5c99-b0b3-4e45-82e8-01ba20a02907-istiod-ca-cert\") pod \"1b4f5c99-b0b3-4e45-82e8-01ba20a02907\" (UID: \"1b4f5c99-b0b3-4e45-82e8-01ba20a02907\") "
Sep 08 16:02:44.954044 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:02:44.953951    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b4f5c99-b0b3-4e45-82e8-01ba20a02907-credential-socket\") pod \"1b4f5c99-b0b3-4e45-82e8-01ba20a02907\" (UID: \"1b4f5c99-b0b3-4e45-82e8-01ba20a02907\") "
Sep 08 16:02:44.954044 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:02:44.953977    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b4f5c99-b0b3-4e45-82e8-01ba20a02907-istio-envoy\") pod \"1b4f5c99-b0b3-4e45-82e8-01ba20a02907\" (UID: \"1b4f5c99-b0b3-4e45-82e8-01ba20a02907\") "
Sep 08 16:02:44.954044 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:02:44.954026    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b4f5c99-b0b3-4e45-82e8-01ba20a02907-workload-certs\") pod \"1b4f5c99-b0b3-4e45-82e8-01ba20a02907\" (UID: \"1b4f5c99-b0b3-4e45-82e8-01ba20a02907\") "
Sep 08 16:02:44.954206 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:02:44.954062    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ttgsq\" (UniqueName: \"kubernetes.io/projected/1b4f5c99-b0b3-4e45-82e8-01ba20a02907-kube-api-access-ttgsq\") pod \"1b4f5c99-b0b3-4e45-82e8-01ba20a02907\" (UID: \"1b4f5c99-b0b3-4e45-82e8-01ba20a02907\") "
Sep 08 16:02:44.954206 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:02:44.954099    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b4f5c99-b0b3-4e45-82e8-01ba20a02907-istio-token\") pod \"1b4f5c99-b0b3-4e45-82e8-01ba20a02907\" (UID: \"1b4f5c99-b0b3-4e45-82e8-01ba20a02907\") "
Sep 08 16:02:44.954316 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:02:44.954197    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b4f5c99-b0b3-4e45-82e8-01ba20a02907-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "1b4f5c99-b0b3-4e45-82e8-01ba20a02907" (UID: "1b4f5c99-b0b3-4e45-82e8-01ba20a02907"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:02:44.954316 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:02:44.954212    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b4f5c99-b0b3-4e45-82e8-01ba20a02907-workload-socket\") pod \"1b4f5c99-b0b3-4e45-82e8-01ba20a02907\" (UID: \"1b4f5c99-b0b3-4e45-82e8-01ba20a02907\") "
Sep 08 16:02:44.954316 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:02:44.954284    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b4f5c99-b0b3-4e45-82e8-01ba20a02907-istio-podinfo\") pod \"1b4f5c99-b0b3-4e45-82e8-01ba20a02907\" (UID: \"1b4f5c99-b0b3-4e45-82e8-01ba20a02907\") "
Sep 08 16:02:44.954488 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:02:44.954321    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b4f5c99-b0b3-4e45-82e8-01ba20a02907-istio-data\") pod \"1b4f5c99-b0b3-4e45-82e8-01ba20a02907\" (UID: \"1b4f5c99-b0b3-4e45-82e8-01ba20a02907\") "
Sep 08 16:02:44.954488 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:02:44.954362    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b4f5c99-b0b3-4e45-82e8-01ba20a02907-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "1b4f5c99-b0b3-4e45-82e8-01ba20a02907" (UID: "1b4f5c99-b0b3-4e45-82e8-01ba20a02907"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:02:44.954488 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:02:44.954420    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b4f5c99-b0b3-4e45-82e8-01ba20a02907-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "1b4f5c99-b0b3-4e45-82e8-01ba20a02907" (UID: "1b4f5c99-b0b3-4e45-82e8-01ba20a02907"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:02:44.954654 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:02:44.954563    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b4f5c99-b0b3-4e45-82e8-01ba20a02907-istio-data" (OuterVolumeSpecName: "istio-data") pod "1b4f5c99-b0b3-4e45-82e8-01ba20a02907" (UID: "1b4f5c99-b0b3-4e45-82e8-01ba20a02907"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:02:44.954654 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:02:44.954608    2549 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b4f5c99-b0b3-4e45-82e8-01ba20a02907-istiod-ca-cert\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 16:02:44.954654 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:02:44.954628    2549 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b4f5c99-b0b3-4e45-82e8-01ba20a02907-credential-socket\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 16:02:44.954654 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:02:44.954637    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b4f5c99-b0b3-4e45-82e8-01ba20a02907-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "1b4f5c99-b0b3-4e45-82e8-01ba20a02907" (UID: "1b4f5c99-b0b3-4e45-82e8-01ba20a02907"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:02:44.954885 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:02:44.954644    2549 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b4f5c99-b0b3-4e45-82e8-01ba20a02907-workload-certs\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 16:02:44.956221 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:02:44.956192    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b4f5c99-b0b3-4e45-82e8-01ba20a02907-istio-token" (OuterVolumeSpecName: "istio-token") pod "1b4f5c99-b0b3-4e45-82e8-01ba20a02907" (UID: "1b4f5c99-b0b3-4e45-82e8-01ba20a02907"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:02:44.956319 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:02:44.956243    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b4f5c99-b0b3-4e45-82e8-01ba20a02907-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "1b4f5c99-b0b3-4e45-82e8-01ba20a02907" (UID: "1b4f5c99-b0b3-4e45-82e8-01ba20a02907"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:02:44.956319 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:02:44.956243    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1b4f5c99-b0b3-4e45-82e8-01ba20a02907-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "1b4f5c99-b0b3-4e45-82e8-01ba20a02907" (UID: "1b4f5c99-b0b3-4e45-82e8-01ba20a02907"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 08 16:02:44.956319 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:02:44.956269    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b4f5c99-b0b3-4e45-82e8-01ba20a02907-kube-api-access-ttgsq" (OuterVolumeSpecName: "kube-api-access-ttgsq") pod "1b4f5c99-b0b3-4e45-82e8-01ba20a02907" (UID: "1b4f5c99-b0b3-4e45-82e8-01ba20a02907"). InnerVolumeSpecName "kube-api-access-ttgsq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:02:45.055116 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:02:45.055055    2549 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b4f5c99-b0b3-4e45-82e8-01ba20a02907-istio-podinfo\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 16:02:45.055116 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:02:45.055079    2549 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b4f5c99-b0b3-4e45-82e8-01ba20a02907-istio-data\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 16:02:45.055116 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:02:45.055088    2549 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b4f5c99-b0b3-4e45-82e8-01ba20a02907-istio-envoy\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 16:02:45.055116 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:02:45.055096    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ttgsq\" (UniqueName: \"kubernetes.io/projected/1b4f5c99-b0b3-4e45-82e8-01ba20a02907-kube-api-access-ttgsq\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 16:02:45.055116 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:02:45.055105    2549 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b4f5c99-b0b3-4e45-82e8-01ba20a02907-istio-token\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 16:02:45.055116 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:02:45.055113    2549 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b4f5c99-b0b3-4e45-82e8-01ba20a02907-workload-socket\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 16:02:45.253126 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:02:45.253095    2549 generic.go:356] "Generic (PLEG): container finished" podID="1b4f5c99-b0b3-4e45-82e8-01ba20a02907" containerID="73205b16d63fff9ba3223c370229309059a49a94ec20c1dbea18740ec3526359" exitCode=0
Sep 08 16:02:45.253287 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:02:45.253152    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-642d3b-openshift-default-596967c6f6-pxbln"
Sep 08 16:02:45.253287 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:02:45.253186    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-642d3b-openshift-default-596967c6f6-pxbln" event={"ID":"1b4f5c99-b0b3-4e45-82e8-01ba20a02907","Type":"ContainerDied","Data":"73205b16d63fff9ba3223c370229309059a49a94ec20c1dbea18740ec3526359"}
Sep 08 16:02:45.253287 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:02:45.253259    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-642d3b-openshift-default-596967c6f6-pxbln" event={"ID":"1b4f5c99-b0b3-4e45-82e8-01ba20a02907","Type":"ContainerDied","Data":"de0904beb3919df912f71934b02011614b0a628092fb937247931cdbaca56772"}
Sep 08 16:02:45.253287 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:02:45.253283    2549 scope.go:117] "RemoveContainer" containerID="73205b16d63fff9ba3223c370229309059a49a94ec20c1dbea18740ec3526359"
Sep 08 16:02:45.262730 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:02:45.262710    2549 scope.go:117] "RemoveContainer" containerID="ad4d9b2b832b1b82773a45d8feb86ae8c664042c35d45d32d6a2c880e7a94814"
Sep 08 16:02:45.270623 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:02:45.270606    2549 scope.go:117] "RemoveContainer" containerID="73205b16d63fff9ba3223c370229309059a49a94ec20c1dbea18740ec3526359"
Sep 08 16:02:45.270883 ip-10-0-143-61 kubenswrapper[2549]: E0908 16:02:45.270840    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73205b16d63fff9ba3223c370229309059a49a94ec20c1dbea18740ec3526359\": container with ID starting with 73205b16d63fff9ba3223c370229309059a49a94ec20c1dbea18740ec3526359 not found: ID does not exist" containerID="73205b16d63fff9ba3223c370229309059a49a94ec20c1dbea18740ec3526359"
Sep 08 16:02:45.270949 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:02:45.270891    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73205b16d63fff9ba3223c370229309059a49a94ec20c1dbea18740ec3526359"} err="failed to get container status \"73205b16d63fff9ba3223c370229309059a49a94ec20c1dbea18740ec3526359\": rpc error: code = NotFound desc = could not find container \"73205b16d63fff9ba3223c370229309059a49a94ec20c1dbea18740ec3526359\": container with ID starting with 73205b16d63fff9ba3223c370229309059a49a94ec20c1dbea18740ec3526359 not found: ID does not exist"
Sep 08 16:02:45.270949 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:02:45.270912    2549 scope.go:117] "RemoveContainer" containerID="ad4d9b2b832b1b82773a45d8feb86ae8c664042c35d45d32d6a2c880e7a94814"
Sep 08 16:02:45.271182 ip-10-0-143-61 kubenswrapper[2549]: E0908 16:02:45.271162    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad4d9b2b832b1b82773a45d8feb86ae8c664042c35d45d32d6a2c880e7a94814\": container with ID starting with ad4d9b2b832b1b82773a45d8feb86ae8c664042c35d45d32d6a2c880e7a94814 not found: ID does not exist" containerID="ad4d9b2b832b1b82773a45d8feb86ae8c664042c35d45d32d6a2c880e7a94814"
Sep 08 16:02:45.271241 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:02:45.271187    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad4d9b2b832b1b82773a45d8feb86ae8c664042c35d45d32d6a2c880e7a94814"} err="failed to get container status \"ad4d9b2b832b1b82773a45d8feb86ae8c664042c35d45d32d6a2c880e7a94814\": rpc error: code = NotFound desc = could not find container \"ad4d9b2b832b1b82773a45d8feb86ae8c664042c35d45d32d6a2c880e7a94814\": container with ID starting with ad4d9b2b832b1b82773a45d8feb86ae8c664042c35d45d32d6a2c880e7a94814 not found: ID does not exist"
Sep 08 16:02:45.274719 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:02:45.274699    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-642d3b-openshift-default-596967c6f6-pxbln"]
Sep 08 16:02:45.278192 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:02:45.278173    2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-642d3b-openshift-default-596967c6f6-pxbln"]
Sep 08 16:02:46.399826 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:02:46.399783    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1b4f5c99-b0b3-4e45-82e8-01ba20a02907" path="/var/lib/kubelet/pods/1b4f5c99-b0b3-4e45-82e8-01ba20a02907/volumes"
Sep 08 16:03:42.658071 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:03:42.658036    2549 scope.go:117] "RemoveContainer" containerID="8ec126a3b72edb520580a77bde0a4c939eccf6645e77e86955b75d02eed41068"
Sep 08 16:06:42.694833 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:06:42.694803    2549 scope.go:117] "RemoveContainer" containerID="965563f3ea75f67dc257cca888ce2827c3d20c91bb1a2d4d4ebb4e7478e48b2c"
Sep 08 16:07:42.508491 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:07:42.508460    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-wpwdq_5f7b0f68-a8e3-4058-b032-15f419682b2a/console-operator/2.log"
Sep 08 16:07:42.509740 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:07:42.509720    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-wpwdq_5f7b0f68-a8e3-4058-b032-15f419682b2a/console-operator/2.log"
Sep 08 16:07:42.514495 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:07:42.514473    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-f4vvt_195f1a74-19bc-49c9-aada-fa1c13488480/ovn-acl-logging/0.log"
Sep 08 16:07:42.515971 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:07:42.515954    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-f4vvt_195f1a74-19bc-49c9-aada-fa1c13488480/ovn-acl-logging/0.log"
Sep 08 16:08:00.239897 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:08:00.239844    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6b689f8c54-wgbx2"]
Sep 08 16:08:00.240356 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:08:00.240197    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b4f5c99-b0b3-4e45-82e8-01ba20a02907" containerName="istio-proxy"
Sep 08 16:08:00.240356 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:08:00.240208    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b4f5c99-b0b3-4e45-82e8-01ba20a02907" containerName="istio-proxy"
Sep 08 16:08:00.240356 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:08:00.240223    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b819926-bbcf-4c06-ba74-670452cff153" containerName="revoke-keys"
Sep 08 16:08:00.240356 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:08:00.240228    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b819926-bbcf-4c06-ba74-670452cff153" containerName="revoke-keys"
Sep 08 16:08:00.240356 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:08:00.240284    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b819926-bbcf-4c06-ba74-670452cff153" containerName="revoke-keys"
Sep 08 16:08:00.240356 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:08:00.240291    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b4f5c99-b0b3-4e45-82e8-01ba20a02907" containerName="istio-proxy"
Sep 08 16:08:00.243108 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:08:00.243092    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6b689f8c54-wgbx2"
Sep 08 16:08:00.247297 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:08:00.247273    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 08 16:08:00.247297 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:08:00.247273    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-tq4f4\""
Sep 08 16:08:00.247493 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:08:00.247275    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 08 16:08:00.247493 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:08:00.247370    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 08 16:08:00.247493 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:08:00.247275    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 08 16:08:00.254383 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:08:00.254360    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6b689f8c54-wgbx2"]
Sep 08 16:08:00.303090 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:08:00.303057    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b7d29b3-d0fa-49d5-b326-db316a8150d2-openshift-service-ca\") pod \"maas-api-6b689f8c54-wgbx2\" (UID: \"8b7d29b3-d0fa-49d5-b326-db316a8150d2\") " pod="odh-ai-gateway-infra/maas-api-6b689f8c54-wgbx2"
Sep 08 16:08:00.303236 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:08:00.303106    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b7d29b3-d0fa-49d5-b326-db316a8150d2-maas-api-tls\") pod \"maas-api-6b689f8c54-wgbx2\" (UID: \"8b7d29b3-d0fa-49d5-b326-db316a8150d2\") " pod="odh-ai-gateway-infra/maas-api-6b689f8c54-wgbx2"
Sep 08 16:08:00.303236 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:08:00.303136    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-splr9\" (UniqueName: \"kubernetes.io/projected/8b7d29b3-d0fa-49d5-b326-db316a8150d2-kube-api-access-splr9\") pod \"maas-api-6b689f8c54-wgbx2\" (UID: \"8b7d29b3-d0fa-49d5-b326-db316a8150d2\") " pod="odh-ai-gateway-infra/maas-api-6b689f8c54-wgbx2"
Sep 08 16:08:00.303236 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:08:00.303158    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b7d29b3-d0fa-49d5-b326-db316a8150d2-metrics-certs\") pod \"maas-api-6b689f8c54-wgbx2\" (UID: \"8b7d29b3-d0fa-49d5-b326-db316a8150d2\") " pod="odh-ai-gateway-infra/maas-api-6b689f8c54-wgbx2"
Sep 08 16:08:00.404251 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:08:00.404221    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b7d29b3-d0fa-49d5-b326-db316a8150d2-openshift-service-ca\") pod \"maas-api-6b689f8c54-wgbx2\" (UID: \"8b7d29b3-d0fa-49d5-b326-db316a8150d2\") " pod="odh-ai-gateway-infra/maas-api-6b689f8c54-wgbx2"
Sep 08 16:08:00.404401 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:08:00.404264    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b7d29b3-d0fa-49d5-b326-db316a8150d2-maas-api-tls\") pod \"maas-api-6b689f8c54-wgbx2\" (UID: \"8b7d29b3-d0fa-49d5-b326-db316a8150d2\") " pod="odh-ai-gateway-infra/maas-api-6b689f8c54-wgbx2"
Sep 08 16:08:00.404401 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:08:00.404283    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-splr9\" (UniqueName: \"kubernetes.io/projected/8b7d29b3-d0fa-49d5-b326-db316a8150d2-kube-api-access-splr9\") pod \"maas-api-6b689f8c54-wgbx2\" (UID: \"8b7d29b3-d0fa-49d5-b326-db316a8150d2\") " pod="odh-ai-gateway-infra/maas-api-6b689f8c54-wgbx2"
Sep 08 16:08:00.404401 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:08:00.404311    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b7d29b3-d0fa-49d5-b326-db316a8150d2-metrics-certs\") pod \"maas-api-6b689f8c54-wgbx2\" (UID: \"8b7d29b3-d0fa-49d5-b326-db316a8150d2\") " pod="odh-ai-gateway-infra/maas-api-6b689f8c54-wgbx2"
Sep 08 16:08:00.404835 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:08:00.404814    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b7d29b3-d0fa-49d5-b326-db316a8150d2-openshift-service-ca\") pod \"maas-api-6b689f8c54-wgbx2\" (UID: \"8b7d29b3-d0fa-49d5-b326-db316a8150d2\") " pod="odh-ai-gateway-infra/maas-api-6b689f8c54-wgbx2"
Sep 08 16:08:00.406654 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:08:00.406632    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b7d29b3-d0fa-49d5-b326-db316a8150d2-maas-api-tls\") pod \"maas-api-6b689f8c54-wgbx2\" (UID: \"8b7d29b3-d0fa-49d5-b326-db316a8150d2\") " pod="odh-ai-gateway-infra/maas-api-6b689f8c54-wgbx2"
Sep 08 16:08:00.406805 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:08:00.406770    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b7d29b3-d0fa-49d5-b326-db316a8150d2-metrics-certs\") pod \"maas-api-6b689f8c54-wgbx2\" (UID: \"8b7d29b3-d0fa-49d5-b326-db316a8150d2\") " pod="odh-ai-gateway-infra/maas-api-6b689f8c54-wgbx2"
Sep 08 16:08:00.412124 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:08:00.412098    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-splr9\" (UniqueName: \"kubernetes.io/projected/8b7d29b3-d0fa-49d5-b326-db316a8150d2-kube-api-access-splr9\") pod \"maas-api-6b689f8c54-wgbx2\" (UID: \"8b7d29b3-d0fa-49d5-b326-db316a8150d2\") " pod="odh-ai-gateway-infra/maas-api-6b689f8c54-wgbx2"
Sep 08 16:08:00.554966 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:08:00.554880    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6b689f8c54-wgbx2"
Sep 08 16:08:00.830463 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:08:00.830440    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6b689f8c54-wgbx2"]
Sep 08 16:08:00.832005 ip-10-0-143-61 kubenswrapper[2549]: W0908 16:08:00.831967    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b7d29b3_d0fa_49d5_b326_db316a8150d2.slice/crio-cd1f9229292c0766a9022ab975607439d9c156bdca0b586b42d4fcd5790471cf WatchSource:0}: Error finding container cd1f9229292c0766a9022ab975607439d9c156bdca0b586b42d4fcd5790471cf: Status 404 returned error can't find the container with id cd1f9229292c0766a9022ab975607439d9c156bdca0b586b42d4fcd5790471cf
Sep 08 16:08:00.833275 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:08:00.833257    2549 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 08 16:08:01.786672 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:08:01.786585    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6b689f8c54-wgbx2" event={"ID":"8b7d29b3-d0fa-49d5-b326-db316a8150d2","Type":"ContainerStarted","Data":"72ccd76516bbe8789257e63b32dfb8102b14639efdab69a5c787ec8fec42a000"}
Sep 08 16:08:01.786672 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:08:01.786643    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6b689f8c54-wgbx2" event={"ID":"8b7d29b3-d0fa-49d5-b326-db316a8150d2","Type":"ContainerStarted","Data":"cd1f9229292c0766a9022ab975607439d9c156bdca0b586b42d4fcd5790471cf"}
Sep 08 16:08:01.786672 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:08:01.786671    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6b689f8c54-wgbx2"
Sep 08 16:08:01.805803 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:08:01.805748    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6b689f8c54-wgbx2" podStartSLOduration=1.172436343 podStartE2EDuration="1.805733392s" totalImagesPullingTime="633.297049ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:08:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:08:01.803800145 +0000 UTC m=+2119.893251235" watchObservedRunningTime="2026-09-08 16:08:01.805733392 +0000 UTC m=+2119.895184501"
Sep 08 16:08:07.796538 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:08:07.796505    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6b689f8c54-wgbx2"
Sep 08 16:08:08.225909 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:08:08.225874    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-d7bcf777d-p9sw9"]
Sep 08 16:08:08.226234 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:08:08.226219    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b4f5c99-b0b3-4e45-82e8-01ba20a02907" containerName="istio-proxy"
Sep 08 16:08:08.226286 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:08:08.226236    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b4f5c99-b0b3-4e45-82e8-01ba20a02907" containerName="istio-proxy"
Sep 08 16:08:08.226328 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:08:08.226301    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b4f5c99-b0b3-4e45-82e8-01ba20a02907" containerName="istio-proxy"
Sep 08 16:08:08.229502 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:08:08.229487    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-d7bcf777d-p9sw9"
Sep 08 16:08:08.239701 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:08:08.239672    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-d7bcf777d-p9sw9"]
Sep 08 16:08:08.366637 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:08:08.366599    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4r5w\" (UniqueName: \"kubernetes.io/projected/a086eccf-4b74-4ab7-b687-ca9008c2d123-kube-api-access-p4r5w\") pod \"maas-api-d7bcf777d-p9sw9\" (UID: \"a086eccf-4b74-4ab7-b687-ca9008c2d123\") " pod="odh-ai-gateway-infra/maas-api-d7bcf777d-p9sw9"
Sep 08 16:08:08.366637 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:08:08.366640    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a086eccf-4b74-4ab7-b687-ca9008c2d123-metrics-certs\") pod \"maas-api-d7bcf777d-p9sw9\" (UID: \"a086eccf-4b74-4ab7-b687-ca9008c2d123\") " pod="odh-ai-gateway-infra/maas-api-d7bcf777d-p9sw9"
Sep 08 16:08:08.366906 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:08:08.366679    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a086eccf-4b74-4ab7-b687-ca9008c2d123-maas-api-tls\") pod \"maas-api-d7bcf777d-p9sw9\" (UID: \"a086eccf-4b74-4ab7-b687-ca9008c2d123\") " pod="odh-ai-gateway-infra/maas-api-d7bcf777d-p9sw9"
Sep 08 16:08:08.366906 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:08:08.366771    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a086eccf-4b74-4ab7-b687-ca9008c2d123-openshift-service-ca\") pod \"maas-api-d7bcf777d-p9sw9\" (UID: \"a086eccf-4b74-4ab7-b687-ca9008c2d123\") " pod="odh-ai-gateway-infra/maas-api-d7bcf777d-p9sw9"
Sep 08 16:08:08.467134 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:08:08.467102    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a086eccf-4b74-4ab7-b687-ca9008c2d123-metrics-certs\") pod \"maas-api-d7bcf777d-p9sw9\" (UID: \"a086eccf-4b74-4ab7-b687-ca9008c2d123\") " pod="odh-ai-gateway-infra/maas-api-d7bcf777d-p9sw9"
Sep 08 16:08:08.467276 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:08:08.467146    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a086eccf-4b74-4ab7-b687-ca9008c2d123-maas-api-tls\") pod \"maas-api-d7bcf777d-p9sw9\" (UID: \"a086eccf-4b74-4ab7-b687-ca9008c2d123\") " pod="odh-ai-gateway-infra/maas-api-d7bcf777d-p9sw9"
Sep 08 16:08:08.467359 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:08:08.467333    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a086eccf-4b74-4ab7-b687-ca9008c2d123-openshift-service-ca\") pod \"maas-api-d7bcf777d-p9sw9\" (UID: \"a086eccf-4b74-4ab7-b687-ca9008c2d123\") " pod="odh-ai-gateway-infra/maas-api-d7bcf777d-p9sw9"
Sep 08 16:08:08.467874 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:08:08.467796    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4r5w\" (UniqueName: \"kubernetes.io/projected/a086eccf-4b74-4ab7-b687-ca9008c2d123-kube-api-access-p4r5w\") pod \"maas-api-d7bcf777d-p9sw9\" (UID: \"a086eccf-4b74-4ab7-b687-ca9008c2d123\") " pod="odh-ai-gateway-infra/maas-api-d7bcf777d-p9sw9"
Sep 08 16:08:08.468270 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:08:08.468242    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a086eccf-4b74-4ab7-b687-ca9008c2d123-openshift-service-ca\") pod \"maas-api-d7bcf777d-p9sw9\" (UID: \"a086eccf-4b74-4ab7-b687-ca9008c2d123\") " pod="odh-ai-gateway-infra/maas-api-d7bcf777d-p9sw9"
Sep 08 16:08:08.469793 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:08:08.469766    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a086eccf-4b74-4ab7-b687-ca9008c2d123-maas-api-tls\") pod \"maas-api-d7bcf777d-p9sw9\" (UID: \"a086eccf-4b74-4ab7-b687-ca9008c2d123\") " pod="odh-ai-gateway-infra/maas-api-d7bcf777d-p9sw9"
Sep 08 16:08:08.469915 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:08:08.469770    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a086eccf-4b74-4ab7-b687-ca9008c2d123-metrics-certs\") pod \"maas-api-d7bcf777d-p9sw9\" (UID: \"a086eccf-4b74-4ab7-b687-ca9008c2d123\") " pod="odh-ai-gateway-infra/maas-api-d7bcf777d-p9sw9"
Sep 08 16:08:08.476182 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:08:08.476115    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4r5w\" (UniqueName: \"kubernetes.io/projected/a086eccf-4b74-4ab7-b687-ca9008c2d123-kube-api-access-p4r5w\") pod \"maas-api-d7bcf777d-p9sw9\" (UID: \"a086eccf-4b74-4ab7-b687-ca9008c2d123\") " pod="odh-ai-gateway-infra/maas-api-d7bcf777d-p9sw9"
Sep 08 16:08:08.539961 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:08:08.539932    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-d7bcf777d-p9sw9"
Sep 08 16:08:08.810159 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:08:08.810133    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-d7bcf777d-p9sw9"]
Sep 08 16:08:08.812067 ip-10-0-143-61 kubenswrapper[2549]: W0908 16:08:08.812038    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda086eccf_4b74_4ab7_b687_ca9008c2d123.slice/crio-859d3e0db186ed2144885c3fc9b73eb9ea33195b7b24080dff20b74d35f902e8 WatchSource:0}: Error finding container 859d3e0db186ed2144885c3fc9b73eb9ea33195b7b24080dff20b74d35f902e8: Status 404 returned error can't find the container with id 859d3e0db186ed2144885c3fc9b73eb9ea33195b7b24080dff20b74d35f902e8
Sep 08 16:08:08.820438 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:08:08.820416    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-d7bcf777d-p9sw9" event={"ID":"a086eccf-4b74-4ab7-b687-ca9008c2d123","Type":"ContainerStarted","Data":"859d3e0db186ed2144885c3fc9b73eb9ea33195b7b24080dff20b74d35f902e8"}
Sep 08 16:08:12.841347 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:08:12.841312    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-d7bcf777d-p9sw9" event={"ID":"a086eccf-4b74-4ab7-b687-ca9008c2d123","Type":"ContainerStarted","Data":"83d97107f23a42c5143a12eda86ae28bedae3bf291a3490a7091c2c56128de1f"}
Sep 08 16:08:12.841738 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:08:12.841439    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-d7bcf777d-p9sw9"
Sep 08 16:08:12.861431 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:08:12.861384    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-d7bcf777d-p9sw9" podStartSLOduration=1.379127232 podStartE2EDuration="4.861370039s" totalImagesPullingTime="3.482242807s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:08:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:08:12.858653419 +0000 UTC m=+2130.948104507" watchObservedRunningTime="2026-09-08 16:08:12.861370039 +0000 UTC m=+2130.950821128"
Sep 08 16:08:18.851550 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:08:18.851522    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-d7bcf777d-p9sw9"
Sep 08 16:08:18.900990 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:08:18.900958    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6b689f8c54-wgbx2"]
Sep 08 16:08:18.901211 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:08:18.901182    2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-6b689f8c54-wgbx2" podUID="8b7d29b3-d0fa-49d5-b326-db316a8150d2" containerName="maas-api" containerID="cri-o://72ccd76516bbe8789257e63b32dfb8102b14639efdab69a5c787ec8fec42a000" gracePeriod=30
Sep 08 16:08:20.131951 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:08:20.131922    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6b689f8c54-wgbx2"
Sep 08 16:08:20.264573 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:08:20.264499    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b7d29b3-d0fa-49d5-b326-db316a8150d2-maas-api-tls\") pod \"8b7d29b3-d0fa-49d5-b326-db316a8150d2\" (UID: \"8b7d29b3-d0fa-49d5-b326-db316a8150d2\") "
Sep 08 16:08:20.264573 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:08:20.264533    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b7d29b3-d0fa-49d5-b326-db316a8150d2-metrics-certs\") pod \"8b7d29b3-d0fa-49d5-b326-db316a8150d2\" (UID: \"8b7d29b3-d0fa-49d5-b326-db316a8150d2\") "
Sep 08 16:08:20.264573 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:08:20.264573    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-splr9\" (UniqueName: \"kubernetes.io/projected/8b7d29b3-d0fa-49d5-b326-db316a8150d2-kube-api-access-splr9\") pod \"8b7d29b3-d0fa-49d5-b326-db316a8150d2\" (UID: \"8b7d29b3-d0fa-49d5-b326-db316a8150d2\") "
Sep 08 16:08:20.264770 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:08:20.264624    2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b7d29b3-d0fa-49d5-b326-db316a8150d2-openshift-service-ca\") pod \"8b7d29b3-d0fa-49d5-b326-db316a8150d2\" (UID: \"8b7d29b3-d0fa-49d5-b326-db316a8150d2\") "
Sep 08 16:08:20.265102 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:08:20.265065    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b7d29b3-d0fa-49d5-b326-db316a8150d2-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8b7d29b3-d0fa-49d5-b326-db316a8150d2" (UID: "8b7d29b3-d0fa-49d5-b326-db316a8150d2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:08:20.266593 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:08:20.266566    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b7d29b3-d0fa-49d5-b326-db316a8150d2-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "8b7d29b3-d0fa-49d5-b326-db316a8150d2" (UID: "8b7d29b3-d0fa-49d5-b326-db316a8150d2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 16:08:20.266711 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:08:20.266666    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b7d29b3-d0fa-49d5-b326-db316a8150d2-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8b7d29b3-d0fa-49d5-b326-db316a8150d2" (UID: "8b7d29b3-d0fa-49d5-b326-db316a8150d2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 16:08:20.266753 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:08:20.266719    2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b7d29b3-d0fa-49d5-b326-db316a8150d2-kube-api-access-splr9" (OuterVolumeSpecName: "kube-api-access-splr9") pod "8b7d29b3-d0fa-49d5-b326-db316a8150d2" (UID: "8b7d29b3-d0fa-49d5-b326-db316a8150d2"). InnerVolumeSpecName "kube-api-access-splr9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:08:20.365365 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:08:20.365333    2549 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b7d29b3-d0fa-49d5-b326-db316a8150d2-openshift-service-ca\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 16:08:20.365365 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:08:20.365360    2549 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b7d29b3-d0fa-49d5-b326-db316a8150d2-maas-api-tls\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 16:08:20.365365 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:08:20.365368    2549 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b7d29b3-d0fa-49d5-b326-db316a8150d2-metrics-certs\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 16:08:20.365564 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:08:20.365377    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-splr9\" (UniqueName: \"kubernetes.io/projected/8b7d29b3-d0fa-49d5-b326-db316a8150d2-kube-api-access-splr9\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Sep 08 16:08:20.884899 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:08:20.884843    2549 generic.go:356] "Generic (PLEG): container finished" podID="8b7d29b3-d0fa-49d5-b326-db316a8150d2" containerID="72ccd76516bbe8789257e63b32dfb8102b14639efdab69a5c787ec8fec42a000" exitCode=0
Sep 08 16:08:20.884899 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:08:20.884901    2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6b689f8c54-wgbx2"
Sep 08 16:08:20.885145 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:08:20.884896    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6b689f8c54-wgbx2" event={"ID":"8b7d29b3-d0fa-49d5-b326-db316a8150d2","Type":"ContainerDied","Data":"72ccd76516bbe8789257e63b32dfb8102b14639efdab69a5c787ec8fec42a000"}
Sep 08 16:08:20.885145 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:08:20.885021    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6b689f8c54-wgbx2" event={"ID":"8b7d29b3-d0fa-49d5-b326-db316a8150d2","Type":"ContainerDied","Data":"cd1f9229292c0766a9022ab975607439d9c156bdca0b586b42d4fcd5790471cf"}
Sep 08 16:08:20.885145 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:08:20.885048    2549 scope.go:117] "RemoveContainer" containerID="72ccd76516bbe8789257e63b32dfb8102b14639efdab69a5c787ec8fec42a000"
Sep 08 16:08:20.893972 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:08:20.893949    2549 scope.go:117] "RemoveContainer" containerID="72ccd76516bbe8789257e63b32dfb8102b14639efdab69a5c787ec8fec42a000"
Sep 08 16:08:20.894267 ip-10-0-143-61 kubenswrapper[2549]: E0908 16:08:20.894247    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72ccd76516bbe8789257e63b32dfb8102b14639efdab69a5c787ec8fec42a000\": container with ID starting with 72ccd76516bbe8789257e63b32dfb8102b14639efdab69a5c787ec8fec42a000 not found: ID does not exist" containerID="72ccd76516bbe8789257e63b32dfb8102b14639efdab69a5c787ec8fec42a000"
Sep 08 16:08:20.894312 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:08:20.894277    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72ccd76516bbe8789257e63b32dfb8102b14639efdab69a5c787ec8fec42a000"} err="failed to get container status \"72ccd76516bbe8789257e63b32dfb8102b14639efdab69a5c787ec8fec42a000\": rpc error: code = NotFound desc = could not find container \"72ccd76516bbe8789257e63b32dfb8102b14639efdab69a5c787ec8fec42a000\": container with ID starting with 72ccd76516bbe8789257e63b32dfb8102b14639efdab69a5c787ec8fec42a000 not found: ID does not exist"
Sep 08 16:08:20.925339 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:08:20.925311    2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6b689f8c54-wgbx2"]
Sep 08 16:08:20.937346 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:08:20.937319    2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-6b689f8c54-wgbx2"]
Sep 08 16:08:22.400847 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:08:22.400801    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8b7d29b3-d0fa-49d5-b326-db316a8150d2" path="/var/lib/kubelet/pods/8b7d29b3-d0fa-49d5-b326-db316a8150d2/volumes"
Sep 08 16:08:36.185660 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:08:36.185582    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-r8vkh_a1b207e9-fabc-4090-8658-040c165b446b/server/0.log"
Sep 08 16:08:36.422174 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:08:36.422141    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-84d887cf8-6rlx5_4fe4fa15-773c-4e7d-a5d9-c5ee9d7755e4/copy-manifests/0.log"
Sep 08 16:08:36.439461 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:08:36.439400    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-84d887cf8-6rlx5_4fe4fa15-773c-4e7d-a5d9-c5ee9d7755e4/manager/0.log"
Sep 08 16:08:37.052250 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:08:37.052220    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-d7bcf777d-p9sw9_a086eccf-4b74-4ab7-b687-ca9008c2d123/maas-api/0.log"
Sep 08 16:08:38.456280 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:08:38.456249    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-p7lvk_eb4fccc1-c0ed-423b-86dc-1226e8f7a666/manager/0.log"
Sep 08 16:08:39.565080 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:08:39.565049    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-trp8s_b22c4ca4-806f-4ab6-8b4c-d934e28a619f/discovery/0.log"
Sep 08 16:08:39.796444 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:08:39.796414    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7c6995f664-vbm7s_1ddcc598-ce1e-427c-9bd2-7bb888ba47a7/kube-auth-proxy/0.log"
Sep 08 16:08:42.721186 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:08:42.721156    2549 scope.go:117] "RemoveContainer" containerID="0c34e3fa1599c8c1c3fb04d9fffce351738427a893725c1e4b2ce4c22c173e0b"
Sep 08 16:08:42.731399 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:08:42.731380    2549 scope.go:117] "RemoveContainer" containerID="41fb839138f6db517d3af8f26a57e690eff8cbb6a68dcebe38874f7efe226d7b"
Sep 08 16:08:50.269488 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:08:50.269455    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-fw5jh_84663ea3-1feb-4c87-8609-c580fa60416e/global-pull-secret-syncer/0.log"
Sep 08 16:08:50.474074 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:08:50.474043    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-st9hm_28cd0557-2fec-4948-af98-6c67fb1e568d/konnectivity-agent/0.log"
Sep 08 16:08:50.547503 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:08:50.547381    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-143-61.ec2.internal_4a20335aaf3733c5ea5fa6f08be94600/haproxy/0.log"
Sep 08 16:08:55.026878 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:08:55.026838    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-p7lvk_eb4fccc1-c0ed-423b-86dc-1226e8f7a666/manager/0.log"
Sep 08 16:08:56.952330 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:08:56.952298    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-4cgxb_09d9c8c2-604a-481d-9e58-57bbb07b2e03/kube-state-metrics/0.log"
Sep 08 16:08:56.971710 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:08:56.971686    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-4cgxb_09d9c8c2-604a-481d-9e58-57bbb07b2e03/kube-rbac-proxy-main/0.log"
Sep 08 16:08:56.991930 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:08:56.991909    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-4cgxb_09d9c8c2-604a-481d-9e58-57bbb07b2e03/kube-rbac-proxy-self/0.log"
Sep 08 16:08:57.020870 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:08:57.020842    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-699f699998-xsc8g_7c03ad4e-85a7-4ca0-ad0a-389282c57543/metrics-server/0.log"
Sep 08 16:08:57.246761 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:08:57.246687    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-phnv9_6865c4b2-f146-4d9f-88cf-2eaa6460880a/node-exporter/0.log"
Sep 08 16:08:57.266709 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:08:57.266687    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-phnv9_6865c4b2-f146-4d9f-88cf-2eaa6460880a/kube-rbac-proxy/0.log"
Sep 08 16:08:57.285787 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:08:57.285768    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-phnv9_6865c4b2-f146-4d9f-88cf-2eaa6460880a/init-textfile/0.log"
Sep 08 16:08:57.317810 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:08:57.317782    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-mpbc4_e942dd64-4a92-449d-b013-7ac71c351448/kube-rbac-proxy-main/0.log"
Sep 08 16:08:57.342458 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:08:57.342436    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-mpbc4_e942dd64-4a92-449d-b013-7ac71c351448/kube-rbac-proxy-self/0.log"
Sep 08 16:08:57.367133 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:08:57.367109    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-mpbc4_e942dd64-4a92-449d-b013-7ac71c351448/openshift-state-metrics/0.log"
Sep 08 16:08:57.791294 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:08:57.791220    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-68f45db6d-mt86k_87979b2e-c099-493e-b4eb-03264e005462/thanos-query/0.log"
Sep 08 16:08:57.811806 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:08:57.811781    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-68f45db6d-mt86k_87979b2e-c099-493e-b4eb-03264e005462/kube-rbac-proxy-web/0.log"
Sep 08 16:08:57.836893 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:08:57.836841    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-68f45db6d-mt86k_87979b2e-c099-493e-b4eb-03264e005462/kube-rbac-proxy/0.log"
Sep 08 16:08:57.858909 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:08:57.858889    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-68f45db6d-mt86k_87979b2e-c099-493e-b4eb-03264e005462/prom-label-proxy/0.log"
Sep 08 16:08:57.883218 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:08:57.883196    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-68f45db6d-mt86k_87979b2e-c099-493e-b4eb-03264e005462/kube-rbac-proxy-rules/0.log"
Sep 08 16:08:57.906104 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:08:57.906087    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-68f45db6d-mt86k_87979b2e-c099-493e-b4eb-03264e005462/kube-rbac-proxy-metrics/0.log"
Sep 08 16:09:00.044625 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:09:00.044581    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-wpwdq_5f7b0f68-a8e3-4058-b032-15f419682b2a/console-operator/2.log"
Sep 08 16:09:00.052220 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:09:00.052179    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-wpwdq_5f7b0f68-a8e3-4058-b032-15f419682b2a/console-operator/3.log"
Sep 08 16:09:00.584993 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:09:00.584966    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-85d98c998f-28wpz_2d27f249-a6aa-47a6-a5cb-a8dcc588390c/download-server/0.log"
Sep 08 16:09:01.216253 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:09:01.216213    2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-zhg9q/perf-node-gather-daemonset-rkvdj"]
Sep 08 16:09:01.216729 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:09:01.216707    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b7d29b3-d0fa-49d5-b326-db316a8150d2" containerName="maas-api"
Sep 08 16:09:01.216808 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:09:01.216732    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b7d29b3-d0fa-49d5-b326-db316a8150d2" containerName="maas-api"
Sep 08 16:09:01.216883 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:09:01.216869    2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b7d29b3-d0fa-49d5-b326-db316a8150d2" containerName="maas-api"
Sep 08 16:09:01.220087 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:09:01.220065    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zhg9q/perf-node-gather-daemonset-rkvdj"
Sep 08 16:09:01.222947 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:09:01.222930    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-zhg9q\"/\"kube-root-ca.crt\""
Sep 08 16:09:01.224640 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:09:01.224264    2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-zhg9q\"/\"openshift-service-ca.crt\""
Sep 08 16:09:01.224640 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:09:01.224290    2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-zhg9q\"/\"default-dockercfg-7qzb5\""
Sep 08 16:09:01.230288 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:09:01.230269    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zhg9q/perf-node-gather-daemonset-rkvdj"]
Sep 08 16:09:01.297088 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:09:01.297061    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2b079027-5c35-49da-9487-3cc19ab232fb-sys\") pod \"perf-node-gather-daemonset-rkvdj\" (UID: \"2b079027-5c35-49da-9487-3cc19ab232fb\") " pod="openshift-must-gather-zhg9q/perf-node-gather-daemonset-rkvdj"
Sep 08 16:09:01.297196 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:09:01.297098    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rj54\" (UniqueName: \"kubernetes.io/projected/2b079027-5c35-49da-9487-3cc19ab232fb-kube-api-access-2rj54\") pod \"perf-node-gather-daemonset-rkvdj\" (UID: \"2b079027-5c35-49da-9487-3cc19ab232fb\") " pod="openshift-must-gather-zhg9q/perf-node-gather-daemonset-rkvdj"
Sep 08 16:09:01.297196 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:09:01.297118    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2b079027-5c35-49da-9487-3cc19ab232fb-lib-modules\") pod \"perf-node-gather-daemonset-rkvdj\" (UID: \"2b079027-5c35-49da-9487-3cc19ab232fb\") " pod="openshift-must-gather-zhg9q/perf-node-gather-daemonset-rkvdj"
Sep 08 16:09:01.297196 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:09:01.297147    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2b079027-5c35-49da-9487-3cc19ab232fb-podres\") pod \"perf-node-gather-daemonset-rkvdj\" (UID: \"2b079027-5c35-49da-9487-3cc19ab232fb\") " pod="openshift-must-gather-zhg9q/perf-node-gather-daemonset-rkvdj"
Sep 08 16:09:01.297196 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:09:01.297189    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2b079027-5c35-49da-9487-3cc19ab232fb-proc\") pod \"perf-node-gather-daemonset-rkvdj\" (UID: \"2b079027-5c35-49da-9487-3cc19ab232fb\") " pod="openshift-must-gather-zhg9q/perf-node-gather-daemonset-rkvdj"
Sep 08 16:09:01.397803 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:09:01.397777    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2b079027-5c35-49da-9487-3cc19ab232fb-proc\") pod \"perf-node-gather-daemonset-rkvdj\" (UID: \"2b079027-5c35-49da-9487-3cc19ab232fb\") " pod="openshift-must-gather-zhg9q/perf-node-gather-daemonset-rkvdj"
Sep 08 16:09:01.397915 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:09:01.397830    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2b079027-5c35-49da-9487-3cc19ab232fb-sys\") pod \"perf-node-gather-daemonset-rkvdj\" (UID: \"2b079027-5c35-49da-9487-3cc19ab232fb\") " pod="openshift-must-gather-zhg9q/perf-node-gather-daemonset-rkvdj"
Sep 08 16:09:01.397915 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:09:01.397875    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rj54\" (UniqueName: \"kubernetes.io/projected/2b079027-5c35-49da-9487-3cc19ab232fb-kube-api-access-2rj54\") pod \"perf-node-gather-daemonset-rkvdj\" (UID: \"2b079027-5c35-49da-9487-3cc19ab232fb\") " pod="openshift-must-gather-zhg9q/perf-node-gather-daemonset-rkvdj"
Sep 08 16:09:01.397915 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:09:01.397905    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2b079027-5c35-49da-9487-3cc19ab232fb-lib-modules\") pod \"perf-node-gather-daemonset-rkvdj\" (UID: \"2b079027-5c35-49da-9487-3cc19ab232fb\") " pod="openshift-must-gather-zhg9q/perf-node-gather-daemonset-rkvdj"
Sep 08 16:09:01.398031 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:09:01.397912    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2b079027-5c35-49da-9487-3cc19ab232fb-sys\") pod \"perf-node-gather-daemonset-rkvdj\" (UID: \"2b079027-5c35-49da-9487-3cc19ab232fb\") " pod="openshift-must-gather-zhg9q/perf-node-gather-daemonset-rkvdj"
Sep 08 16:09:01.398031 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:09:01.397925    2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2b079027-5c35-49da-9487-3cc19ab232fb-podres\") pod \"perf-node-gather-daemonset-rkvdj\" (UID: \"2b079027-5c35-49da-9487-3cc19ab232fb\") " pod="openshift-must-gather-zhg9q/perf-node-gather-daemonset-rkvdj"
Sep 08 16:09:01.398031 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:09:01.397914    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2b079027-5c35-49da-9487-3cc19ab232fb-proc\") pod \"perf-node-gather-daemonset-rkvdj\" (UID: \"2b079027-5c35-49da-9487-3cc19ab232fb\") " pod="openshift-must-gather-zhg9q/perf-node-gather-daemonset-rkvdj"
Sep 08 16:09:01.398031 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:09:01.398018    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2b079027-5c35-49da-9487-3cc19ab232fb-lib-modules\") pod \"perf-node-gather-daemonset-rkvdj\" (UID: \"2b079027-5c35-49da-9487-3cc19ab232fb\") " pod="openshift-must-gather-zhg9q/perf-node-gather-daemonset-rkvdj"
Sep 08 16:09:01.398160 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:09:01.398022    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2b079027-5c35-49da-9487-3cc19ab232fb-podres\") pod \"perf-node-gather-daemonset-rkvdj\" (UID: \"2b079027-5c35-49da-9487-3cc19ab232fb\") " pod="openshift-must-gather-zhg9q/perf-node-gather-daemonset-rkvdj"
Sep 08 16:09:01.406268 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:09:01.406248    2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rj54\" (UniqueName: \"kubernetes.io/projected/2b079027-5c35-49da-9487-3cc19ab232fb-kube-api-access-2rj54\") pod \"perf-node-gather-daemonset-rkvdj\" (UID: \"2b079027-5c35-49da-9487-3cc19ab232fb\") " pod="openshift-must-gather-zhg9q/perf-node-gather-daemonset-rkvdj"
Sep 08 16:09:01.531201 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:09:01.531125    2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zhg9q/perf-node-gather-daemonset-rkvdj"
Sep 08 16:09:01.799593 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:09:01.799566    2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zhg9q/perf-node-gather-daemonset-rkvdj"]
Sep 08 16:09:01.801616 ip-10-0-143-61 kubenswrapper[2549]: W0908 16:09:01.801582    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod2b079027_5c35_49da_9487_3cc19ab232fb.slice/crio-d681cd22dad449b030938fba3b0b72319fe83427f72046ded784d3e4133aaec1 WatchSource:0}: Error finding container d681cd22dad449b030938fba3b0b72319fe83427f72046ded784d3e4133aaec1: Status 404 returned error can't find the container with id d681cd22dad449b030938fba3b0b72319fe83427f72046ded784d3e4133aaec1
Sep 08 16:09:02.090872 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:09:02.090762    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zhg9q/perf-node-gather-daemonset-rkvdj" event={"ID":"2b079027-5c35-49da-9487-3cc19ab232fb","Type":"ContainerStarted","Data":"d08e268ee032c2ee2c6da15f042e60d63ef7af5cbe71ff07de53921243b5bb25"}
Sep 08 16:09:02.090872 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:09:02.090837    2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zhg9q/perf-node-gather-daemonset-rkvdj" event={"ID":"2b079027-5c35-49da-9487-3cc19ab232fb","Type":"ContainerStarted","Data":"d681cd22dad449b030938fba3b0b72319fe83427f72046ded784d3e4133aaec1"}
Sep 08 16:09:02.091090 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:09:02.090909    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-zhg9q/perf-node-gather-daemonset-rkvdj"
Sep 08 16:09:02.109611 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:09:02.109566    2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-zhg9q/perf-node-gather-daemonset-rkvdj" podStartSLOduration=1.109552429 podStartE2EDuration="1.109552429s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 16:09:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:09:02.107665067 +0000 UTC m=+2180.197116156" watchObservedRunningTime="2026-09-08 16:09:02.109552429 +0000 UTC m=+2180.199003518"
Sep 08 16:09:02.504547 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:09:02.504495    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-vhsng_57cef312-9005-43bc-9e2f-b59db9e864c3/dns/0.log"
Sep 08 16:09:02.523429 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:09:02.523406    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-vhsng_57cef312-9005-43bc-9e2f-b59db9e864c3/kube-rbac-proxy/0.log"
Sep 08 16:09:02.645535 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:09:02.645516    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-vp9nk_a955f56c-5ac8-422a-8ddc-ce17a8361eb6/dns-node-resolver/0.log"
Sep 08 16:09:03.167943 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:09:03.167906    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-5fc68db7dd-pl9pd_f95cc865-fd51-4726-9804-71f9b0b63ce7/registry/0.log"
Sep 08 16:09:03.190714 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:09:03.190691    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-6q7cq_5f9598ad-fb27-4dda-ab17-98d54cbd30aa/node-ca/0.log"
Sep 08 16:09:04.707157 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:09:04.707113    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-trp8s_b22c4ca4-806f-4ab6-8b4c-d934e28a619f/discovery/0.log"
Sep 08 16:09:04.756676 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:09:04.756648    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7c6995f664-vbm7s_1ddcc598-ce1e-427c-9bd2-7bb888ba47a7/kube-auth-proxy/0.log"
Sep 08 16:09:05.782031 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:09:05.781998    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-tt5hr_0e985291-302f-4109-b4ba-8569a56235e4/serve-healthcheck-canary/0.log"
Sep 08 16:09:06.287515 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:09:06.287486    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-6tn6s_3b1fd2bf-0e75-4f18-b224-db2a474aba0e/kube-rbac-proxy/0.log"
Sep 08 16:09:06.306495 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:09:06.306474    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-6tn6s_3b1fd2bf-0e75-4f18-b224-db2a474aba0e/exporter/0.log"
Sep 08 16:09:06.327141 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:09:06.327115    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-6tn6s_3b1fd2bf-0e75-4f18-b224-db2a474aba0e/extractor/0.log"
Sep 08 16:09:08.105729 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:09:08.105699    2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-zhg9q/perf-node-gather-daemonset-rkvdj"
Sep 08 16:09:08.993984 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:09:08.993949    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-r8vkh_a1b207e9-fabc-4090-8658-040c165b446b/server/0.log"
Sep 08 16:09:09.242791 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:09:09.242761    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-84d887cf8-6rlx5_4fe4fa15-773c-4e7d-a5d9-c5ee9d7755e4/manager/0.log"
Sep 08 16:09:09.258249 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:09:09.258179    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-84d887cf8-6rlx5_4fe4fa15-773c-4e7d-a5d9-c5ee9d7755e4/copy-manifests/0.log"
Sep 08 16:09:10.705058 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:09:10.705032    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-66d9b677c7-dhthn_4de71f3c-7786-45f7-b075-765ecbdaa643/manager/0.log"
Sep 08 16:09:11.620888 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:09:11.620786    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-trp8s_b22c4ca4-806f-4ab6-8b4c-d934e28a619f/discovery/0.log"
Sep 08 16:09:11.642870 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:09:11.642840    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7c6995f664-vbm7s_1ddcc598-ce1e-427c-9bd2-7bb888ba47a7/kube-auth-proxy/0.log"
Sep 08 16:09:13.862210 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:09:13.862177    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-trp8s_b22c4ca4-806f-4ab6-8b4c-d934e28a619f/discovery/0.log"
Sep 08 16:09:13.896695 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:09:13.896660    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7c6995f664-vbm7s_1ddcc598-ce1e-427c-9bd2-7bb888ba47a7/kube-auth-proxy/0.log"
Sep 08 16:09:16.228077 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:09:16.228039    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-8d7dd4675-7mfs7_d1d7e194-2f46-486c-aa08-f87ef377f54f/kube-storage-version-migrator-operator/1.log"
Sep 08 16:09:16.229723 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:09:16.229696    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-8d7dd4675-7mfs7_d1d7e194-2f46-486c-aa08-f87ef377f54f/kube-storage-version-migrator-operator/0.log"
Sep 08 16:09:17.214337 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:09:17.214295    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-95s5s_95c9958b-a005-439c-b856-0cbb99f08700/kube-multus/0.log"
Sep 08 16:09:17.394940 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:09:17.394910    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-rpthc_3236f6c9-60de-47f9-84a9-5980933c58cd/kube-multus-additional-cni-plugins/0.log"
Sep 08 16:09:17.417934 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:09:17.417898    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-rpthc_3236f6c9-60de-47f9-84a9-5980933c58cd/egress-router-binary-copy/0.log"
Sep 08 16:09:17.438768 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:09:17.438741    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-rpthc_3236f6c9-60de-47f9-84a9-5980933c58cd/cni-plugins/0.log"
Sep 08 16:09:17.462206 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:09:17.462144    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-rpthc_3236f6c9-60de-47f9-84a9-5980933c58cd/bond-cni-plugin/0.log"
Sep 08 16:09:17.483601 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:09:17.483577    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-rpthc_3236f6c9-60de-47f9-84a9-5980933c58cd/routeoverride-cni/0.log"
Sep 08 16:09:17.503654 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:09:17.503634    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-rpthc_3236f6c9-60de-47f9-84a9-5980933c58cd/whereabouts-cni-bincopy/0.log"
Sep 08 16:09:17.558275 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:09:17.558256    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-rpthc_3236f6c9-60de-47f9-84a9-5980933c58cd/whereabouts-cni/0.log"
Sep 08 16:09:18.048295 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:09:18.048263    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-h4cml_7d4d759e-9d3c-497a-b546-462ae2b48785/network-metrics-daemon/0.log"
Sep 08 16:09:18.065795 ip-10-0-143-61 kubenswrapper[2549]: I0908 16:09:18.065765    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-h4cml_7d4d759e-9d3c-497a-b546-462ae2b48785/kube-rbac-proxy/0.log"