Sep 11 13:19:27.414613 ip-10-0-131-86 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 11 13:19:27.414625 ip-10-0-131-86 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 11 13:19:27.414635 ip-10-0-131-86 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 11 13:19:27.414991 ip-10-0-131-86 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 11 13:19:37.489448 ip-10-0-131-86 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 11 13:19:37.489471 ip-10-0-131-86 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot c779c6a994cf4399a4013abf7cce5549 --
Sep 11 13:21:32.497266 ip-10-0-131-86 systemd[1]: Starting Kubernetes Kubelet...
Sep 11 13:21:32.888047 ip-10-0-131-86 kubenswrapper[2578]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 11 13:21:32.888047 ip-10-0-131-86 kubenswrapper[2578]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 11 13:21:32.888047 ip-10-0-131-86 kubenswrapper[2578]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 11 13:21:32.888047 ip-10-0-131-86 kubenswrapper[2578]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 11 13:21:32.888047 ip-10-0-131-86 kubenswrapper[2578]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 11 13:21:32.893898 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.893746    2578 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 11 13:21:32.898003 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.897977    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 13:21:32.898003 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.898003    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 13:21:32.898003 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.898008    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 13:21:32.898126 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.898012    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 13:21:32.898126 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.898015    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 13:21:32.898126 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.898019    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 13:21:32.898126 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.898022    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 13:21:32.898126 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.898025    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 13:21:32.898126 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.898028    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 13:21:32.898126 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.898031    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 13:21:32.898126 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.898034    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 13:21:32.898126 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.898037    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 13:21:32.898126 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.898039    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 13:21:32.898126 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.898042    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 13:21:32.898126 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.898045    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 13:21:32.898126 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.898047    2578 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 13:21:32.898126 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.898050    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 13:21:32.898126 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.898052    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 13:21:32.898126 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.898055    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 13:21:32.898126 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.898058    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 13:21:32.898126 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.898062    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 13:21:32.898126 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.898065    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 13:21:32.898580 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.898068    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 13:21:32.898580 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.898071    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 13:21:32.898580 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.898073    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 13:21:32.898580 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.898076    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 13:21:32.898580 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.898079    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 13:21:32.898580 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.898081    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 13:21:32.898580 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.898084    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 13:21:32.898580 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.898087    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 13:21:32.898580 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.898089    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 13:21:32.898580 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.898092    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 13:21:32.898580 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.898095    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 13:21:32.898580 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.898097    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 13:21:32.898580 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.898101    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 13:21:32.898580 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.898107    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 13:21:32.898580 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.898111    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 13:21:32.898580 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.898115    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 13:21:32.898580 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.898118    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 13:21:32.898580 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.898121    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 13:21:32.898580 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.898124    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 13:21:32.898580 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.898128    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 13:21:32.899094 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.898131    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 13:21:32.899094 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.898133    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 13:21:32.899094 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.898136    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 13:21:32.899094 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.898139    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 13:21:32.899094 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.898142    2578 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 13:21:32.899094 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.898144    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 13:21:32.899094 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.898147    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 13:21:32.899094 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.898149    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 13:21:32.899094 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.898152    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 13:21:32.899094 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.898154    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 13:21:32.899094 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.898157    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 13:21:32.899094 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.898160    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 13:21:32.899094 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.898162    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 13:21:32.899094 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.898165    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 13:21:32.899094 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.898167    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 13:21:32.899094 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.898170    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 13:21:32.899094 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.898173    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 13:21:32.899094 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.898177    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 13:21:32.899094 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.898181    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 13:21:32.899094 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.898185    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 13:21:32.899583 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.898188    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 13:21:32.899583 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.898191    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 13:21:32.899583 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.898194    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 13:21:32.899583 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.898196    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 13:21:32.899583 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.898199    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 13:21:32.899583 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.898203    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 13:21:32.899583 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.898206    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 13:21:32.899583 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.898208    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 13:21:32.899583 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.898211    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 13:21:32.899583 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.898215    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 13:21:32.899583 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.898218    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 13:21:32.899583 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.898220    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 13:21:32.899583 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.898223    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 13:21:32.899583 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.898226    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 13:21:32.899583 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.898229    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 13:21:32.899583 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.898231    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 13:21:32.899583 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.898234    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 13:21:32.899583 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.898237    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 13:21:32.899583 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.898239    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 13:21:32.899583 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.898242    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 13:21:32.900104 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.898244    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 13:21:32.900104 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.898247    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 13:21:32.900104 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.898250    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 13:21:32.900104 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.898252    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 13:21:32.900104 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.898256    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 13:21:32.900104 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.898260    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 13:21:32.900104 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.898263    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 13:21:32.900104 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.898267    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 13:21:32.900104 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.898270    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 13:21:32.900104 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.898272    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 13:21:32.900104 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.898275    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 13:21:32.900104 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.898278    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 13:21:32.900104 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.898281    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 13:21:32.900104 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.898284    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 13:21:32.900104 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.898286    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 13:21:32.900104 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.898289    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 13:21:32.900104 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.898293    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 13:21:32.900104 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.899102    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 13:21:32.900104 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.899110    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 13:21:32.900561 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.899113    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 13:21:32.900561 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.899116    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 13:21:32.900561 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.899119    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 13:21:32.900561 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.899122    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 13:21:32.900561 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.899125    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 13:21:32.900561 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.899128    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 13:21:32.900561 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.899131    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 13:21:32.900561 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.899134    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 13:21:32.900561 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.899137    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 13:21:32.900561 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.899139    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 13:21:32.900561 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.899142    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 13:21:32.900561 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.899145    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 13:21:32.900561 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.899147    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 13:21:32.900561 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.899150    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 13:21:32.900561 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.899153    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 13:21:32.900561 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.899155    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 13:21:32.900561 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.899158    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 13:21:32.900561 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.899160    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 13:21:32.900561 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.899163    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 13:21:32.900561 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.899166    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 13:21:32.901138 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.899168    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 13:21:32.901138 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.899170    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 13:21:32.901138 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.899173    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 13:21:32.901138 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.899175    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 13:21:32.901138 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.899178    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 13:21:32.901138 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.899181    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 13:21:32.901138 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.899184    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 13:21:32.901138 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.899187    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 13:21:32.901138 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.899190    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 13:21:32.901138 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.899195    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 13:21:32.901138 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.899198    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 13:21:32.901138 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.899201    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 13:21:32.901138 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.899203    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 13:21:32.901138 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.899206    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 13:21:32.901138 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.899208    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 13:21:32.901138 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.899211    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 13:21:32.901138 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.899214    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 13:21:32.901138 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.899216    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 13:21:32.901138 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.899219    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 13:21:32.901633 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.899222    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 13:21:32.901633 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.899224    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 13:21:32.901633 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.899227    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 13:21:32.901633 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.899229    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 13:21:32.901633 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.899232    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 13:21:32.901633 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.899235    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 13:21:32.901633 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.899237    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 13:21:32.901633 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.899240    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 13:21:32.901633 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.899242    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 13:21:32.901633 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.899245    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 13:21:32.901633 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.899248    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 13:21:32.901633 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.899250    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 13:21:32.901633 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.899252    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 13:21:32.901633 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.899255    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 13:21:32.901633 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.899257    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 13:21:32.901633 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.899261    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 13:21:32.901633 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.899263    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 13:21:32.901633 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.899266    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 13:21:32.901633 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.899271    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 13:21:32.902164 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.899276    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 13:21:32.902164 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.899279    2578 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 13:21:32.902164 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.899282    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 13:21:32.902164 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.899287    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 13:21:32.902164 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.899290    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 13:21:32.902164 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.899293    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 13:21:32.902164 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.899295    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 13:21:32.902164 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.899298    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 13:21:32.902164 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.899301    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 13:21:32.902164 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.899303    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 13:21:32.902164 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.899306    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 13:21:32.902164 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.899309    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 13:21:32.902164 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.899312    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 13:21:32.902164 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.899315    2578 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 13:21:32.902164 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.899318    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 13:21:32.902164 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.899320    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 13:21:32.902164 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.899323    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 13:21:32.902164 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.899325    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 13:21:32.902164 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.899328    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 13:21:32.902164 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.899332    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 13:21:32.902660 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.899335    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 13:21:32.902660 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.899338    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 13:21:32.902660 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.899340    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 13:21:32.902660 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.899343    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 13:21:32.902660 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.899345    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 13:21:32.902660 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.899348    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 13:21:32.902660 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.899350    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 13:21:32.902660 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.899352    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 13:21:32.902660 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.899356    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 13:21:32.902660 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.899358    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 13:21:32.902660 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.899361    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 13:21:32.902660 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.899364    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 13:21:32.902660 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.899367    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 13:21:32.902660 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.899371    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 13:21:32.902660 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.899375    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 13:21:32.902660 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.899380    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 13:21:32.902660 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.899383    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 13:21:32.902660 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.899386    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 13:21:32.902660 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.899388    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 13:21:32.902660 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.899842    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 13:21:32.903188 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.899848    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 13:21:32.903188 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.899851    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 13:21:32.903188 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.899854    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 13:21:32.903188 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.899858    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 13:21:32.903188 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.899861    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 13:21:32.903188 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.899863    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 13:21:32.903188 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.899866    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 13:21:32.903188 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.899869    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 13:21:32.903188 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.899872    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 13:21:32.903188 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.899874    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 13:21:32.903188 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.899877    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 13:21:32.903188 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.899879    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 13:21:32.903188 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.899882    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 13:21:32.903188 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.899885    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 13:21:32.903188 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.899888    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 13:21:32.903188 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.899890    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 13:21:32.903188 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.899893    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 13:21:32.903188 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.899895    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 13:21:32.903188 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.899897    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 13:21:32.903188 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.899900    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 13:21:32.903723 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.899904    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 13:21:32.903723 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.899907    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 13:21:32.903723 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.899909    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 13:21:32.903723 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.899913    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 13:21:32.903723 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.899915    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 13:21:32.903723 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.899918    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 13:21:32.903723 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.899920    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 13:21:32.903723 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.899923    2578 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 13:21:32.903723 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.899926    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 13:21:32.903723 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.899928    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 13:21:32.903723 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.899930    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 13:21:32.903723 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.899933    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 13:21:32.903723 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.899935    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 13:21:32.903723 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.899938    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 13:21:32.903723 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.899940    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 13:21:32.903723 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.899944    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 13:21:32.903723 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.899948    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 13:21:32.903723 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.899950    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 13:21:32.903723 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.899952    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 13:21:32.903723 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.899955    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 13:21:32.904237 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.899957    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 13:21:32.904237 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.899959    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 13:21:32.904237 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.899962    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 13:21:32.904237 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.899964    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 13:21:32.904237 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.899967    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 13:21:32.904237 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.899969    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 13:21:32.904237 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.899971    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 13:21:32.904237 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.899974    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 13:21:32.904237 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.899977    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 13:21:32.904237 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.899979    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 13:21:32.904237 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.899982    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 13:21:32.904237 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.899984    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 13:21:32.904237 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.899987    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 13:21:32.904237 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.899989    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 13:21:32.904237 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.899992    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 13:21:32.904237 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.899995    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 13:21:32.904237 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.899997    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 13:21:32.904237 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.900002    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 13:21:32.904683 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.900005    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 13:21:32.904683 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.900009    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 13:21:32.904683 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.900012    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 13:21:32.904683 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.900015    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 13:21:32.904683 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.900017    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 13:21:32.904683 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.900020    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 13:21:32.904683 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.900023    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 13:21:32.904683 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.900025    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 13:21:32.904683 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.900028    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 13:21:32.904683 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.900032    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 13:21:32.904683 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.900034    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 13:21:32.904683 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.900038    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 13:21:32.904683 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.900042    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 13:21:32.904683 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.900044    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 13:21:32.904683 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.900047    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 13:21:32.904683 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.900051    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 13:21:32.904683 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.900053    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 13:21:32.904683 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.900056    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 13:21:32.904683 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.900059    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 13:21:32.904683 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.900061    2578 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 13:21:32.905271 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.900064    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 13:21:32.905271 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.900067    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 13:21:32.905271 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.900069    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 13:21:32.905271 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.900072    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 13:21:32.905271 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.900074    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 13:21:32.905271 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.900076    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 13:21:32.905271 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.900079    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 13:21:32.905271 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.900081    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 13:21:32.905271 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.900084    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 13:21:32.905271 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.900086    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 13:21:32.905271 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.900089    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 13:21:32.905271 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.900092    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 13:21:32.905271 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.900094    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 13:21:32.905271 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.900097    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 13:21:32.905271 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.900100    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 13:21:32.905271 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.900102    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 13:21:32.905271 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.900105    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 13:21:32.905271 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.900108    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 13:21:32.905271 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.900110    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 13:21:32.905271 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.900113    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 13:21:32.905774 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.900209    2578 flags.go:64] FLAG: --address="0.0.0.0"
Sep 11 13:21:32.905774 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.900249    2578 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 11 13:21:32.905774 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.900257    2578 flags.go:64] FLAG: --anonymous-auth="true"
Sep 11 13:21:32.905774 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.900262    2578 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 11 13:21:32.905774 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.900267    2578 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 11 13:21:32.905774 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.900271    2578 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 11 13:21:32.905774 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.900277    2578 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 11 13:21:32.905774 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.900282    2578 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 11 13:21:32.905774 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.900286    2578 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 11 13:21:32.905774 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.900289    2578 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 11 13:21:32.905774 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.900293    2578 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 11 13:21:32.905774 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.900297    2578 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 11 13:21:32.905774 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.900300    2578 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 11 13:21:32.905774 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.900304    2578 flags.go:64] FLAG: --cgroup-root=""
Sep 11 13:21:32.905774 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.900307    2578 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 11 13:21:32.905774 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.900310    2578 flags.go:64] FLAG: --client-ca-file=""
Sep 11 13:21:32.905774 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.900313    2578 flags.go:64] FLAG: --cloud-provider="external"
Sep 11 13:21:32.905774 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.900316    2578 flags.go:64] FLAG: --cluster-dns="[]"
Sep 11 13:21:32.905774 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.900322    2578 flags.go:64] FLAG: --cluster-domain=""
Sep 11 13:21:32.905774 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.900325    2578 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 11 13:21:32.905774 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.900328    2578 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 11 13:21:32.905774 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.900332    2578 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 11 13:21:32.905774 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.900336    2578 flags.go:64] FLAG: --container-log-max-files="5"
Sep 11 13:21:32.905774 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.900340    2578 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 11 13:21:32.906404 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.900343    2578 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 11 13:21:32.906404 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.900346    2578 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 11 13:21:32.906404 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.900350    2578 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 11 13:21:32.906404 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.900353    2578 flags.go:64] FLAG: --contention-profiling="false"
Sep 11 13:21:32.906404 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.900356    2578 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 11 13:21:32.906404 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.900359    2578 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 11 13:21:32.906404 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.900362    2578 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 11 13:21:32.906404 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.900365    2578 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 11 13:21:32.906404 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.900369    2578 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 11 13:21:32.906404 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.900372    2578 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 11 13:21:32.906404 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.900375    2578 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 11 13:21:32.906404 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.900378    2578 flags.go:64] FLAG: --enable-load-reader="false"
Sep 11 13:21:32.906404 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.900381    2578 flags.go:64] FLAG: --enable-server="true"
Sep 11 13:21:32.906404 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.900385    2578 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 11 13:21:32.906404 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.900389    2578 flags.go:64] FLAG: --event-burst="100"
Sep 11 13:21:32.906404 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.900393    2578 flags.go:64] FLAG: --event-qps="50"
Sep 11 13:21:32.906404 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.900395    2578 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 11 13:21:32.906404 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.900401    2578 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 11 13:21:32.906404 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.900404    2578 flags.go:64] FLAG: --eviction-hard=""
Sep 11 13:21:32.906404 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.900408    2578 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 11 13:21:32.906404 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.900411    2578 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 11 13:21:32.906404 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.900414    2578 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 11 13:21:32.906404 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.900417    2578 flags.go:64] FLAG: --eviction-soft=""
Sep 11 13:21:32.906404 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.900420    2578 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 11 13:21:32.906404 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.900423    2578 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 11 13:21:32.907105 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.900426    2578 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 11 13:21:32.907105 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.900429    2578 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 11 13:21:32.907105 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.900432    2578 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 11 13:21:32.907105 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.900437    2578 flags.go:64] FLAG: --fail-swap-on="true"
Sep 11 13:21:32.907105 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.900440    2578 flags.go:64] FLAG: --feature-gates=""
Sep 11 13:21:32.907105 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.900444    2578 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 11 13:21:32.907105 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.900447    2578 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 11 13:21:32.907105 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.900450    2578 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 11 13:21:32.907105 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.900454    2578 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 11 13:21:32.907105 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.900457    2578 flags.go:64] FLAG: --healthz-port="10248"
Sep 11 13:21:32.907105 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.900460    2578 flags.go:64] FLAG: --help="false"
Sep 11 13:21:32.907105 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.900463    2578 flags.go:64] FLAG: --hostname-override="ip-10-0-131-86.ec2.internal"
Sep 11 13:21:32.907105 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.900467    2578 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 11 13:21:32.907105 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.900470    2578 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 11 13:21:32.907105 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.900473    2578 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 11 13:21:32.907105 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.900477    2578 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 11 13:21:32.907105 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.900480    2578 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 11 13:21:32.907105 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.900483    2578 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 11 13:21:32.907105 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.900486    2578 flags.go:64] FLAG: --image-service-endpoint=""
Sep 11 13:21:32.907105 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.900489    2578 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 11 13:21:32.907105 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.900492    2578 flags.go:64] FLAG: --kube-api-burst="100"
Sep 11 13:21:32.907105 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.900496    2578 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 11 13:21:32.907105 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.900499    2578 flags.go:64] FLAG: --kube-api-qps="50"
Sep 11 13:21:32.907105 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.900502    2578 flags.go:64] FLAG: --kube-reserved=""
Sep 11 13:21:32.907734 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.900507    2578 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 11 13:21:32.907734 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.900510    2578 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 11 13:21:32.907734 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.900513    2578 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 11 13:21:32.907734 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.900516    2578 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 11 13:21:32.907734 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.900519    2578 flags.go:64] FLAG: --lock-file=""
Sep 11 13:21:32.907734 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.900521    2578 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 11 13:21:32.907734 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.900524    2578 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 11 13:21:32.907734 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.900528    2578 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 11 13:21:32.907734 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.900533    2578 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 11 13:21:32.907734 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.900536    2578 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 11 13:21:32.907734 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.900539    2578 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 11 13:21:32.907734 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.900544    2578 flags.go:64] FLAG: --logging-format="text"
Sep 11 13:21:32.907734 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.900547    2578 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 11 13:21:32.907734 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.900551    2578 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 11 13:21:32.907734 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.900554    2578 flags.go:64] FLAG: --manifest-url=""
Sep 11 13:21:32.907734 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.900556    2578 flags.go:64] FLAG: --manifest-url-header=""
Sep 11 13:21:32.907734 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.900561    2578 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 11 13:21:32.907734 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.900565    2578 flags.go:64] FLAG: --max-open-files="1000000"
Sep 11 13:21:32.907734 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.900569    2578 flags.go:64] FLAG: --max-pods="110"
Sep 11 13:21:32.907734 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.900572    2578 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 11 13:21:32.907734 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.900576    2578 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 11 13:21:32.907734 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.900579    2578 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 11 13:21:32.907734 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.900582    2578 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 11 13:21:32.907734 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.900585    2578 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 11 13:21:32.907734 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.900588    2578 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 11 13:21:32.908396 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.900591    2578 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 11 13:21:32.908396 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.900598    2578 flags.go:64] FLAG: --node-status-max-images="50"
Sep 11 13:21:32.908396 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.900601    2578 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 11 13:21:32.908396 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.900604    2578 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 11 13:21:32.908396 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.900607    2578 flags.go:64] FLAG: --pod-cidr=""
Sep 11 13:21:32.908396 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.900615    2578 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 11 13:21:32.908396 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.900622    2578 flags.go:64] FLAG: --pod-manifest-path=""
Sep 11 13:21:32.908396 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.900626    2578 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 11 13:21:32.908396 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.900630    2578 flags.go:64] FLAG: --pods-per-core="0"
Sep 11 13:21:32.908396 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.900633    2578 flags.go:64] FLAG: --port="10250"
Sep 11 13:21:32.908396 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.900636    2578 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 11 13:21:32.908396 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.900639    2578 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0582d2de7739b11bd"
Sep 11 13:21:32.908396 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.900642    2578 flags.go:64] FLAG: --qos-reserved=""
Sep 11 13:21:32.908396 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.900645    2578 flags.go:64] FLAG: --read-only-port="10255"
Sep 11 13:21:32.908396 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.900648    2578 flags.go:64] FLAG: --register-node="true"
Sep 11 13:21:32.908396 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.900651    2578 flags.go:64] FLAG: --register-with-taints=""
Sep 11 13:21:32.908396 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.900655    2578 flags.go:64] FLAG: --registry-burst="10"
Sep 11 13:21:32.908396 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.900658    2578 flags.go:64] FLAG: --registry-qps="5"
Sep 11 13:21:32.908396 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.900660    2578 flags.go:64] FLAG: --reserved-cpus=""
Sep 11 13:21:32.908396 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.900665    2578 flags.go:64] FLAG: --reserved-memory=""
Sep 11 13:21:32.908396 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.900669    2578 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 11 13:21:32.908396 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.900672    2578 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 11 13:21:32.908396 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.900675    2578 flags.go:64] FLAG: --rotate-certificates="false"
Sep 11 13:21:32.908396 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.900678    2578 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 11 13:21:32.908396 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.900681    2578 flags.go:64] FLAG: --runonce="false"
Sep 11 13:21:32.909077 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.900683    2578 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 11 13:21:32.909077 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.900686    2578 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 11 13:21:32.909077 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.900689    2578 flags.go:64] FLAG: --seccomp-default="false"
Sep 11 13:21:32.909077 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.900692    2578 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 11 13:21:32.909077 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.900695    2578 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 11 13:21:32.909077 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.900698    2578 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 11 13:21:32.909077 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.900701    2578 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 11 13:21:32.909077 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.900704    2578 flags.go:64] FLAG: --storage-driver-password="root"
Sep 11 13:21:32.909077 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.900707    2578 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 11 13:21:32.909077 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.900709    2578 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 11 13:21:32.909077 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.900712    2578 flags.go:64] FLAG: --storage-driver-user="root"
Sep 11 13:21:32.909077 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.900715    2578 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 11 13:21:32.909077 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.900718    2578 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 11 13:21:32.909077 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.900721    2578 flags.go:64] FLAG: --system-cgroups=""
Sep 11 13:21:32.909077 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.900726    2578 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 11 13:21:32.909077 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.900731    2578 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 11 13:21:32.909077 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.900734    2578 flags.go:64] FLAG: --tls-cert-file=""
Sep 11 13:21:32.909077 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.900737    2578 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 11 13:21:32.909077 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.900742    2578 flags.go:64] FLAG: --tls-min-version=""
Sep 11 13:21:32.909077 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.900744    2578 flags.go:64] FLAG: --tls-private-key-file=""
Sep 11 13:21:32.909077 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.900747    2578 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 11 13:21:32.909077 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.900750    2578 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 11 13:21:32.909077 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.900753    2578 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 11 13:21:32.909077 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.900756    2578 flags.go:64] FLAG: --v="2"
Sep 11 13:21:32.909077 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.900760    2578 flags.go:64] FLAG: --version="false"
Sep 11 13:21:32.909767 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.900764    2578 flags.go:64] FLAG: --vmodule=""
Sep 11 13:21:32.909767 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.900770    2578 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 11 13:21:32.909767 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.900773    2578 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 11 13:21:32.909767 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.900912    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 13:21:32.909767 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.900917    2578 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 13:21:32.909767 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.900920    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 13:21:32.909767 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.900923    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 13:21:32.909767 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.900926    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 13:21:32.909767 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.900929    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 13:21:32.909767 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.900931    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 13:21:32.909767 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.900934    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 13:21:32.909767 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.900936    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 13:21:32.909767 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.900939    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 13:21:32.909767 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.900941    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 13:21:32.909767 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.900944    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 13:21:32.909767 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.900947    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 13:21:32.909767 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.900950    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 13:21:32.909767 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.900952    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 13:21:32.909767 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.900956    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 13:21:32.909767 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.900960    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 13:21:32.909767 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.900964    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 13:21:32.910320 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.900971    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 13:21:32.910320 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.900975    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 13:21:32.910320 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.900978    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 13:21:32.910320 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.900982    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 13:21:32.910320 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.900986    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 13:21:32.910320 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.900989    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 13:21:32.910320 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.900992    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 13:21:32.910320 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.900995    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 13:21:32.910320 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.900998    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 13:21:32.910320 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.901002    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 13:21:32.910320 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.901005    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 13:21:32.910320 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.901008    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 13:21:32.910320 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.901013    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 13:21:32.910320 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.901016    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 13:21:32.910320 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.901019    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 13:21:32.910320 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.901022    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 13:21:32.910320 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.901025    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 13:21:32.910320 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.901027    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 13:21:32.910320 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.901030    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 13:21:32.910824 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.901033    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 13:21:32.910824 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.901036    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 13:21:32.910824 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.901039    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 13:21:32.910824 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.901041    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 13:21:32.910824 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.901044    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 13:21:32.910824 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.901046    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 13:21:32.910824 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.901049    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 13:21:32.910824 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.901052    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 13:21:32.910824 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.901055    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 13:21:32.910824 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.901057    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 13:21:32.910824 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.901060    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 13:21:32.910824 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.901063    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 13:21:32.910824 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.901065    2578 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 13:21:32.910824 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.901069    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 13:21:32.910824 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.901081    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 13:21:32.910824 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.901084    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 13:21:32.910824 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.901087    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 13:21:32.910824 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.901089    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 13:21:32.910824 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.901092    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 13:21:32.910824 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.901094    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 13:21:32.911489 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.901097    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 13:21:32.911489 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.901099    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 13:21:32.911489 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.901102    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 13:21:32.911489 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.901104    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 13:21:32.911489 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.901107    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 13:21:32.911489 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.901111    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 13:21:32.911489 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.901114    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 13:21:32.911489 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.901118    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 13:21:32.911489 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.901121    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 13:21:32.911489 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.901124    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 13:21:32.911489 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.901127    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 13:21:32.911489 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.901130    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 13:21:32.911489 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.901132    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 13:21:32.911489 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.901135    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 13:21:32.911489 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.901139    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 13:21:32.911489 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.901143    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 13:21:32.911489 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.901146    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 13:21:32.911489 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.901149    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 13:21:32.911489 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.901151    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 13:21:32.911995 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.901155    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 13:21:32.911995 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.901157    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 13:21:32.911995 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.901160    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 13:21:32.911995 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.901162    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 13:21:32.911995 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.901165    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 13:21:32.911995 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.901167    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 13:21:32.911995 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.901171    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 13:21:32.911995 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.901174    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 13:21:32.911995 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.901194    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 13:21:32.911995 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.901199    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 13:21:32.911995 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.901202    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 13:21:32.911995 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.901204    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 13:21:32.911995 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.901207    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 13:21:32.911995 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.901209    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 13:21:32.911995 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.901212    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 13:21:32.911995 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.901214    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 13:21:32.911995 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.901217    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 13:21:32.911995 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.901220    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 13:21:32.911995 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.901224    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 13:21:32.911995 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.901227    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 13:21:32.912499 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.901229    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 13:21:32.912499 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.901231    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 13:21:32.912499 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.901234    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 13:21:32.912499 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.901786    2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 11 13:21:32.912499 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.911142    2578 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 11 13:21:32.912499 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.911170    2578 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 11 13:21:32.912499 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.911197    2578 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 11 13:21:32.912499 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.911208    2578 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 11 13:21:32.912499 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.911291    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 13:21:32.912499 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.911297    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 13:21:32.912499 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.911300    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 13:21:32.912499 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.911304    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 13:21:32.912499 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.911307    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 13:21:32.912499 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.911310    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 13:21:32.912499 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.911313    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 13:21:32.912920 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.911316    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 13:21:32.912920 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.911319    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 13:21:32.912920 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.911321    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 13:21:32.912920 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.911324    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 13:21:32.912920 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.911328    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 13:21:32.912920 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.911331    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 13:21:32.912920 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.911334    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 13:21:32.912920 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.911337    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 13:21:32.912920 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.911339    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 13:21:32.912920 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.911342    2578 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 13:21:32.912920 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.911345    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 13:21:32.912920 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.911348    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 13:21:32.912920 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.911351    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 13:21:32.912920 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.911354    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 13:21:32.912920 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.911356    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 13:21:32.912920 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.911359    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 13:21:32.912920 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.911362    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 13:21:32.912920 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.911364    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 13:21:32.912920 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.911367    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 13:21:32.912920 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.911370    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 13:21:32.913438 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.911372    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 13:21:32.913438 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.911375    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 13:21:32.913438 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.911382    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 13:21:32.913438 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.911385    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 13:21:32.913438 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.911390    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 13:21:32.913438 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.911393    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 13:21:32.913438 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.911395    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 13:21:32.913438 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.911398    2578 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 13:21:32.913438 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.911401    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 13:21:32.913438 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.911404    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 13:21:32.913438 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.911409    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 13:21:32.913438 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.911413    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 13:21:32.913438 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.911416    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 13:21:32.913438 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.911419    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 13:21:32.913438 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.911422    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 13:21:32.913438 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.911425    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 13:21:32.913438 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.911427    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 13:21:32.913438 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.911430    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 13:21:32.913438 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.911433    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 13:21:32.914131 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.911436    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 13:21:32.914131 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.911439    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 13:21:32.914131 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.911442    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 13:21:32.914131 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.911444    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 13:21:32.914131 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.911447    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 13:21:32.914131 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.911450    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 13:21:32.914131 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.911453    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 13:21:32.914131 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.911455    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 13:21:32.914131 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.911458    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 13:21:32.914131 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.911460    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 13:21:32.914131 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.911463    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 13:21:32.914131 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.911466    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 13:21:32.914131 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.911468    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 13:21:32.914131 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.911471    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 13:21:32.914131 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.911473    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 13:21:32.914131 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.911478    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 13:21:32.914131 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.911481    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 13:21:32.914131 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.911484    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 13:21:32.914131 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.911486    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 13:21:32.914131 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.911490    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 13:21:32.914620 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.911495    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 13:21:32.914620 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.911498    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 13:21:32.914620 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.911501    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 13:21:32.914620 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.911504    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 13:21:32.914620 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.911507    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 13:21:32.914620 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.911510    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 13:21:32.914620 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.911514    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 13:21:32.914620 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.911518    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 13:21:32.914620 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.911522    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 13:21:32.914620 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.911524    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 13:21:32.914620 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.911527    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 13:21:32.914620 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.911529    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 13:21:32.914620 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.911532    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 13:21:32.914620 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.911536    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 13:21:32.914620 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.911539    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 13:21:32.914620 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.911541    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 13:21:32.914620 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.911544    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 13:21:32.914620 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.911546    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 13:21:32.914620 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.911549    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 13:21:32.915204 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.911551    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 13:21:32.915204 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.911553    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 13:21:32.915204 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.911556    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 13:21:32.915204 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.911558    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 13:21:32.915204 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.911561    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 13:21:32.915204 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.911564    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 13:21:32.915204 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.911566    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 13:21:32.915204 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.911569    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 13:21:32.915204 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.911572    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 13:21:32.915204 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.911575    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 13:21:32.915204 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.911578    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 13:21:32.915204 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.911581    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 13:21:32.915204 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.911583    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 13:21:32.915204 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.911586    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 13:21:32.915583 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.911591    2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 11 13:21:32.915583 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.911764    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 13:21:32.915583 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.911770    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 13:21:32.915583 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.911773    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 13:21:32.915583 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.911776    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 13:21:32.915583 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.911778    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 13:21:32.915583 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.911781    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 13:21:32.915583 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.911784    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 13:21:32.915583 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.911787    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 13:21:32.915583 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.911790    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 13:21:32.915583 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.911792    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 13:21:32.915583 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.911795    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 13:21:32.915583 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.911798    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 13:21:32.915583 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.911800    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 13:21:32.915973 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.911803    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 13:21:32.915973 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.911806    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 13:21:32.915973 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.911809    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 13:21:32.915973 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.911811    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 13:21:32.915973 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.911814    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 13:21:32.915973 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.911816    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 13:21:32.915973 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.911819    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 13:21:32.915973 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.911822    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 13:21:32.915973 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.911824    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 13:21:32.915973 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.911841    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 13:21:32.915973 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.911844    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 13:21:32.915973 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.911848    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 13:21:32.915973 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.911852    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 13:21:32.915973 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.911855    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 13:21:32.915973 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.911858    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 13:21:32.915973 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.911862    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 13:21:32.915973 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.911865    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 13:21:32.915973 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.911868    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 13:21:32.915973 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.911870    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 13:21:32.915973 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.911873    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 13:21:32.916594 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.911876    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 13:21:32.916594 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.911878    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 13:21:32.916594 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.911881    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 13:21:32.916594 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.911883    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 13:21:32.916594 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.911886    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 13:21:32.916594 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.911888    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 13:21:32.916594 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.911891    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 13:21:32.916594 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.911894    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 13:21:32.916594 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.911897    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 13:21:32.916594 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.911899    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 13:21:32.916594 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.911902    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 13:21:32.916594 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.911904    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 13:21:32.916594 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.911906    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 13:21:32.916594 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.911910    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 13:21:32.916594 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.911912    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 13:21:32.916594 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.911916    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 13:21:32.916594 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.911921    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 13:21:32.916594 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.911924    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 13:21:32.916594 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.911927    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 13:21:32.917145 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.911930    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 13:21:32.917145 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.911933    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 13:21:32.917145 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.911936    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 13:21:32.917145 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.911939    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 13:21:32.917145 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.911942    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 13:21:32.917145 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.911945    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 13:21:32.917145 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.911947    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 13:21:32.917145 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.911950    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 13:21:32.917145 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.911953    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 13:21:32.917145 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.911957    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 13:21:32.917145 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.911961    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 13:21:32.917145 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.911964    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 13:21:32.917145 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.911967    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 13:21:32.917145 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.911970    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 13:21:32.917145 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.911973    2578 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 13:21:32.917145 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.911975    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 13:21:32.917145 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.911978    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 13:21:32.917145 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.911981    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 13:21:32.917145 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.911984    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 13:21:32.917916 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.911987    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 13:21:32.917916 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.911989    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 13:21:32.917916 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.911992    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 13:21:32.917916 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.911994    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 13:21:32.917916 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.911998    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 13:21:32.917916 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.912000    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 13:21:32.917916 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.912003    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 13:21:32.917916 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.912006    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 13:21:32.917916 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.912009    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 13:21:32.917916 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.912011    2578 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 13:21:32.917916 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.912014    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 13:21:32.917916 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.912016    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 13:21:32.917916 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.912019    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 13:21:32.917916 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.912022    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 13:21:32.917916 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.912025    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 13:21:32.917916 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.912028    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 13:21:32.917916 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.912030    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 13:21:32.917916 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.912033    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 13:21:32.917916 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.912036    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 13:21:32.917916 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.912039    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 13:21:32.918680 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.912042    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 13:21:32.918680 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.912046    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 13:21:32.918680 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.912049    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 13:21:32.918680 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.912053    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 13:21:32.918680 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.912056    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 13:21:32.918680 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.912059    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 13:21:32.918680 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.912062    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 13:21:32.918680 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:32.912065    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 13:21:32.918680 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.912070    2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 11 13:21:32.918680 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.912252    2578 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 11 13:21:32.918680 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.914866    2578 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 11 13:21:32.918680 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.916063    2578 server.go:1019] "Starting client certificate rotation"
Sep 11 13:21:32.918680 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.916205    2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 11 13:21:32.919175 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.916266    2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 11 13:21:32.919631 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.919610    2578 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 11 13:21:32.925440 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.925407    2578 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 11 13:21:32.938167 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.938136    2578 log.go:25] "Validated CRI v1 runtime API"
Sep 11 13:21:32.941819 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.941791    2578 log.go:25] "Validated CRI v1 image API"
Sep 11 13:21:32.943011 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.942988    2578 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 11 13:21:32.947657 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.947620    2578 fs.go:135] Filesystem UUIDs: map[6a5ef070-80c6-4492-957f-9c93f9ad77f3:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 ea32cb62-2d08-4f4e-9ca8-a8f1edefaff7:/dev/nvme0n1p4]
Sep 11 13:21:32.947657 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.947653    2578 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 11 13:21:32.953492 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.953318    2578 manager.go:216] Machine: {Timestamp:2026-09-11 13:21:32.951771597 +0000 UTC m=+0.337874771 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3095664 MemoryCapacity:32812150784 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec23d0c616d422ed60dd48be481c256e SystemUUID:ec23d0c6-16d4-22ed-60dd-48be481c256e BootID:c779c6a9-94cf-4399-a401-3abf7cce5549 Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406073344 Type:vfs Inodes:4005389 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562430976 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406077440 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:5e:22:c6:fb:f9 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:5e:22:c6:fb:f9 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:0e:20:a7:4d:89:cf Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812150784 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 11 13:21:32.953492 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.953480    2578 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 11 13:21:32.953636 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.953613    2578 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 11 13:21:32.957392 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.957347    2578 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 11 13:21:32.957600 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.957400    2578 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-131-86.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 11 13:21:32.958324 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.958313    2578 topology_manager.go:138] "Creating topology manager with none policy"
Sep 11 13:21:32.958366 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.958329    2578 container_manager_linux.go:309] "Creating device plugin manager"
Sep 11 13:21:32.958366 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.958339    2578 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 11 13:21:32.959911 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.959897    2578 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 11 13:21:32.961101 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.961088    2578 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 11 13:21:32.961862 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.961802    2578 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 11 13:21:32.964868 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.964852    2578 state_mem.go:36] "Initialized new in-memory state store"
Sep 11 13:21:32.965038 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.965028    2578 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 11 13:21:32.965074 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.965067    2578 kubelet.go:486] "Attempting to sync node with API server"
Sep 11 13:21:32.965102 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.965085    2578 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 11 13:21:32.965133 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.965103    2578 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 11 13:21:32.965133 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.965116    2578 kubelet.go:398] "Adding apiserver pod source"
Sep 11 13:21:32.965133 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.965130    2578 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 11 13:21:32.965753 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.965737    2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 11 13:21:32.965858 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.965797    2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 11 13:21:32.968530 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.968501    2578 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 11 13:21:32.971804 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.971783    2578 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 11 13:21:32.971945 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.971824    2578 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 11 13:21:32.973874 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.973856    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 11 13:21:32.973986 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.973883    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 11 13:21:32.973986 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.973894    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 11 13:21:32.973986 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.973902    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 11 13:21:32.973986 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.973917    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 11 13:21:32.973986 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.973929    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 11 13:21:32.973986 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.973954    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 11 13:21:32.973986 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.973962    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 11 13:21:32.973986 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.973975    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 11 13:21:32.973986 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.973989    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 11 13:21:32.974257 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.974006    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 11 13:21:32.974257 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.974018    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 11 13:21:32.974891 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.974878    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 11 13:21:32.974949 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.974895    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 11 13:21:32.978119 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.978072    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-131-86.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 11 13:21:32.979283 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:21:32.979232    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 11 13:21:32.979397 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:21:32.979350    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-131-86.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 11 13:21:32.980857 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.980807    2578 server.go:1268] "Started kubelet"
Sep 11 13:21:32.980972 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.980887    2578 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 11 13:21:32.981034 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.980989    2578 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 11 13:21:32.981079 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.981062    2578 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 11 13:21:32.982091 ip-10-0-131-86 systemd[1]: Started Kubernetes Kubelet.
Sep 11 13:21:32.983630 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.983373    2578 server.go:310] "Adding debug handlers to kubelet server"
Sep 11 13:21:32.985845 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.985806    2578 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 11 13:21:32.985945 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:21:32.985891    2578 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 11 13:21:32.991877 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.991855    2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 11 13:21:32.992560 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.992543    2578 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 11 13:21:32.992663 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:21:32.992646    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-86.ec2.internal\" not found"
Sep 11 13:21:32.992757 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.992738    2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-8n7cn"
Sep 11 13:21:32.993386 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.993358    2578 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 11 13:21:32.993481 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.993391    2578 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 11 13:21:32.993481 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.993357    2578 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 11 13:21:32.993575 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.993487    2578 reconstruct.go:97] "Volume reconstruction finished"
Sep 11 13:21:32.993575 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.993494    2578 reconciler.go:29] "Reconciler: start to sync state"
Sep 11 13:21:32.993755 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.993742    2578 factory.go:55] Registering systemd factory
Sep 11 13:21:32.993808 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.993759    2578 factory.go:223] Registration of the systemd container factory successfully
Sep 11 13:21:32.994056 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.994039    2578 factory.go:153] Registering CRI-O factory
Sep 11 13:21:32.994056 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.994056    2578 factory.go:223] Registration of the crio container factory successfully
Sep 11 13:21:32.994169 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.994108    2578 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 11 13:21:32.994169 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.994124    2578 factory.go:103] Registering Raw factory
Sep 11 13:21:32.994169 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.994137    2578 manager.go:1196] Started watching for new ooms in manager
Sep 11 13:21:32.994474 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:32.994463    2578 manager.go:349] Starting recovery of all containers
Sep 11 13:21:33.002744 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:33.002691    2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 11 13:21:33.002900 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:21:33.002871    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 11 13:21:33.003033 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:21:33.003006    2578 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-131-86.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 11 13:21:33.003926 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:21:33.002725    2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-86.ec2.internal.18d446e30e15edfc  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-86.ec2.internal,UID:ip-10-0-131-86.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-131-86.ec2.internal,},FirstTimestamp:2026-09-11 13:21:32.980768252 +0000 UTC m=+0.366871459,LastTimestamp:2026-09-11 13:21:32.980768252 +0000 UTC m=+0.366871459,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-86.ec2.internal,}"
Sep 11 13:21:33.004941 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:33.004850    2578 manager.go:354] Recovery completed
Sep 11 13:21:33.011276 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:33.011246    2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-8n7cn"
Sep 11 13:21:33.013757 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:33.013742    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 13:21:33.016402 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:33.016378    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-86.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 13:21:33.016504 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:33.016422    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-86.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 13:21:33.016504 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:33.016438    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-86.ec2.internal" event="NodeHasSufficientPID"
Sep 11 13:21:33.017249 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:33.017233    2578 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 11 13:21:33.017249 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:33.017246    2578 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 11 13:21:33.017397 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:33.017264    2578 state_mem.go:36] "Initialized new in-memory state store"
Sep 11 13:21:33.017397 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:33.017302    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 13:21:33.018199 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:33.018180    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-86.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 13:21:33.018283 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:33.018222    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-86.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 13:21:33.018283 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:33.018238    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-86.ec2.internal" event="NodeHasSufficientPID"
Sep 11 13:21:33.018410 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:33.018396    2578 policy_none.go:49] "None policy: Start"
Sep 11 13:21:33.018462 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:33.018416    2578 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 11 13:21:33.018462 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:33.018429    2578 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 11 13:21:33.019190 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:21:33.019095    2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-86.ec2.internal.18d446e31035b1a7  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-86.ec2.internal,UID:ip-10-0-131-86.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-131-86.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-131-86.ec2.internal,},FirstTimestamp:2026-09-11 13:21:33.016404391 +0000 UTC m=+0.402507547,LastTimestamp:2026-09-11 13:21:33.016404391 +0000 UTC m=+0.402507547,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-86.ec2.internal,}"
Sep 11 13:21:33.019648 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:33.019635    2578 policy_none.go:47] "Start"
Sep 11 13:21:33.053547 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:33.053525    2578 manager.go:337] "Starting Device Plugin manager"
Sep 11 13:21:33.067963 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:21:33.053576    2578 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 11 13:21:33.067963 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:33.053591    2578 server.go:87] "Starting device plugin registration server"
Sep 11 13:21:33.067963 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:33.054027    2578 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 11 13:21:33.067963 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:33.054039    2578 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 11 13:21:33.067963 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:33.054131    2578 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 11 13:21:33.067963 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:33.054219    2578 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 11 13:21:33.067963 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:33.054226    2578 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 11 13:21:33.067963 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:21:33.057056    2578 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 11 13:21:33.067963 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:21:33.057107    2578 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-131-86.ec2.internal\" not found"
Sep 11 13:21:33.099296 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:33.099256    2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 11 13:21:33.099491 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:33.099331    2578 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 11 13:21:33.099491 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:33.099368    2578 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 11 13:21:33.099491 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:21:33.099418    2578 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 11 13:21:33.104046 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:33.104005    2578 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 13:21:33.155251 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:33.155104    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 13:21:33.156884 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:33.156864    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-86.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 13:21:33.156981 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:33.156904    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-86.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 13:21:33.156981 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:33.156915    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-86.ec2.internal" event="NodeHasSufficientPID"
Sep 11 13:21:33.156981 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:33.156948    2578 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-131-86.ec2.internal"
Sep 11 13:21:33.168434 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:33.168403    2578 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-131-86.ec2.internal"
Sep 11 13:21:33.168502 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:21:33.168443    2578 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-131-86.ec2.internal\": node \"ip-10-0-131-86.ec2.internal\" not found"
Sep 11 13:21:33.199782 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:33.199731    2578 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-86.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-131-86.ec2.internal"]
Sep 11 13:21:33.199987 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:33.199869    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 13:21:33.201855 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:33.201814    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-86.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 13:21:33.202019 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:33.201875    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-86.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 13:21:33.202019 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:33.201887    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-86.ec2.internal" event="NodeHasSufficientPID"
Sep 11 13:21:33.203375 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:33.203356    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 13:21:33.203544 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:33.203523    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-86.ec2.internal"
Sep 11 13:21:33.203589 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:33.203573    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 13:21:33.204407 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:33.204384    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-86.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 13:21:33.204407 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:33.204413    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-86.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 13:21:33.204578 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:33.204422    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-86.ec2.internal" event="NodeHasSufficientPID"
Sep 11 13:21:33.204578 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:33.204385    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-86.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 13:21:33.204578 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:33.204450    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-86.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 13:21:33.204578 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:33.204465    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-86.ec2.internal" event="NodeHasSufficientPID"
Sep 11 13:21:33.206015 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:33.205991    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-86.ec2.internal"
Sep 11 13:21:33.206089 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:33.206027    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 13:21:33.206980 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:33.206961    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-86.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 13:21:33.207103 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:33.206997    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-86.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 13:21:33.207103 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:33.207008    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-86.ec2.internal" event="NodeHasSufficientPID"
Sep 11 13:21:33.219037 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:21:33.219003    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-86.ec2.internal\" not found"
Sep 11 13:21:33.238150 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:21:33.238125    2578 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-86.ec2.internal\" not found" node="ip-10-0-131-86.ec2.internal"
Sep 11 13:21:33.243613 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:21:33.243584    2578 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-86.ec2.internal\" not found" node="ip-10-0-131-86.ec2.internal"
Sep 11 13:21:33.319271 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:21:33.319228    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-86.ec2.internal\" not found"
Sep 11 13:21:33.394396 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:33.394347    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/76de799c7422d703cdded403fdfd9c70-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-86.ec2.internal\" (UID: \"76de799c7422d703cdded403fdfd9c70\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-86.ec2.internal"
Sep 11 13:21:33.394396 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:33.394390    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/76de799c7422d703cdded403fdfd9c70-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-86.ec2.internal\" (UID: \"76de799c7422d703cdded403fdfd9c70\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-86.ec2.internal"
Sep 11 13:21:33.394635 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:33.394411    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ac822255a7ab3b2619823b5af72a01ec-config\") pod \"kube-apiserver-proxy-ip-10-0-131-86.ec2.internal\" (UID: \"ac822255a7ab3b2619823b5af72a01ec\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-86.ec2.internal"
Sep 11 13:21:33.420383 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:21:33.420283    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-86.ec2.internal\" not found"
Sep 11 13:21:33.495079 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:33.495033    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/76de799c7422d703cdded403fdfd9c70-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-86.ec2.internal\" (UID: \"76de799c7422d703cdded403fdfd9c70\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-86.ec2.internal"
Sep 11 13:21:33.495079 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:33.495064    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/76de799c7422d703cdded403fdfd9c70-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-86.ec2.internal\" (UID: \"76de799c7422d703cdded403fdfd9c70\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-86.ec2.internal"
Sep 11 13:21:33.495372 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:33.495107    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/76de799c7422d703cdded403fdfd9c70-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-86.ec2.internal\" (UID: \"76de799c7422d703cdded403fdfd9c70\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-86.ec2.internal"
Sep 11 13:21:33.495372 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:33.495140    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ac822255a7ab3b2619823b5af72a01ec-config\") pod \"kube-apiserver-proxy-ip-10-0-131-86.ec2.internal\" (UID: \"ac822255a7ab3b2619823b5af72a01ec\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-86.ec2.internal"
Sep 11 13:21:33.495372 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:33.495155    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/76de799c7422d703cdded403fdfd9c70-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-86.ec2.internal\" (UID: \"76de799c7422d703cdded403fdfd9c70\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-86.ec2.internal"
Sep 11 13:21:33.495372 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:33.495177    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ac822255a7ab3b2619823b5af72a01ec-config\") pod \"kube-apiserver-proxy-ip-10-0-131-86.ec2.internal\" (UID: \"ac822255a7ab3b2619823b5af72a01ec\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-86.ec2.internal"
Sep 11 13:21:33.521048 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:21:33.521010    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-86.ec2.internal\" not found"
Sep 11 13:21:33.542236 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:33.542198    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-86.ec2.internal"
Sep 11 13:21:33.547154 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:33.546566    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-86.ec2.internal"
Sep 11 13:21:33.621822 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:21:33.621763    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-86.ec2.internal\" not found"
Sep 11 13:21:33.722410 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:21:33.722284    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-86.ec2.internal\" not found"
Sep 11 13:21:33.822879 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:21:33.822816    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-86.ec2.internal\" not found"
Sep 11 13:21:33.915453 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:33.915414    2578 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 11 13:21:33.916032 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:33.915607    2578 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 11 13:21:33.923709 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:21:33.923681    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-86.ec2.internal\" not found"
Sep 11 13:21:34.017005 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:34.016901    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-10 13:16:33 +0000 UTC" deadline="2028-05-19 00:54:19.37796917 +0000 UTC"
Sep 11 13:21:34.017005 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:34.016948    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14771h32m45.361024497s"
Sep 11 13:21:34.024306 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:21:34.024274    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-86.ec2.internal\" not found"
Sep 11 13:21:34.124631 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:21:34.124564    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-86.ec2.internal\" not found"
Sep 11 13:21:34.162194 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:34.162165    2578 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 13:21:34.174015 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:34.173987    2578 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 13:21:34.183201 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:34.183152    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac822255a7ab3b2619823b5af72a01ec.slice/crio-3365623e79628f5e7bae26be055d03aba10924a9f3e81db4cd781677d06c9040 WatchSource:0}: Error finding container 3365623e79628f5e7bae26be055d03aba10924a9f3e81db4cd781677d06c9040: Status 404 returned error can't find the container with id 3365623e79628f5e7bae26be055d03aba10924a9f3e81db4cd781677d06c9040
Sep 11 13:21:34.184068 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:34.184033    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76de799c7422d703cdded403fdfd9c70.slice/crio-1da064c293c97edad648465b2cb3c25496df5249b84c9d9b59a7518179c904e5 WatchSource:0}: Error finding container 1da064c293c97edad648465b2cb3c25496df5249b84c9d9b59a7518179c904e5: Status 404 returned error can't find the container with id 1da064c293c97edad648465b2cb3c25496df5249b84c9d9b59a7518179c904e5
Sep 11 13:21:34.189344 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:34.189320    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 13:21:34.193442 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:34.193416    2578 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-86.ec2.internal"
Sep 11 13:21:34.206588 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:34.206552    2578 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 11 13:21:34.207697 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:34.207673    2578 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-86.ec2.internal"
Sep 11 13:21:34.223853 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:34.223801    2578 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 11 13:21:34.397947 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:34.397816    2578 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 13:21:34.887552 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:34.887453    2578 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 13:21:34.967512 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:34.967464    2578 apiserver.go:52] "Watching apiserver"
Sep 11 13:21:34.980453 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:34.980402    2578 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 11 13:21:34.981857 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:34.981085    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-zb2qw","openshift-network-diagnostics/network-check-target-6frmx","kube-system/global-pull-secret-syncer-m6d9g","kube-system/konnectivity-agent-v874k","openshift-dns/node-resolver-z8p25","openshift-image-registry/node-ca-8jj8l","openshift-network-operator/iptables-alerter-k27nx","openshift-ovn-kubernetes/ovnkube-node-6gltj","kube-system/kube-apiserver-proxy-ip-10-0-131-86.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kp24z","openshift-cluster-node-tuning-operator/tuned-smdjz","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-86.ec2.internal","openshift-multus/multus-25zm6","openshift-multus/multus-additional-cni-plugins-qcfls"]
Sep 11 13:21:34.983378 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:34.983343    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-6gltj"
Sep 11 13:21:34.985587 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:34.985551    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6frmx"
Sep 11 13:21:34.985795 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:21:34.985671    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6frmx" podUID="1004285c-1dd5-49bf-be06-8549f85cdeda"
Sep 11 13:21:34.987820 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:34.987470    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-25zm6"
Sep 11 13:21:34.988225 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:34.988194    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 11 13:21:34.989028 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:34.988396    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 11 13:21:34.989028 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:34.988524    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 11 13:21:34.989028 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:34.988762    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 11 13:21:34.989028 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:34.988889    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 11 13:21:34.989309 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:34.989145    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-zdc6h\""
Sep 11 13:21:34.989309 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:34.989212    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 11 13:21:34.989420 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:34.989390    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-smdjz"
Sep 11 13:21:34.991346 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:34.990635    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-z8p25"
Sep 11 13:21:34.994368 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:34.992377    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-8jj8l"
Sep 11 13:21:34.994368 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:34.992976    2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 11 13:21:34.994368 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:34.993899    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kp24z"
Sep 11 13:21:34.996547 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:34.995676    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-v874k"
Sep 11 13:21:34.996547 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:34.995798    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-ptcgg\""
Sep 11 13:21:34.996547 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:34.995891    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 11 13:21:34.996547 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:34.996092    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 11 13:21:34.996547 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:34.996348    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 11 13:21:34.996547 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:34.996393    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 11 13:21:34.996547 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:34.996406    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 11 13:21:34.997055 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:34.996661    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-9q8ns\""
Sep 11 13:21:34.997055 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:34.996996    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-k27nx"
Sep 11 13:21:34.999216 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:34.998161    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zb2qw"
Sep 11 13:21:34.999216 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:21:34.998252    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zb2qw" podUID="a2354ca1-4b4e-4a17-a17e-52d61af0e80c"
Sep 11 13:21:34.999216 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:34.999119    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 11 13:21:34.999502 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:34.999262    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-x4j72\""
Sep 11 13:21:35.001144 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.000564    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m6d9g"
Sep 11 13:21:35.001144 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:21:35.000649    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m6d9g" podUID="be2cb948-420f-4f91-b56b-a292efb25181"
Sep 11 13:21:35.002629 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.001950    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-qcfls"
Sep 11 13:21:35.006943 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.003529    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/eeb9973c-b94d-4b62-8ce1-3cdf114016f2-host\") pod \"tuned-smdjz\" (UID: \"eeb9973c-b94d-4b62-8ce1-3cdf114016f2\") " pod="openshift-cluster-node-tuning-operator/tuned-smdjz"
Sep 11 13:21:35.006943 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.003570    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7ab88b5-7073-460e-b3d4-a9841f50b185-tmp-dir\") pod \"node-resolver-z8p25\" (UID: \"f7ab88b5-7073-460e-b3d4-a9841f50b185\") " pod="openshift-dns/node-resolver-z8p25"
Sep 11 13:21:35.006943 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.003603    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/86a1ce82-9c9a-4a59-8633-96606b53067a-socket-dir\") pod \"aws-ebs-csi-driver-node-kp24z\" (UID: \"86a1ce82-9c9a-4a59-8633-96606b53067a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kp24z"
Sep 11 13:21:35.006943 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.003633    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/86a1ce82-9c9a-4a59-8633-96606b53067a-sys-fs\") pod \"aws-ebs-csi-driver-node-kp24z\" (UID: \"86a1ce82-9c9a-4a59-8633-96606b53067a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kp24z"
Sep 11 13:21:35.006943 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.003661    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e384caae-3cb1-4264-8a25-55b40ac8ac45-log-socket\") pod \"ovnkube-node-6gltj\" (UID: \"e384caae-3cb1-4264-8a25-55b40ac8ac45\") " pod="openshift-ovn-kubernetes/ovnkube-node-6gltj"
Sep 11 13:21:35.006943 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.003688    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4925d62a-ba17-4740-84c4-c7cf29069831-hostroot\") pod \"multus-25zm6\" (UID: \"4925d62a-ba17-4740-84c4-c7cf29069831\") " pod="openshift-multus/multus-25zm6"
Sep 11 13:21:35.006943 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.003717    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e384caae-3cb1-4264-8a25-55b40ac8ac45-host-cni-netd\") pod \"ovnkube-node-6gltj\" (UID: \"e384caae-3cb1-4264-8a25-55b40ac8ac45\") " pod="openshift-ovn-kubernetes/ovnkube-node-6gltj"
Sep 11 13:21:35.006943 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.003744    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4925d62a-ba17-4740-84c4-c7cf29069831-multus-cni-dir\") pod \"multus-25zm6\" (UID: \"4925d62a-ba17-4740-84c4-c7cf29069831\") " pod="openshift-multus/multus-25zm6"
Sep 11 13:21:35.006943 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.003799    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4925d62a-ba17-4740-84c4-c7cf29069831-multus-socket-dir-parent\") pod \"multus-25zm6\" (UID: \"4925d62a-ba17-4740-84c4-c7cf29069831\") " pod="openshift-multus/multus-25zm6"
Sep 11 13:21:35.006943 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.003831    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4925d62a-ba17-4740-84c4-c7cf29069831-host-run-netns\") pod \"multus-25zm6\" (UID: \"4925d62a-ba17-4740-84c4-c7cf29069831\") " pod="openshift-multus/multus-25zm6"
Sep 11 13:21:35.006943 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.003859    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58f48\" (UniqueName: \"kubernetes.io/projected/1004285c-1dd5-49bf-be06-8549f85cdeda-kube-api-access-58f48\") pod \"network-check-target-6frmx\" (UID: \"1004285c-1dd5-49bf-be06-8549f85cdeda\") " pod="openshift-network-diagnostics/network-check-target-6frmx"
Sep 11 13:21:35.006943 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.003885    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d9bf058c-d89b-4236-86ce-3dee6f389711-host\") pod \"node-ca-8jj8l\" (UID: \"d9bf058c-d89b-4236-86ce-3dee6f389711\") " pod="openshift-image-registry/node-ca-8jj8l"
Sep 11 13:21:35.006943 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.003932    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/86a1ce82-9c9a-4a59-8633-96606b53067a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kp24z\" (UID: \"86a1ce82-9c9a-4a59-8633-96606b53067a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kp24z"
Sep 11 13:21:35.006943 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.003976    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4925d62a-ba17-4740-84c4-c7cf29069831-host-var-lib-kubelet\") pod \"multus-25zm6\" (UID: \"4925d62a-ba17-4740-84c4-c7cf29069831\") " pod="openshift-multus/multus-25zm6"
Sep 11 13:21:35.006943 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.004003    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eeb9973c-b94d-4b62-8ce1-3cdf114016f2-tmp\") pod \"tuned-smdjz\" (UID: \"eeb9973c-b94d-4b62-8ce1-3cdf114016f2\") " pod="openshift-cluster-node-tuning-operator/tuned-smdjz"
Sep 11 13:21:35.006943 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.004032    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zn7r7\" (UniqueName: \"kubernetes.io/projected/f7ab88b5-7073-460e-b3d4-a9841f50b185-kube-api-access-zn7r7\") pod \"node-resolver-z8p25\" (UID: \"f7ab88b5-7073-460e-b3d4-a9841f50b185\") " pod="openshift-dns/node-resolver-z8p25"
Sep 11 13:21:35.006943 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.004058    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e384caae-3cb1-4264-8a25-55b40ac8ac45-etc-openvswitch\") pod \"ovnkube-node-6gltj\" (UID: \"e384caae-3cb1-4264-8a25-55b40ac8ac45\") " pod="openshift-ovn-kubernetes/ovnkube-node-6gltj"
Sep 11 13:21:35.007702 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.004089    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e384caae-3cb1-4264-8a25-55b40ac8ac45-host-run-ovn-kubernetes\") pod \"ovnkube-node-6gltj\" (UID: \"e384caae-3cb1-4264-8a25-55b40ac8ac45\") " pod="openshift-ovn-kubernetes/ovnkube-node-6gltj"
Sep 11 13:21:35.007702 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.004116    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4925d62a-ba17-4740-84c4-c7cf29069831-host-run-k8s-cni-cncf-io\") pod \"multus-25zm6\" (UID: \"4925d62a-ba17-4740-84c4-c7cf29069831\") " pod="openshift-multus/multus-25zm6"
Sep 11 13:21:35.007702 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.004151    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4925d62a-ba17-4740-84c4-c7cf29069831-host-var-lib-cni-multus\") pod \"multus-25zm6\" (UID: \"4925d62a-ba17-4740-84c4-c7cf29069831\") " pod="openshift-multus/multus-25zm6"
Sep 11 13:21:35.007702 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.004179    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/61643763-a1e1-4b2f-8e7c-548d0618d90c-konnectivity-ca\") pod \"konnectivity-agent-v874k\" (UID: \"61643763-a1e1-4b2f-8e7c-548d0618d90c\") " pod="kube-system/konnectivity-agent-v874k"
Sep 11 13:21:35.007702 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.004207    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e384caae-3cb1-4264-8a25-55b40ac8ac45-systemd-units\") pod \"ovnkube-node-6gltj\" (UID: \"e384caae-3cb1-4264-8a25-55b40ac8ac45\") " pod="openshift-ovn-kubernetes/ovnkube-node-6gltj"
Sep 11 13:21:35.007702 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.004245    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e384caae-3cb1-4264-8a25-55b40ac8ac45-env-overrides\") pod \"ovnkube-node-6gltj\" (UID: \"e384caae-3cb1-4264-8a25-55b40ac8ac45\") " pod="openshift-ovn-kubernetes/ovnkube-node-6gltj"
Sep 11 13:21:35.007702 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.004274    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4925d62a-ba17-4740-84c4-c7cf29069831-cni-binary-copy\") pod \"multus-25zm6\" (UID: \"4925d62a-ba17-4740-84c4-c7cf29069831\") " pod="openshift-multus/multus-25zm6"
Sep 11 13:21:35.007702 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.004307    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e384caae-3cb1-4264-8a25-55b40ac8ac45-run-ovn\") pod \"ovnkube-node-6gltj\" (UID: \"e384caae-3cb1-4264-8a25-55b40ac8ac45\") " pod="openshift-ovn-kubernetes/ovnkube-node-6gltj"
Sep 11 13:21:35.007702 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.004335    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4925d62a-ba17-4740-84c4-c7cf29069831-etc-kubernetes\") pod \"multus-25zm6\" (UID: \"4925d62a-ba17-4740-84c4-c7cf29069831\") " pod="openshift-multus/multus-25zm6"
Sep 11 13:21:35.007702 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.004363    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/eeb9973c-b94d-4b62-8ce1-3cdf114016f2-etc-sysconfig\") pod \"tuned-smdjz\" (UID: \"eeb9973c-b94d-4b62-8ce1-3cdf114016f2\") " pod="openshift-cluster-node-tuning-operator/tuned-smdjz"
Sep 11 13:21:35.007702 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.004398    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f7ab88b5-7073-460e-b3d4-a9841f50b185-hosts-file\") pod \"node-resolver-z8p25\" (UID: \"f7ab88b5-7073-460e-b3d4-a9841f50b185\") " pod="openshift-dns/node-resolver-z8p25"
Sep 11 13:21:35.007702 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.004427    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5bd5\" (UniqueName: \"kubernetes.io/projected/86a1ce82-9c9a-4a59-8633-96606b53067a-kube-api-access-k5bd5\") pod \"aws-ebs-csi-driver-node-kp24z\" (UID: \"86a1ce82-9c9a-4a59-8633-96606b53067a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kp24z"
Sep 11 13:21:35.007702 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.004467    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e384caae-3cb1-4264-8a25-55b40ac8ac45-host-kubelet\") pod \"ovnkube-node-6gltj\" (UID: \"e384caae-3cb1-4264-8a25-55b40ac8ac45\") " pod="openshift-ovn-kubernetes/ovnkube-node-6gltj"
Sep 11 13:21:35.007702 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.004494    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4925d62a-ba17-4740-84c4-c7cf29069831-system-cni-dir\") pod \"multus-25zm6\" (UID: \"4925d62a-ba17-4740-84c4-c7cf29069831\") " pod="openshift-multus/multus-25zm6"
Sep 11 13:21:35.007702 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.004521    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4925d62a-ba17-4740-84c4-c7cf29069831-cnibin\") pod \"multus-25zm6\" (UID: \"4925d62a-ba17-4740-84c4-c7cf29069831\") " pod="openshift-multus/multus-25zm6"
Sep 11 13:21:35.007702 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.004549    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/eeb9973c-b94d-4b62-8ce1-3cdf114016f2-etc-sysctl-conf\") pod \"tuned-smdjz\" (UID: \"eeb9973c-b94d-4b62-8ce1-3cdf114016f2\") " pod="openshift-cluster-node-tuning-operator/tuned-smdjz"
Sep 11 13:21:35.007702 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.004576    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eeb9973c-b94d-4b62-8ce1-3cdf114016f2-sys\") pod \"tuned-smdjz\" (UID: \"eeb9973c-b94d-4b62-8ce1-3cdf114016f2\") " pod="openshift-cluster-node-tuning-operator/tuned-smdjz"
Sep 11 13:21:35.008383 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.004602    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d9bf058c-d89b-4236-86ce-3dee6f389711-serviceca\") pod \"node-ca-8jj8l\" (UID: \"d9bf058c-d89b-4236-86ce-3dee6f389711\") " pod="openshift-image-registry/node-ca-8jj8l"
Sep 11 13:21:35.008383 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.004635    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e384caae-3cb1-4264-8a25-55b40ac8ac45-host-slash\") pod \"ovnkube-node-6gltj\" (UID: \"e384caae-3cb1-4264-8a25-55b40ac8ac45\") " pod="openshift-ovn-kubernetes/ovnkube-node-6gltj"
Sep 11 13:21:35.008383 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.004663    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9p5d\" (UniqueName: \"kubernetes.io/projected/4925d62a-ba17-4740-84c4-c7cf29069831-kube-api-access-z9p5d\") pod \"multus-25zm6\" (UID: \"4925d62a-ba17-4740-84c4-c7cf29069831\") " pod="openshift-multus/multus-25zm6"
Sep 11 13:21:35.008383 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.004693    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e384caae-3cb1-4264-8a25-55b40ac8ac45-node-log\") pod \"ovnkube-node-6gltj\" (UID: \"e384caae-3cb1-4264-8a25-55b40ac8ac45\") " pod="openshift-ovn-kubernetes/ovnkube-node-6gltj"
Sep 11 13:21:35.008383 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.004720    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e384caae-3cb1-4264-8a25-55b40ac8ac45-host-cni-bin\") pod \"ovnkube-node-6gltj\" (UID: \"e384caae-3cb1-4264-8a25-55b40ac8ac45\") " pod="openshift-ovn-kubernetes/ovnkube-node-6gltj"
Sep 11 13:21:35.008383 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.004749    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jjg9\" (UniqueName: \"kubernetes.io/projected/e384caae-3cb1-4264-8a25-55b40ac8ac45-kube-api-access-5jjg9\") pod \"ovnkube-node-6gltj\" (UID: \"e384caae-3cb1-4264-8a25-55b40ac8ac45\") " pod="openshift-ovn-kubernetes/ovnkube-node-6gltj"
Sep 11 13:21:35.008383 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.004808    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/eeb9973c-b94d-4b62-8ce1-3cdf114016f2-etc-sysctl-d\") pod \"tuned-smdjz\" (UID: \"eeb9973c-b94d-4b62-8ce1-3cdf114016f2\") " pod="openshift-cluster-node-tuning-operator/tuned-smdjz"
Sep 11 13:21:35.008383 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.004836    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/eeb9973c-b94d-4b62-8ce1-3cdf114016f2-run\") pod \"tuned-smdjz\" (UID: \"eeb9973c-b94d-4b62-8ce1-3cdf114016f2\") " pod="openshift-cluster-node-tuning-operator/tuned-smdjz"
Sep 11 13:21:35.008383 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.004866    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/86a1ce82-9c9a-4a59-8633-96606b53067a-device-dir\") pod \"aws-ebs-csi-driver-node-kp24z\" (UID: \"86a1ce82-9c9a-4a59-8633-96606b53067a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kp24z"
Sep 11 13:21:35.008383 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.004895    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/61643763-a1e1-4b2f-8e7c-548d0618d90c-agent-certs\") pod \"konnectivity-agent-v874k\" (UID: \"61643763-a1e1-4b2f-8e7c-548d0618d90c\") " pod="kube-system/konnectivity-agent-v874k"
Sep 11 13:21:35.008383 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.004945    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e384caae-3cb1-4264-8a25-55b40ac8ac45-host-run-netns\") pod \"ovnkube-node-6gltj\" (UID: \"e384caae-3cb1-4264-8a25-55b40ac8ac45\") " pod="openshift-ovn-kubernetes/ovnkube-node-6gltj"
Sep 11 13:21:35.008383 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.004976    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e384caae-3cb1-4264-8a25-55b40ac8ac45-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6gltj\" (UID: \"e384caae-3cb1-4264-8a25-55b40ac8ac45\") " pod="openshift-ovn-kubernetes/ovnkube-node-6gltj"
Sep 11 13:21:35.008383 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.005008    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e384caae-3cb1-4264-8a25-55b40ac8ac45-ovn-node-metrics-cert\") pod \"ovnkube-node-6gltj\" (UID: \"e384caae-3cb1-4264-8a25-55b40ac8ac45\") " pod="openshift-ovn-kubernetes/ovnkube-node-6gltj"
Sep 11 13:21:35.008383 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.005036    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4925d62a-ba17-4740-84c4-c7cf29069831-host-var-lib-cni-bin\") pod \"multus-25zm6\" (UID: \"4925d62a-ba17-4740-84c4-c7cf29069831\") " pod="openshift-multus/multus-25zm6"
Sep 11 13:21:35.008383 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.005069    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4925d62a-ba17-4740-84c4-c7cf29069831-host-run-multus-certs\") pod \"multus-25zm6\" (UID: \"4925d62a-ba17-4740-84c4-c7cf29069831\") " pod="openshift-multus/multus-25zm6"
Sep 11 13:21:35.008383 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.005128    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/86a1ce82-9c9a-4a59-8633-96606b53067a-etc-selinux\") pod \"aws-ebs-csi-driver-node-kp24z\" (UID: \"86a1ce82-9c9a-4a59-8633-96606b53067a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kp24z"
Sep 11 13:21:35.009071 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.005157    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e384caae-3cb1-4264-8a25-55b40ac8ac45-var-lib-openvswitch\") pod \"ovnkube-node-6gltj\" (UID: \"e384caae-3cb1-4264-8a25-55b40ac8ac45\") " pod="openshift-ovn-kubernetes/ovnkube-node-6gltj"
Sep 11 13:21:35.009071 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.005200    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e384caae-3cb1-4264-8a25-55b40ac8ac45-ovnkube-config\") pod \"ovnkube-node-6gltj\" (UID: \"e384caae-3cb1-4264-8a25-55b40ac8ac45\") " pod="openshift-ovn-kubernetes/ovnkube-node-6gltj"
Sep 11 13:21:35.009071 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.005231    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/eeb9973c-b94d-4b62-8ce1-3cdf114016f2-var-lib-kubelet\") pod \"tuned-smdjz\" (UID: \"eeb9973c-b94d-4b62-8ce1-3cdf114016f2\") " pod="openshift-cluster-node-tuning-operator/tuned-smdjz"
Sep 11 13:21:35.009071 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.005265    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/eeb9973c-b94d-4b62-8ce1-3cdf114016f2-etc-tuned\") pod \"tuned-smdjz\" (UID: \"eeb9973c-b94d-4b62-8ce1-3cdf114016f2\") " pod="openshift-cluster-node-tuning-operator/tuned-smdjz"
Sep 11 13:21:35.009071 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.005293    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4925d62a-ba17-4740-84c4-c7cf29069831-multus-conf-dir\") pod \"multus-25zm6\" (UID: \"4925d62a-ba17-4740-84c4-c7cf29069831\") " pod="openshift-multus/multus-25zm6"
Sep 11 13:21:35.009071 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.005335    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/eeb9973c-b94d-4b62-8ce1-3cdf114016f2-etc-systemd\") pod \"tuned-smdjz\" (UID: \"eeb9973c-b94d-4b62-8ce1-3cdf114016f2\") " pod="openshift-cluster-node-tuning-operator/tuned-smdjz"
Sep 11 13:21:35.009071 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.005364    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e384caae-3cb1-4264-8a25-55b40ac8ac45-run-openvswitch\") pod \"ovnkube-node-6gltj\" (UID: \"e384caae-3cb1-4264-8a25-55b40ac8ac45\") " pod="openshift-ovn-kubernetes/ovnkube-node-6gltj"
Sep 11 13:21:35.009071 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.005412    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/86a1ce82-9c9a-4a59-8633-96606b53067a-registration-dir\") pod \"aws-ebs-csi-driver-node-kp24z\" (UID: \"86a1ce82-9c9a-4a59-8633-96606b53067a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kp24z"
Sep 11 13:21:35.009071 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.005447    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e384caae-3cb1-4264-8a25-55b40ac8ac45-run-systemd\") pod \"ovnkube-node-6gltj\" (UID: \"e384caae-3cb1-4264-8a25-55b40ac8ac45\") " pod="openshift-ovn-kubernetes/ovnkube-node-6gltj"
Sep 11 13:21:35.009071 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.005495    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/eeb9973c-b94d-4b62-8ce1-3cdf114016f2-etc-modprobe-d\") pod \"tuned-smdjz\" (UID: \"eeb9973c-b94d-4b62-8ce1-3cdf114016f2\") " pod="openshift-cluster-node-tuning-operator/tuned-smdjz"
Sep 11 13:21:35.009071 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.005528    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eeb9973c-b94d-4b62-8ce1-3cdf114016f2-etc-kubernetes\") pod \"tuned-smdjz\" (UID: \"eeb9973c-b94d-4b62-8ce1-3cdf114016f2\") " pod="openshift-cluster-node-tuning-operator/tuned-smdjz"
Sep 11 13:21:35.009071 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.005557    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpcp7\" (UniqueName: \"kubernetes.io/projected/d9bf058c-d89b-4236-86ce-3dee6f389711-kube-api-access-gpcp7\") pod \"node-ca-8jj8l\" (UID: \"d9bf058c-d89b-4236-86ce-3dee6f389711\") " pod="openshift-image-registry/node-ca-8jj8l"
Sep 11 13:21:35.009071 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.005593    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbk9b\" (UniqueName: \"kubernetes.io/projected/eeb9973c-b94d-4b62-8ce1-3cdf114016f2-kube-api-access-sbk9b\") pod \"tuned-smdjz\" (UID: \"eeb9973c-b94d-4b62-8ce1-3cdf114016f2\") " pod="openshift-cluster-node-tuning-operator/tuned-smdjz"
Sep 11 13:21:35.009071 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.005624    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e384caae-3cb1-4264-8a25-55b40ac8ac45-ovnkube-script-lib\") pod \"ovnkube-node-6gltj\" (UID: \"e384caae-3cb1-4264-8a25-55b40ac8ac45\") " pod="openshift-ovn-kubernetes/ovnkube-node-6gltj"
Sep 11 13:21:35.009071 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.005655    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4925d62a-ba17-4740-84c4-c7cf29069831-multus-daemon-config\") pod \"multus-25zm6\" (UID: \"4925d62a-ba17-4740-84c4-c7cf29069831\") " pod="openshift-multus/multus-25zm6"
Sep 11 13:21:35.009071 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.005686    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/eeb9973c-b94d-4b62-8ce1-3cdf114016f2-lib-modules\") pod \"tuned-smdjz\" (UID: \"eeb9973c-b94d-4b62-8ce1-3cdf114016f2\") " pod="openshift-cluster-node-tuning-operator/tuned-smdjz"
Sep 11 13:21:35.012050 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.012019    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 11 13:21:35.012698 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.012680    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 11 13:21:35.014448 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.014420    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-58zbx\""
Sep 11 13:21:35.018147 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.018111    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 11 13:21:35.019036 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.019008    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 11 13:21:35.019188 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.019161    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 11 13:21:35.019188 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.019185    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 11 13:21:35.019351 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.019332    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 11 13:21:35.019435 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.019414    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-2crdk\""
Sep 11 13:21:35.019543 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.019523    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 11 13:21:35.019597 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.019338    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-gsww4\""
Sep 11 13:21:35.019782 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.018836    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 11 13:21:35.019855 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.019828    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 11 13:21:35.019980 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.019963    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 11 13:21:35.020310 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.020289    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-q5hwt\""
Sep 11 13:21:35.020418 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.020335    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-bmpx7\""
Sep 11 13:21:35.020495 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.020479    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 11 13:21:35.020670 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.020655    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 11 13:21:35.020813 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.020796    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 11 13:21:35.021210 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.021192    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 11 13:21:35.045224 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.045192    2578 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 11 13:21:35.051425 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.051392    2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-m9wsj"
Sep 11 13:21:35.064971 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.064939    2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-m9wsj"
Sep 11 13:21:35.095084 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.095032    2578 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 11 13:21:35.105966 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.105881    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-86.ec2.internal" event={"ID":"ac822255a7ab3b2619823b5af72a01ec","Type":"ContainerStarted","Data":"3365623e79628f5e7bae26be055d03aba10924a9f3e81db4cd781677d06c9040"}
Sep 11 13:21:35.106237 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.106199    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/eeb9973c-b94d-4b62-8ce1-3cdf114016f2-run\") pod \"tuned-smdjz\" (UID: \"eeb9973c-b94d-4b62-8ce1-3cdf114016f2\") " pod="openshift-cluster-node-tuning-operator/tuned-smdjz"
Sep 11 13:21:35.106383 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.106240    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/86a1ce82-9c9a-4a59-8633-96606b53067a-device-dir\") pod \"aws-ebs-csi-driver-node-kp24z\" (UID: \"86a1ce82-9c9a-4a59-8633-96606b53067a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kp24z"
Sep 11 13:21:35.106383 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.106276    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2354ca1-4b4e-4a17-a17e-52d61af0e80c-metrics-certs\") pod \"network-metrics-daemon-zb2qw\" (UID: \"a2354ca1-4b4e-4a17-a17e-52d61af0e80c\") " pod="openshift-multus/network-metrics-daemon-zb2qw"
Sep 11 13:21:35.106383 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.106305    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4d117442-da24-4a63-8182-041e7522635d-cnibin\") pod \"multus-additional-cni-plugins-qcfls\" (UID: \"4d117442-da24-4a63-8182-041e7522635d\") " pod="openshift-multus/multus-additional-cni-plugins-qcfls"
Sep 11 13:21:35.106383 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.106329    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/eeb9973c-b94d-4b62-8ce1-3cdf114016f2-run\") pod \"tuned-smdjz\" (UID: \"eeb9973c-b94d-4b62-8ce1-3cdf114016f2\") " pod="openshift-cluster-node-tuning-operator/tuned-smdjz"
Sep 11 13:21:35.106383 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.106332    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4d117442-da24-4a63-8182-041e7522635d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qcfls\" (UID: \"4d117442-da24-4a63-8182-041e7522635d\") " pod="openshift-multus/multus-additional-cni-plugins-qcfls"
Sep 11 13:21:35.106652 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.106392    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4d117442-da24-4a63-8182-041e7522635d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qcfls\" (UID: \"4d117442-da24-4a63-8182-041e7522635d\") " pod="openshift-multus/multus-additional-cni-plugins-qcfls"
Sep 11 13:21:35.106652 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.106416    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/86a1ce82-9c9a-4a59-8633-96606b53067a-device-dir\") pod \"aws-ebs-csi-driver-node-kp24z\" (UID: \"86a1ce82-9c9a-4a59-8633-96606b53067a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kp24z"
Sep 11 13:21:35.106652 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.106426    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/61643763-a1e1-4b2f-8e7c-548d0618d90c-agent-certs\") pod \"konnectivity-agent-v874k\" (UID: \"61643763-a1e1-4b2f-8e7c-548d0618d90c\") " pod="kube-system/konnectivity-agent-v874k"
Sep 11 13:21:35.106652 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.106446    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e384caae-3cb1-4264-8a25-55b40ac8ac45-host-run-netns\") pod \"ovnkube-node-6gltj\" (UID: \"e384caae-3cb1-4264-8a25-55b40ac8ac45\") " pod="openshift-ovn-kubernetes/ovnkube-node-6gltj"
Sep 11 13:21:35.106652 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.106473    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e384caae-3cb1-4264-8a25-55b40ac8ac45-host-run-netns\") pod \"ovnkube-node-6gltj\" (UID: \"e384caae-3cb1-4264-8a25-55b40ac8ac45\") " pod="openshift-ovn-kubernetes/ovnkube-node-6gltj"
Sep 11 13:21:35.106652 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.106476    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e384caae-3cb1-4264-8a25-55b40ac8ac45-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6gltj\" (UID: \"e384caae-3cb1-4264-8a25-55b40ac8ac45\") " pod="openshift-ovn-kubernetes/ovnkube-node-6gltj"
Sep 11 13:21:35.106652 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.106502    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e384caae-3cb1-4264-8a25-55b40ac8ac45-ovn-node-metrics-cert\") pod \"ovnkube-node-6gltj\" (UID: \"e384caae-3cb1-4264-8a25-55b40ac8ac45\") " pod="openshift-ovn-kubernetes/ovnkube-node-6gltj"
Sep 11 13:21:35.107024 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.106680    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4925d62a-ba17-4740-84c4-c7cf29069831-host-var-lib-cni-bin\") pod \"multus-25zm6\" (UID: \"4925d62a-ba17-4740-84c4-c7cf29069831\") " pod="openshift-multus/multus-25zm6"
Sep 11 13:21:35.107024 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.106715    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4925d62a-ba17-4740-84c4-c7cf29069831-host-run-multus-certs\") pod \"multus-25zm6\" (UID: \"4925d62a-ba17-4740-84c4-c7cf29069831\") " pod="openshift-multus/multus-25zm6"
Sep 11 13:21:35.107024 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.106743    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/86a1ce82-9c9a-4a59-8633-96606b53067a-etc-selinux\") pod \"aws-ebs-csi-driver-node-kp24z\" (UID: \"86a1ce82-9c9a-4a59-8633-96606b53067a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kp24z"
Sep 11 13:21:35.107024 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.106790    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sktn9\" (UniqueName: \"kubernetes.io/projected/7205c288-a931-457b-bd3b-d68f4440176a-kube-api-access-sktn9\") pod \"iptables-alerter-k27nx\" (UID: \"7205c288-a931-457b-bd3b-d68f4440176a\") " pod="openshift-network-operator/iptables-alerter-k27nx"
Sep 11 13:21:35.107024 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.106821    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e384caae-3cb1-4264-8a25-55b40ac8ac45-var-lib-openvswitch\") pod \"ovnkube-node-6gltj\" (UID: \"e384caae-3cb1-4264-8a25-55b40ac8ac45\") " pod="openshift-ovn-kubernetes/ovnkube-node-6gltj"
Sep 11 13:21:35.107024 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.106848    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e384caae-3cb1-4264-8a25-55b40ac8ac45-ovnkube-config\") pod \"ovnkube-node-6gltj\" (UID: \"e384caae-3cb1-4264-8a25-55b40ac8ac45\") " pod="openshift-ovn-kubernetes/ovnkube-node-6gltj"
Sep 11 13:21:35.107024 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.106888    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/eeb9973c-b94d-4b62-8ce1-3cdf114016f2-var-lib-kubelet\") pod \"tuned-smdjz\" (UID: \"eeb9973c-b94d-4b62-8ce1-3cdf114016f2\") " pod="openshift-cluster-node-tuning-operator/tuned-smdjz"
Sep 11 13:21:35.107024 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.106947    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/eeb9973c-b94d-4b62-8ce1-3cdf114016f2-etc-tuned\") pod \"tuned-smdjz\" (UID: \"eeb9973c-b94d-4b62-8ce1-3cdf114016f2\") " pod="openshift-cluster-node-tuning-operator/tuned-smdjz"
Sep 11 13:21:35.107024 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.106975    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4925d62a-ba17-4740-84c4-c7cf29069831-multus-conf-dir\") pod \"multus-25zm6\" (UID: \"4925d62a-ba17-4740-84c4-c7cf29069831\") " pod="openshift-multus/multus-25zm6"
Sep 11 13:21:35.107024 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.107001    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/eeb9973c-b94d-4b62-8ce1-3cdf114016f2-etc-systemd\") pod \"tuned-smdjz\" (UID: \"eeb9973c-b94d-4b62-8ce1-3cdf114016f2\") " pod="openshift-cluster-node-tuning-operator/tuned-smdjz"
Sep 11 13:21:35.107513 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.107068    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/eeb9973c-b94d-4b62-8ce1-3cdf114016f2-etc-systemd\") pod \"tuned-smdjz\" (UID: \"eeb9973c-b94d-4b62-8ce1-3cdf114016f2\") " pod="openshift-cluster-node-tuning-operator/tuned-smdjz"
Sep 11 13:21:35.107513 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.107100    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e384caae-3cb1-4264-8a25-55b40ac8ac45-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6gltj\" (UID: \"e384caae-3cb1-4264-8a25-55b40ac8ac45\") " pod="openshift-ovn-kubernetes/ovnkube-node-6gltj"
Sep 11 13:21:35.107513 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.107163    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4925d62a-ba17-4740-84c4-c7cf29069831-host-var-lib-cni-bin\") pod \"multus-25zm6\" (UID: \"4925d62a-ba17-4740-84c4-c7cf29069831\") " pod="openshift-multus/multus-25zm6"
Sep 11 13:21:35.107513 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.107214    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4925d62a-ba17-4740-84c4-c7cf29069831-host-run-multus-certs\") pod \"multus-25zm6\" (UID: \"4925d62a-ba17-4740-84c4-c7cf29069831\") " pod="openshift-multus/multus-25zm6"
Sep 11 13:21:35.107513 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.107321    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/86a1ce82-9c9a-4a59-8633-96606b53067a-etc-selinux\") pod \"aws-ebs-csi-driver-node-kp24z\" (UID: \"86a1ce82-9c9a-4a59-8633-96606b53067a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kp24z"
Sep 11 13:21:35.107513 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.107350    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e384caae-3cb1-4264-8a25-55b40ac8ac45-run-openvswitch\") pod \"ovnkube-node-6gltj\" (UID: \"e384caae-3cb1-4264-8a25-55b40ac8ac45\") " pod="openshift-ovn-kubernetes/ovnkube-node-6gltj"
Sep 11 13:21:35.107513 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.107396    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/86a1ce82-9c9a-4a59-8633-96606b53067a-registration-dir\") pod \"aws-ebs-csi-driver-node-kp24z\" (UID: \"86a1ce82-9c9a-4a59-8633-96606b53067a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kp24z"
Sep 11 13:21:35.107513 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.107433    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e384caae-3cb1-4264-8a25-55b40ac8ac45-run-systemd\") pod \"ovnkube-node-6gltj\" (UID: \"e384caae-3cb1-4264-8a25-55b40ac8ac45\") " pod="openshift-ovn-kubernetes/ovnkube-node-6gltj"
Sep 11 13:21:35.107513 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.107466    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/eeb9973c-b94d-4b62-8ce1-3cdf114016f2-etc-modprobe-d\") pod \"tuned-smdjz\" (UID: \"eeb9973c-b94d-4b62-8ce1-3cdf114016f2\") " pod="openshift-cluster-node-tuning-operator/tuned-smdjz"
Sep 11 13:21:35.107513 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.107511    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eeb9973c-b94d-4b62-8ce1-3cdf114016f2-etc-kubernetes\") pod \"tuned-smdjz\" (UID: \"eeb9973c-b94d-4b62-8ce1-3cdf114016f2\") " pod="openshift-cluster-node-tuning-operator/tuned-smdjz"
Sep 11 13:21:35.108064 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.107540    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpcp7\" (UniqueName: \"kubernetes.io/projected/d9bf058c-d89b-4236-86ce-3dee6f389711-kube-api-access-gpcp7\") pod \"node-ca-8jj8l\" (UID: \"d9bf058c-d89b-4236-86ce-3dee6f389711\") " pod="openshift-image-registry/node-ca-8jj8l"
Sep 11 13:21:35.108064 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.107574    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4d117442-da24-4a63-8182-041e7522635d-system-cni-dir\") pod \"multus-additional-cni-plugins-qcfls\" (UID: \"4d117442-da24-4a63-8182-041e7522635d\") " pod="openshift-multus/multus-additional-cni-plugins-qcfls"
Sep 11 13:21:35.108064 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.107595    2578 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 11 13:21:35.108064 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.107608    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4d117442-da24-4a63-8182-041e7522635d-cni-binary-copy\") pod \"multus-additional-cni-plugins-qcfls\" (UID: \"4d117442-da24-4a63-8182-041e7522635d\") " pod="openshift-multus/multus-additional-cni-plugins-qcfls"
Sep 11 13:21:35.108064 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.107638    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbk9b\" (UniqueName: \"kubernetes.io/projected/eeb9973c-b94d-4b62-8ce1-3cdf114016f2-kube-api-access-sbk9b\") pod \"tuned-smdjz\" (UID: \"eeb9973c-b94d-4b62-8ce1-3cdf114016f2\") " pod="openshift-cluster-node-tuning-operator/tuned-smdjz"
Sep 11 13:21:35.108064 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.107668    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e384caae-3cb1-4264-8a25-55b40ac8ac45-ovnkube-script-lib\") pod \"ovnkube-node-6gltj\" (UID: \"e384caae-3cb1-4264-8a25-55b40ac8ac45\") " pod="openshift-ovn-kubernetes/ovnkube-node-6gltj"
Sep 11 13:21:35.108064 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.107695    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4925d62a-ba17-4740-84c4-c7cf29069831-multus-daemon-config\") pod \"multus-25zm6\" (UID: \"4925d62a-ba17-4740-84c4-c7cf29069831\") " pod="openshift-multus/multus-25zm6"
Sep 11 13:21:35.108064 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.107722    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/eeb9973c-b94d-4b62-8ce1-3cdf114016f2-lib-modules\") pod \"tuned-smdjz\" (UID: \"eeb9973c-b94d-4b62-8ce1-3cdf114016f2\") " pod="openshift-cluster-node-tuning-operator/tuned-smdjz"
Sep 11 13:21:35.108064 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.107748    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/eeb9973c-b94d-4b62-8ce1-3cdf114016f2-host\") pod \"tuned-smdjz\" (UID: \"eeb9973c-b94d-4b62-8ce1-3cdf114016f2\") " pod="openshift-cluster-node-tuning-operator/tuned-smdjz"
Sep 11 13:21:35.108064 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.107777    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7ab88b5-7073-460e-b3d4-a9841f50b185-tmp-dir\") pod \"node-resolver-z8p25\" (UID: \"f7ab88b5-7073-460e-b3d4-a9841f50b185\") " pod="openshift-dns/node-resolver-z8p25"
Sep 11 13:21:35.108064 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.107805    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/86a1ce82-9c9a-4a59-8633-96606b53067a-socket-dir\") pod \"aws-ebs-csi-driver-node-kp24z\" (UID: \"86a1ce82-9c9a-4a59-8633-96606b53067a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kp24z"
Sep 11 13:21:35.108064 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.107833    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/86a1ce82-9c9a-4a59-8633-96606b53067a-sys-fs\") pod \"aws-ebs-csi-driver-node-kp24z\" (UID: \"86a1ce82-9c9a-4a59-8633-96606b53067a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kp24z"
Sep 11 13:21:35.108064 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.107863    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e384caae-3cb1-4264-8a25-55b40ac8ac45-log-socket\") pod \"ovnkube-node-6gltj\" (UID: \"e384caae-3cb1-4264-8a25-55b40ac8ac45\") " pod="openshift-ovn-kubernetes/ovnkube-node-6gltj"
Sep 11 13:21:35.108064 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.107888    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4925d62a-ba17-4740-84c4-c7cf29069831-hostroot\") pod \"multus-25zm6\" (UID: \"4925d62a-ba17-4740-84c4-c7cf29069831\") " pod="openshift-multus/multus-25zm6"
Sep 11 13:21:35.108064 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.107903    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e384caae-3cb1-4264-8a25-55b40ac8ac45-ovnkube-config\") pod \"ovnkube-node-6gltj\" (UID: \"e384caae-3cb1-4264-8a25-55b40ac8ac45\") " pod="openshift-ovn-kubernetes/ovnkube-node-6gltj"
Sep 11 13:21:35.108064 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.107943    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e384caae-3cb1-4264-8a25-55b40ac8ac45-host-cni-netd\") pod \"ovnkube-node-6gltj\" (UID: \"e384caae-3cb1-4264-8a25-55b40ac8ac45\") " pod="openshift-ovn-kubernetes/ovnkube-node-6gltj"
Sep 11 13:21:35.108064 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.107993    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4925d62a-ba17-4740-84c4-c7cf29069831-multus-cni-dir\") pod \"multus-25zm6\" (UID: \"4925d62a-ba17-4740-84c4-c7cf29069831\") " pod="openshift-multus/multus-25zm6"
Sep 11 13:21:35.108064 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.107991    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e384caae-3cb1-4264-8a25-55b40ac8ac45-host-cni-netd\") pod \"ovnkube-node-6gltj\" (UID: \"e384caae-3cb1-4264-8a25-55b40ac8ac45\") " pod="openshift-ovn-kubernetes/ovnkube-node-6gltj"
Sep 11 13:21:35.109059 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.108029    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4925d62a-ba17-4740-84c4-c7cf29069831-multus-socket-dir-parent\") pod \"multus-25zm6\" (UID: \"4925d62a-ba17-4740-84c4-c7cf29069831\") " pod="openshift-multus/multus-25zm6"
Sep 11 13:21:35.109059 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.108067    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e384caae-3cb1-4264-8a25-55b40ac8ac45-run-openvswitch\") pod \"ovnkube-node-6gltj\" (UID: \"e384caae-3cb1-4264-8a25-55b40ac8ac45\") " pod="openshift-ovn-kubernetes/ovnkube-node-6gltj"
Sep 11 13:21:35.109059 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.108059    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4925d62a-ba17-4740-84c4-c7cf29069831-host-run-netns\") pod \"multus-25zm6\" (UID: \"4925d62a-ba17-4740-84c4-c7cf29069831\") " pod="openshift-multus/multus-25zm6"
Sep 11 13:21:35.109059 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.108116    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58f48\" (UniqueName: \"kubernetes.io/projected/1004285c-1dd5-49bf-be06-8549f85cdeda-kube-api-access-58f48\") pod \"network-check-target-6frmx\" (UID: \"1004285c-1dd5-49bf-be06-8549f85cdeda\") " pod="openshift-network-diagnostics/network-check-target-6frmx"
Sep 11 13:21:35.109059 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.108127    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/86a1ce82-9c9a-4a59-8633-96606b53067a-registration-dir\") pod \"aws-ebs-csi-driver-node-kp24z\" (UID: \"86a1ce82-9c9a-4a59-8633-96606b53067a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kp24z"
Sep 11 13:21:35.109059 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.108143    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d9bf058c-d89b-4236-86ce-3dee6f389711-host\") pod \"node-ca-8jj8l\" (UID: \"d9bf058c-d89b-4236-86ce-3dee6f389711\") " pod="openshift-image-registry/node-ca-8jj8l"
Sep 11 13:21:35.109059 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.108171    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e384caae-3cb1-4264-8a25-55b40ac8ac45-run-systemd\") pod \"ovnkube-node-6gltj\" (UID: \"e384caae-3cb1-4264-8a25-55b40ac8ac45\") " pod="openshift-ovn-kubernetes/ovnkube-node-6gltj"
Sep 11 13:21:35.109059 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.108187    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/86a1ce82-9c9a-4a59-8633-96606b53067a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kp24z\" (UID: \"86a1ce82-9c9a-4a59-8633-96606b53067a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kp24z"
Sep 11 13:21:35.109059 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.108221    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/be2cb948-420f-4f91-b56b-a292efb25181-kubelet-config\") pod \"global-pull-secret-syncer-m6d9g\" (UID: \"be2cb948-420f-4f91-b56b-a292efb25181\") " pod="kube-system/global-pull-secret-syncer-m6d9g"
Sep 11 13:21:35.109059 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.108252    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4925d62a-ba17-4740-84c4-c7cf29069831-host-var-lib-kubelet\") pod \"multus-25zm6\" (UID: \"4925d62a-ba17-4740-84c4-c7cf29069831\") " pod="openshift-multus/multus-25zm6"
Sep 11 13:21:35.109059 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.108292    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eeb9973c-b94d-4b62-8ce1-3cdf114016f2-tmp\") pod \"tuned-smdjz\" (UID: \"eeb9973c-b94d-4b62-8ce1-3cdf114016f2\") " pod="openshift-cluster-node-tuning-operator/tuned-smdjz"
Sep 11 13:21:35.109059 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.108285    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/eeb9973c-b94d-4b62-8ce1-3cdf114016f2-etc-modprobe-d\") pod \"tuned-smdjz\" (UID: \"eeb9973c-b94d-4b62-8ce1-3cdf114016f2\") " pod="openshift-cluster-node-tuning-operator/tuned-smdjz"
Sep 11 13:21:35.109059 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.108324    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zn7r7\" (UniqueName: \"kubernetes.io/projected/f7ab88b5-7073-460e-b3d4-a9841f50b185-kube-api-access-zn7r7\") pod \"node-resolver-z8p25\" (UID: \"f7ab88b5-7073-460e-b3d4-a9841f50b185\") " pod="openshift-dns/node-resolver-z8p25"
Sep 11 13:21:35.109059 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.108360    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9vs6\" (UniqueName: \"kubernetes.io/projected/a2354ca1-4b4e-4a17-a17e-52d61af0e80c-kube-api-access-s9vs6\") pod \"network-metrics-daemon-zb2qw\" (UID: \"a2354ca1-4b4e-4a17-a17e-52d61af0e80c\") " pod="openshift-multus/network-metrics-daemon-zb2qw"
Sep 11 13:21:35.109059 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.108380    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4d117442-da24-4a63-8182-041e7522635d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qcfls\" (UID: \"4d117442-da24-4a63-8182-041e7522635d\") " pod="openshift-multus/multus-additional-cni-plugins-qcfls"
Sep 11 13:21:35.109059 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.108403    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eeb9973c-b94d-4b62-8ce1-3cdf114016f2-etc-kubernetes\") pod \"tuned-smdjz\" (UID: \"eeb9973c-b94d-4b62-8ce1-3cdf114016f2\") " pod="openshift-cluster-node-tuning-operator/tuned-smdjz"
Sep 11 13:21:35.109059 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.108400    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e384caae-3cb1-4264-8a25-55b40ac8ac45-etc-openvswitch\") pod \"ovnkube-node-6gltj\" (UID: \"e384caae-3cb1-4264-8a25-55b40ac8ac45\") " pod="openshift-ovn-kubernetes/ovnkube-node-6gltj"
Sep 11 13:21:35.109899 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.108443    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e384caae-3cb1-4264-8a25-55b40ac8ac45-etc-openvswitch\") pod \"ovnkube-node-6gltj\" (UID: \"e384caae-3cb1-4264-8a25-55b40ac8ac45\") " pod="openshift-ovn-kubernetes/ovnkube-node-6gltj"
Sep 11 13:21:35.109899 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.108454    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e384caae-3cb1-4264-8a25-55b40ac8ac45-host-run-ovn-kubernetes\") pod \"ovnkube-node-6gltj\" (UID: \"e384caae-3cb1-4264-8a25-55b40ac8ac45\") " pod="openshift-ovn-kubernetes/ovnkube-node-6gltj"
Sep 11 13:21:35.109899 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.108478    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4925d62a-ba17-4740-84c4-c7cf29069831-multus-conf-dir\") pod \"multus-25zm6\" (UID: \"4925d62a-ba17-4740-84c4-c7cf29069831\") " pod="openshift-multus/multus-25zm6"
Sep 11 13:21:35.109899 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.108488    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4925d62a-ba17-4740-84c4-c7cf29069831-host-run-k8s-cni-cncf-io\") pod \"multus-25zm6\" (UID: \"4925d62a-ba17-4740-84c4-c7cf29069831\") " pod="openshift-multus/multus-25zm6"
Sep 11 13:21:35.109899 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.108515    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e384caae-3cb1-4264-8a25-55b40ac8ac45-var-lib-openvswitch\") pod \"ovnkube-node-6gltj\" (UID: \"e384caae-3cb1-4264-8a25-55b40ac8ac45\") " pod="openshift-ovn-kubernetes/ovnkube-node-6gltj"
Sep 11 13:21:35.109899 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.108517    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4925d62a-ba17-4740-84c4-c7cf29069831-host-var-lib-cni-multus\") pod \"multus-25zm6\" (UID: \"4925d62a-ba17-4740-84c4-c7cf29069831\") " pod="openshift-multus/multus-25zm6"
Sep 11 13:21:35.109899 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.108551    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4925d62a-ba17-4740-84c4-c7cf29069831-host-var-lib-cni-multus\") pod \"multus-25zm6\" (UID: \"4925d62a-ba17-4740-84c4-c7cf29069831\") " pod="openshift-multus/multus-25zm6"
Sep 11 13:21:35.109899 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.108557    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dlgl\" (UniqueName: \"kubernetes.io/projected/4d117442-da24-4a63-8182-041e7522635d-kube-api-access-9dlgl\") pod \"multus-additional-cni-plugins-qcfls\" (UID: \"4d117442-da24-4a63-8182-041e7522635d\") " pod="openshift-multus/multus-additional-cni-plugins-qcfls"
Sep 11 13:21:35.109899 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.108585    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/61643763-a1e1-4b2f-8e7c-548d0618d90c-konnectivity-ca\") pod \"konnectivity-agent-v874k\" (UID: \"61643763-a1e1-4b2f-8e7c-548d0618d90c\") " pod="kube-system/konnectivity-agent-v874k"
Sep 11 13:21:35.109899 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.108608    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e384caae-3cb1-4264-8a25-55b40ac8ac45-systemd-units\") pod \"ovnkube-node-6gltj\" (UID: \"e384caae-3cb1-4264-8a25-55b40ac8ac45\") " pod="openshift-ovn-kubernetes/ovnkube-node-6gltj"
Sep 11 13:21:35.109899 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.108627    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e384caae-3cb1-4264-8a25-55b40ac8ac45-env-overrides\") pod \"ovnkube-node-6gltj\" (UID: \"e384caae-3cb1-4264-8a25-55b40ac8ac45\") " pod="openshift-ovn-kubernetes/ovnkube-node-6gltj"
Sep 11 13:21:35.109899 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.108655    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4925d62a-ba17-4740-84c4-c7cf29069831-cni-binary-copy\") pod \"multus-25zm6\" (UID: \"4925d62a-ba17-4740-84c4-c7cf29069831\") " pod="openshift-multus/multus-25zm6"
Sep 11 13:21:35.109899 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.108683    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/be2cb948-420f-4f91-b56b-a292efb25181-dbus\") pod \"global-pull-secret-syncer-m6d9g\" (UID: \"be2cb948-420f-4f91-b56b-a292efb25181\") " pod="kube-system/global-pull-secret-syncer-m6d9g"
Sep 11 13:21:35.109899 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.108712    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7205c288-a931-457b-bd3b-d68f4440176a-host-slash\") pod \"iptables-alerter-k27nx\" (UID: \"7205c288-a931-457b-bd3b-d68f4440176a\") " pod="openshift-network-operator/iptables-alerter-k27nx"
Sep 11 13:21:35.109899 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.108744    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e384caae-3cb1-4264-8a25-55b40ac8ac45-run-ovn\") pod \"ovnkube-node-6gltj\" (UID: \"e384caae-3cb1-4264-8a25-55b40ac8ac45\") " pod="openshift-ovn-kubernetes/ovnkube-node-6gltj"
Sep 11 13:21:35.109899 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.108765    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4925d62a-ba17-4740-84c4-c7cf29069831-etc-kubernetes\") pod \"multus-25zm6\" (UID: \"4925d62a-ba17-4740-84c4-c7cf29069831\") " pod="openshift-multus/multus-25zm6"
Sep 11 13:21:35.109899 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.108786    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/eeb9973c-b94d-4b62-8ce1-3cdf114016f2-etc-sysconfig\") pod \"tuned-smdjz\" (UID: \"eeb9973c-b94d-4b62-8ce1-3cdf114016f2\") " pod="openshift-cluster-node-tuning-operator/tuned-smdjz"
Sep 11 13:21:35.113447 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.108817    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f7ab88b5-7073-460e-b3d4-a9841f50b185-hosts-file\") pod \"node-resolver-z8p25\" (UID: \"f7ab88b5-7073-460e-b3d4-a9841f50b185\") " pod="openshift-dns/node-resolver-z8p25"
Sep 11 13:21:35.113447 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.108850    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5bd5\" (UniqueName: \"kubernetes.io/projected/86a1ce82-9c9a-4a59-8633-96606b53067a-kube-api-access-k5bd5\") pod \"aws-ebs-csi-driver-node-kp24z\" (UID: \"86a1ce82-9c9a-4a59-8633-96606b53067a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kp24z"
Sep 11 13:21:35.113447 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.108883    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e384caae-3cb1-4264-8a25-55b40ac8ac45-host-kubelet\") pod \"ovnkube-node-6gltj\" (UID: \"e384caae-3cb1-4264-8a25-55b40ac8ac45\") " pod="openshift-ovn-kubernetes/ovnkube-node-6gltj"
Sep 11 13:21:35.113447 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.108927    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4925d62a-ba17-4740-84c4-c7cf29069831-system-cni-dir\") pod \"multus-25zm6\" (UID: \"4925d62a-ba17-4740-84c4-c7cf29069831\") " pod="openshift-multus/multus-25zm6"
Sep 11 13:21:35.113447 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.108952    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4925d62a-ba17-4740-84c4-c7cf29069831-cnibin\") pod \"multus-25zm6\" (UID: \"4925d62a-ba17-4740-84c4-c7cf29069831\") " pod="openshift-multus/multus-25zm6"
Sep 11 13:21:35.113447 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.108974    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/eeb9973c-b94d-4b62-8ce1-3cdf114016f2-etc-sysctl-conf\") pod \"tuned-smdjz\" (UID: \"eeb9973c-b94d-4b62-8ce1-3cdf114016f2\") " pod="openshift-cluster-node-tuning-operator/tuned-smdjz"
Sep 11 13:21:35.113447 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.108999    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eeb9973c-b94d-4b62-8ce1-3cdf114016f2-sys\") pod \"tuned-smdjz\" (UID: \"eeb9973c-b94d-4b62-8ce1-3cdf114016f2\") " pod="openshift-cluster-node-tuning-operator/tuned-smdjz"
Sep 11 13:21:35.113447 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.109022    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d9bf058c-d89b-4236-86ce-3dee6f389711-serviceca\") pod \"node-ca-8jj8l\" (UID: \"d9bf058c-d89b-4236-86ce-3dee6f389711\") " pod="openshift-image-registry/node-ca-8jj8l"
Sep 11 13:21:35.113447 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.109042    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e384caae-3cb1-4264-8a25-55b40ac8ac45-host-slash\") pod \"ovnkube-node-6gltj\" (UID: \"e384caae-3cb1-4264-8a25-55b40ac8ac45\") " pod="openshift-ovn-kubernetes/ovnkube-node-6gltj"
Sep 11 13:21:35.113447 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.109045    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/eeb9973c-b94d-4b62-8ce1-3cdf114016f2-var-lib-kubelet\") pod \"tuned-smdjz\" (UID: \"eeb9973c-b94d-4b62-8ce1-3cdf114016f2\") " pod="openshift-cluster-node-tuning-operator/tuned-smdjz"
Sep 11 13:21:35.113447 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.109061    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9p5d\" (UniqueName: \"kubernetes.io/projected/4925d62a-ba17-4740-84c4-c7cf29069831-kube-api-access-z9p5d\") pod \"multus-25zm6\" (UID: \"4925d62a-ba17-4740-84c4-c7cf29069831\") " pod="openshift-multus/multus-25zm6"
Sep 11 13:21:35.113447 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.109097    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/be2cb948-420f-4f91-b56b-a292efb25181-original-pull-secret\") pod \"global-pull-secret-syncer-m6d9g\" (UID: \"be2cb948-420f-4f91-b56b-a292efb25181\") " pod="kube-system/global-pull-secret-syncer-m6d9g"
Sep 11 13:21:35.113447 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.109119    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7205c288-a931-457b-bd3b-d68f4440176a-iptables-alerter-script\") pod \"iptables-alerter-k27nx\" (UID: \"7205c288-a931-457b-bd3b-d68f4440176a\") " pod="openshift-network-operator/iptables-alerter-k27nx"
Sep 11 13:21:35.113447 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.109139    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e384caae-3cb1-4264-8a25-55b40ac8ac45-node-log\") pod \"ovnkube-node-6gltj\" (UID: \"e384caae-3cb1-4264-8a25-55b40ac8ac45\") " pod="openshift-ovn-kubernetes/ovnkube-node-6gltj"
Sep 11 13:21:35.113447 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.109172    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e384caae-3cb1-4264-8a25-55b40ac8ac45-host-cni-bin\") pod \"ovnkube-node-6gltj\" (UID: \"e384caae-3cb1-4264-8a25-55b40ac8ac45\") " pod="openshift-ovn-kubernetes/ovnkube-node-6gltj"
Sep 11 13:21:35.113447 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.109193    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jjg9\" (UniqueName: \"kubernetes.io/projected/e384caae-3cb1-4264-8a25-55b40ac8ac45-kube-api-access-5jjg9\") pod \"ovnkube-node-6gltj\" (UID: \"e384caae-3cb1-4264-8a25-55b40ac8ac45\") " pod="openshift-ovn-kubernetes/ovnkube-node-6gltj"
Sep 11 13:21:35.113447 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.109187    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-86.ec2.internal" event={"ID":"76de799c7422d703cdded403fdfd9c70","Type":"ContainerStarted","Data":"1da064c293c97edad648465b2cb3c25496df5249b84c9d9b59a7518179c904e5"}
Sep 11 13:21:35.114340 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.109212    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/eeb9973c-b94d-4b62-8ce1-3cdf114016f2-etc-sysctl-d\") pod \"tuned-smdjz\" (UID: \"eeb9973c-b94d-4b62-8ce1-3cdf114016f2\") " pod="openshift-cluster-node-tuning-operator/tuned-smdjz"
Sep 11 13:21:35.114340 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.109304    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4925d62a-ba17-4740-84c4-c7cf29069831-multus-cni-dir\") pod \"multus-25zm6\" (UID: \"4925d62a-ba17-4740-84c4-c7cf29069831\") " pod="openshift-multus/multus-25zm6"
Sep 11 13:21:35.114340 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.109344    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4925d62a-ba17-4740-84c4-c7cf29069831-multus-socket-dir-parent\") pod \"multus-25zm6\" (UID: \"4925d62a-ba17-4740-84c4-c7cf29069831\") " pod="openshift-multus/multus-25zm6"
Sep 11 13:21:35.114340 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.109356    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/eeb9973c-b94d-4b62-8ce1-3cdf114016f2-etc-sysctl-d\") pod \"tuned-smdjz\" (UID: \"eeb9973c-b94d-4b62-8ce1-3cdf114016f2\") " pod="openshift-cluster-node-tuning-operator/tuned-smdjz"
Sep 11 13:21:35.114340 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.109400    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e384caae-3cb1-4264-8a25-55b40ac8ac45-run-ovn\") pod \"ovnkube-node-6gltj\" (UID: \"e384caae-3cb1-4264-8a25-55b40ac8ac45\") " pod="openshift-ovn-kubernetes/ovnkube-node-6gltj"
Sep 11 13:21:35.114340 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.109402    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d9bf058c-d89b-4236-86ce-3dee6f389711-host\") pod \"node-ca-8jj8l\" (UID: \"d9bf058c-d89b-4236-86ce-3dee6f389711\") " pod="openshift-image-registry/node-ca-8jj8l"
Sep 11 13:21:35.114340 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.109430    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4925d62a-ba17-4740-84c4-c7cf29069831-etc-kubernetes\") pod \"multus-25zm6\" (UID: \"4925d62a-ba17-4740-84c4-c7cf29069831\") " pod="openshift-multus/multus-25zm6"
Sep 11 13:21:35.114340 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.109440    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/86a1ce82-9c9a-4a59-8633-96606b53067a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kp24z\" (UID: \"86a1ce82-9c9a-4a59-8633-96606b53067a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kp24z"
Sep 11 13:21:35.114340 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.109458    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4925d62a-ba17-4740-84c4-c7cf29069831-host-run-netns\") pod \"multus-25zm6\" (UID: \"4925d62a-ba17-4740-84c4-c7cf29069831\") " pod="openshift-multus/multus-25zm6"
Sep 11 13:21:35.114340 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.109489    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4925d62a-ba17-4740-84c4-c7cf29069831-host-var-lib-kubelet\") pod \"multus-25zm6\" (UID: \"4925d62a-ba17-4740-84c4-c7cf29069831\") " pod="openshift-multus/multus-25zm6"
Sep 11 13:21:35.114340 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.109512    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f7ab88b5-7073-460e-b3d4-a9841f50b185-hosts-file\") pod \"node-resolver-z8p25\" (UID: \"f7ab88b5-7073-460e-b3d4-a9841f50b185\") " pod="openshift-dns/node-resolver-z8p25"
Sep 11 13:21:35.114340 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.109529    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/eeb9973c-b94d-4b62-8ce1-3cdf114016f2-etc-sysconfig\") pod \"tuned-smdjz\" (UID: \"eeb9973c-b94d-4b62-8ce1-3cdf114016f2\") " pod="openshift-cluster-node-tuning-operator/tuned-smdjz"
Sep 11 13:21:35.114340 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.109814    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e384caae-3cb1-4264-8a25-55b40ac8ac45-host-kubelet\") pod \"ovnkube-node-6gltj\" (UID: \"e384caae-3cb1-4264-8a25-55b40ac8ac45\") " pod="openshift-ovn-kubernetes/ovnkube-node-6gltj"
Sep 11 13:21:35.114340 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.109888    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4925d62a-ba17-4740-84c4-c7cf29069831-system-cni-dir\") pod \"multus-25zm6\" (UID: \"4925d62a-ba17-4740-84c4-c7cf29069831\") " pod="openshift-multus/multus-25zm6"
Sep 11 13:21:35.114340 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.109965    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4925d62a-ba17-4740-84c4-c7cf29069831-cnibin\") pod \"multus-25zm6\" (UID: \"4925d62a-ba17-4740-84c4-c7cf29069831\") " pod="openshift-multus/multus-25zm6"
Sep 11 13:21:35.114340 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.110093    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/eeb9973c-b94d-4b62-8ce1-3cdf114016f2-etc-sysctl-conf\") pod \"tuned-smdjz\" (UID: \"eeb9973c-b94d-4b62-8ce1-3cdf114016f2\") " pod="openshift-cluster-node-tuning-operator/tuned-smdjz"
Sep 11 13:21:35.114340 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.110150    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eeb9973c-b94d-4b62-8ce1-3cdf114016f2-sys\") pod \"tuned-smdjz\" (UID: \"eeb9973c-b94d-4b62-8ce1-3cdf114016f2\") " pod="openshift-cluster-node-tuning-operator/tuned-smdjz"
Sep 11 13:21:35.114340 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.110645    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d9bf058c-d89b-4236-86ce-3dee6f389711-serviceca\") pod \"node-ca-8jj8l\" (UID: \"d9bf058c-d89b-4236-86ce-3dee6f389711\") " pod="openshift-image-registry/node-ca-8jj8l"
Sep 11 13:21:35.115419 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.110718    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e384caae-3cb1-4264-8a25-55b40ac8ac45-host-slash\") pod \"ovnkube-node-6gltj\" (UID: \"e384caae-3cb1-4264-8a25-55b40ac8ac45\") " pod="openshift-ovn-kubernetes/ovnkube-node-6gltj"
Sep 11 13:21:35.115419 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.110883    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e384caae-3cb1-4264-8a25-55b40ac8ac45-ovnkube-script-lib\") pod \"ovnkube-node-6gltj\" (UID: \"e384caae-3cb1-4264-8a25-55b40ac8ac45\") " pod="openshift-ovn-kubernetes/ovnkube-node-6gltj"
Sep 11 13:21:35.115419 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.110968    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4925d62a-ba17-4740-84c4-c7cf29069831-multus-daemon-config\") pod \"multus-25zm6\" (UID: \"4925d62a-ba17-4740-84c4-c7cf29069831\") " pod="openshift-multus/multus-25zm6"
Sep 11 13:21:35.115419 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.111087    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e384caae-3cb1-4264-8a25-55b40ac8ac45-host-run-ovn-kubernetes\") pod \"ovnkube-node-6gltj\" (UID: \"e384caae-3cb1-4264-8a25-55b40ac8ac45\") " pod="openshift-ovn-kubernetes/ovnkube-node-6gltj"
Sep 11 13:21:35.115419 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.111131    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/86a1ce82-9c9a-4a59-8633-96606b53067a-socket-dir\") pod \"aws-ebs-csi-driver-node-kp24z\" (UID: \"86a1ce82-9c9a-4a59-8633-96606b53067a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kp24z"
Sep 11 13:21:35.115419 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.111125    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e384caae-3cb1-4264-8a25-55b40ac8ac45-node-log\") pod \"ovnkube-node-6gltj\" (UID: \"e384caae-3cb1-4264-8a25-55b40ac8ac45\") " pod="openshift-ovn-kubernetes/ovnkube-node-6gltj"
Sep 11 13:21:35.115419 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.111147    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/eeb9973c-b94d-4b62-8ce1-3cdf114016f2-lib-modules\") pod \"tuned-smdjz\" (UID: \"eeb9973c-b94d-4b62-8ce1-3cdf114016f2\") " pod="openshift-cluster-node-tuning-operator/tuned-smdjz"
Sep 11 13:21:35.115419 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.111183    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/eeb9973c-b94d-4b62-8ce1-3cdf114016f2-host\") pod \"tuned-smdjz\" (UID: \"eeb9973c-b94d-4b62-8ce1-3cdf114016f2\") " pod="openshift-cluster-node-tuning-operator/tuned-smdjz"
Sep 11 13:21:35.115419 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.111185    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e384caae-3cb1-4264-8a25-55b40ac8ac45-host-cni-bin\") pod \"ovnkube-node-6gltj\" (UID: \"e384caae-3cb1-4264-8a25-55b40ac8ac45\") " pod="openshift-ovn-kubernetes/ovnkube-node-6gltj"
Sep 11 13:21:35.115419 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.111241    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4925d62a-ba17-4740-84c4-c7cf29069831-hostroot\") pod \"multus-25zm6\" (UID: \"4925d62a-ba17-4740-84c4-c7cf29069831\") " pod="openshift-multus/multus-25zm6"
Sep 11 13:21:35.115419 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.111288    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e384caae-3cb1-4264-8a25-55b40ac8ac45-log-socket\") pod \"ovnkube-node-6gltj\" (UID: \"e384caae-3cb1-4264-8a25-55b40ac8ac45\") " pod="openshift-ovn-kubernetes/ovnkube-node-6gltj"
Sep 11 13:21:35.115419 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.111437    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7ab88b5-7073-460e-b3d4-a9841f50b185-tmp-dir\") pod \"node-resolver-z8p25\" (UID: \"f7ab88b5-7073-460e-b3d4-a9841f50b185\") " pod="openshift-dns/node-resolver-z8p25"
Sep 11 13:21:35.115419 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.111566    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/86a1ce82-9c9a-4a59-8633-96606b53067a-sys-fs\") pod \"aws-ebs-csi-driver-node-kp24z\" (UID: \"86a1ce82-9c9a-4a59-8633-96606b53067a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kp24z"
Sep 11 13:21:35.115419 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.111624    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e384caae-3cb1-4264-8a25-55b40ac8ac45-env-overrides\") pod \"ovnkube-node-6gltj\" (UID: \"e384caae-3cb1-4264-8a25-55b40ac8ac45\") " pod="openshift-ovn-kubernetes/ovnkube-node-6gltj"
Sep 11 13:21:35.115419 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.111640    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e384caae-3cb1-4264-8a25-55b40ac8ac45-systemd-units\") pod \"ovnkube-node-6gltj\" (UID: \"e384caae-3cb1-4264-8a25-55b40ac8ac45\") " pod="openshift-ovn-kubernetes/ovnkube-node-6gltj"
Sep 11 13:21:35.115419 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.111691    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4925d62a-ba17-4740-84c4-c7cf29069831-host-run-k8s-cni-cncf-io\") pod \"multus-25zm6\" (UID: \"4925d62a-ba17-4740-84c4-c7cf29069831\") " pod="openshift-multus/multus-25zm6"
Sep 11 13:21:35.115419 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.111906    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/61643763-a1e1-4b2f-8e7c-548d0618d90c-konnectivity-ca\") pod \"konnectivity-agent-v874k\" (UID: \"61643763-a1e1-4b2f-8e7c-548d0618d90c\") " pod="kube-system/konnectivity-agent-v874k"
Sep 11 13:21:35.115419 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.112224    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4925d62a-ba17-4740-84c4-c7cf29069831-cni-binary-copy\") pod \"multus-25zm6\" (UID: \"4925d62a-ba17-4740-84c4-c7cf29069831\") " pod="openshift-multus/multus-25zm6"
Sep 11 13:21:35.116405 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.114605    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/eeb9973c-b94d-4b62-8ce1-3cdf114016f2-etc-tuned\") pod \"tuned-smdjz\" (UID: \"eeb9973c-b94d-4b62-8ce1-3cdf114016f2\") " pod="openshift-cluster-node-tuning-operator/tuned-smdjz"
Sep 11 13:21:35.116405 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.114873    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e384caae-3cb1-4264-8a25-55b40ac8ac45-ovn-node-metrics-cert\") pod \"ovnkube-node-6gltj\" (UID: \"e384caae-3cb1-4264-8a25-55b40ac8ac45\") " pod="openshift-ovn-kubernetes/ovnkube-node-6gltj"
Sep 11 13:21:35.116405 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.115487    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/61643763-a1e1-4b2f-8e7c-548d0618d90c-agent-certs\") pod \"konnectivity-agent-v874k\" (UID: \"61643763-a1e1-4b2f-8e7c-548d0618d90c\") " pod="kube-system/konnectivity-agent-v874k"
Sep 11 13:21:35.116405 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.116255    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eeb9973c-b94d-4b62-8ce1-3cdf114016f2-tmp\") pod \"tuned-smdjz\" (UID: \"eeb9973c-b94d-4b62-8ce1-3cdf114016f2\") " pod="openshift-cluster-node-tuning-operator/tuned-smdjz"
Sep 11 13:21:35.127788 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.127742    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9p5d\" (UniqueName: \"kubernetes.io/projected/4925d62a-ba17-4740-84c4-c7cf29069831-kube-api-access-z9p5d\") pod \"multus-25zm6\" (UID: \"4925d62a-ba17-4740-84c4-c7cf29069831\") " pod="openshift-multus/multus-25zm6"
Sep 11 13:21:35.138066 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:21:35.137937    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 13:21:35.138066 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:21:35.137988    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 13:21:35.138066 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:21:35.138008    2578 projected.go:196] Error preparing data for projected volume kube-api-access-58f48 for pod openshift-network-diagnostics/network-check-target-6frmx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 13:21:35.138416 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:21:35.138195    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1004285c-1dd5-49bf-be06-8549f85cdeda-kube-api-access-58f48 podName:1004285c-1dd5-49bf-be06-8549f85cdeda nodeName:}" failed. No retries permitted until 2026-09-11 13:21:35.638145407 +0000 UTC m=+3.024248572 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-58f48" (UniqueName: "kubernetes.io/projected/1004285c-1dd5-49bf-be06-8549f85cdeda-kube-api-access-58f48") pod "network-check-target-6frmx" (UID: "1004285c-1dd5-49bf-be06-8549f85cdeda") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 13:21:35.144646 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.144599    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbk9b\" (UniqueName: \"kubernetes.io/projected/eeb9973c-b94d-4b62-8ce1-3cdf114016f2-kube-api-access-sbk9b\") pod \"tuned-smdjz\" (UID: \"eeb9973c-b94d-4b62-8ce1-3cdf114016f2\") " pod="openshift-cluster-node-tuning-operator/tuned-smdjz"
Sep 11 13:21:35.153264 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.153223    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zn7r7\" (UniqueName: \"kubernetes.io/projected/f7ab88b5-7073-460e-b3d4-a9841f50b185-kube-api-access-zn7r7\") pod \"node-resolver-z8p25\" (UID: \"f7ab88b5-7073-460e-b3d4-a9841f50b185\") " pod="openshift-dns/node-resolver-z8p25"
Sep 11 13:21:35.156226 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.156190    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpcp7\" (UniqueName: \"kubernetes.io/projected/d9bf058c-d89b-4236-86ce-3dee6f389711-kube-api-access-gpcp7\") pod \"node-ca-8jj8l\" (UID: \"d9bf058c-d89b-4236-86ce-3dee6f389711\") " pod="openshift-image-registry/node-ca-8jj8l"
Sep 11 13:21:35.158188 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.158145    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5bd5\" (UniqueName: \"kubernetes.io/projected/86a1ce82-9c9a-4a59-8633-96606b53067a-kube-api-access-k5bd5\") pod \"aws-ebs-csi-driver-node-kp24z\" (UID: \"86a1ce82-9c9a-4a59-8633-96606b53067a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kp24z"
Sep 11 13:21:35.159483 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.159439    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jjg9\" (UniqueName: \"kubernetes.io/projected/e384caae-3cb1-4264-8a25-55b40ac8ac45-kube-api-access-5jjg9\") pod \"ovnkube-node-6gltj\" (UID: \"e384caae-3cb1-4264-8a25-55b40ac8ac45\") " pod="openshift-ovn-kubernetes/ovnkube-node-6gltj"
Sep 11 13:21:35.209689 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.209636    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sktn9\" (UniqueName: \"kubernetes.io/projected/7205c288-a931-457b-bd3b-d68f4440176a-kube-api-access-sktn9\") pod \"iptables-alerter-k27nx\" (UID: \"7205c288-a931-457b-bd3b-d68f4440176a\") " pod="openshift-network-operator/iptables-alerter-k27nx"
Sep 11 13:21:35.209947 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.209713    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4d117442-da24-4a63-8182-041e7522635d-system-cni-dir\") pod \"multus-additional-cni-plugins-qcfls\" (UID: \"4d117442-da24-4a63-8182-041e7522635d\") " pod="openshift-multus/multus-additional-cni-plugins-qcfls"
Sep 11 13:21:35.209947 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.209740    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4d117442-da24-4a63-8182-041e7522635d-cni-binary-copy\") pod \"multus-additional-cni-plugins-qcfls\" (UID: \"4d117442-da24-4a63-8182-041e7522635d\") " pod="openshift-multus/multus-additional-cni-plugins-qcfls"
Sep 11 13:21:35.209947 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.209864    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4d117442-da24-4a63-8182-041e7522635d-system-cni-dir\") pod \"multus-additional-cni-plugins-qcfls\" (UID: \"4d117442-da24-4a63-8182-041e7522635d\") " pod="openshift-multus/multus-additional-cni-plugins-qcfls"
Sep 11 13:21:35.210159 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.209960    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/be2cb948-420f-4f91-b56b-a292efb25181-kubelet-config\") pod \"global-pull-secret-syncer-m6d9g\" (UID: \"be2cb948-420f-4f91-b56b-a292efb25181\") " pod="kube-system/global-pull-secret-syncer-m6d9g"
Sep 11 13:21:35.210159 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.210008    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9vs6\" (UniqueName: \"kubernetes.io/projected/a2354ca1-4b4e-4a17-a17e-52d61af0e80c-kube-api-access-s9vs6\") pod \"network-metrics-daemon-zb2qw\" (UID: \"a2354ca1-4b4e-4a17-a17e-52d61af0e80c\") " pod="openshift-multus/network-metrics-daemon-zb2qw"
Sep 11 13:21:35.210159 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.210036    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4d117442-da24-4a63-8182-041e7522635d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qcfls\" (UID: \"4d117442-da24-4a63-8182-041e7522635d\") " pod="openshift-multus/multus-additional-cni-plugins-qcfls"
Sep 11 13:21:35.210159 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.210072    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dlgl\" (UniqueName: \"kubernetes.io/projected/4d117442-da24-4a63-8182-041e7522635d-kube-api-access-9dlgl\") pod \"multus-additional-cni-plugins-qcfls\" (UID: \"4d117442-da24-4a63-8182-041e7522635d\") " pod="openshift-multus/multus-additional-cni-plugins-qcfls"
Sep 11 13:21:35.210159 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.210095    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/be2cb948-420f-4f91-b56b-a292efb25181-kubelet-config\") pod \"global-pull-secret-syncer-m6d9g\" (UID: \"be2cb948-420f-4f91-b56b-a292efb25181\") " pod="kube-system/global-pull-secret-syncer-m6d9g"
Sep 11 13:21:35.210159 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.210106    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/be2cb948-420f-4f91-b56b-a292efb25181-dbus\") pod \"global-pull-secret-syncer-m6d9g\" (UID: \"be2cb948-420f-4f91-b56b-a292efb25181\") " pod="kube-system/global-pull-secret-syncer-m6d9g"
Sep 11 13:21:35.210159 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.210141    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7205c288-a931-457b-bd3b-d68f4440176a-host-slash\") pod \"iptables-alerter-k27nx\" (UID: \"7205c288-a931-457b-bd3b-d68f4440176a\") " pod="openshift-network-operator/iptables-alerter-k27nx"
Sep 11 13:21:35.210535 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.210173    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/be2cb948-420f-4f91-b56b-a292efb25181-original-pull-secret\") pod \"global-pull-secret-syncer-m6d9g\" (UID: \"be2cb948-420f-4f91-b56b-a292efb25181\") " pod="kube-system/global-pull-secret-syncer-m6d9g"
Sep 11 13:21:35.210535 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.210191    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7205c288-a931-457b-bd3b-d68f4440176a-iptables-alerter-script\") pod \"iptables-alerter-k27nx\" (UID: \"7205c288-a931-457b-bd3b-d68f4440176a\") " pod="openshift-network-operator/iptables-alerter-k27nx"
Sep 11 13:21:35.210535 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.210214    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2354ca1-4b4e-4a17-a17e-52d61af0e80c-metrics-certs\") pod \"network-metrics-daemon-zb2qw\" (UID: \"a2354ca1-4b4e-4a17-a17e-52d61af0e80c\") " pod="openshift-multus/network-metrics-daemon-zb2qw"
Sep 11 13:21:35.210535 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.210231    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4d117442-da24-4a63-8182-041e7522635d-cnibin\") pod \"multus-additional-cni-plugins-qcfls\" (UID: \"4d117442-da24-4a63-8182-041e7522635d\") " pod="openshift-multus/multus-additional-cni-plugins-qcfls"
Sep 11 13:21:35.210535 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.210248    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4d117442-da24-4a63-8182-041e7522635d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qcfls\" (UID: \"4d117442-da24-4a63-8182-041e7522635d\") " pod="openshift-multus/multus-additional-cni-plugins-qcfls"
Sep 11 13:21:35.210535 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.210268    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4d117442-da24-4a63-8182-041e7522635d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qcfls\" (UID: \"4d117442-da24-4a63-8182-041e7522635d\") " pod="openshift-multus/multus-additional-cni-plugins-qcfls"
Sep 11 13:21:35.210535 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.210268    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/be2cb948-420f-4f91-b56b-a292efb25181-dbus\") pod \"global-pull-secret-syncer-m6d9g\" (UID: \"be2cb948-420f-4f91-b56b-a292efb25181\") " pod="kube-system/global-pull-secret-syncer-m6d9g"
Sep 11 13:21:35.210535 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.210330    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7205c288-a931-457b-bd3b-d68f4440176a-host-slash\") pod \"iptables-alerter-k27nx\" (UID: \"7205c288-a931-457b-bd3b-d68f4440176a\") " pod="openshift-network-operator/iptables-alerter-k27nx"
Sep 11 13:21:35.210535 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:21:35.210413    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 13:21:35.210535 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.210435    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4d117442-da24-4a63-8182-041e7522635d-cni-binary-copy\") pod \"multus-additional-cni-plugins-qcfls\" (UID: \"4d117442-da24-4a63-8182-041e7522635d\") " pod="openshift-multus/multus-additional-cni-plugins-qcfls"
Sep 11 13:21:35.210535 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:21:35.210474    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be2cb948-420f-4f91-b56b-a292efb25181-original-pull-secret podName:be2cb948-420f-4f91-b56b-a292efb25181 nodeName:}" failed. No retries permitted until 2026-09-11 13:21:35.710456586 +0000 UTC m=+3.096559730 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/be2cb948-420f-4f91-b56b-a292efb25181-original-pull-secret") pod "global-pull-secret-syncer-m6d9g" (UID: "be2cb948-420f-4f91-b56b-a292efb25181") : object "kube-system"/"original-pull-secret" not registered
Sep 11 13:21:35.211184 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.210605    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4d117442-da24-4a63-8182-041e7522635d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qcfls\" (UID: \"4d117442-da24-4a63-8182-041e7522635d\") " pod="openshift-multus/multus-additional-cni-plugins-qcfls"
Sep 11 13:21:35.211184 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:21:35.210948    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 13:21:35.211184 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:21:35.211005    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a2354ca1-4b4e-4a17-a17e-52d61af0e80c-metrics-certs podName:a2354ca1-4b4e-4a17-a17e-52d61af0e80c nodeName:}" failed. No retries permitted until 2026-09-11 13:21:35.710986748 +0000 UTC m=+3.097089891 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a2354ca1-4b4e-4a17-a17e-52d61af0e80c-metrics-certs") pod "network-metrics-daemon-zb2qw" (UID: "a2354ca1-4b4e-4a17-a17e-52d61af0e80c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 13:21:35.211184 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.211025    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7205c288-a931-457b-bd3b-d68f4440176a-iptables-alerter-script\") pod \"iptables-alerter-k27nx\" (UID: \"7205c288-a931-457b-bd3b-d68f4440176a\") " pod="openshift-network-operator/iptables-alerter-k27nx"
Sep 11 13:21:35.211184 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.211048    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4d117442-da24-4a63-8182-041e7522635d-cnibin\") pod \"multus-additional-cni-plugins-qcfls\" (UID: \"4d117442-da24-4a63-8182-041e7522635d\") " pod="openshift-multus/multus-additional-cni-plugins-qcfls"
Sep 11 13:21:35.211469 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.211197    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4d117442-da24-4a63-8182-041e7522635d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qcfls\" (UID: \"4d117442-da24-4a63-8182-041e7522635d\") " pod="openshift-multus/multus-additional-cni-plugins-qcfls"
Sep 11 13:21:35.211469 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.211361    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4d117442-da24-4a63-8182-041e7522635d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qcfls\" (UID: \"4d117442-da24-4a63-8182-041e7522635d\") " pod="openshift-multus/multus-additional-cni-plugins-qcfls"
Sep 11 13:21:35.227098 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.227055    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sktn9\" (UniqueName: \"kubernetes.io/projected/7205c288-a931-457b-bd3b-d68f4440176a-kube-api-access-sktn9\") pod \"iptables-alerter-k27nx\" (UID: \"7205c288-a931-457b-bd3b-d68f4440176a\") " pod="openshift-network-operator/iptables-alerter-k27nx"
Sep 11 13:21:35.227665 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.227632    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9vs6\" (UniqueName: \"kubernetes.io/projected/a2354ca1-4b4e-4a17-a17e-52d61af0e80c-kube-api-access-s9vs6\") pod \"network-metrics-daemon-zb2qw\" (UID: \"a2354ca1-4b4e-4a17-a17e-52d61af0e80c\") " pod="openshift-multus/network-metrics-daemon-zb2qw"
Sep 11 13:21:35.227791 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.227639    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dlgl\" (UniqueName: \"kubernetes.io/projected/4d117442-da24-4a63-8182-041e7522635d-kube-api-access-9dlgl\") pod \"multus-additional-cni-plugins-qcfls\" (UID: \"4d117442-da24-4a63-8182-041e7522635d\") " pod="openshift-multus/multus-additional-cni-plugins-qcfls"
Sep 11 13:21:35.307592 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.307543    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-6gltj"
Sep 11 13:21:35.316023 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.315980    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-25zm6"
Sep 11 13:21:35.332882 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.332830    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-smdjz"
Sep 11 13:21:35.336231 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.336195    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-z8p25"
Sep 11 13:21:35.345658 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.345620    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-8jj8l"
Sep 11 13:21:35.354034 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.353991    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kp24z"
Sep 11 13:21:35.362589 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.362558    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-v874k"
Sep 11 13:21:35.374959 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.374899    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-k27nx"
Sep 11 13:21:35.381481 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.381435    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-qcfls"
Sep 11 13:21:35.714831 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.714713    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58f48\" (UniqueName: \"kubernetes.io/projected/1004285c-1dd5-49bf-be06-8549f85cdeda-kube-api-access-58f48\") pod \"network-check-target-6frmx\" (UID: \"1004285c-1dd5-49bf-be06-8549f85cdeda\") " pod="openshift-network-diagnostics/network-check-target-6frmx"
Sep 11 13:21:35.715067 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.714891    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/be2cb948-420f-4f91-b56b-a292efb25181-original-pull-secret\") pod \"global-pull-secret-syncer-m6d9g\" (UID: \"be2cb948-420f-4f91-b56b-a292efb25181\") " pod="kube-system/global-pull-secret-syncer-m6d9g"
Sep 11 13:21:35.715067 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:35.714939    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2354ca1-4b4e-4a17-a17e-52d61af0e80c-metrics-certs\") pod \"network-metrics-daemon-zb2qw\" (UID: \"a2354ca1-4b4e-4a17-a17e-52d61af0e80c\") " pod="openshift-multus/network-metrics-daemon-zb2qw"
Sep 11 13:21:35.715067 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:21:35.714972    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 13:21:35.715067 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:21:35.715003    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 13:21:35.715067 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:21:35.715016    2578 projected.go:196] Error preparing data for projected volume kube-api-access-58f48 for pod openshift-network-diagnostics/network-check-target-6frmx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 13:21:35.715322 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:21:35.715071    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 13:21:35.715322 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:21:35.715071    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 13:21:35.715322 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:21:35.715099    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1004285c-1dd5-49bf-be06-8549f85cdeda-kube-api-access-58f48 podName:1004285c-1dd5-49bf-be06-8549f85cdeda nodeName:}" failed. No retries permitted until 2026-09-11 13:21:36.715075631 +0000 UTC m=+4.101178800 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-58f48" (UniqueName: "kubernetes.io/projected/1004285c-1dd5-49bf-be06-8549f85cdeda-kube-api-access-58f48") pod "network-check-target-6frmx" (UID: "1004285c-1dd5-49bf-be06-8549f85cdeda") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 13:21:35.715322 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:21:35.715129    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be2cb948-420f-4f91-b56b-a292efb25181-original-pull-secret podName:be2cb948-420f-4f91-b56b-a292efb25181 nodeName:}" failed. No retries permitted until 2026-09-11 13:21:36.715112572 +0000 UTC m=+4.101215731 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/be2cb948-420f-4f91-b56b-a292efb25181-original-pull-secret") pod "global-pull-secret-syncer-m6d9g" (UID: "be2cb948-420f-4f91-b56b-a292efb25181") : object "kube-system"/"original-pull-secret" not registered
Sep 11 13:21:35.715322 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:21:35.715152    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a2354ca1-4b4e-4a17-a17e-52d61af0e80c-metrics-certs podName:a2354ca1-4b4e-4a17-a17e-52d61af0e80c nodeName:}" failed. No retries permitted until 2026-09-11 13:21:36.715140533 +0000 UTC m=+4.101243677 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a2354ca1-4b4e-4a17-a17e-52d61af0e80c-metrics-certs") pod "network-metrics-daemon-zb2qw" (UID: "a2354ca1-4b4e-4a17-a17e-52d61af0e80c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 13:21:36.069345 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:36.069241    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-10 13:16:35 +0000 UTC" deadline="2028-03-21 12:07:26.231211029 +0000 UTC"
Sep 11 13:21:36.069345 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:36.069282    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13366h45m50.161933721s"
Sep 11 13:21:36.100304 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:36.100256    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zb2qw"
Sep 11 13:21:36.100548 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:21:36.100415    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zb2qw" podUID="a2354ca1-4b4e-4a17-a17e-52d61af0e80c"
Sep 11 13:21:36.406283 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:36.406248    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9bf058c_d89b_4236_86ce_3dee6f389711.slice/crio-87e91637ce93966f4a1df7e32a68f59878e166076ca4fd313116494d712539b9 WatchSource:0}: Error finding container 87e91637ce93966f4a1df7e32a68f59878e166076ca4fd313116494d712539b9: Status 404 returned error can't find the container with id 87e91637ce93966f4a1df7e32a68f59878e166076ca4fd313116494d712539b9
Sep 11 13:21:36.408453 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:36.408415    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode384caae_3cb1_4264_8a25_55b40ac8ac45.slice/crio-1186ff2c9d202b7ffb91a6f5d9811331a7e04534bee0311ba982a3cb50490a52 WatchSource:0}: Error finding container 1186ff2c9d202b7ffb91a6f5d9811331a7e04534bee0311ba982a3cb50490a52: Status 404 returned error can't find the container with id 1186ff2c9d202b7ffb91a6f5d9811331a7e04534bee0311ba982a3cb50490a52
Sep 11 13:21:36.410385 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:36.410357    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86a1ce82_9c9a_4a59_8633_96606b53067a.slice/crio-61af54b4138831c586826097fe4ca2419709ffff8da4e6c669617d814e08339c WatchSource:0}: Error finding container 61af54b4138831c586826097fe4ca2419709ffff8da4e6c669617d814e08339c: Status 404 returned error can't find the container with id 61af54b4138831c586826097fe4ca2419709ffff8da4e6c669617d814e08339c
Sep 11 13:21:36.411319 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:36.411254    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7ab88b5_7073_460e_b3d4_a9841f50b185.slice/crio-e6b1ae3576881663d5351be99193010d1b92cb6770f6cd9de84fbc68bd06c78c WatchSource:0}: Error finding container e6b1ae3576881663d5351be99193010d1b92cb6770f6cd9de84fbc68bd06c78c: Status 404 returned error can't find the container with id e6b1ae3576881663d5351be99193010d1b92cb6770f6cd9de84fbc68bd06c78c
Sep 11 13:21:36.412434 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:21:36.412364    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7205c288_a931_457b_bd3b_d68f4440176a.slice/crio-795030f40cea311a76abf37d104b9a20d43784c3db72ea589302052fef9e193a WatchSource:0}: Error finding container 795030f40cea311a76abf37d104b9a20d43784c3db72ea589302052fef9e193a: Status 404 returned error can't find the container with id 795030f40cea311a76abf37d104b9a20d43784c3db72ea589302052fef9e193a
Sep 11 13:21:36.724517 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:36.724461    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/be2cb948-420f-4f91-b56b-a292efb25181-original-pull-secret\") pod \"global-pull-secret-syncer-m6d9g\" (UID: \"be2cb948-420f-4f91-b56b-a292efb25181\") " pod="kube-system/global-pull-secret-syncer-m6d9g"
Sep 11 13:21:36.724517 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:36.724522    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2354ca1-4b4e-4a17-a17e-52d61af0e80c-metrics-certs\") pod \"network-metrics-daemon-zb2qw\" (UID: \"a2354ca1-4b4e-4a17-a17e-52d61af0e80c\") " pod="openshift-multus/network-metrics-daemon-zb2qw"
Sep 11 13:21:36.724783 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:36.724562    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58f48\" (UniqueName: \"kubernetes.io/projected/1004285c-1dd5-49bf-be06-8549f85cdeda-kube-api-access-58f48\") pod \"network-check-target-6frmx\" (UID: \"1004285c-1dd5-49bf-be06-8549f85cdeda\") " pod="openshift-network-diagnostics/network-check-target-6frmx"
Sep 11 13:21:36.724783 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:21:36.724664    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 13:21:36.724783 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:21:36.724685    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 13:21:36.724783 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:21:36.724717    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 13:21:36.724783 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:21:36.724734    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 13:21:36.724783 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:21:36.724746    2578 projected.go:196] Error preparing data for projected volume kube-api-access-58f48 for pod openshift-network-diagnostics/network-check-target-6frmx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 13:21:36.724783 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:21:36.724759    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be2cb948-420f-4f91-b56b-a292efb25181-original-pull-secret podName:be2cb948-420f-4f91-b56b-a292efb25181 nodeName:}" failed. No retries permitted until 2026-09-11 13:21:38.724734639 +0000 UTC m=+6.110837803 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/be2cb948-420f-4f91-b56b-a292efb25181-original-pull-secret") pod "global-pull-secret-syncer-m6d9g" (UID: "be2cb948-420f-4f91-b56b-a292efb25181") : object "kube-system"/"original-pull-secret" not registered
Sep 11 13:21:36.724783 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:21:36.724784    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1004285c-1dd5-49bf-be06-8549f85cdeda-kube-api-access-58f48 podName:1004285c-1dd5-49bf-be06-8549f85cdeda nodeName:}" failed. No retries permitted until 2026-09-11 13:21:38.724770124 +0000 UTC m=+6.110873267 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-58f48" (UniqueName: "kubernetes.io/projected/1004285c-1dd5-49bf-be06-8549f85cdeda-kube-api-access-58f48") pod "network-check-target-6frmx" (UID: "1004285c-1dd5-49bf-be06-8549f85cdeda") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 13:21:36.725105 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:21:36.724800    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a2354ca1-4b4e-4a17-a17e-52d61af0e80c-metrics-certs podName:a2354ca1-4b4e-4a17-a17e-52d61af0e80c nodeName:}" failed. No retries permitted until 2026-09-11 13:21:38.724793488 +0000 UTC m=+6.110896631 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a2354ca1-4b4e-4a17-a17e-52d61af0e80c-metrics-certs") pod "network-metrics-daemon-zb2qw" (UID: "a2354ca1-4b4e-4a17-a17e-52d61af0e80c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 13:21:37.070496 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:37.070341    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-10 13:16:35 +0000 UTC" deadline="2028-06-06 05:13:42.494630166 +0000 UTC"
Sep 11 13:21:37.070496 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:37.070429    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15207h52m5.42420579s"
Sep 11 13:21:37.103619 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:37.103579    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m6d9g"
Sep 11 13:21:37.103864 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:21:37.103746    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m6d9g" podUID="be2cb948-420f-4f91-b56b-a292efb25181"
Sep 11 13:21:37.103962 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:37.103939    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6frmx"
Sep 11 13:21:37.104057 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:21:37.104035    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6frmx" podUID="1004285c-1dd5-49bf-be06-8549f85cdeda"
Sep 11 13:21:37.126800 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:37.125605    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-86.ec2.internal" event={"ID":"ac822255a7ab3b2619823b5af72a01ec","Type":"ContainerStarted","Data":"8c2126b137e9f3c0d76495f30c3a6a8c03229cd1839cd052a382b5b3fd0c1750"}
Sep 11 13:21:37.128210 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:37.128168    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-smdjz" event={"ID":"eeb9973c-b94d-4b62-8ce1-3cdf114016f2","Type":"ContainerStarted","Data":"315f5ff2a3910b0e33911f2740947d2438a15e1a8a48d85b2157d374919c5420"}
Sep 11 13:21:37.143737 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:37.143685    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-25zm6" event={"ID":"4925d62a-ba17-4740-84c4-c7cf29069831","Type":"ContainerStarted","Data":"d099e30518359513c09762757edb2212a398adcd83ae841df0c47b45199e045a"}
Sep 11 13:21:37.146227 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:37.146154    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-86.ec2.internal" podStartSLOduration=3.146131376 podStartE2EDuration="3.146131376s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 13:21:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:21:37.145447743 +0000 UTC m=+4.531550928" watchObservedRunningTime="2026-09-11 13:21:37.146131376 +0000 UTC m=+4.532234544"
Sep 11 13:21:37.151812 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:37.151771    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-z8p25" event={"ID":"f7ab88b5-7073-460e-b3d4-a9841f50b185","Type":"ContainerStarted","Data":"e6b1ae3576881663d5351be99193010d1b92cb6770f6cd9de84fbc68bd06c78c"}
Sep 11 13:21:37.156124 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:37.156064    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kp24z" event={"ID":"86a1ce82-9c9a-4a59-8633-96606b53067a","Type":"ContainerStarted","Data":"61af54b4138831c586826097fe4ca2419709ffff8da4e6c669617d814e08339c"}
Sep 11 13:21:37.162386 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:37.162257    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6gltj" event={"ID":"e384caae-3cb1-4264-8a25-55b40ac8ac45","Type":"ContainerStarted","Data":"1186ff2c9d202b7ffb91a6f5d9811331a7e04534bee0311ba982a3cb50490a52"}
Sep 11 13:21:37.165990 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:37.164825    2578 generic.go:356] "Generic (PLEG): container finished" podID="76de799c7422d703cdded403fdfd9c70" containerID="9b8d07bd74ce837bf67c033b8e51158f09c85fed48e2a4879e2b626f17cc12b1" exitCode=0
Sep 11 13:21:37.165990 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:37.164958    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-86.ec2.internal" event={"ID":"76de799c7422d703cdded403fdfd9c70","Type":"ContainerDied","Data":"9b8d07bd74ce837bf67c033b8e51158f09c85fed48e2a4879e2b626f17cc12b1"}
Sep 11 13:21:37.172270 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:37.171996    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-v874k" event={"ID":"61643763-a1e1-4b2f-8e7c-548d0618d90c","Type":"ContainerStarted","Data":"4830133312b124a184e7f834bea66872f65f04ecf190b1411850b86a3f3ecbaf"}
Sep 11 13:21:37.189890 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:37.189764    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qcfls" event={"ID":"4d117442-da24-4a63-8182-041e7522635d","Type":"ContainerStarted","Data":"ef299fc81b72ab497bd8450df2e16937e6c3220db3dbd28148e278a0c3f99aad"}
Sep 11 13:21:37.207842 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:37.207779    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-k27nx" event={"ID":"7205c288-a931-457b-bd3b-d68f4440176a","Type":"ContainerStarted","Data":"795030f40cea311a76abf37d104b9a20d43784c3db72ea589302052fef9e193a"}
Sep 11 13:21:37.217022 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:37.216956    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-8jj8l" event={"ID":"d9bf058c-d89b-4236-86ce-3dee6f389711","Type":"ContainerStarted","Data":"87e91637ce93966f4a1df7e32a68f59878e166076ca4fd313116494d712539b9"}
Sep 11 13:21:38.100503 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:38.100452    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zb2qw"
Sep 11 13:21:38.101215 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:21:38.100705    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zb2qw" podUID="a2354ca1-4b4e-4a17-a17e-52d61af0e80c"
Sep 11 13:21:38.275700 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:38.275240    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-86.ec2.internal" event={"ID":"76de799c7422d703cdded403fdfd9c70","Type":"ContainerStarted","Data":"aac05ce72ba162ea20bacfb1cfbe6db7cddceb3bcdae88938845ae0074e72563"}
Sep 11 13:21:38.743862 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:38.743818    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/be2cb948-420f-4f91-b56b-a292efb25181-original-pull-secret\") pod \"global-pull-secret-syncer-m6d9g\" (UID: \"be2cb948-420f-4f91-b56b-a292efb25181\") " pod="kube-system/global-pull-secret-syncer-m6d9g"
Sep 11 13:21:38.744184 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:38.743891    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2354ca1-4b4e-4a17-a17e-52d61af0e80c-metrics-certs\") pod \"network-metrics-daemon-zb2qw\" (UID: \"a2354ca1-4b4e-4a17-a17e-52d61af0e80c\") " pod="openshift-multus/network-metrics-daemon-zb2qw"
Sep 11 13:21:38.744184 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:38.743936    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58f48\" (UniqueName: \"kubernetes.io/projected/1004285c-1dd5-49bf-be06-8549f85cdeda-kube-api-access-58f48\") pod \"network-check-target-6frmx\" (UID: \"1004285c-1dd5-49bf-be06-8549f85cdeda\") " pod="openshift-network-diagnostics/network-check-target-6frmx"
Sep 11 13:21:38.744184 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:21:38.744104    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 13:21:38.744184 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:21:38.744122    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 13:21:38.744184 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:21:38.744134    2578 projected.go:196] Error preparing data for projected volume kube-api-access-58f48 for pod openshift-network-diagnostics/network-check-target-6frmx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 13:21:38.744454 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:21:38.744210    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1004285c-1dd5-49bf-be06-8549f85cdeda-kube-api-access-58f48 podName:1004285c-1dd5-49bf-be06-8549f85cdeda nodeName:}" failed. No retries permitted until 2026-09-11 13:21:42.744183092 +0000 UTC m=+10.130286236 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-58f48" (UniqueName: "kubernetes.io/projected/1004285c-1dd5-49bf-be06-8549f85cdeda-kube-api-access-58f48") pod "network-check-target-6frmx" (UID: "1004285c-1dd5-49bf-be06-8549f85cdeda") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 13:21:38.744454 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:21:38.744291    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 13:21:38.744454 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:21:38.744325    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be2cb948-420f-4f91-b56b-a292efb25181-original-pull-secret podName:be2cb948-420f-4f91-b56b-a292efb25181 nodeName:}" failed. No retries permitted until 2026-09-11 13:21:42.744316874 +0000 UTC m=+10.130420017 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/be2cb948-420f-4f91-b56b-a292efb25181-original-pull-secret") pod "global-pull-secret-syncer-m6d9g" (UID: "be2cb948-420f-4f91-b56b-a292efb25181") : object "kube-system"/"original-pull-secret" not registered
Sep 11 13:21:38.744454 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:21:38.744372    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 13:21:38.744454 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:21:38.744395    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a2354ca1-4b4e-4a17-a17e-52d61af0e80c-metrics-certs podName:a2354ca1-4b4e-4a17-a17e-52d61af0e80c nodeName:}" failed. No retries permitted until 2026-09-11 13:21:42.74438779 +0000 UTC m=+10.130490934 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a2354ca1-4b4e-4a17-a17e-52d61af0e80c-metrics-certs") pod "network-metrics-daemon-zb2qw" (UID: "a2354ca1-4b4e-4a17-a17e-52d61af0e80c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 13:21:39.103293 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:39.103180    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6frmx"
Sep 11 13:21:39.103293 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:39.103252    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m6d9g"
Sep 11 13:21:39.104114 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:21:39.103362    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6frmx" podUID="1004285c-1dd5-49bf-be06-8549f85cdeda"
Sep 11 13:21:39.104114 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:21:39.103441    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m6d9g" podUID="be2cb948-420f-4f91-b56b-a292efb25181"
Sep 11 13:21:40.100338 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:40.100278    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zb2qw"
Sep 11 13:21:40.100624 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:21:40.100485    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zb2qw" podUID="a2354ca1-4b4e-4a17-a17e-52d61af0e80c"
Sep 11 13:21:41.102103 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:41.102059    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m6d9g"
Sep 11 13:21:41.102103 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:41.102108    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6frmx"
Sep 11 13:21:41.102779 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:21:41.102255    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m6d9g" podUID="be2cb948-420f-4f91-b56b-a292efb25181"
Sep 11 13:21:41.102779 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:21:41.102380    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6frmx" podUID="1004285c-1dd5-49bf-be06-8549f85cdeda"
Sep 11 13:21:42.100325 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:42.099989    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zb2qw"
Sep 11 13:21:42.100595 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:21:42.100513    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zb2qw" podUID="a2354ca1-4b4e-4a17-a17e-52d61af0e80c"
Sep 11 13:21:42.785108 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:42.785061    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58f48\" (UniqueName: \"kubernetes.io/projected/1004285c-1dd5-49bf-be06-8549f85cdeda-kube-api-access-58f48\") pod \"network-check-target-6frmx\" (UID: \"1004285c-1dd5-49bf-be06-8549f85cdeda\") " pod="openshift-network-diagnostics/network-check-target-6frmx"
Sep 11 13:21:42.785681 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:42.785133    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/be2cb948-420f-4f91-b56b-a292efb25181-original-pull-secret\") pod \"global-pull-secret-syncer-m6d9g\" (UID: \"be2cb948-420f-4f91-b56b-a292efb25181\") " pod="kube-system/global-pull-secret-syncer-m6d9g"
Sep 11 13:21:42.785681 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:42.785161    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2354ca1-4b4e-4a17-a17e-52d61af0e80c-metrics-certs\") pod \"network-metrics-daemon-zb2qw\" (UID: \"a2354ca1-4b4e-4a17-a17e-52d61af0e80c\") " pod="openshift-multus/network-metrics-daemon-zb2qw"
Sep 11 13:21:42.785681 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:21:42.785218    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 13:21:42.785681 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:21:42.785245    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 13:21:42.785681 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:21:42.785260    2578 projected.go:196] Error preparing data for projected volume kube-api-access-58f48 for pod openshift-network-diagnostics/network-check-target-6frmx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 13:21:42.785681 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:21:42.785286    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 13:21:42.785681 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:21:42.785322    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1004285c-1dd5-49bf-be06-8549f85cdeda-kube-api-access-58f48 podName:1004285c-1dd5-49bf-be06-8549f85cdeda nodeName:}" failed. No retries permitted until 2026-09-11 13:21:50.785305696 +0000 UTC m=+18.171408842 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-58f48" (UniqueName: "kubernetes.io/projected/1004285c-1dd5-49bf-be06-8549f85cdeda-kube-api-access-58f48") pod "network-check-target-6frmx" (UID: "1004285c-1dd5-49bf-be06-8549f85cdeda") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 13:21:42.785681 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:21:42.785341    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a2354ca1-4b4e-4a17-a17e-52d61af0e80c-metrics-certs podName:a2354ca1-4b4e-4a17-a17e-52d61af0e80c nodeName:}" failed. No retries permitted until 2026-09-11 13:21:50.785332826 +0000 UTC m=+18.171435971 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a2354ca1-4b4e-4a17-a17e-52d61af0e80c-metrics-certs") pod "network-metrics-daemon-zb2qw" (UID: "a2354ca1-4b4e-4a17-a17e-52d61af0e80c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 13:21:42.785681 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:21:42.785378    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 13:21:42.785681 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:21:42.785419    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be2cb948-420f-4f91-b56b-a292efb25181-original-pull-secret podName:be2cb948-420f-4f91-b56b-a292efb25181 nodeName:}" failed. No retries permitted until 2026-09-11 13:21:50.785401777 +0000 UTC m=+18.171504924 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/be2cb948-420f-4f91-b56b-a292efb25181-original-pull-secret") pod "global-pull-secret-syncer-m6d9g" (UID: "be2cb948-420f-4f91-b56b-a292efb25181") : object "kube-system"/"original-pull-secret" not registered
Sep 11 13:21:43.101220 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:43.101045    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m6d9g"
Sep 11 13:21:43.101220 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:43.101101    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6frmx"
Sep 11 13:21:43.101220 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:21:43.101186    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m6d9g" podUID="be2cb948-420f-4f91-b56b-a292efb25181"
Sep 11 13:21:43.101574 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:21:43.101281    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6frmx" podUID="1004285c-1dd5-49bf-be06-8549f85cdeda"
Sep 11 13:21:44.100674 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:44.100622    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zb2qw"
Sep 11 13:21:44.101204 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:21:44.100844    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zb2qw" podUID="a2354ca1-4b4e-4a17-a17e-52d61af0e80c"
Sep 11 13:21:45.100296 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:45.100246    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6frmx"
Sep 11 13:21:45.100296 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:45.100293    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m6d9g"
Sep 11 13:21:45.100617 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:21:45.100447    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6frmx" podUID="1004285c-1dd5-49bf-be06-8549f85cdeda"
Sep 11 13:21:45.101093 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:21:45.101059    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m6d9g" podUID="be2cb948-420f-4f91-b56b-a292efb25181"
Sep 11 13:21:46.100013 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:46.099963    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zb2qw"
Sep 11 13:21:46.100276 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:21:46.100151    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zb2qw" podUID="a2354ca1-4b4e-4a17-a17e-52d61af0e80c"
Sep 11 13:21:47.100085 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:47.100046    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m6d9g"
Sep 11 13:21:47.100618 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:47.100046    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6frmx"
Sep 11 13:21:47.100618 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:21:47.100208    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m6d9g" podUID="be2cb948-420f-4f91-b56b-a292efb25181"
Sep 11 13:21:47.100618 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:21:47.100260    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6frmx" podUID="1004285c-1dd5-49bf-be06-8549f85cdeda"
Sep 11 13:21:48.099960 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:48.099928    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zb2qw"
Sep 11 13:21:48.100173 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:21:48.100064    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zb2qw" podUID="a2354ca1-4b4e-4a17-a17e-52d61af0e80c"
Sep 11 13:21:49.100169 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:49.100129    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6frmx"
Sep 11 13:21:49.100169 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:49.100191    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m6d9g"
Sep 11 13:21:49.100696 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:21:49.100313    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6frmx" podUID="1004285c-1dd5-49bf-be06-8549f85cdeda"
Sep 11 13:21:49.100696 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:21:49.100436    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m6d9g" podUID="be2cb948-420f-4f91-b56b-a292efb25181"
Sep 11 13:21:50.100291 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:50.100234    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zb2qw"
Sep 11 13:21:50.100951 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:21:50.100418    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zb2qw" podUID="a2354ca1-4b4e-4a17-a17e-52d61af0e80c"
Sep 11 13:21:50.856739 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:50.856696    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/be2cb948-420f-4f91-b56b-a292efb25181-original-pull-secret\") pod \"global-pull-secret-syncer-m6d9g\" (UID: \"be2cb948-420f-4f91-b56b-a292efb25181\") " pod="kube-system/global-pull-secret-syncer-m6d9g"
Sep 11 13:21:50.856739 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:50.856740    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2354ca1-4b4e-4a17-a17e-52d61af0e80c-metrics-certs\") pod \"network-metrics-daemon-zb2qw\" (UID: \"a2354ca1-4b4e-4a17-a17e-52d61af0e80c\") " pod="openshift-multus/network-metrics-daemon-zb2qw"
Sep 11 13:21:50.857032 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:21:50.856862    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 13:21:50.857032 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:21:50.856864    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 13:21:50.857032 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:50.856880    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58f48\" (UniqueName: \"kubernetes.io/projected/1004285c-1dd5-49bf-be06-8549f85cdeda-kube-api-access-58f48\") pod \"network-check-target-6frmx\" (UID: \"1004285c-1dd5-49bf-be06-8549f85cdeda\") " pod="openshift-network-diagnostics/network-check-target-6frmx"
Sep 11 13:21:50.857032 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:21:50.856913    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a2354ca1-4b4e-4a17-a17e-52d61af0e80c-metrics-certs podName:a2354ca1-4b4e-4a17-a17e-52d61af0e80c nodeName:}" failed. No retries permitted until 2026-09-11 13:22:06.856897476 +0000 UTC m=+34.243000618 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a2354ca1-4b4e-4a17-a17e-52d61af0e80c-metrics-certs") pod "network-metrics-daemon-zb2qw" (UID: "a2354ca1-4b4e-4a17-a17e-52d61af0e80c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 13:21:50.857032 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:21:50.856938    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be2cb948-420f-4f91-b56b-a292efb25181-original-pull-secret podName:be2cb948-420f-4f91-b56b-a292efb25181 nodeName:}" failed. No retries permitted until 2026-09-11 13:22:06.856931833 +0000 UTC m=+34.243034975 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/be2cb948-420f-4f91-b56b-a292efb25181-original-pull-secret") pod "global-pull-secret-syncer-m6d9g" (UID: "be2cb948-420f-4f91-b56b-a292efb25181") : object "kube-system"/"original-pull-secret" not registered
Sep 11 13:21:50.857032 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:21:50.856991    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 13:21:50.857032 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:21:50.857012    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 13:21:50.857032 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:21:50.857025    2578 projected.go:196] Error preparing data for projected volume kube-api-access-58f48 for pod openshift-network-diagnostics/network-check-target-6frmx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 13:21:50.857307 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:21:50.857079    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1004285c-1dd5-49bf-be06-8549f85cdeda-kube-api-access-58f48 podName:1004285c-1dd5-49bf-be06-8549f85cdeda nodeName:}" failed. No retries permitted until 2026-09-11 13:22:06.857063429 +0000 UTC m=+34.243166589 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-58f48" (UniqueName: "kubernetes.io/projected/1004285c-1dd5-49bf-be06-8549f85cdeda-kube-api-access-58f48") pod "network-check-target-6frmx" (UID: "1004285c-1dd5-49bf-be06-8549f85cdeda") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 13:21:51.100091 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:51.100029    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m6d9g"
Sep 11 13:21:51.100391 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:21:51.100207    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m6d9g" podUID="be2cb948-420f-4f91-b56b-a292efb25181"
Sep 11 13:21:51.100814 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:51.100738    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6frmx"
Sep 11 13:21:51.100906 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:21:51.100876    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6frmx" podUID="1004285c-1dd5-49bf-be06-8549f85cdeda"
Sep 11 13:21:52.100038 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:52.099995    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zb2qw"
Sep 11 13:21:52.100277 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:21:52.100174    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zb2qw" podUID="a2354ca1-4b4e-4a17-a17e-52d61af0e80c"
Sep 11 13:21:53.100468 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:53.100425    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6frmx"
Sep 11 13:21:53.100876 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:21:53.100563    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6frmx" podUID="1004285c-1dd5-49bf-be06-8549f85cdeda"
Sep 11 13:21:53.100876 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:53.100612    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m6d9g"
Sep 11 13:21:53.100876 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:21:53.100708    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m6d9g" podUID="be2cb948-420f-4f91-b56b-a292efb25181"
Sep 11 13:21:54.100859 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:54.100784    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zb2qw"
Sep 11 13:21:54.101512 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:21:54.101003    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zb2qw" podUID="a2354ca1-4b4e-4a17-a17e-52d61af0e80c"
Sep 11 13:21:55.101388 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:55.101299    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m6d9g"
Sep 11 13:21:55.102910 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:21:55.101485    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m6d9g" podUID="be2cb948-420f-4f91-b56b-a292efb25181"
Sep 11 13:21:55.104583 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:55.103335    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6frmx"
Sep 11 13:21:55.104583 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:21:55.103482    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6frmx" podUID="1004285c-1dd5-49bf-be06-8549f85cdeda"
Sep 11 13:21:55.329678 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:55.329373    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-8jj8l" event={"ID":"d9bf058c-d89b-4236-86ce-3dee6f389711","Type":"ContainerStarted","Data":"88f6fce6483e39d20ee97bbd4306bfe176fe3ab7d75e28c680ab7c9c9dd13d13"}
Sep 11 13:21:55.331598 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:55.331570    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-smdjz" event={"ID":"eeb9973c-b94d-4b62-8ce1-3cdf114016f2","Type":"ContainerStarted","Data":"228c1589592fb20ab796022b0a0a3555a43cb19f946813e642c46022dc38023f"}
Sep 11 13:21:55.333456 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:55.333429    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-25zm6" event={"ID":"4925d62a-ba17-4740-84c4-c7cf29069831","Type":"ContainerStarted","Data":"7118611bebdf6c18d3f72e4f97a20e1e10217c4c8c33c4e489661a803fea7d55"}
Sep 11 13:21:55.335191 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:55.335162    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-z8p25" event={"ID":"f7ab88b5-7073-460e-b3d4-a9841f50b185","Type":"ContainerStarted","Data":"583f8e41ad5fd3b30489c660989753b4f68b8f333de2badc66358caf6aa2a912"}
Sep 11 13:21:55.336915 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:55.336884    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kp24z" event={"ID":"86a1ce82-9c9a-4a59-8633-96606b53067a","Type":"ContainerStarted","Data":"34b608e8c167b5321408153535c3660597e7768adf52ff1cb91236066f191945"}
Sep 11 13:21:55.339039 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:55.339019    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6gltj_e384caae-3cb1-4264-8a25-55b40ac8ac45/ovn-acl-logging/0.log"
Sep 11 13:21:55.339421 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:55.339392    2578 generic.go:356] "Generic (PLEG): container finished" podID="e384caae-3cb1-4264-8a25-55b40ac8ac45" containerID="b6cc6bd0fd0feb0b50b9b475b78398902327b8f0e7e042cd46fe0c0333c3bc54" exitCode=1
Sep 11 13:21:55.339498 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:55.339465    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6gltj" event={"ID":"e384caae-3cb1-4264-8a25-55b40ac8ac45","Type":"ContainerStarted","Data":"e89f9ca273fd4faa5ac841acfb7ade8f06134e4afad6e1847fc2f2edc3db7448"}
Sep 11 13:21:55.339568 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:55.339552    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6gltj" event={"ID":"e384caae-3cb1-4264-8a25-55b40ac8ac45","Type":"ContainerDied","Data":"b6cc6bd0fd0feb0b50b9b475b78398902327b8f0e7e042cd46fe0c0333c3bc54"}
Sep 11 13:21:55.339628 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:55.339616    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6gltj" event={"ID":"e384caae-3cb1-4264-8a25-55b40ac8ac45","Type":"ContainerStarted","Data":"f1b831214b2af21c9a413e7d96345b4f376d56eba4b65f6ce886d2420bfde169"}
Sep 11 13:21:55.341082 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:55.341053    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-v874k" event={"ID":"61643763-a1e1-4b2f-8e7c-548d0618d90c","Type":"ContainerStarted","Data":"10b58cc7c038ed78731535301f438286db76127e2b23ea853b24db75263b24c5"}
Sep 11 13:21:55.342645 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:55.342618    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qcfls" event={"ID":"4d117442-da24-4a63-8182-041e7522635d","Type":"ContainerStarted","Data":"36bc89f356fffe39041d339ae4dd510caaf75562b11df76414d70506e5be0a2c"}
Sep 11 13:21:55.348379 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:55.348315    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-8jj8l" podStartSLOduration=3.878744708 podStartE2EDuration="22.348290613s" totalImagesPullingTime="18.469545905s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 13:21:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:21:55.347957444 +0000 UTC m=+22.734060679" watchObservedRunningTime="2026-09-11 13:21:55.348290613 +0000 UTC m=+22.734393778"
Sep 11 13:21:55.349001 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:55.348956    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-86.ec2.internal" podStartSLOduration=21.34894173 podStartE2EDuration="21.34894173s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 13:21:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:21:38.306284738 +0000 UTC m=+5.692387909" watchObservedRunningTime="2026-09-11 13:21:55.34894173 +0000 UTC m=+22.735044898"
Sep 11 13:21:55.371514 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:55.371425    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-v874k" podStartSLOduration=2.92031384 podStartE2EDuration="21.371399627s" totalImagesPullingTime="18.451085787s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:21:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:21:55.370283486 +0000 UTC m=+22.756386663" watchObservedRunningTime="2026-09-11 13:21:55.371399627 +0000 UTC m=+22.757502825"
Sep 11 13:21:55.408412 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:55.408340    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-25zm6" podStartSLOduration=3.894190764 podStartE2EDuration="22.408318914s" totalImagesPullingTime="18.51412815s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 13:21:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:21:55.408046175 +0000 UTC m=+22.794149340" watchObservedRunningTime="2026-09-11 13:21:55.408318914 +0000 UTC m=+22.794422077"
Sep 11 13:21:55.426033 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:55.425954    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-smdjz" podStartSLOduration=3.968632947 podStartE2EDuration="22.425927621s" totalImagesPullingTime="18.457294674s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 13:21:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:21:55.425272506 +0000 UTC m=+22.811375671" watchObservedRunningTime="2026-09-11 13:21:55.425927621 +0000 UTC m=+22.812030784"
Sep 11 13:21:55.444182 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:55.444122    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-z8p25" podStartSLOduration=3.963688199 podStartE2EDuration="22.444107894s" totalImagesPullingTime="18.480419695s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 13:21:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:21:55.443581516 +0000 UTC m=+22.829684699" watchObservedRunningTime="2026-09-11 13:21:55.444107894 +0000 UTC m=+22.830211058"
Sep 11 13:21:56.099643 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:56.099599    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zb2qw"
Sep 11 13:21:56.100008 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:21:56.099783    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zb2qw" podUID="a2354ca1-4b4e-4a17-a17e-52d61af0e80c"
Sep 11 13:21:56.190929 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:56.190875    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-v874k"
Sep 11 13:21:56.191893 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:56.191868    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-v874k"
Sep 11 13:21:56.352449 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:56.352344    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6gltj_e384caae-3cb1-4264-8a25-55b40ac8ac45/ovn-acl-logging/0.log"
Sep 11 13:21:56.352846 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:56.352796    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6gltj" event={"ID":"e384caae-3cb1-4264-8a25-55b40ac8ac45","Type":"ContainerStarted","Data":"99c0575eee217cd2395346cdaa1c64785a4132f9fe4093e992baf40c74454cc8"}
Sep 11 13:21:56.353004 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:56.352967    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6gltj" event={"ID":"e384caae-3cb1-4264-8a25-55b40ac8ac45","Type":"ContainerStarted","Data":"a5f2e67833a2f8d7009c8c3456c38e3460f2a3199b3fc4ba4aa543e278548746"}
Sep 11 13:21:56.353067 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:56.353027    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6gltj" event={"ID":"e384caae-3cb1-4264-8a25-55b40ac8ac45","Type":"ContainerStarted","Data":"f903da59188008c889a5ba67753721df537f8f9ba4bc201973ea124178f3ad94"}
Sep 11 13:21:56.354517 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:56.354477    2578 generic.go:356] "Generic (PLEG): container finished" podID="4d117442-da24-4a63-8182-041e7522635d" containerID="36bc89f356fffe39041d339ae4dd510caaf75562b11df76414d70506e5be0a2c" exitCode=0
Sep 11 13:21:56.354663 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:56.354550    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qcfls" event={"ID":"4d117442-da24-4a63-8182-041e7522635d","Type":"ContainerDied","Data":"36bc89f356fffe39041d339ae4dd510caaf75562b11df76414d70506e5be0a2c"}
Sep 11 13:21:56.532524 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:56.532307    2578 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 11 13:21:57.066889 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:57.066742    2578 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-11T13:21:56.53251267Z","UUID":"255fb566-abb5-4b78-8da7-b916e8c2338d","Handler":null,"Name":"","Endpoint":""}
Sep 11 13:21:57.069077 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:57.069043    2578 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 11 13:21:57.069293 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:57.069092    2578 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 11 13:21:57.099984 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:57.099948    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6frmx"
Sep 11 13:21:57.100163 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:57.100009    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m6d9g"
Sep 11 13:21:57.100163 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:21:57.100118    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6frmx" podUID="1004285c-1dd5-49bf-be06-8549f85cdeda"
Sep 11 13:21:57.100426 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:21:57.100400    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m6d9g" podUID="be2cb948-420f-4f91-b56b-a292efb25181"
Sep 11 13:21:57.359948 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:57.359824    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kp24z" event={"ID":"86a1ce82-9c9a-4a59-8633-96606b53067a","Type":"ContainerStarted","Data":"580ede694e230a8601a4aba775b66e3bb35852706d45aca490af70528db55c0f"}
Sep 11 13:21:57.362250 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:57.362209    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-k27nx" event={"ID":"7205c288-a931-457b-bd3b-d68f4440176a","Type":"ContainerStarted","Data":"119a6b19f71d5816aef04373bf4c445dcea3b6372d4d2aedc7d463767256720f"}
Sep 11 13:21:57.362250 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:57.362241    2578 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 11 13:21:58.100165 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:58.100027    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zb2qw"
Sep 11 13:21:58.100423 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:21:58.100221    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zb2qw" podUID="a2354ca1-4b4e-4a17-a17e-52d61af0e80c"
Sep 11 13:21:58.367481 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:58.367353    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kp24z" event={"ID":"86a1ce82-9c9a-4a59-8633-96606b53067a","Type":"ContainerStarted","Data":"58f1cbe5e91f81a3c6c72a74ad883d6fa20875e37d881304e7599972116db0fc"}
Sep 11 13:21:58.371744 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:58.371715    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6gltj_e384caae-3cb1-4264-8a25-55b40ac8ac45/ovn-acl-logging/0.log"
Sep 11 13:21:58.372655 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:58.372206    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6gltj" event={"ID":"e384caae-3cb1-4264-8a25-55b40ac8ac45","Type":"ContainerStarted","Data":"96b9e9e8ccc590bba1e0b6f14e2167c789d8c40a29e5b2414f19d224f651df69"}
Sep 11 13:21:58.399606 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:58.399530    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-k27nx" podStartSLOduration=5.946300878 podStartE2EDuration="24.399505071s" totalImagesPullingTime="18.453204193s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 13:21:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:21:57.381153959 +0000 UTC m=+24.767257122" watchObservedRunningTime="2026-09-11 13:21:58.399505071 +0000 UTC m=+25.785608235"
Sep 11 13:21:58.399918 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:58.399885    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kp24z" podStartSLOduration=4.691579327 podStartE2EDuration="25.399872338s" totalImagesPullingTime="20.708293011s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 13:21:33 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:21:58.399049175 +0000 UTC m=+25.785152364" watchObservedRunningTime="2026-09-11 13:21:58.399872338 +0000 UTC m=+25.785975501"
Sep 11 13:21:59.100253 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:59.100197    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m6d9g"
Sep 11 13:21:59.100584 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:21:59.100371    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m6d9g" podUID="be2cb948-420f-4f91-b56b-a292efb25181"
Sep 11 13:21:59.100961 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:21:59.100932    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6frmx"
Sep 11 13:21:59.101087 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:21:59.101062    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6frmx" podUID="1004285c-1dd5-49bf-be06-8549f85cdeda"
Sep 11 13:22:00.099722 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:22:00.099613    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zb2qw"
Sep 11 13:22:00.100184 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:22:00.099760    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zb2qw" podUID="a2354ca1-4b4e-4a17-a17e-52d61af0e80c"
Sep 11 13:22:01.100244 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:22:01.100187    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m6d9g"
Sep 11 13:22:01.100244 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:22:01.100230    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6frmx"
Sep 11 13:22:01.100932 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:22:01.100359    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m6d9g" podUID="be2cb948-420f-4f91-b56b-a292efb25181"
Sep 11 13:22:01.100932 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:22:01.100507    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6frmx" podUID="1004285c-1dd5-49bf-be06-8549f85cdeda"
Sep 11 13:22:01.388643 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:22:01.388302    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6gltj_e384caae-3cb1-4264-8a25-55b40ac8ac45/ovn-acl-logging/0.log"
Sep 11 13:22:01.389353 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:22:01.389276    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6gltj" event={"ID":"e384caae-3cb1-4264-8a25-55b40ac8ac45","Type":"ContainerStarted","Data":"dc665ed313dd3e9fab2267552a6b01a09a2861fa2c86ab0d22874a37702f5a6d"}
Sep 11 13:22:01.389793 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:22:01.389766    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-6gltj"
Sep 11 13:22:01.389948 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:22:01.389880    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-6gltj"
Sep 11 13:22:01.389948 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:22:01.389939    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-6gltj"
Sep 11 13:22:01.390028 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:22:01.390004    2578 scope.go:117] "RemoveContainer" containerID="b6cc6bd0fd0feb0b50b9b475b78398902327b8f0e7e042cd46fe0c0333c3bc54"
Sep 11 13:22:01.411722 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:22:01.411674    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6gltj"
Sep 11 13:22:01.417546 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:22:01.417513    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6gltj"
Sep 11 13:22:02.100363 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:22:02.100250    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zb2qw"
Sep 11 13:22:02.101068 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:22:02.100388    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zb2qw" podUID="a2354ca1-4b4e-4a17-a17e-52d61af0e80c"
Sep 11 13:22:02.400667 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:22:02.400556    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6gltj_e384caae-3cb1-4264-8a25-55b40ac8ac45/ovn-acl-logging/0.log"
Sep 11 13:22:02.401083 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:22:02.401056    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6gltj" event={"ID":"e384caae-3cb1-4264-8a25-55b40ac8ac45","Type":"ContainerStarted","Data":"f09540d187bca559c934cae27e898b8a3227828959455b3e3917eea171a826df"}
Sep 11 13:22:02.403054 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:22:02.403024    2578 generic.go:356] "Generic (PLEG): container finished" podID="4d117442-da24-4a63-8182-041e7522635d" containerID="f8bcd600bac3d56144ff9c65e4a535ef4c81e28a4c937fa3a3603328981287b6" exitCode=0
Sep 11 13:22:02.403174 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:22:02.403068    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qcfls" event={"ID":"4d117442-da24-4a63-8182-041e7522635d","Type":"ContainerDied","Data":"f8bcd600bac3d56144ff9c65e4a535ef4c81e28a4c937fa3a3603328981287b6"}
Sep 11 13:22:02.447151 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:22:02.447075    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-6gltj" podStartSLOduration=10.921872675 podStartE2EDuration="29.44704888s" totalImagesPullingTime="18.525176205s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 13:21:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:22:02.441279233 +0000 UTC m=+29.827382408" watchObservedRunningTime="2026-09-11 13:22:02.44704888 +0000 UTC m=+29.833152045"
Sep 11 13:22:03.101693 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:22:03.100928    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m6d9g"
Sep 11 13:22:03.101693 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:22:03.101082    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m6d9g" podUID="be2cb948-420f-4f91-b56b-a292efb25181"
Sep 11 13:22:03.101693 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:22:03.100932    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6frmx"
Sep 11 13:22:03.101693 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:22:03.101194    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6frmx" podUID="1004285c-1dd5-49bf-be06-8549f85cdeda"
Sep 11 13:22:03.381020 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:22:03.380674    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-m6d9g"]
Sep 11 13:22:03.384887 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:22:03.384850    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-zb2qw"]
Sep 11 13:22:03.385092 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:22:03.385012    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zb2qw"
Sep 11 13:22:03.385143 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:22:03.385121    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zb2qw" podUID="a2354ca1-4b4e-4a17-a17e-52d61af0e80c"
Sep 11 13:22:03.407260 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:22:03.407195    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-6frmx"]
Sep 11 13:22:03.408730 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:22:03.408694    2578 generic.go:356] "Generic (PLEG): container finished" podID="4d117442-da24-4a63-8182-041e7522635d" containerID="8d1efb5c7b770bad2b0194081c2f9528106365ff609ac1e6af90b38025c4727b" exitCode=0
Sep 11 13:22:03.408905 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:22:03.408807    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qcfls" event={"ID":"4d117442-da24-4a63-8182-041e7522635d","Type":"ContainerDied","Data":"8d1efb5c7b770bad2b0194081c2f9528106365ff609ac1e6af90b38025c4727b"}
Sep 11 13:22:03.408983 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:22:03.408962    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6frmx"
Sep 11 13:22:03.409031 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:22:03.408993    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m6d9g"
Sep 11 13:22:03.409126 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:22:03.409106    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6frmx" podUID="1004285c-1dd5-49bf-be06-8549f85cdeda"
Sep 11 13:22:03.409179 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:22:03.409156    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m6d9g" podUID="be2cb948-420f-4f91-b56b-a292efb25181"
Sep 11 13:22:03.970464 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:22:03.970416    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-v874k"
Sep 11 13:22:03.970772 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:22:03.970755    2578 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 11 13:22:03.971436 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:22:03.971310    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-v874k"
Sep 11 13:22:04.414692 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:22:04.414636    2578 generic.go:356] "Generic (PLEG): container finished" podID="4d117442-da24-4a63-8182-041e7522635d" containerID="a1fc828e25f5499e6770f675336b95b64a13c9364a09cadd04c58d36ad0ba8c2" exitCode=0
Sep 11 13:22:04.415379 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:22:04.414769    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qcfls" event={"ID":"4d117442-da24-4a63-8182-041e7522635d","Type":"ContainerDied","Data":"a1fc828e25f5499e6770f675336b95b64a13c9364a09cadd04c58d36ad0ba8c2"}
Sep 11 13:22:05.100692 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:22:05.100636    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zb2qw"
Sep 11 13:22:05.100692 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:22:05.100678    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m6d9g"
Sep 11 13:22:05.100692 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:22:05.100636    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6frmx"
Sep 11 13:22:05.101047 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:22:05.100890    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6frmx" podUID="1004285c-1dd5-49bf-be06-8549f85cdeda"
Sep 11 13:22:05.101047 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:22:05.100890    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m6d9g" podUID="be2cb948-420f-4f91-b56b-a292efb25181"
Sep 11 13:22:05.101047 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:22:05.100990    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zb2qw" podUID="a2354ca1-4b4e-4a17-a17e-52d61af0e80c"
Sep 11 13:22:06.892484 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:22:06.892427    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/be2cb948-420f-4f91-b56b-a292efb25181-original-pull-secret\") pod \"global-pull-secret-syncer-m6d9g\" (UID: \"be2cb948-420f-4f91-b56b-a292efb25181\") " pod="kube-system/global-pull-secret-syncer-m6d9g"
Sep 11 13:22:06.892484 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:22:06.892494    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2354ca1-4b4e-4a17-a17e-52d61af0e80c-metrics-certs\") pod \"network-metrics-daemon-zb2qw\" (UID: \"a2354ca1-4b4e-4a17-a17e-52d61af0e80c\") " pod="openshift-multus/network-metrics-daemon-zb2qw"
Sep 11 13:22:06.893218 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:22:06.892558    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58f48\" (UniqueName: \"kubernetes.io/projected/1004285c-1dd5-49bf-be06-8549f85cdeda-kube-api-access-58f48\") pod \"network-check-target-6frmx\" (UID: \"1004285c-1dd5-49bf-be06-8549f85cdeda\") " pod="openshift-network-diagnostics/network-check-target-6frmx"
Sep 11 13:22:06.893218 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:22:06.892648    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 13:22:06.893218 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:22:06.892654    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 13:22:06.893218 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:22:06.892712    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 13:22:06.893218 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:22:06.892735    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 13:22:06.893218 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:22:06.892748    2578 projected.go:196] Error preparing data for projected volume kube-api-access-58f48 for pod openshift-network-diagnostics/network-check-target-6frmx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 13:22:06.893218 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:22:06.892767    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a2354ca1-4b4e-4a17-a17e-52d61af0e80c-metrics-certs podName:a2354ca1-4b4e-4a17-a17e-52d61af0e80c nodeName:}" failed. No retries permitted until 2026-09-11 13:22:38.892740914 +0000 UTC m=+66.278844057 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a2354ca1-4b4e-4a17-a17e-52d61af0e80c-metrics-certs") pod "network-metrics-daemon-zb2qw" (UID: "a2354ca1-4b4e-4a17-a17e-52d61af0e80c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 13:22:06.893218 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:22:06.892799    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1004285c-1dd5-49bf-be06-8549f85cdeda-kube-api-access-58f48 podName:1004285c-1dd5-49bf-be06-8549f85cdeda nodeName:}" failed. No retries permitted until 2026-09-11 13:22:38.892782854 +0000 UTC m=+66.278886011 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-58f48" (UniqueName: "kubernetes.io/projected/1004285c-1dd5-49bf-be06-8549f85cdeda-kube-api-access-58f48") pod "network-check-target-6frmx" (UID: "1004285c-1dd5-49bf-be06-8549f85cdeda") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 13:22:06.893218 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:22:06.893186    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be2cb948-420f-4f91-b56b-a292efb25181-original-pull-secret podName:be2cb948-420f-4f91-b56b-a292efb25181 nodeName:}" failed. No retries permitted until 2026-09-11 13:22:38.893167945 +0000 UTC m=+66.279271088 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/be2cb948-420f-4f91-b56b-a292efb25181-original-pull-secret") pod "global-pull-secret-syncer-m6d9g" (UID: "be2cb948-420f-4f91-b56b-a292efb25181") : object "kube-system"/"original-pull-secret" not registered
Sep 11 13:22:07.100237 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:22:07.100107    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6frmx"
Sep 11 13:22:07.100237 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:22:07.100107    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zb2qw"
Sep 11 13:22:07.100546 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:22:07.100280    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6frmx" podUID="1004285c-1dd5-49bf-be06-8549f85cdeda"
Sep 11 13:22:07.100546 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:22:07.100107    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m6d9g"
Sep 11 13:22:07.100546 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:22:07.100319    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zb2qw" podUID="a2354ca1-4b4e-4a17-a17e-52d61af0e80c"
Sep 11 13:22:07.100546 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:22:07.100398    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m6d9g" podUID="be2cb948-420f-4f91-b56b-a292efb25181"
Sep 11 13:22:08.937974 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:22:08.937894    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-86.ec2.internal" event="NodeReady"
Sep 11 13:22:08.938424 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:22:08.938058    2578 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 11 13:22:08.994325 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:22:08.994285    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-pfffb"]
Sep 11 13:22:09.007730 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:22:09.007686    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-dcgnn"]
Sep 11 13:22:09.007924 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:22:09.007845    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-pfffb"
Sep 11 13:22:09.010379 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:22:09.010350    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 11 13:22:09.010683 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:22:09.010662    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-vqfbt\""
Sep 11 13:22:09.010946 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:22:09.010933    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 11 13:22:09.028961 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:22:09.028925    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-pfffb"]
Sep 11 13:22:09.029152 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:22:09.028983    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-dcgnn"]
Sep 11 13:22:09.029152 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:22:09.029099    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-dcgnn"
Sep 11 13:22:09.032204 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:22:09.032172    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 11 13:22:09.032344 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:22:09.032224    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 11 13:22:09.032344 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:22:09.032226    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 11 13:22:09.032344 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:22:09.032281    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-5ng6x\""
Sep 11 13:22:09.099979 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:22:09.099943    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6frmx"
Sep 11 13:22:09.100177 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:22:09.099943    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m6d9g"
Sep 11 13:22:09.100177 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:22:09.099943    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zb2qw"
Sep 11 13:22:09.105218 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:22:09.105196    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 11 13:22:09.106333 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:22:09.106051    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-tkfcx\""
Sep 11 13:22:09.106333 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:22:09.106139    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 11 13:22:09.106333 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:22:09.106224    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 11 13:22:09.106333 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:22:09.106143    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-j2l67\""
Sep 11 13:22:09.107368 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:22:09.107347    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 11 13:22:09.110465 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:22:09.110436    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58vpf\" (UniqueName: \"kubernetes.io/projected/5ab38ce4-40c2-4630-98f3-62d1b84532a5-kube-api-access-58vpf\") pod \"dns-default-pfffb\" (UID: \"5ab38ce4-40c2-4630-98f3-62d1b84532a5\") " pod="openshift-dns/dns-default-pfffb"
Sep 11 13:22:09.110581 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:22:09.110522    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5ab38ce4-40c2-4630-98f3-62d1b84532a5-config-volume\") pod \"dns-default-pfffb\" (UID: \"5ab38ce4-40c2-4630-98f3-62d1b84532a5\") " pod="openshift-dns/dns-default-pfffb"
Sep 11 13:22:09.110581 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:22:09.110550    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5ab38ce4-40c2-4630-98f3-62d1b84532a5-metrics-tls\") pod \"dns-default-pfffb\" (UID: \"5ab38ce4-40c2-4630-98f3-62d1b84532a5\") " pod="openshift-dns/dns-default-pfffb"
Sep 11 13:22:09.110581 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:22:09.110566    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ab38ce4-40c2-4630-98f3-62d1b84532a5-tmp-dir\") pod \"dns-default-pfffb\" (UID: \"5ab38ce4-40c2-4630-98f3-62d1b84532a5\") " pod="openshift-dns/dns-default-pfffb"
Sep 11 13:22:09.110725 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:22:09.110586    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/42df0901-f4f9-4a7e-85da-4551acd80be7-cert\") pod \"ingress-canary-dcgnn\" (UID: \"42df0901-f4f9-4a7e-85da-4551acd80be7\") " pod="openshift-ingress-canary/ingress-canary-dcgnn"
Sep 11 13:22:09.110725 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:22:09.110697    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlxw8\" (UniqueName: \"kubernetes.io/projected/42df0901-f4f9-4a7e-85da-4551acd80be7-kube-api-access-qlxw8\") pod \"ingress-canary-dcgnn\" (UID: \"42df0901-f4f9-4a7e-85da-4551acd80be7\") " pod="openshift-ingress-canary/ingress-canary-dcgnn"
Sep 11 13:22:09.212093 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:22:09.211978    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/42df0901-f4f9-4a7e-85da-4551acd80be7-cert\") pod \"ingress-canary-dcgnn\" (UID: \"42df0901-f4f9-4a7e-85da-4551acd80be7\") " pod="openshift-ingress-canary/ingress-canary-dcgnn"
Sep 11 13:22:09.212093 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:22:09.212087    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlxw8\" (UniqueName: \"kubernetes.io/projected/42df0901-f4f9-4a7e-85da-4551acd80be7-kube-api-access-qlxw8\") pod \"ingress-canary-dcgnn\" (UID: \"42df0901-f4f9-4a7e-85da-4551acd80be7\") " pod="openshift-ingress-canary/ingress-canary-dcgnn"
Sep 11 13:22:09.212380 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:22:09.212126    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58vpf\" (UniqueName: \"kubernetes.io/projected/5ab38ce4-40c2-4630-98f3-62d1b84532a5-kube-api-access-58vpf\") pod \"dns-default-pfffb\" (UID: \"5ab38ce4-40c2-4630-98f3-62d1b84532a5\") " pod="openshift-dns/dns-default-pfffb"
Sep 11 13:22:09.212380 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:22:09.212203    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5ab38ce4-40c2-4630-98f3-62d1b84532a5-config-volume\") pod \"dns-default-pfffb\" (UID: \"5ab38ce4-40c2-4630-98f3-62d1b84532a5\") " pod="openshift-dns/dns-default-pfffb"
Sep 11 13:22:09.212380 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:22:09.212240    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5ab38ce4-40c2-4630-98f3-62d1b84532a5-metrics-tls\") pod \"dns-default-pfffb\" (UID: \"5ab38ce4-40c2-4630-98f3-62d1b84532a5\") " pod="openshift-dns/dns-default-pfffb"
Sep 11 13:22:09.212380 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:22:09.212263    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ab38ce4-40c2-4630-98f3-62d1b84532a5-tmp-dir\") pod \"dns-default-pfffb\" (UID: \"5ab38ce4-40c2-4630-98f3-62d1b84532a5\") " pod="openshift-dns/dns-default-pfffb"
Sep 11 13:22:09.212380 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:22:09.212341    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 13:22:09.212589 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:22:09.212442    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ab38ce4-40c2-4630-98f3-62d1b84532a5-metrics-tls podName:5ab38ce4-40c2-4630-98f3-62d1b84532a5 nodeName:}" failed. No retries permitted until 2026-09-11 13:22:09.712413475 +0000 UTC m=+37.098516623 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5ab38ce4-40c2-4630-98f3-62d1b84532a5-metrics-tls") pod "dns-default-pfffb" (UID: "5ab38ce4-40c2-4630-98f3-62d1b84532a5") : secret "dns-default-metrics-tls" not found
Sep 11 13:22:09.213072 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:22:09.213048    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ab38ce4-40c2-4630-98f3-62d1b84532a5-tmp-dir\") pod \"dns-default-pfffb\" (UID: \"5ab38ce4-40c2-4630-98f3-62d1b84532a5\") " pod="openshift-dns/dns-default-pfffb"
Sep 11 13:22:09.213330 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:22:09.213296    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5ab38ce4-40c2-4630-98f3-62d1b84532a5-config-volume\") pod \"dns-default-pfffb\" (UID: \"5ab38ce4-40c2-4630-98f3-62d1b84532a5\") " pod="openshift-dns/dns-default-pfffb"
Sep 11 13:22:09.218645 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:22:09.218317    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/42df0901-f4f9-4a7e-85da-4551acd80be7-cert\") pod \"ingress-canary-dcgnn\" (UID: \"42df0901-f4f9-4a7e-85da-4551acd80be7\") " pod="openshift-ingress-canary/ingress-canary-dcgnn"
Sep 11 13:22:09.226204 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:22:09.226161    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlxw8\" (UniqueName: \"kubernetes.io/projected/42df0901-f4f9-4a7e-85da-4551acd80be7-kube-api-access-qlxw8\") pod \"ingress-canary-dcgnn\" (UID: \"42df0901-f4f9-4a7e-85da-4551acd80be7\") " pod="openshift-ingress-canary/ingress-canary-dcgnn"
Sep 11 13:22:09.226762 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:22:09.226720    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-58vpf\" (UniqueName: \"kubernetes.io/projected/5ab38ce4-40c2-4630-98f3-62d1b84532a5-kube-api-access-58vpf\") pod \"dns-default-pfffb\" (UID: \"5ab38ce4-40c2-4630-98f3-62d1b84532a5\") " pod="openshift-dns/dns-default-pfffb"
Sep 11 13:22:09.338392 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:22:09.338342    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-dcgnn"
Sep 11 13:22:09.434358 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:22:09.434312    2578 generic.go:356] "Generic (PLEG): container finished" podID="4d117442-da24-4a63-8182-041e7522635d" containerID="21626fa4a57e5d91d7994468dbc030255b6cf5adac41f15b9dea8d56b63d39b0" exitCode=0
Sep 11 13:22:09.435335 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:22:09.435287    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qcfls" event={"ID":"4d117442-da24-4a63-8182-041e7522635d","Type":"ContainerDied","Data":"21626fa4a57e5d91d7994468dbc030255b6cf5adac41f15b9dea8d56b63d39b0"}
Sep 11 13:22:09.661582 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:22:09.661490    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-dcgnn"]
Sep 11 13:22:09.686796 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:22:09.686759    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42df0901_f4f9_4a7e_85da_4551acd80be7.slice/crio-8aa9375b63fdb5b5086a4dd805a8cf1ba892dfeddd50397bafc115918ebcf4f2 WatchSource:0}: Error finding container 8aa9375b63fdb5b5086a4dd805a8cf1ba892dfeddd50397bafc115918ebcf4f2: Status 404 returned error can't find the container with id 8aa9375b63fdb5b5086a4dd805a8cf1ba892dfeddd50397bafc115918ebcf4f2
Sep 11 13:22:09.717003 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:22:09.716953    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5ab38ce4-40c2-4630-98f3-62d1b84532a5-metrics-tls\") pod \"dns-default-pfffb\" (UID: \"5ab38ce4-40c2-4630-98f3-62d1b84532a5\") " pod="openshift-dns/dns-default-pfffb"
Sep 11 13:22:09.717220 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:22:09.717142    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 13:22:09.717288 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:22:09.717232    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ab38ce4-40c2-4630-98f3-62d1b84532a5-metrics-tls podName:5ab38ce4-40c2-4630-98f3-62d1b84532a5 nodeName:}" failed. No retries permitted until 2026-09-11 13:22:10.717209976 +0000 UTC m=+38.103313120 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5ab38ce4-40c2-4630-98f3-62d1b84532a5-metrics-tls") pod "dns-default-pfffb" (UID: "5ab38ce4-40c2-4630-98f3-62d1b84532a5") : secret "dns-default-metrics-tls" not found
Sep 11 13:22:10.442273 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:22:10.442231    2578 generic.go:356] "Generic (PLEG): container finished" podID="4d117442-da24-4a63-8182-041e7522635d" containerID="ff63fd9585b17ac79b911b4e8fa3a2e63f60efbdaf55d05f77220fc6b10ea49b" exitCode=0
Sep 11 13:22:10.443420 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:22:10.442348    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qcfls" event={"ID":"4d117442-da24-4a63-8182-041e7522635d","Type":"ContainerDied","Data":"ff63fd9585b17ac79b911b4e8fa3a2e63f60efbdaf55d05f77220fc6b10ea49b"}
Sep 11 13:22:10.444054 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:22:10.443801    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-dcgnn" event={"ID":"42df0901-f4f9-4a7e-85da-4551acd80be7","Type":"ContainerStarted","Data":"8aa9375b63fdb5b5086a4dd805a8cf1ba892dfeddd50397bafc115918ebcf4f2"}
Sep 11 13:22:10.726414 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:22:10.726037    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5ab38ce4-40c2-4630-98f3-62d1b84532a5-metrics-tls\") pod \"dns-default-pfffb\" (UID: \"5ab38ce4-40c2-4630-98f3-62d1b84532a5\") " pod="openshift-dns/dns-default-pfffb"
Sep 11 13:22:10.726668 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:22:10.726246    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 13:22:10.726668 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:22:10.726584    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ab38ce4-40c2-4630-98f3-62d1b84532a5-metrics-tls podName:5ab38ce4-40c2-4630-98f3-62d1b84532a5 nodeName:}" failed. No retries permitted until 2026-09-11 13:22:12.726548968 +0000 UTC m=+40.112652114 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5ab38ce4-40c2-4630-98f3-62d1b84532a5-metrics-tls") pod "dns-default-pfffb" (UID: "5ab38ce4-40c2-4630-98f3-62d1b84532a5") : secret "dns-default-metrics-tls" not found
Sep 11 13:22:11.452881 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:22:11.452809    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qcfls" event={"ID":"4d117442-da24-4a63-8182-041e7522635d","Type":"ContainerStarted","Data":"b42061011f18cf6dfb067ae4d28023333d88bffff223a2db8bc8bda03812dc60"}
Sep 11 13:22:11.493960 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:22:11.493874    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-qcfls" podStartSLOduration=7.944855958 podStartE2EDuration="37.493826447s" totalImagesPullingTime="29.548970489s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 13:21:34 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:22:11.493668046 +0000 UTC m=+38.879771234" watchObservedRunningTime="2026-09-11 13:22:11.493826447 +0000 UTC m=+38.879929612"
Sep 11 13:22:12.457113 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:22:12.456999    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-dcgnn" event={"ID":"42df0901-f4f9-4a7e-85da-4551acd80be7","Type":"ContainerStarted","Data":"d4cda4f053b679f859b042d5ac595eca9ebcdfff2dfe0751e1b1421aef93fffa"}
Sep 11 13:22:12.478700 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:22:12.478645    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-dcgnn" podStartSLOduration=2.049194874 podStartE2EDuration="4.478625063s" totalImagesPullingTime="2.429430189s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:22:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:22:12.477148213 +0000 UTC m=+39.863251376" watchObservedRunningTime="2026-09-11 13:22:12.478625063 +0000 UTC m=+39.864728223"
Sep 11 13:22:12.742566 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:22:12.742507    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5ab38ce4-40c2-4630-98f3-62d1b84532a5-metrics-tls\") pod \"dns-default-pfffb\" (UID: \"5ab38ce4-40c2-4630-98f3-62d1b84532a5\") " pod="openshift-dns/dns-default-pfffb"
Sep 11 13:22:12.742765 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:22:12.742669    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 13:22:12.742765 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:22:12.742735    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ab38ce4-40c2-4630-98f3-62d1b84532a5-metrics-tls podName:5ab38ce4-40c2-4630-98f3-62d1b84532a5 nodeName:}" failed. No retries permitted until 2026-09-11 13:22:16.742716615 +0000 UTC m=+44.128819776 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5ab38ce4-40c2-4630-98f3-62d1b84532a5-metrics-tls") pod "dns-default-pfffb" (UID: "5ab38ce4-40c2-4630-98f3-62d1b84532a5") : secret "dns-default-metrics-tls" not found
Sep 11 13:22:16.770786 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:22:16.770709    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5ab38ce4-40c2-4630-98f3-62d1b84532a5-metrics-tls\") pod \"dns-default-pfffb\" (UID: \"5ab38ce4-40c2-4630-98f3-62d1b84532a5\") " pod="openshift-dns/dns-default-pfffb"
Sep 11 13:22:16.771574 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:22:16.770957    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 13:22:16.771574 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:22:16.771065    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ab38ce4-40c2-4630-98f3-62d1b84532a5-metrics-tls podName:5ab38ce4-40c2-4630-98f3-62d1b84532a5 nodeName:}" failed. No retries permitted until 2026-09-11 13:22:24.771035795 +0000 UTC m=+52.157138937 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5ab38ce4-40c2-4630-98f3-62d1b84532a5-metrics-tls") pod "dns-default-pfffb" (UID: "5ab38ce4-40c2-4630-98f3-62d1b84532a5") : secret "dns-default-metrics-tls" not found
Sep 11 13:22:24.834169 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:22:24.834121    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5ab38ce4-40c2-4630-98f3-62d1b84532a5-metrics-tls\") pod \"dns-default-pfffb\" (UID: \"5ab38ce4-40c2-4630-98f3-62d1b84532a5\") " pod="openshift-dns/dns-default-pfffb"
Sep 11 13:22:24.834721 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:22:24.834301    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 13:22:24.834721 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:22:24.834383    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ab38ce4-40c2-4630-98f3-62d1b84532a5-metrics-tls podName:5ab38ce4-40c2-4630-98f3-62d1b84532a5 nodeName:}" failed. No retries permitted until 2026-09-11 13:22:40.834361699 +0000 UTC m=+68.220464840 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5ab38ce4-40c2-4630-98f3-62d1b84532a5-metrics-tls") pod "dns-default-pfffb" (UID: "5ab38ce4-40c2-4630-98f3-62d1b84532a5") : secret "dns-default-metrics-tls" not found
Sep 11 13:22:33.421858 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:22:33.421801    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6gltj"
Sep 11 13:22:38.930561 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:22:38.930488    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/be2cb948-420f-4f91-b56b-a292efb25181-original-pull-secret\") pod \"global-pull-secret-syncer-m6d9g\" (UID: \"be2cb948-420f-4f91-b56b-a292efb25181\") " pod="kube-system/global-pull-secret-syncer-m6d9g"
Sep 11 13:22:38.930561 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:22:38.930556    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2354ca1-4b4e-4a17-a17e-52d61af0e80c-metrics-certs\") pod \"network-metrics-daemon-zb2qw\" (UID: \"a2354ca1-4b4e-4a17-a17e-52d61af0e80c\") " pod="openshift-multus/network-metrics-daemon-zb2qw"
Sep 11 13:22:38.931186 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:22:38.930592    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58f48\" (UniqueName: \"kubernetes.io/projected/1004285c-1dd5-49bf-be06-8549f85cdeda-kube-api-access-58f48\") pod \"network-check-target-6frmx\" (UID: \"1004285c-1dd5-49bf-be06-8549f85cdeda\") " pod="openshift-network-diagnostics/network-check-target-6frmx"
Sep 11 13:22:38.933546 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:22:38.933515    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 11 13:22:38.933655 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:22:38.933583    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 11 13:22:38.933765 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:22:38.933743    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 11 13:22:38.941526 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:22:38.941485    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 11 13:22:38.941699 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:22:38.941596    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a2354ca1-4b4e-4a17-a17e-52d61af0e80c-metrics-certs podName:a2354ca1-4b4e-4a17-a17e-52d61af0e80c nodeName:}" failed. No retries permitted until 2026-09-11 13:23:42.941572503 +0000 UTC m=+130.327675645 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a2354ca1-4b4e-4a17-a17e-52d61af0e80c-metrics-certs") pod "network-metrics-daemon-zb2qw" (UID: "a2354ca1-4b4e-4a17-a17e-52d61af0e80c") : secret "metrics-daemon-secret" not found
Sep 11 13:22:38.943948 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:22:38.943922    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 11 13:22:38.946193 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:22:38.946169    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/be2cb948-420f-4f91-b56b-a292efb25181-original-pull-secret\") pod \"global-pull-secret-syncer-m6d9g\" (UID: \"be2cb948-420f-4f91-b56b-a292efb25181\") " pod="kube-system/global-pull-secret-syncer-m6d9g"
Sep 11 13:22:38.955058 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:22:38.955031    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-58f48\" (UniqueName: \"kubernetes.io/projected/1004285c-1dd5-49bf-be06-8549f85cdeda-kube-api-access-58f48\") pod \"network-check-target-6frmx\" (UID: \"1004285c-1dd5-49bf-be06-8549f85cdeda\") " pod="openshift-network-diagnostics/network-check-target-6frmx"
Sep 11 13:22:39.116244 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:22:39.116206    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-tkfcx\""
Sep 11 13:22:39.119269 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:22:39.119244    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m6d9g"
Sep 11 13:22:39.124470 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:22:39.124434    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6frmx"
Sep 11 13:22:39.425361 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:22:39.425309    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-m6d9g"]
Sep 11 13:22:39.428511 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:22:39.428484    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-6frmx"]
Sep 11 13:22:39.433804 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:22:39.433756    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1004285c_1dd5_49bf_be06_8549f85cdeda.slice/crio-b6de4cefa5cc5de272b3aa91869336ebb833a7bb92f32c58edcc4a7e4db5554e WatchSource:0}: Error finding container b6de4cefa5cc5de272b3aa91869336ebb833a7bb92f32c58edcc4a7e4db5554e: Status 404 returned error can't find the container with id b6de4cefa5cc5de272b3aa91869336ebb833a7bb92f32c58edcc4a7e4db5554e
Sep 11 13:22:39.527487 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:22:39.527432    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-m6d9g" event={"ID":"be2cb948-420f-4f91-b56b-a292efb25181","Type":"ContainerStarted","Data":"b474718c9db4045d81137a31c108068a8ed54d95fb352e56ce8c745140d90220"}
Sep 11 13:22:39.528735 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:22:39.528704    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-6frmx" event={"ID":"1004285c-1dd5-49bf-be06-8549f85cdeda","Type":"ContainerStarted","Data":"b6de4cefa5cc5de272b3aa91869336ebb833a7bb92f32c58edcc4a7e4db5554e"}
Sep 11 13:22:40.844532 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:22:40.844488    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5ab38ce4-40c2-4630-98f3-62d1b84532a5-metrics-tls\") pod \"dns-default-pfffb\" (UID: \"5ab38ce4-40c2-4630-98f3-62d1b84532a5\") " pod="openshift-dns/dns-default-pfffb"
Sep 11 13:22:40.845242 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:22:40.844672    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 13:22:40.845242 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:22:40.844753    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ab38ce4-40c2-4630-98f3-62d1b84532a5-metrics-tls podName:5ab38ce4-40c2-4630-98f3-62d1b84532a5 nodeName:}" failed. No retries permitted until 2026-09-11 13:23:12.844728783 +0000 UTC m=+100.230831931 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5ab38ce4-40c2-4630-98f3-62d1b84532a5-metrics-tls") pod "dns-default-pfffb" (UID: "5ab38ce4-40c2-4630-98f3-62d1b84532a5") : secret "dns-default-metrics-tls" not found
Sep 11 13:22:44.546223 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:22:44.546164    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-6frmx" event={"ID":"1004285c-1dd5-49bf-be06-8549f85cdeda","Type":"ContainerStarted","Data":"27bc1a24976c887a059fac4b9e34ce9317de3af2e4bf20127455fc5f511acfc2"}
Sep 11 13:22:44.546894 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:22:44.546299    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-6frmx"
Sep 11 13:22:44.548065 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:22:44.548032    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-m6d9g" event={"ID":"be2cb948-420f-4f91-b56b-a292efb25181","Type":"ContainerStarted","Data":"77d82e936a087c7bb05317c73feec03400e41cea128805ff3071e0c31ccca7e9"}
Sep 11 13:22:44.568397 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:22:44.568312    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-6frmx" podStartSLOduration=66.961935171 podStartE2EDuration="1m11.568286231s" totalImagesPullingTime="4.60635106s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:21:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:22:44.567023735 +0000 UTC m=+71.953126915" watchObservedRunningTime="2026-09-11 13:22:44.568286231 +0000 UTC m=+71.954389396"
Sep 11 13:22:44.589634 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:22:44.589574    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-m6d9g" podStartSLOduration=65.975343808 podStartE2EDuration="1m10.589549359s" totalImagesPullingTime="4.614205551s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 13:21:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:22:44.589224259 +0000 UTC m=+71.975327424" watchObservedRunningTime="2026-09-11 13:22:44.589549359 +0000 UTC m=+71.975652523"
Sep 11 13:23:01.678688 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:01.678527    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5844b856db-m686n"]
Sep 11 13:23:01.683233 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:01.683209    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-m686n"
Sep 11 13:23:01.685687 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:01.685646    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-mc7pn\""
Sep 11 13:23:01.686224 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:01.686196    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 11 13:23:01.686412 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:01.686270    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Sep 11 13:23:01.686412 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:01.686284    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Sep 11 13:23:01.686412 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:01.686284    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 11 13:23:01.693347 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:01.693315    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Sep 11 13:23:01.694939 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:01.694888    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-m686n"]
Sep 11 13:23:01.793563 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:01.793513    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-7798cf8dd4-5hzkk"]
Sep 11 13:23:01.796749 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:01.796724    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7798cf8dd4-5hzkk"
Sep 11 13:23:01.799888 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:01.799853    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Sep 11 13:23:01.800217 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:01.800132    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 11 13:23:01.800217 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:01.800144    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Sep 11 13:23:01.800453 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:01.800211    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 11 13:23:01.800453 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:01.800295    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-4sphs\""
Sep 11 13:23:01.800453 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:01.800332    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Sep 11 13:23:01.800453 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:01.800359    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Sep 11 13:23:01.802563 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:01.802543    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f833a58e-8d54-42fc-848c-bea07b176b0b-tmp\") pod \"insights-operator-5844b856db-m686n\" (UID: \"f833a58e-8d54-42fc-848c-bea07b176b0b\") " pod="openshift-insights/insights-operator-5844b856db-m686n"
Sep 11 13:23:01.802624 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:01.802580    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxkbg\" (UniqueName: \"kubernetes.io/projected/f833a58e-8d54-42fc-848c-bea07b176b0b-kube-api-access-vxkbg\") pod \"insights-operator-5844b856db-m686n\" (UID: \"f833a58e-8d54-42fc-848c-bea07b176b0b\") " pod="openshift-insights/insights-operator-5844b856db-m686n"
Sep 11 13:23:01.802624 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:01.802611    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/f833a58e-8d54-42fc-848c-bea07b176b0b-snapshots\") pod \"insights-operator-5844b856db-m686n\" (UID: \"f833a58e-8d54-42fc-848c-bea07b176b0b\") " pod="openshift-insights/insights-operator-5844b856db-m686n"
Sep 11 13:23:01.802719 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:01.802700    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f833a58e-8d54-42fc-848c-bea07b176b0b-service-ca-bundle\") pod \"insights-operator-5844b856db-m686n\" (UID: \"f833a58e-8d54-42fc-848c-bea07b176b0b\") " pod="openshift-insights/insights-operator-5844b856db-m686n"
Sep 11 13:23:01.802779 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:01.802766    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f833a58e-8d54-42fc-848c-bea07b176b0b-trusted-ca-bundle\") pod \"insights-operator-5844b856db-m686n\" (UID: \"f833a58e-8d54-42fc-848c-bea07b176b0b\") " pod="openshift-insights/insights-operator-5844b856db-m686n"
Sep 11 13:23:01.802816 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:01.802800    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f833a58e-8d54-42fc-848c-bea07b176b0b-serving-cert\") pod \"insights-operator-5844b856db-m686n\" (UID: \"f833a58e-8d54-42fc-848c-bea07b176b0b\") " pod="openshift-insights/insights-operator-5844b856db-m686n"
Sep 11 13:23:01.808196 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:01.808170    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-7798cf8dd4-5hzkk"]
Sep 11 13:23:01.903496 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:01.903448    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/a93d3879-2973-4fa1-bd7d-bc855a257ddf-stats-auth\") pod \"router-default-7798cf8dd4-5hzkk\" (UID: \"a93d3879-2973-4fa1-bd7d-bc855a257ddf\") " pod="openshift-ingress/router-default-7798cf8dd4-5hzkk"
Sep 11 13:23:01.903496 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:01.903488    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-975tf\" (UniqueName: \"kubernetes.io/projected/a93d3879-2973-4fa1-bd7d-bc855a257ddf-kube-api-access-975tf\") pod \"router-default-7798cf8dd4-5hzkk\" (UID: \"a93d3879-2973-4fa1-bd7d-bc855a257ddf\") " pod="openshift-ingress/router-default-7798cf8dd4-5hzkk"
Sep 11 13:23:01.903884 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:01.903514    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f833a58e-8d54-42fc-848c-bea07b176b0b-serving-cert\") pod \"insights-operator-5844b856db-m686n\" (UID: \"f833a58e-8d54-42fc-848c-bea07b176b0b\") " pod="openshift-insights/insights-operator-5844b856db-m686n"
Sep 11 13:23:01.903884 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:01.903535    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f833a58e-8d54-42fc-848c-bea07b176b0b-tmp\") pod \"insights-operator-5844b856db-m686n\" (UID: \"f833a58e-8d54-42fc-848c-bea07b176b0b\") " pod="openshift-insights/insights-operator-5844b856db-m686n"
Sep 11 13:23:01.903884 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:01.903610    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a93d3879-2973-4fa1-bd7d-bc855a257ddf-metrics-certs\") pod \"router-default-7798cf8dd4-5hzkk\" (UID: \"a93d3879-2973-4fa1-bd7d-bc855a257ddf\") " pod="openshift-ingress/router-default-7798cf8dd4-5hzkk"
Sep 11 13:23:01.903884 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:01.903654    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxkbg\" (UniqueName: \"kubernetes.io/projected/f833a58e-8d54-42fc-848c-bea07b176b0b-kube-api-access-vxkbg\") pod \"insights-operator-5844b856db-m686n\" (UID: \"f833a58e-8d54-42fc-848c-bea07b176b0b\") " pod="openshift-insights/insights-operator-5844b856db-m686n"
Sep 11 13:23:01.903884 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:01.903685    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/f833a58e-8d54-42fc-848c-bea07b176b0b-snapshots\") pod \"insights-operator-5844b856db-m686n\" (UID: \"f833a58e-8d54-42fc-848c-bea07b176b0b\") " pod="openshift-insights/insights-operator-5844b856db-m686n"
Sep 11 13:23:01.903884 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:01.903742    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f833a58e-8d54-42fc-848c-bea07b176b0b-service-ca-bundle\") pod \"insights-operator-5844b856db-m686n\" (UID: \"f833a58e-8d54-42fc-848c-bea07b176b0b\") " pod="openshift-insights/insights-operator-5844b856db-m686n"
Sep 11 13:23:01.903884 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:01.903771    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/a93d3879-2973-4fa1-bd7d-bc855a257ddf-default-certificate\") pod \"router-default-7798cf8dd4-5hzkk\" (UID: \"a93d3879-2973-4fa1-bd7d-bc855a257ddf\") " pod="openshift-ingress/router-default-7798cf8dd4-5hzkk"
Sep 11 13:23:01.903884 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:01.903796    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a93d3879-2973-4fa1-bd7d-bc855a257ddf-service-ca-bundle\") pod \"router-default-7798cf8dd4-5hzkk\" (UID: \"a93d3879-2973-4fa1-bd7d-bc855a257ddf\") " pod="openshift-ingress/router-default-7798cf8dd4-5hzkk"
Sep 11 13:23:01.904228 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:01.903923    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f833a58e-8d54-42fc-848c-bea07b176b0b-trusted-ca-bundle\") pod \"insights-operator-5844b856db-m686n\" (UID: \"f833a58e-8d54-42fc-848c-bea07b176b0b\") " pod="openshift-insights/insights-operator-5844b856db-m686n"
Sep 11 13:23:01.904228 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:01.903927    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f833a58e-8d54-42fc-848c-bea07b176b0b-tmp\") pod \"insights-operator-5844b856db-m686n\" (UID: \"f833a58e-8d54-42fc-848c-bea07b176b0b\") " pod="openshift-insights/insights-operator-5844b856db-m686n"
Sep 11 13:23:01.904392 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:01.904365    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f833a58e-8d54-42fc-848c-bea07b176b0b-service-ca-bundle\") pod \"insights-operator-5844b856db-m686n\" (UID: \"f833a58e-8d54-42fc-848c-bea07b176b0b\") " pod="openshift-insights/insights-operator-5844b856db-m686n"
Sep 11 13:23:01.904538 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:01.904520    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/f833a58e-8d54-42fc-848c-bea07b176b0b-snapshots\") pod \"insights-operator-5844b856db-m686n\" (UID: \"f833a58e-8d54-42fc-848c-bea07b176b0b\") " pod="openshift-insights/insights-operator-5844b856db-m686n"
Sep 11 13:23:01.904644 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:01.904626    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f833a58e-8d54-42fc-848c-bea07b176b0b-trusted-ca-bundle\") pod \"insights-operator-5844b856db-m686n\" (UID: \"f833a58e-8d54-42fc-848c-bea07b176b0b\") " pod="openshift-insights/insights-operator-5844b856db-m686n"
Sep 11 13:23:01.906603 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:01.906582    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f833a58e-8d54-42fc-848c-bea07b176b0b-serving-cert\") pod \"insights-operator-5844b856db-m686n\" (UID: \"f833a58e-8d54-42fc-848c-bea07b176b0b\") " pod="openshift-insights/insights-operator-5844b856db-m686n"
Sep 11 13:23:01.915718 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:01.915670    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxkbg\" (UniqueName: \"kubernetes.io/projected/f833a58e-8d54-42fc-848c-bea07b176b0b-kube-api-access-vxkbg\") pod \"insights-operator-5844b856db-m686n\" (UID: \"f833a58e-8d54-42fc-848c-bea07b176b0b\") " pod="openshift-insights/insights-operator-5844b856db-m686n"
Sep 11 13:23:01.999451 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:01.999401    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-m686n"
Sep 11 13:23:02.005263 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:02.005207    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/a93d3879-2973-4fa1-bd7d-bc855a257ddf-stats-auth\") pod \"router-default-7798cf8dd4-5hzkk\" (UID: \"a93d3879-2973-4fa1-bd7d-bc855a257ddf\") " pod="openshift-ingress/router-default-7798cf8dd4-5hzkk"
Sep 11 13:23:02.005263 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:02.005260    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-975tf\" (UniqueName: \"kubernetes.io/projected/a93d3879-2973-4fa1-bd7d-bc855a257ddf-kube-api-access-975tf\") pod \"router-default-7798cf8dd4-5hzkk\" (UID: \"a93d3879-2973-4fa1-bd7d-bc855a257ddf\") " pod="openshift-ingress/router-default-7798cf8dd4-5hzkk"
Sep 11 13:23:02.005507 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:02.005298    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a93d3879-2973-4fa1-bd7d-bc855a257ddf-metrics-certs\") pod \"router-default-7798cf8dd4-5hzkk\" (UID: \"a93d3879-2973-4fa1-bd7d-bc855a257ddf\") " pod="openshift-ingress/router-default-7798cf8dd4-5hzkk"
Sep 11 13:23:02.005507 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:02.005406    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/a93d3879-2973-4fa1-bd7d-bc855a257ddf-default-certificate\") pod \"router-default-7798cf8dd4-5hzkk\" (UID: \"a93d3879-2973-4fa1-bd7d-bc855a257ddf\") " pod="openshift-ingress/router-default-7798cf8dd4-5hzkk"
Sep 11 13:23:02.005507 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:02.005439    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a93d3879-2973-4fa1-bd7d-bc855a257ddf-service-ca-bundle\") pod \"router-default-7798cf8dd4-5hzkk\" (UID: \"a93d3879-2973-4fa1-bd7d-bc855a257ddf\") " pod="openshift-ingress/router-default-7798cf8dd4-5hzkk"
Sep 11 13:23:02.005507 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:23:02.005413    2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 11 13:23:02.005709 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:23:02.005602    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a93d3879-2973-4fa1-bd7d-bc855a257ddf-metrics-certs podName:a93d3879-2973-4fa1-bd7d-bc855a257ddf nodeName:}" failed. No retries permitted until 2026-09-11 13:23:02.505569733 +0000 UTC m=+89.891672875 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a93d3879-2973-4fa1-bd7d-bc855a257ddf-metrics-certs") pod "router-default-7798cf8dd4-5hzkk" (UID: "a93d3879-2973-4fa1-bd7d-bc855a257ddf") : secret "router-metrics-certs-default" not found
Sep 11 13:23:02.005709 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:23:02.005644    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a93d3879-2973-4fa1-bd7d-bc855a257ddf-service-ca-bundle podName:a93d3879-2973-4fa1-bd7d-bc855a257ddf nodeName:}" failed. No retries permitted until 2026-09-11 13:23:02.505624726 +0000 UTC m=+89.891727888 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/a93d3879-2973-4fa1-bd7d-bc855a257ddf-service-ca-bundle") pod "router-default-7798cf8dd4-5hzkk" (UID: "a93d3879-2973-4fa1-bd7d-bc855a257ddf") : configmap references non-existent config key: service-ca.crt
Sep 11 13:23:02.008573 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:02.008537    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/a93d3879-2973-4fa1-bd7d-bc855a257ddf-stats-auth\") pod \"router-default-7798cf8dd4-5hzkk\" (UID: \"a93d3879-2973-4fa1-bd7d-bc855a257ddf\") " pod="openshift-ingress/router-default-7798cf8dd4-5hzkk"
Sep 11 13:23:02.008736 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:02.008576    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/a93d3879-2973-4fa1-bd7d-bc855a257ddf-default-certificate\") pod \"router-default-7798cf8dd4-5hzkk\" (UID: \"a93d3879-2973-4fa1-bd7d-bc855a257ddf\") " pod="openshift-ingress/router-default-7798cf8dd4-5hzkk"
Sep 11 13:23:02.015601 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:02.015565    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-975tf\" (UniqueName: \"kubernetes.io/projected/a93d3879-2973-4fa1-bd7d-bc855a257ddf-kube-api-access-975tf\") pod \"router-default-7798cf8dd4-5hzkk\" (UID: \"a93d3879-2973-4fa1-bd7d-bc855a257ddf\") " pod="openshift-ingress/router-default-7798cf8dd4-5hzkk"
Sep 11 13:23:02.282218 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:02.282095    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-m686n"]
Sep 11 13:23:02.286046 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:23:02.286003    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf833a58e_8d54_42fc_848c_bea07b176b0b.slice/crio-73c7dd0e0a448dd9077129d6f82abaa863a8a70e049f3025d3245f2ce1a06f55 WatchSource:0}: Error finding container 73c7dd0e0a448dd9077129d6f82abaa863a8a70e049f3025d3245f2ce1a06f55: Status 404 returned error can't find the container with id 73c7dd0e0a448dd9077129d6f82abaa863a8a70e049f3025d3245f2ce1a06f55
Sep 11 13:23:02.509995 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:02.509949    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a93d3879-2973-4fa1-bd7d-bc855a257ddf-service-ca-bundle\") pod \"router-default-7798cf8dd4-5hzkk\" (UID: \"a93d3879-2973-4fa1-bd7d-bc855a257ddf\") " pod="openshift-ingress/router-default-7798cf8dd4-5hzkk"
Sep 11 13:23:02.510225 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:02.510034    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a93d3879-2973-4fa1-bd7d-bc855a257ddf-metrics-certs\") pod \"router-default-7798cf8dd4-5hzkk\" (UID: \"a93d3879-2973-4fa1-bd7d-bc855a257ddf\") " pod="openshift-ingress/router-default-7798cf8dd4-5hzkk"
Sep 11 13:23:02.510225 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:23:02.510139    2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 11 13:23:02.510225 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:23:02.510157    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a93d3879-2973-4fa1-bd7d-bc855a257ddf-service-ca-bundle podName:a93d3879-2973-4fa1-bd7d-bc855a257ddf nodeName:}" failed. No retries permitted until 2026-09-11 13:23:03.510137915 +0000 UTC m=+90.896241057 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/a93d3879-2973-4fa1-bd7d-bc855a257ddf-service-ca-bundle") pod "router-default-7798cf8dd4-5hzkk" (UID: "a93d3879-2973-4fa1-bd7d-bc855a257ddf") : configmap references non-existent config key: service-ca.crt
Sep 11 13:23:02.510225 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:23:02.510186    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a93d3879-2973-4fa1-bd7d-bc855a257ddf-metrics-certs podName:a93d3879-2973-4fa1-bd7d-bc855a257ddf nodeName:}" failed. No retries permitted until 2026-09-11 13:23:03.510174858 +0000 UTC m=+90.896278018 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a93d3879-2973-4fa1-bd7d-bc855a257ddf-metrics-certs") pod "router-default-7798cf8dd4-5hzkk" (UID: "a93d3879-2973-4fa1-bd7d-bc855a257ddf") : secret "router-metrics-certs-default" not found
Sep 11 13:23:02.599172 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:02.599081    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-m686n" event={"ID":"f833a58e-8d54-42fc-848c-bea07b176b0b","Type":"ContainerStarted","Data":"73c7dd0e0a448dd9077129d6f82abaa863a8a70e049f3025d3245f2ce1a06f55"}
Sep 11 13:23:03.517399 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:03.517351    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a93d3879-2973-4fa1-bd7d-bc855a257ddf-metrics-certs\") pod \"router-default-7798cf8dd4-5hzkk\" (UID: \"a93d3879-2973-4fa1-bd7d-bc855a257ddf\") " pod="openshift-ingress/router-default-7798cf8dd4-5hzkk"
Sep 11 13:23:03.517992 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:03.517444    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a93d3879-2973-4fa1-bd7d-bc855a257ddf-service-ca-bundle\") pod \"router-default-7798cf8dd4-5hzkk\" (UID: \"a93d3879-2973-4fa1-bd7d-bc855a257ddf\") " pod="openshift-ingress/router-default-7798cf8dd4-5hzkk"
Sep 11 13:23:03.517992 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:23:03.517528    2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 11 13:23:03.517992 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:23:03.517596    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a93d3879-2973-4fa1-bd7d-bc855a257ddf-service-ca-bundle podName:a93d3879-2973-4fa1-bd7d-bc855a257ddf nodeName:}" failed. No retries permitted until 2026-09-11 13:23:05.517575276 +0000 UTC m=+92.903678432 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/a93d3879-2973-4fa1-bd7d-bc855a257ddf-service-ca-bundle") pod "router-default-7798cf8dd4-5hzkk" (UID: "a93d3879-2973-4fa1-bd7d-bc855a257ddf") : configmap references non-existent config key: service-ca.crt
Sep 11 13:23:03.517992 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:23:03.517649    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a93d3879-2973-4fa1-bd7d-bc855a257ddf-metrics-certs podName:a93d3879-2973-4fa1-bd7d-bc855a257ddf nodeName:}" failed. No retries permitted until 2026-09-11 13:23:05.517637693 +0000 UTC m=+92.903740835 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a93d3879-2973-4fa1-bd7d-bc855a257ddf-metrics-certs") pod "router-default-7798cf8dd4-5hzkk" (UID: "a93d3879-2973-4fa1-bd7d-bc855a257ddf") : secret "router-metrics-certs-default" not found
Sep 11 13:23:04.610145 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:04.610085    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-m686n" event={"ID":"f833a58e-8d54-42fc-848c-bea07b176b0b","Type":"ContainerStarted","Data":"cc8ed32e78e27a2c821e92dbfd2970302c108db7cbb3cfb5d523948014426323"}
Sep 11 13:23:04.628199 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:04.628138    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5844b856db-m686n" podStartSLOduration=1.455297731 podStartE2EDuration="3.628121502s" totalImagesPullingTime="2.172823771s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:23:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:23:04.627753896 +0000 UTC m=+92.013857060" watchObservedRunningTime="2026-09-11 13:23:04.628121502 +0000 UTC m=+92.014224688"
Sep 11 13:23:05.534015 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:05.533957    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a93d3879-2973-4fa1-bd7d-bc855a257ddf-service-ca-bundle\") pod \"router-default-7798cf8dd4-5hzkk\" (UID: \"a93d3879-2973-4fa1-bd7d-bc855a257ddf\") " pod="openshift-ingress/router-default-7798cf8dd4-5hzkk"
Sep 11 13:23:05.534227 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:05.534055    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a93d3879-2973-4fa1-bd7d-bc855a257ddf-metrics-certs\") pod \"router-default-7798cf8dd4-5hzkk\" (UID: \"a93d3879-2973-4fa1-bd7d-bc855a257ddf\") " pod="openshift-ingress/router-default-7798cf8dd4-5hzkk"
Sep 11 13:23:05.534227 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:23:05.534165    2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 11 13:23:05.534227 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:23:05.534194    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a93d3879-2973-4fa1-bd7d-bc855a257ddf-service-ca-bundle podName:a93d3879-2973-4fa1-bd7d-bc855a257ddf nodeName:}" failed. No retries permitted until 2026-09-11 13:23:09.53417347 +0000 UTC m=+96.920276628 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/a93d3879-2973-4fa1-bd7d-bc855a257ddf-service-ca-bundle") pod "router-default-7798cf8dd4-5hzkk" (UID: "a93d3879-2973-4fa1-bd7d-bc855a257ddf") : configmap references non-existent config key: service-ca.crt
Sep 11 13:23:05.534227 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:23:05.534222    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a93d3879-2973-4fa1-bd7d-bc855a257ddf-metrics-certs podName:a93d3879-2973-4fa1-bd7d-bc855a257ddf nodeName:}" failed. No retries permitted until 2026-09-11 13:23:09.534215784 +0000 UTC m=+96.920318926 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a93d3879-2973-4fa1-bd7d-bc855a257ddf-metrics-certs") pod "router-default-7798cf8dd4-5hzkk" (UID: "a93d3879-2973-4fa1-bd7d-bc855a257ddf") : secret "router-metrics-certs-default" not found
Sep 11 13:23:05.978252 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:05.978209    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-dsppd"]
Sep 11 13:23:05.981726 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:05.981699    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-dsppd"
Sep 11 13:23:05.988514 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:05.988473    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 11 13:23:05.988767 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:05.988542    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 11 13:23:05.988767 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:05.988614    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-kd6rn\""
Sep 11 13:23:06.001386 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:06.001350    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-dsppd"]
Sep 11 13:23:06.038258 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:06.038198    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9d1ee163-9447-4e3f-a7d5-90f8210d9c2f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dsppd\" (UID: \"9d1ee163-9447-4e3f-a7d5-90f8210d9c2f\") " pod="openshift-insights/insights-runtime-extractor-dsppd"
Sep 11 13:23:06.038529 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:06.038293    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9d1ee163-9447-4e3f-a7d5-90f8210d9c2f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dsppd\" (UID: \"9d1ee163-9447-4e3f-a7d5-90f8210d9c2f\") " pod="openshift-insights/insights-runtime-extractor-dsppd"
Sep 11 13:23:06.038529 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:06.038357    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9d1ee163-9447-4e3f-a7d5-90f8210d9c2f-crio-socket\") pod \"insights-runtime-extractor-dsppd\" (UID: \"9d1ee163-9447-4e3f-a7d5-90f8210d9c2f\") " pod="openshift-insights/insights-runtime-extractor-dsppd"
Sep 11 13:23:06.038529 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:06.038418    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9d1ee163-9447-4e3f-a7d5-90f8210d9c2f-data-volume\") pod \"insights-runtime-extractor-dsppd\" (UID: \"9d1ee163-9447-4e3f-a7d5-90f8210d9c2f\") " pod="openshift-insights/insights-runtime-extractor-dsppd"
Sep 11 13:23:06.038529 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:06.038454    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkwzn\" (UniqueName: \"kubernetes.io/projected/9d1ee163-9447-4e3f-a7d5-90f8210d9c2f-kube-api-access-vkwzn\") pod \"insights-runtime-extractor-dsppd\" (UID: \"9d1ee163-9447-4e3f-a7d5-90f8210d9c2f\") " pod="openshift-insights/insights-runtime-extractor-dsppd"
Sep 11 13:23:06.139053 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:06.138993    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9d1ee163-9447-4e3f-a7d5-90f8210d9c2f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dsppd\" (UID: \"9d1ee163-9447-4e3f-a7d5-90f8210d9c2f\") " pod="openshift-insights/insights-runtime-extractor-dsppd"
Sep 11 13:23:06.139053 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:06.139049    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9d1ee163-9447-4e3f-a7d5-90f8210d9c2f-crio-socket\") pod \"insights-runtime-extractor-dsppd\" (UID: \"9d1ee163-9447-4e3f-a7d5-90f8210d9c2f\") " pod="openshift-insights/insights-runtime-extractor-dsppd"
Sep 11 13:23:06.139333 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:06.139092    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9d1ee163-9447-4e3f-a7d5-90f8210d9c2f-data-volume\") pod \"insights-runtime-extractor-dsppd\" (UID: \"9d1ee163-9447-4e3f-a7d5-90f8210d9c2f\") " pod="openshift-insights/insights-runtime-extractor-dsppd"
Sep 11 13:23:06.139333 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:06.139112    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkwzn\" (UniqueName: \"kubernetes.io/projected/9d1ee163-9447-4e3f-a7d5-90f8210d9c2f-kube-api-access-vkwzn\") pod \"insights-runtime-extractor-dsppd\" (UID: \"9d1ee163-9447-4e3f-a7d5-90f8210d9c2f\") " pod="openshift-insights/insights-runtime-extractor-dsppd"
Sep 11 13:23:06.139333 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:06.139160    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9d1ee163-9447-4e3f-a7d5-90f8210d9c2f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dsppd\" (UID: \"9d1ee163-9447-4e3f-a7d5-90f8210d9c2f\") " pod="openshift-insights/insights-runtime-extractor-dsppd"
Sep 11 13:23:06.139333 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:23:06.139175    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 13:23:06.139333 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:06.139214    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9d1ee163-9447-4e3f-a7d5-90f8210d9c2f-crio-socket\") pod \"insights-runtime-extractor-dsppd\" (UID: \"9d1ee163-9447-4e3f-a7d5-90f8210d9c2f\") " pod="openshift-insights/insights-runtime-extractor-dsppd"
Sep 11 13:23:06.139333 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:23:06.139261    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 13:23:06.139333 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:23:06.139265    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d1ee163-9447-4e3f-a7d5-90f8210d9c2f-insights-runtime-extractor-tls podName:9d1ee163-9447-4e3f-a7d5-90f8210d9c2f nodeName:}" failed. No retries permitted until 2026-09-11 13:23:06.639245596 +0000 UTC m=+94.025348754 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9d1ee163-9447-4e3f-a7d5-90f8210d9c2f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dsppd" (UID: "9d1ee163-9447-4e3f-a7d5-90f8210d9c2f") : secret "insights-runtime-extractor-tls" not found
Sep 11 13:23:06.139674 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:23:06.139346    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9d1ee163-9447-4e3f-a7d5-90f8210d9c2f-kube-rbac-proxy-cm podName:9d1ee163-9447-4e3f-a7d5-90f8210d9c2f nodeName:}" failed. No retries permitted until 2026-09-11 13:23:06.639326123 +0000 UTC m=+94.025429271 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9d1ee163-9447-4e3f-a7d5-90f8210d9c2f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dsppd" (UID: "9d1ee163-9447-4e3f-a7d5-90f8210d9c2f") : configmap "kube-rbac-proxy" not found
Sep 11 13:23:06.139674 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:06.139581    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9d1ee163-9447-4e3f-a7d5-90f8210d9c2f-data-volume\") pod \"insights-runtime-extractor-dsppd\" (UID: \"9d1ee163-9447-4e3f-a7d5-90f8210d9c2f\") " pod="openshift-insights/insights-runtime-extractor-dsppd"
Sep 11 13:23:06.151712 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:06.151667    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkwzn\" (UniqueName: \"kubernetes.io/projected/9d1ee163-9447-4e3f-a7d5-90f8210d9c2f-kube-api-access-vkwzn\") pod \"insights-runtime-extractor-dsppd\" (UID: \"9d1ee163-9447-4e3f-a7d5-90f8210d9c2f\") " pod="openshift-insights/insights-runtime-extractor-dsppd"
Sep 11 13:23:06.643515 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:06.643440    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9d1ee163-9447-4e3f-a7d5-90f8210d9c2f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dsppd\" (UID: \"9d1ee163-9447-4e3f-a7d5-90f8210d9c2f\") " pod="openshift-insights/insights-runtime-extractor-dsppd"
Sep 11 13:23:06.643859 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:06.643596    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9d1ee163-9447-4e3f-a7d5-90f8210d9c2f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dsppd\" (UID: \"9d1ee163-9447-4e3f-a7d5-90f8210d9c2f\") " pod="openshift-insights/insights-runtime-extractor-dsppd"
Sep 11 13:23:06.643859 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:23:06.643616    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 13:23:06.643859 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:23:06.643696    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d1ee163-9447-4e3f-a7d5-90f8210d9c2f-insights-runtime-extractor-tls podName:9d1ee163-9447-4e3f-a7d5-90f8210d9c2f nodeName:}" failed. No retries permitted until 2026-09-11 13:23:07.643675916 +0000 UTC m=+95.029779092 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9d1ee163-9447-4e3f-a7d5-90f8210d9c2f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dsppd" (UID: "9d1ee163-9447-4e3f-a7d5-90f8210d9c2f") : secret "insights-runtime-extractor-tls" not found
Sep 11 13:23:06.643859 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:23:06.643713    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 13:23:06.643859 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:23:06.643765    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9d1ee163-9447-4e3f-a7d5-90f8210d9c2f-kube-rbac-proxy-cm podName:9d1ee163-9447-4e3f-a7d5-90f8210d9c2f nodeName:}" failed. No retries permitted until 2026-09-11 13:23:07.643752574 +0000 UTC m=+95.029855716 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9d1ee163-9447-4e3f-a7d5-90f8210d9c2f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dsppd" (UID: "9d1ee163-9447-4e3f-a7d5-90f8210d9c2f") : configmap "kube-rbac-proxy" not found
Sep 11 13:23:07.651188 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:07.651137    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9d1ee163-9447-4e3f-a7d5-90f8210d9c2f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dsppd\" (UID: \"9d1ee163-9447-4e3f-a7d5-90f8210d9c2f\") " pod="openshift-insights/insights-runtime-extractor-dsppd"
Sep 11 13:23:07.651188 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:07.651200    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9d1ee163-9447-4e3f-a7d5-90f8210d9c2f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dsppd\" (UID: \"9d1ee163-9447-4e3f-a7d5-90f8210d9c2f\") " pod="openshift-insights/insights-runtime-extractor-dsppd"
Sep 11 13:23:07.651982 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:23:07.651290    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 13:23:07.651982 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:23:07.651326    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 13:23:07.651982 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:23:07.651387    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9d1ee163-9447-4e3f-a7d5-90f8210d9c2f-kube-rbac-proxy-cm podName:9d1ee163-9447-4e3f-a7d5-90f8210d9c2f nodeName:}" failed. No retries permitted until 2026-09-11 13:23:09.651359046 +0000 UTC m=+97.037462192 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9d1ee163-9447-4e3f-a7d5-90f8210d9c2f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dsppd" (UID: "9d1ee163-9447-4e3f-a7d5-90f8210d9c2f") : configmap "kube-rbac-proxy" not found
Sep 11 13:23:07.651982 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:23:07.651408    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d1ee163-9447-4e3f-a7d5-90f8210d9c2f-insights-runtime-extractor-tls podName:9d1ee163-9447-4e3f-a7d5-90f8210d9c2f nodeName:}" failed. No retries permitted until 2026-09-11 13:23:09.651399455 +0000 UTC m=+97.037502597 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9d1ee163-9447-4e3f-a7d5-90f8210d9c2f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dsppd" (UID: "9d1ee163-9447-4e3f-a7d5-90f8210d9c2f") : secret "insights-runtime-extractor-tls" not found
Sep 11 13:23:07.948167 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:07.948065    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-z8p25_f7ab88b5-7073-460e-b3d4-a9841f50b185/dns-node-resolver/0.log"
Sep 11 13:23:08.326872 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:08.326815    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-8jj8l_d9bf058c-d89b-4236-86ce-3dee6f389711/node-ca/0.log"
Sep 11 13:23:09.326682 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:09.326637    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-dcgnn_42df0901-f4f9-4a7e-85da-4551acd80be7/serve-healthcheck-canary/0.log"
Sep 11 13:23:09.569493 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:09.569446    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a93d3879-2973-4fa1-bd7d-bc855a257ddf-metrics-certs\") pod \"router-default-7798cf8dd4-5hzkk\" (UID: \"a93d3879-2973-4fa1-bd7d-bc855a257ddf\") " pod="openshift-ingress/router-default-7798cf8dd4-5hzkk"
Sep 11 13:23:09.569724 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:09.569524    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a93d3879-2973-4fa1-bd7d-bc855a257ddf-service-ca-bundle\") pod \"router-default-7798cf8dd4-5hzkk\" (UID: \"a93d3879-2973-4fa1-bd7d-bc855a257ddf\") " pod="openshift-ingress/router-default-7798cf8dd4-5hzkk"
Sep 11 13:23:09.569724 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:23:09.569632    2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 11 13:23:09.569724 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:23:09.569718    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a93d3879-2973-4fa1-bd7d-bc855a257ddf-metrics-certs podName:a93d3879-2973-4fa1-bd7d-bc855a257ddf nodeName:}" failed. No retries permitted until 2026-09-11 13:23:17.56969791 +0000 UTC m=+104.955801055 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a93d3879-2973-4fa1-bd7d-bc855a257ddf-metrics-certs") pod "router-default-7798cf8dd4-5hzkk" (UID: "a93d3879-2973-4fa1-bd7d-bc855a257ddf") : secret "router-metrics-certs-default" not found
Sep 11 13:23:09.569868 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:23:09.569737    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a93d3879-2973-4fa1-bd7d-bc855a257ddf-service-ca-bundle podName:a93d3879-2973-4fa1-bd7d-bc855a257ddf nodeName:}" failed. No retries permitted until 2026-09-11 13:23:17.569730007 +0000 UTC m=+104.955833148 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/a93d3879-2973-4fa1-bd7d-bc855a257ddf-service-ca-bundle") pod "router-default-7798cf8dd4-5hzkk" (UID: "a93d3879-2973-4fa1-bd7d-bc855a257ddf") : configmap references non-existent config key: service-ca.crt
Sep 11 13:23:09.670214 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:09.670105    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9d1ee163-9447-4e3f-a7d5-90f8210d9c2f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dsppd\" (UID: \"9d1ee163-9447-4e3f-a7d5-90f8210d9c2f\") " pod="openshift-insights/insights-runtime-extractor-dsppd"
Sep 11 13:23:09.670214 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:09.670209    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9d1ee163-9447-4e3f-a7d5-90f8210d9c2f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dsppd\" (UID: \"9d1ee163-9447-4e3f-a7d5-90f8210d9c2f\") " pod="openshift-insights/insights-runtime-extractor-dsppd"
Sep 11 13:23:09.670435 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:23:09.670262    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 13:23:09.670435 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:23:09.670291    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 13:23:09.670435 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:23:09.670341    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d1ee163-9447-4e3f-a7d5-90f8210d9c2f-insights-runtime-extractor-tls podName:9d1ee163-9447-4e3f-a7d5-90f8210d9c2f nodeName:}" failed. No retries permitted until 2026-09-11 13:23:13.670321077 +0000 UTC m=+101.056424219 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9d1ee163-9447-4e3f-a7d5-90f8210d9c2f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dsppd" (UID: "9d1ee163-9447-4e3f-a7d5-90f8210d9c2f") : secret "insights-runtime-extractor-tls" not found
Sep 11 13:23:09.670435 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:23:09.670357    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9d1ee163-9447-4e3f-a7d5-90f8210d9c2f-kube-rbac-proxy-cm podName:9d1ee163-9447-4e3f-a7d5-90f8210d9c2f nodeName:}" failed. No retries permitted until 2026-09-11 13:23:13.670349805 +0000 UTC m=+101.056452947 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9d1ee163-9447-4e3f-a7d5-90f8210d9c2f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dsppd" (UID: "9d1ee163-9447-4e3f-a7d5-90f8210d9c2f") : configmap "kube-rbac-proxy" not found
Sep 11 13:23:12.898749 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:12.898692    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5ab38ce4-40c2-4630-98f3-62d1b84532a5-metrics-tls\") pod \"dns-default-pfffb\" (UID: \"5ab38ce4-40c2-4630-98f3-62d1b84532a5\") " pod="openshift-dns/dns-default-pfffb"
Sep 11 13:23:12.899354 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:23:12.898968    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 13:23:12.899354 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:23:12.899070    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ab38ce4-40c2-4630-98f3-62d1b84532a5-metrics-tls podName:5ab38ce4-40c2-4630-98f3-62d1b84532a5 nodeName:}" failed. No retries permitted until 2026-09-11 13:24:16.899047394 +0000 UTC m=+164.285150552 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5ab38ce4-40c2-4630-98f3-62d1b84532a5-metrics-tls") pod "dns-default-pfffb" (UID: "5ab38ce4-40c2-4630-98f3-62d1b84532a5") : secret "dns-default-metrics-tls" not found
Sep 11 13:23:13.523817 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:13.523769    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-75fbbd45d6-tcd9m"]
Sep 11 13:23:13.545710 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:13.545656    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-75fbbd45d6-tcd9m"
Sep 11 13:23:13.546590 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:13.546553    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-75fbbd45d6-tcd9m"]
Sep 11 13:23:13.548536 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:13.548512    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-zq2ct\""
Sep 11 13:23:13.548782 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:13.548759    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 11 13:23:13.548957 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:13.548938    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 11 13:23:13.549045 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:13.548979    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 11 13:23:13.558543 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:13.558515    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 11 13:23:13.704219 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:13.704164    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c2d12c02-c63a-4cfe-a074-2d48bc8f3b3a-registry-tls\") pod \"image-registry-75fbbd45d6-tcd9m\" (UID: \"c2d12c02-c63a-4cfe-a074-2d48bc8f3b3a\") " pod="openshift-image-registry/image-registry-75fbbd45d6-tcd9m"
Sep 11 13:23:13.704219 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:13.704217    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c2d12c02-c63a-4cfe-a074-2d48bc8f3b3a-installation-pull-secrets\") pod \"image-registry-75fbbd45d6-tcd9m\" (UID: \"c2d12c02-c63a-4cfe-a074-2d48bc8f3b3a\") " pod="openshift-image-registry/image-registry-75fbbd45d6-tcd9m"
Sep 11 13:23:13.704472 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:13.704256    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c2d12c02-c63a-4cfe-a074-2d48bc8f3b3a-bound-sa-token\") pod \"image-registry-75fbbd45d6-tcd9m\" (UID: \"c2d12c02-c63a-4cfe-a074-2d48bc8f3b3a\") " pod="openshift-image-registry/image-registry-75fbbd45d6-tcd9m"
Sep 11 13:23:13.704472 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:13.704272    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c2d12c02-c63a-4cfe-a074-2d48bc8f3b3a-ca-trust-extracted\") pod \"image-registry-75fbbd45d6-tcd9m\" (UID: \"c2d12c02-c63a-4cfe-a074-2d48bc8f3b3a\") " pod="openshift-image-registry/image-registry-75fbbd45d6-tcd9m"
Sep 11 13:23:13.704472 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:13.704316    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c2d12c02-c63a-4cfe-a074-2d48bc8f3b3a-image-registry-private-configuration\") pod \"image-registry-75fbbd45d6-tcd9m\" (UID: \"c2d12c02-c63a-4cfe-a074-2d48bc8f3b3a\") " pod="openshift-image-registry/image-registry-75fbbd45d6-tcd9m"
Sep 11 13:23:13.704472 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:13.704344    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c2d12c02-c63a-4cfe-a074-2d48bc8f3b3a-registry-certificates\") pod \"image-registry-75fbbd45d6-tcd9m\" (UID: \"c2d12c02-c63a-4cfe-a074-2d48bc8f3b3a\") " pod="openshift-image-registry/image-registry-75fbbd45d6-tcd9m"
Sep 11 13:23:13.704472 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:13.704381    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9d1ee163-9447-4e3f-a7d5-90f8210d9c2f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dsppd\" (UID: \"9d1ee163-9447-4e3f-a7d5-90f8210d9c2f\") " pod="openshift-insights/insights-runtime-extractor-dsppd"
Sep 11 13:23:13.704472 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:23:13.704466    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 13:23:13.704672 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:13.704468    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c2d12c02-c63a-4cfe-a074-2d48bc8f3b3a-trusted-ca\") pod \"image-registry-75fbbd45d6-tcd9m\" (UID: \"c2d12c02-c63a-4cfe-a074-2d48bc8f3b3a\") " pod="openshift-image-registry/image-registry-75fbbd45d6-tcd9m"
Sep 11 13:23:13.704672 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:23:13.704528    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9d1ee163-9447-4e3f-a7d5-90f8210d9c2f-kube-rbac-proxy-cm podName:9d1ee163-9447-4e3f-a7d5-90f8210d9c2f nodeName:}" failed. No retries permitted until 2026-09-11 13:23:21.704512002 +0000 UTC m=+109.090615143 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9d1ee163-9447-4e3f-a7d5-90f8210d9c2f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dsppd" (UID: "9d1ee163-9447-4e3f-a7d5-90f8210d9c2f") : configmap "kube-rbac-proxy" not found
Sep 11 13:23:13.704672 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:13.704543    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9d1ee163-9447-4e3f-a7d5-90f8210d9c2f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dsppd\" (UID: \"9d1ee163-9447-4e3f-a7d5-90f8210d9c2f\") " pod="openshift-insights/insights-runtime-extractor-dsppd"
Sep 11 13:23:13.704672 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:13.704568    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmnb2\" (UniqueName: \"kubernetes.io/projected/c2d12c02-c63a-4cfe-a074-2d48bc8f3b3a-kube-api-access-jmnb2\") pod \"image-registry-75fbbd45d6-tcd9m\" (UID: \"c2d12c02-c63a-4cfe-a074-2d48bc8f3b3a\") " pod="openshift-image-registry/image-registry-75fbbd45d6-tcd9m"
Sep 11 13:23:13.704868 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:23:13.704682    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 13:23:13.704868 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:23:13.704741    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d1ee163-9447-4e3f-a7d5-90f8210d9c2f-insights-runtime-extractor-tls podName:9d1ee163-9447-4e3f-a7d5-90f8210d9c2f nodeName:}" failed. No retries permitted until 2026-09-11 13:23:21.704726958 +0000 UTC m=+109.090830100 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9d1ee163-9447-4e3f-a7d5-90f8210d9c2f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dsppd" (UID: "9d1ee163-9447-4e3f-a7d5-90f8210d9c2f") : secret "insights-runtime-extractor-tls" not found
Sep 11 13:23:13.806089 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:13.805987    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c2d12c02-c63a-4cfe-a074-2d48bc8f3b3a-trusted-ca\") pod \"image-registry-75fbbd45d6-tcd9m\" (UID: \"c2d12c02-c63a-4cfe-a074-2d48bc8f3b3a\") " pod="openshift-image-registry/image-registry-75fbbd45d6-tcd9m"
Sep 11 13:23:13.806089 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:13.806054    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmnb2\" (UniqueName: \"kubernetes.io/projected/c2d12c02-c63a-4cfe-a074-2d48bc8f3b3a-kube-api-access-jmnb2\") pod \"image-registry-75fbbd45d6-tcd9m\" (UID: \"c2d12c02-c63a-4cfe-a074-2d48bc8f3b3a\") " pod="openshift-image-registry/image-registry-75fbbd45d6-tcd9m"
Sep 11 13:23:13.806089 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:13.806082    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c2d12c02-c63a-4cfe-a074-2d48bc8f3b3a-registry-tls\") pod \"image-registry-75fbbd45d6-tcd9m\" (UID: \"c2d12c02-c63a-4cfe-a074-2d48bc8f3b3a\") " pod="openshift-image-registry/image-registry-75fbbd45d6-tcd9m"
Sep 11 13:23:13.806415 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:13.806099    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c2d12c02-c63a-4cfe-a074-2d48bc8f3b3a-installation-pull-secrets\") pod \"image-registry-75fbbd45d6-tcd9m\" (UID: \"c2d12c02-c63a-4cfe-a074-2d48bc8f3b3a\") " pod="openshift-image-registry/image-registry-75fbbd45d6-tcd9m"
Sep 11 13:23:13.806415 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:13.806121    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c2d12c02-c63a-4cfe-a074-2d48bc8f3b3a-bound-sa-token\") pod \"image-registry-75fbbd45d6-tcd9m\" (UID: \"c2d12c02-c63a-4cfe-a074-2d48bc8f3b3a\") " pod="openshift-image-registry/image-registry-75fbbd45d6-tcd9m"
Sep 11 13:23:13.806415 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:13.806136    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c2d12c02-c63a-4cfe-a074-2d48bc8f3b3a-ca-trust-extracted\") pod \"image-registry-75fbbd45d6-tcd9m\" (UID: \"c2d12c02-c63a-4cfe-a074-2d48bc8f3b3a\") " pod="openshift-image-registry/image-registry-75fbbd45d6-tcd9m"
Sep 11 13:23:13.806415 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:13.806228    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c2d12c02-c63a-4cfe-a074-2d48bc8f3b3a-image-registry-private-configuration\") pod \"image-registry-75fbbd45d6-tcd9m\" (UID: \"c2d12c02-c63a-4cfe-a074-2d48bc8f3b3a\") " pod="openshift-image-registry/image-registry-75fbbd45d6-tcd9m"
Sep 11 13:23:13.806415 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:13.806274    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c2d12c02-c63a-4cfe-a074-2d48bc8f3b3a-registry-certificates\") pod \"image-registry-75fbbd45d6-tcd9m\" (UID: \"c2d12c02-c63a-4cfe-a074-2d48bc8f3b3a\") " pod="openshift-image-registry/image-registry-75fbbd45d6-tcd9m"
Sep 11 13:23:13.806415 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:23:13.806373    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 13:23:13.806415 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:23:13.806393    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-75fbbd45d6-tcd9m: secret "image-registry-tls" not found
Sep 11 13:23:13.806736 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:23:13.806475    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c2d12c02-c63a-4cfe-a074-2d48bc8f3b3a-registry-tls podName:c2d12c02-c63a-4cfe-a074-2d48bc8f3b3a nodeName:}" failed. No retries permitted until 2026-09-11 13:23:14.306452787 +0000 UTC m=+101.692555942 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c2d12c02-c63a-4cfe-a074-2d48bc8f3b3a-registry-tls") pod "image-registry-75fbbd45d6-tcd9m" (UID: "c2d12c02-c63a-4cfe-a074-2d48bc8f3b3a") : secret "image-registry-tls" not found
Sep 11 13:23:13.806736 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:13.806645    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c2d12c02-c63a-4cfe-a074-2d48bc8f3b3a-ca-trust-extracted\") pod \"image-registry-75fbbd45d6-tcd9m\" (UID: \"c2d12c02-c63a-4cfe-a074-2d48bc8f3b3a\") " pod="openshift-image-registry/image-registry-75fbbd45d6-tcd9m"
Sep 11 13:23:13.806942 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:13.806923    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c2d12c02-c63a-4cfe-a074-2d48bc8f3b3a-registry-certificates\") pod \"image-registry-75fbbd45d6-tcd9m\" (UID: \"c2d12c02-c63a-4cfe-a074-2d48bc8f3b3a\") " pod="openshift-image-registry/image-registry-75fbbd45d6-tcd9m"
Sep 11 13:23:13.807371 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:13.807342    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c2d12c02-c63a-4cfe-a074-2d48bc8f3b3a-trusted-ca\") pod \"image-registry-75fbbd45d6-tcd9m\" (UID: \"c2d12c02-c63a-4cfe-a074-2d48bc8f3b3a\") " pod="openshift-image-registry/image-registry-75fbbd45d6-tcd9m"
Sep 11 13:23:13.809371 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:13.809332    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c2d12c02-c63a-4cfe-a074-2d48bc8f3b3a-installation-pull-secrets\") pod \"image-registry-75fbbd45d6-tcd9m\" (UID: \"c2d12c02-c63a-4cfe-a074-2d48bc8f3b3a\") " pod="openshift-image-registry/image-registry-75fbbd45d6-tcd9m"
Sep 11 13:23:13.809499 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:13.809449    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c2d12c02-c63a-4cfe-a074-2d48bc8f3b3a-image-registry-private-configuration\") pod \"image-registry-75fbbd45d6-tcd9m\" (UID: \"c2d12c02-c63a-4cfe-a074-2d48bc8f3b3a\") " pod="openshift-image-registry/image-registry-75fbbd45d6-tcd9m"
Sep 11 13:23:13.814988 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:13.814965    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmnb2\" (UniqueName: \"kubernetes.io/projected/c2d12c02-c63a-4cfe-a074-2d48bc8f3b3a-kube-api-access-jmnb2\") pod \"image-registry-75fbbd45d6-tcd9m\" (UID: \"c2d12c02-c63a-4cfe-a074-2d48bc8f3b3a\") " pod="openshift-image-registry/image-registry-75fbbd45d6-tcd9m"
Sep 11 13:23:13.817874 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:13.817822    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c2d12c02-c63a-4cfe-a074-2d48bc8f3b3a-bound-sa-token\") pod \"image-registry-75fbbd45d6-tcd9m\" (UID: \"c2d12c02-c63a-4cfe-a074-2d48bc8f3b3a\") " pod="openshift-image-registry/image-registry-75fbbd45d6-tcd9m"
Sep 11 13:23:14.311525 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:14.311468    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c2d12c02-c63a-4cfe-a074-2d48bc8f3b3a-registry-tls\") pod \"image-registry-75fbbd45d6-tcd9m\" (UID: \"c2d12c02-c63a-4cfe-a074-2d48bc8f3b3a\") " pod="openshift-image-registry/image-registry-75fbbd45d6-tcd9m"
Sep 11 13:23:14.312076 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:23:14.311669    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 13:23:14.312076 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:23:14.311689    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-75fbbd45d6-tcd9m: secret "image-registry-tls" not found
Sep 11 13:23:14.312076 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:23:14.311765    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c2d12c02-c63a-4cfe-a074-2d48bc8f3b3a-registry-tls podName:c2d12c02-c63a-4cfe-a074-2d48bc8f3b3a nodeName:}" failed. No retries permitted until 2026-09-11 13:23:15.31174259 +0000 UTC m=+102.697845754 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c2d12c02-c63a-4cfe-a074-2d48bc8f3b3a-registry-tls") pod "image-registry-75fbbd45d6-tcd9m" (UID: "c2d12c02-c63a-4cfe-a074-2d48bc8f3b3a") : secret "image-registry-tls" not found
Sep 11 13:23:15.321470 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:15.321423    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c2d12c02-c63a-4cfe-a074-2d48bc8f3b3a-registry-tls\") pod \"image-registry-75fbbd45d6-tcd9m\" (UID: \"c2d12c02-c63a-4cfe-a074-2d48bc8f3b3a\") " pod="openshift-image-registry/image-registry-75fbbd45d6-tcd9m"
Sep 11 13:23:15.321928 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:23:15.321592    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 13:23:15.321928 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:23:15.321611    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-75fbbd45d6-tcd9m: secret "image-registry-tls" not found
Sep 11 13:23:15.321928 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:23:15.321673    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c2d12c02-c63a-4cfe-a074-2d48bc8f3b3a-registry-tls podName:c2d12c02-c63a-4cfe-a074-2d48bc8f3b3a nodeName:}" failed. No retries permitted until 2026-09-11 13:23:17.321657535 +0000 UTC m=+104.707760676 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c2d12c02-c63a-4cfe-a074-2d48bc8f3b3a-registry-tls") pod "image-registry-75fbbd45d6-tcd9m" (UID: "c2d12c02-c63a-4cfe-a074-2d48bc8f3b3a") : secret "image-registry-tls" not found
Sep 11 13:23:15.553452 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:15.553410    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-6frmx"
Sep 11 13:23:17.340586 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:17.340539    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c2d12c02-c63a-4cfe-a074-2d48bc8f3b3a-registry-tls\") pod \"image-registry-75fbbd45d6-tcd9m\" (UID: \"c2d12c02-c63a-4cfe-a074-2d48bc8f3b3a\") " pod="openshift-image-registry/image-registry-75fbbd45d6-tcd9m"
Sep 11 13:23:17.341073 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:23:17.340710    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 13:23:17.341073 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:23:17.340733    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-75fbbd45d6-tcd9m: secret "image-registry-tls" not found
Sep 11 13:23:17.341073 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:23:17.340791    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c2d12c02-c63a-4cfe-a074-2d48bc8f3b3a-registry-tls podName:c2d12c02-c63a-4cfe-a074-2d48bc8f3b3a nodeName:}" failed. No retries permitted until 2026-09-11 13:23:21.340776178 +0000 UTC m=+108.726879319 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c2d12c02-c63a-4cfe-a074-2d48bc8f3b3a-registry-tls") pod "image-registry-75fbbd45d6-tcd9m" (UID: "c2d12c02-c63a-4cfe-a074-2d48bc8f3b3a") : secret "image-registry-tls" not found
Sep 11 13:23:17.643585 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:17.643454    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a93d3879-2973-4fa1-bd7d-bc855a257ddf-service-ca-bundle\") pod \"router-default-7798cf8dd4-5hzkk\" (UID: \"a93d3879-2973-4fa1-bd7d-bc855a257ddf\") " pod="openshift-ingress/router-default-7798cf8dd4-5hzkk"
Sep 11 13:23:17.643585 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:17.643561    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a93d3879-2973-4fa1-bd7d-bc855a257ddf-metrics-certs\") pod \"router-default-7798cf8dd4-5hzkk\" (UID: \"a93d3879-2973-4fa1-bd7d-bc855a257ddf\") " pod="openshift-ingress/router-default-7798cf8dd4-5hzkk"
Sep 11 13:23:17.643870 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:23:17.643687    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a93d3879-2973-4fa1-bd7d-bc855a257ddf-service-ca-bundle podName:a93d3879-2973-4fa1-bd7d-bc855a257ddf nodeName:}" failed. No retries permitted until 2026-09-11 13:23:33.643654595 +0000 UTC m=+121.029757754 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/a93d3879-2973-4fa1-bd7d-bc855a257ddf-service-ca-bundle") pod "router-default-7798cf8dd4-5hzkk" (UID: "a93d3879-2973-4fa1-bd7d-bc855a257ddf") : configmap references non-existent config key: service-ca.crt
Sep 11 13:23:17.643870 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:23:17.643729    2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 11 13:23:17.643870 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:23:17.643778    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a93d3879-2973-4fa1-bd7d-bc855a257ddf-metrics-certs podName:a93d3879-2973-4fa1-bd7d-bc855a257ddf nodeName:}" failed. No retries permitted until 2026-09-11 13:23:33.643766843 +0000 UTC m=+121.029869990 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a93d3879-2973-4fa1-bd7d-bc855a257ddf-metrics-certs") pod "router-default-7798cf8dd4-5hzkk" (UID: "a93d3879-2973-4fa1-bd7d-bc855a257ddf") : secret "router-metrics-certs-default" not found
Sep 11 13:23:21.377417 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:21.377358    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c2d12c02-c63a-4cfe-a074-2d48bc8f3b3a-registry-tls\") pod \"image-registry-75fbbd45d6-tcd9m\" (UID: \"c2d12c02-c63a-4cfe-a074-2d48bc8f3b3a\") " pod="openshift-image-registry/image-registry-75fbbd45d6-tcd9m"
Sep 11 13:23:21.377911 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:23:21.377543    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 13:23:21.377911 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:23:21.377564    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-75fbbd45d6-tcd9m: secret "image-registry-tls" not found
Sep 11 13:23:21.377911 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:23:21.377636    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c2d12c02-c63a-4cfe-a074-2d48bc8f3b3a-registry-tls podName:c2d12c02-c63a-4cfe-a074-2d48bc8f3b3a nodeName:}" failed. No retries permitted until 2026-09-11 13:23:29.37761808 +0000 UTC m=+116.763721227 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c2d12c02-c63a-4cfe-a074-2d48bc8f3b3a-registry-tls") pod "image-registry-75fbbd45d6-tcd9m" (UID: "c2d12c02-c63a-4cfe-a074-2d48bc8f3b3a") : secret "image-registry-tls" not found
Sep 11 13:23:21.781175 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:21.781121    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9d1ee163-9447-4e3f-a7d5-90f8210d9c2f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dsppd\" (UID: \"9d1ee163-9447-4e3f-a7d5-90f8210d9c2f\") " pod="openshift-insights/insights-runtime-extractor-dsppd"
Sep 11 13:23:21.781405 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:21.781243    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9d1ee163-9447-4e3f-a7d5-90f8210d9c2f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dsppd\" (UID: \"9d1ee163-9447-4e3f-a7d5-90f8210d9c2f\") " pod="openshift-insights/insights-runtime-extractor-dsppd"
Sep 11 13:23:21.781405 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:23:21.781297    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 13:23:21.781405 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:23:21.781380    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d1ee163-9447-4e3f-a7d5-90f8210d9c2f-insights-runtime-extractor-tls podName:9d1ee163-9447-4e3f-a7d5-90f8210d9c2f nodeName:}" failed. No retries permitted until 2026-09-11 13:23:37.781362669 +0000 UTC m=+125.167465811 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9d1ee163-9447-4e3f-a7d5-90f8210d9c2f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dsppd" (UID: "9d1ee163-9447-4e3f-a7d5-90f8210d9c2f") : secret "insights-runtime-extractor-tls" not found
Sep 11 13:23:21.781794 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:21.781772    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9d1ee163-9447-4e3f-a7d5-90f8210d9c2f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dsppd\" (UID: \"9d1ee163-9447-4e3f-a7d5-90f8210d9c2f\") " pod="openshift-insights/insights-runtime-extractor-dsppd"
Sep 11 13:23:29.448581 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:29.448534    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c2d12c02-c63a-4cfe-a074-2d48bc8f3b3a-registry-tls\") pod \"image-registry-75fbbd45d6-tcd9m\" (UID: \"c2d12c02-c63a-4cfe-a074-2d48bc8f3b3a\") " pod="openshift-image-registry/image-registry-75fbbd45d6-tcd9m"
Sep 11 13:23:29.451354 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:29.451330    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c2d12c02-c63a-4cfe-a074-2d48bc8f3b3a-registry-tls\") pod \"image-registry-75fbbd45d6-tcd9m\" (UID: \"c2d12c02-c63a-4cfe-a074-2d48bc8f3b3a\") " pod="openshift-image-registry/image-registry-75fbbd45d6-tcd9m"
Sep 11 13:23:29.460225 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:29.460186    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-75fbbd45d6-tcd9m"
Sep 11 13:23:29.749115 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:29.749058    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-75fbbd45d6-tcd9m"]
Sep 11 13:23:29.753894 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:23:29.753856    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2d12c02_c63a_4cfe_a074_2d48bc8f3b3a.slice/crio-0c20f6d9ea532f1a883ad1e489f7ade214d4ba7c32fc6d356970e4d2a5404c66 WatchSource:0}: Error finding container 0c20f6d9ea532f1a883ad1e489f7ade214d4ba7c32fc6d356970e4d2a5404c66: Status 404 returned error can't find the container with id 0c20f6d9ea532f1a883ad1e489f7ade214d4ba7c32fc6d356970e4d2a5404c66
Sep 11 13:23:30.684356 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:30.684308    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-75fbbd45d6-tcd9m" event={"ID":"c2d12c02-c63a-4cfe-a074-2d48bc8f3b3a","Type":"ContainerStarted","Data":"d8cd060cec2e03e880091016b056b46962e0ada9f7703ec1a3099d11a57cea8a"}
Sep 11 13:23:30.685027 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:30.684375    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-75fbbd45d6-tcd9m" event={"ID":"c2d12c02-c63a-4cfe-a074-2d48bc8f3b3a","Type":"ContainerStarted","Data":"0c20f6d9ea532f1a883ad1e489f7ade214d4ba7c32fc6d356970e4d2a5404c66"}
Sep 11 13:23:30.685027 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:30.684445    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-75fbbd45d6-tcd9m"
Sep 11 13:23:30.706439 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:30.706357    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-75fbbd45d6-tcd9m" podStartSLOduration=17.70633054 podStartE2EDuration="17.70633054s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:23:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:23:30.705220409 +0000 UTC m=+118.091323588" watchObservedRunningTime="2026-09-11 13:23:30.70633054 +0000 UTC m=+118.092433705"
Sep 11 13:23:33.686620 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:33.686523    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a93d3879-2973-4fa1-bd7d-bc855a257ddf-service-ca-bundle\") pod \"router-default-7798cf8dd4-5hzkk\" (UID: \"a93d3879-2973-4fa1-bd7d-bc855a257ddf\") " pod="openshift-ingress/router-default-7798cf8dd4-5hzkk"
Sep 11 13:23:33.686620 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:33.686642    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a93d3879-2973-4fa1-bd7d-bc855a257ddf-metrics-certs\") pod \"router-default-7798cf8dd4-5hzkk\" (UID: \"a93d3879-2973-4fa1-bd7d-bc855a257ddf\") " pod="openshift-ingress/router-default-7798cf8dd4-5hzkk"
Sep 11 13:23:33.687349 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:33.687322    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a93d3879-2973-4fa1-bd7d-bc855a257ddf-service-ca-bundle\") pod \"router-default-7798cf8dd4-5hzkk\" (UID: \"a93d3879-2973-4fa1-bd7d-bc855a257ddf\") " pod="openshift-ingress/router-default-7798cf8dd4-5hzkk"
Sep 11 13:23:33.689575 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:33.689545    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a93d3879-2973-4fa1-bd7d-bc855a257ddf-metrics-certs\") pod \"router-default-7798cf8dd4-5hzkk\" (UID: \"a93d3879-2973-4fa1-bd7d-bc855a257ddf\") " pod="openshift-ingress/router-default-7798cf8dd4-5hzkk"
Sep 11 13:23:33.909263 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:33.909225    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-4sphs\""
Sep 11 13:23:33.918062 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:33.918015    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7798cf8dd4-5hzkk"
Sep 11 13:23:34.196628 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:34.196583    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-7798cf8dd4-5hzkk"]
Sep 11 13:23:34.200062 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:23:34.200012    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda93d3879_2973_4fa1_bd7d_bc855a257ddf.slice/crio-72509c9aec9d21991e38ccc0dadb38967e3215b1f84fe52522294274010398d2 WatchSource:0}: Error finding container 72509c9aec9d21991e38ccc0dadb38967e3215b1f84fe52522294274010398d2: Status 404 returned error can't find the container with id 72509c9aec9d21991e38ccc0dadb38967e3215b1f84fe52522294274010398d2
Sep 11 13:23:34.700858 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:34.700715    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-7798cf8dd4-5hzkk" event={"ID":"a93d3879-2973-4fa1-bd7d-bc855a257ddf","Type":"ContainerStarted","Data":"53018c07a88e62cec274e8ce3ae17d3d6bbb7f7a6947cabb21e061bb92571c61"}
Sep 11 13:23:34.700858 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:34.700779    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-7798cf8dd4-5hzkk" event={"ID":"a93d3879-2973-4fa1-bd7d-bc855a257ddf","Type":"ContainerStarted","Data":"72509c9aec9d21991e38ccc0dadb38967e3215b1f84fe52522294274010398d2"}
Sep 11 13:23:34.722780 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:34.722706    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-7798cf8dd4-5hzkk" podStartSLOduration=33.722676079 podStartE2EDuration="33.722676079s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:23:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:23:34.721544829 +0000 UTC m=+122.107647993" watchObservedRunningTime="2026-09-11 13:23:34.722676079 +0000 UTC m=+122.108779245"
Sep 11 13:23:34.918686 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:34.918620    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-7798cf8dd4-5hzkk"
Sep 11 13:23:34.921935 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:34.921892    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-7798cf8dd4-5hzkk"
Sep 11 13:23:35.704207 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:35.704152    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-7798cf8dd4-5hzkk"
Sep 11 13:23:35.705878 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:35.705845    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-7798cf8dd4-5hzkk"
Sep 11 13:23:37.824617 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:37.824554    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9d1ee163-9447-4e3f-a7d5-90f8210d9c2f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dsppd\" (UID: \"9d1ee163-9447-4e3f-a7d5-90f8210d9c2f\") " pod="openshift-insights/insights-runtime-extractor-dsppd"
Sep 11 13:23:37.827412 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:37.827380    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9d1ee163-9447-4e3f-a7d5-90f8210d9c2f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dsppd\" (UID: \"9d1ee163-9447-4e3f-a7d5-90f8210d9c2f\") " pod="openshift-insights/insights-runtime-extractor-dsppd"
Sep 11 13:23:38.095178 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:38.095082    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-kd6rn\""
Sep 11 13:23:38.103365 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:38.103317    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-dsppd"
Sep 11 13:23:38.401157 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:38.401040    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-dsppd"]
Sep 11 13:23:38.405017 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:23:38.404978    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d1ee163_9447_4e3f_a7d5_90f8210d9c2f.slice/crio-d64d1f4752c80c30e214836a23aefa299926202b53416a22587c4d12adc50afb WatchSource:0}: Error finding container d64d1f4752c80c30e214836a23aefa299926202b53416a22587c4d12adc50afb: Status 404 returned error can't find the container with id d64d1f4752c80c30e214836a23aefa299926202b53416a22587c4d12adc50afb
Sep 11 13:23:38.720451 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:38.720401    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dsppd" event={"ID":"9d1ee163-9447-4e3f-a7d5-90f8210d9c2f","Type":"ContainerStarted","Data":"08bb3f66360b33069c2e4d652165f528b104a8a621eb1865f9867389e20566d6"}
Sep 11 13:23:38.720673 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:38.720509    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dsppd" event={"ID":"9d1ee163-9447-4e3f-a7d5-90f8210d9c2f","Type":"ContainerStarted","Data":"d64d1f4752c80c30e214836a23aefa299926202b53416a22587c4d12adc50afb"}
Sep 11 13:23:39.728316 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:39.728256    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dsppd" event={"ID":"9d1ee163-9447-4e3f-a7d5-90f8210d9c2f","Type":"ContainerStarted","Data":"339557145ace347705b58ae3673993a367405c4a549e47fa326fb3bfd6084371"}
Sep 11 13:23:40.736261 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:40.736217    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dsppd" event={"ID":"9d1ee163-9447-4e3f-a7d5-90f8210d9c2f","Type":"ContainerStarted","Data":"4828ec607c98ee9e3061158b14917888edf90d3dbe0ff2219dcbf46ac16083be"}
Sep 11 13:23:40.755290 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:40.755224    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-dsppd" podStartSLOduration=33.796267471 podStartE2EDuration="35.755204399s" totalImagesPullingTime="1.958936928s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 13:23:05 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:23:40.755168866 +0000 UTC m=+128.141272039" watchObservedRunningTime="2026-09-11 13:23:40.755204399 +0000 UTC m=+128.141307564"
Sep 11 13:23:42.828639 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:42.828592    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-75fbbd45d6-tcd9m"]
Sep 11 13:23:42.877356 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:42.877300    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-vz6d4"]
Sep 11 13:23:42.880742 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:42.880716    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-vz6d4"
Sep 11 13:23:42.884630 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:42.884593    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Sep 11 13:23:42.886084 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:42.886057    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-4zk2c\""
Sep 11 13:23:42.897683 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:42.897645    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-vz6d4"]
Sep 11 13:23:42.967198 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:42.967157    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/cd48dfdc-fee8-46a7-b05f-b1180b7bec63-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-vz6d4\" (UID: \"cd48dfdc-fee8-46a7-b05f-b1180b7bec63\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-vz6d4"
Sep 11 13:23:42.967445 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:42.967220    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2354ca1-4b4e-4a17-a17e-52d61af0e80c-metrics-certs\") pod \"network-metrics-daemon-zb2qw\" (UID: \"a2354ca1-4b4e-4a17-a17e-52d61af0e80c\") " pod="openshift-multus/network-metrics-daemon-zb2qw"
Sep 11 13:23:42.970134 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:42.970105    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2354ca1-4b4e-4a17-a17e-52d61af0e80c-metrics-certs\") pod \"network-metrics-daemon-zb2qw\" (UID: \"a2354ca1-4b4e-4a17-a17e-52d61af0e80c\") " pod="openshift-multus/network-metrics-daemon-zb2qw"
Sep 11 13:23:43.027525 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:43.027410    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-j2l67\""
Sep 11 13:23:43.036127 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:43.036081    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zb2qw"
Sep 11 13:23:43.067881 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:43.067805    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/cd48dfdc-fee8-46a7-b05f-b1180b7bec63-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-vz6d4\" (UID: \"cd48dfdc-fee8-46a7-b05f-b1180b7bec63\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-vz6d4"
Sep 11 13:23:43.070993 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:43.070969    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/cd48dfdc-fee8-46a7-b05f-b1180b7bec63-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-vz6d4\" (UID: \"cd48dfdc-fee8-46a7-b05f-b1180b7bec63\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-vz6d4"
Sep 11 13:23:43.191903 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:43.191795    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-vz6d4"
Sep 11 13:23:43.317662 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:43.317555    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-zb2qw"]
Sep 11 13:23:43.321049 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:23:43.321014    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2354ca1_4b4e_4a17_a17e_52d61af0e80c.slice/crio-7ac6fa1c094346343cbfec2f029f5450cc09b292e5c15d55225d07c8500b9186 WatchSource:0}: Error finding container 7ac6fa1c094346343cbfec2f029f5450cc09b292e5c15d55225d07c8500b9186: Status 404 returned error can't find the container with id 7ac6fa1c094346343cbfec2f029f5450cc09b292e5c15d55225d07c8500b9186
Sep 11 13:23:43.472223 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:43.472174    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-vz6d4"]
Sep 11 13:23:43.476267 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:23:43.476236    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd48dfdc_fee8_46a7_b05f_b1180b7bec63.slice/crio-ccffdcaf99d2818c38e22d7620adbc55b48eb8f8343cbba29ca2e0d48783eb96 WatchSource:0}: Error finding container ccffdcaf99d2818c38e22d7620adbc55b48eb8f8343cbba29ca2e0d48783eb96: Status 404 returned error can't find the container with id ccffdcaf99d2818c38e22d7620adbc55b48eb8f8343cbba29ca2e0d48783eb96
Sep 11 13:23:43.749466 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:43.749406    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zb2qw" event={"ID":"a2354ca1-4b4e-4a17-a17e-52d61af0e80c","Type":"ContainerStarted","Data":"7ac6fa1c094346343cbfec2f029f5450cc09b292e5c15d55225d07c8500b9186"}
Sep 11 13:23:43.751153 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:43.751108    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-vz6d4" event={"ID":"cd48dfdc-fee8-46a7-b05f-b1180b7bec63","Type":"ContainerStarted","Data":"ccffdcaf99d2818c38e22d7620adbc55b48eb8f8343cbba29ca2e0d48783eb96"}
Sep 11 13:23:44.757871 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:44.757807    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zb2qw" event={"ID":"a2354ca1-4b4e-4a17-a17e-52d61af0e80c","Type":"ContainerStarted","Data":"97a3123d38bbd3c68a630b156ce2f69d10edd1e723565774c09f2b8cf63333f1"}
Sep 11 13:23:44.758364 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:44.757897    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zb2qw" event={"ID":"a2354ca1-4b4e-4a17-a17e-52d61af0e80c","Type":"ContainerStarted","Data":"794097442fe220885c655183fc48f610e9e41ddeebd3d97787fffae8884bf66c"}
Sep 11 13:23:44.759650 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:44.759610    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-vz6d4" event={"ID":"cd48dfdc-fee8-46a7-b05f-b1180b7bec63","Type":"ContainerStarted","Data":"cf3f1e83bfbca9e77b18d1fb71787f52b2443c4a3fb2284a893bba159b7abea9"}
Sep 11 13:23:44.759861 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:44.759822    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-vz6d4"
Sep 11 13:23:44.765364 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:44.765330    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-vz6d4"
Sep 11 13:23:44.774418 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:44.774311    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-zb2qw" podStartSLOduration=129.575355339 podStartE2EDuration="2m10.774290916s" totalImagesPullingTime="1.198935577s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:21:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:23:44.773346873 +0000 UTC m=+132.159450064" watchObservedRunningTime="2026-09-11 13:23:44.774290916 +0000 UTC m=+132.160394079"
Sep 11 13:23:44.791911 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:44.791642    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-vz6d4" podStartSLOduration=1.745691634 podStartE2EDuration="2.791615371s" totalImagesPullingTime="1.045923737s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:23:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:23:44.791012308 +0000 UTC m=+132.177115490" watchObservedRunningTime="2026-09-11 13:23:44.791615371 +0000 UTC m=+132.177718535"
Sep 11 13:23:45.544448 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:45.544412    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-hqsnj"]
Sep 11 13:23:45.547839 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:45.547808    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-hqsnj"
Sep 11 13:23:45.550090 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:45.550055    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Sep 11 13:23:45.550272 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:45.550055    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Sep 11 13:23:45.550889 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:45.550860    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 11 13:23:45.551007 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:45.550948    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-d277m\""
Sep 11 13:23:45.551152 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:45.551130    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 11 13:23:45.551239 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:45.551173    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 11 13:23:45.556962 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:45.556938    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-hqsnj"]
Sep 11 13:23:45.593135 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:45.593080    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/64ef96a4-569e-4a9c-9f3c-cc72a2db18eb-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-hqsnj\" (UID: \"64ef96a4-569e-4a9c-9f3c-cc72a2db18eb\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-hqsnj"
Sep 11 13:23:45.593359 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:45.593160    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/64ef96a4-569e-4a9c-9f3c-cc72a2db18eb-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-hqsnj\" (UID: \"64ef96a4-569e-4a9c-9f3c-cc72a2db18eb\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-hqsnj"
Sep 11 13:23:45.593359 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:45.593198    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfbtg\" (UniqueName: \"kubernetes.io/projected/64ef96a4-569e-4a9c-9f3c-cc72a2db18eb-kube-api-access-zfbtg\") pod \"prometheus-operator-68bcd7f987-hqsnj\" (UID: \"64ef96a4-569e-4a9c-9f3c-cc72a2db18eb\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-hqsnj"
Sep 11 13:23:45.593359 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:45.593316    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/64ef96a4-569e-4a9c-9f3c-cc72a2db18eb-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-hqsnj\" (UID: \"64ef96a4-569e-4a9c-9f3c-cc72a2db18eb\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-hqsnj"
Sep 11 13:23:45.693930 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:45.693880    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/64ef96a4-569e-4a9c-9f3c-cc72a2db18eb-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-hqsnj\" (UID: \"64ef96a4-569e-4a9c-9f3c-cc72a2db18eb\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-hqsnj"
Sep 11 13:23:45.694112 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:45.693938    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/64ef96a4-569e-4a9c-9f3c-cc72a2db18eb-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-hqsnj\" (UID: \"64ef96a4-569e-4a9c-9f3c-cc72a2db18eb\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-hqsnj"
Sep 11 13:23:45.694112 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:45.693988    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/64ef96a4-569e-4a9c-9f3c-cc72a2db18eb-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-hqsnj\" (UID: \"64ef96a4-569e-4a9c-9f3c-cc72a2db18eb\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-hqsnj"
Sep 11 13:23:45.694112 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:45.694018    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfbtg\" (UniqueName: \"kubernetes.io/projected/64ef96a4-569e-4a9c-9f3c-cc72a2db18eb-kube-api-access-zfbtg\") pod \"prometheus-operator-68bcd7f987-hqsnj\" (UID: \"64ef96a4-569e-4a9c-9f3c-cc72a2db18eb\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-hqsnj"
Sep 11 13:23:45.694750 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:45.694679    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/64ef96a4-569e-4a9c-9f3c-cc72a2db18eb-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-hqsnj\" (UID: \"64ef96a4-569e-4a9c-9f3c-cc72a2db18eb\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-hqsnj"
Sep 11 13:23:45.696855 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:45.696802    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/64ef96a4-569e-4a9c-9f3c-cc72a2db18eb-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-hqsnj\" (UID: \"64ef96a4-569e-4a9c-9f3c-cc72a2db18eb\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-hqsnj"
Sep 11 13:23:45.696981 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:45.696903    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/64ef96a4-569e-4a9c-9f3c-cc72a2db18eb-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-hqsnj\" (UID: \"64ef96a4-569e-4a9c-9f3c-cc72a2db18eb\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-hqsnj"
Sep 11 13:23:45.703298 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:45.703268    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfbtg\" (UniqueName: \"kubernetes.io/projected/64ef96a4-569e-4a9c-9f3c-cc72a2db18eb-kube-api-access-zfbtg\") pod \"prometheus-operator-68bcd7f987-hqsnj\" (UID: \"64ef96a4-569e-4a9c-9f3c-cc72a2db18eb\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-hqsnj"
Sep 11 13:23:45.858480 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:45.858355    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-hqsnj"
Sep 11 13:23:46.148992 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:46.148886    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-hqsnj"]
Sep 11 13:23:46.152233 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:23:46.152186    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64ef96a4_569e_4a9c_9f3c_cc72a2db18eb.slice/crio-d1fc6b50371a61bac25f554e4fd81bf5ff42edf1d3697f814da68771750c28cb WatchSource:0}: Error finding container d1fc6b50371a61bac25f554e4fd81bf5ff42edf1d3697f814da68771750c28cb: Status 404 returned error can't find the container with id d1fc6b50371a61bac25f554e4fd81bf5ff42edf1d3697f814da68771750c28cb
Sep 11 13:23:46.775217 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:46.775172    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-hqsnj" event={"ID":"64ef96a4-569e-4a9c-9f3c-cc72a2db18eb","Type":"ContainerStarted","Data":"d1fc6b50371a61bac25f554e4fd81bf5ff42edf1d3697f814da68771750c28cb"}
Sep 11 13:23:47.781045 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:47.780997    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-hqsnj" event={"ID":"64ef96a4-569e-4a9c-9f3c-cc72a2db18eb","Type":"ContainerStarted","Data":"0c537231de49568fe149bbd37a2fa56e9a120d543e33b092a85d5eaa2ae31bb1"}
Sep 11 13:23:47.781563 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:47.781070    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-hqsnj" event={"ID":"64ef96a4-569e-4a9c-9f3c-cc72a2db18eb","Type":"ContainerStarted","Data":"2db8484c177dcf3959c741ae794da20fed87d3120689c05397027ba2420a2d83"}
Sep 11 13:23:47.801004 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:47.800928    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-68bcd7f987-hqsnj" podStartSLOduration=1.5170759390000002 podStartE2EDuration="2.800901965s" totalImagesPullingTime="1.283826026s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:23:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:23:47.8003182 +0000 UTC m=+135.186421390" watchObservedRunningTime="2026-09-11 13:23:47.800901965 +0000 UTC m=+135.187005130"
Sep 11 13:23:49.981545 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:49.981506    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-jnwg7"]
Sep 11 13:23:49.985233 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:49.985192    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-jnwg7"
Sep 11 13:23:49.986419 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:49.986389    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-ldgml"]
Sep 11 13:23:49.987529 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:49.987504    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Sep 11 13:23:49.987658 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:49.987643    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Sep 11 13:23:49.987726 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:49.987693    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-kwbnx\""
Sep 11 13:23:49.989542 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:49.989522    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-ldgml"
Sep 11 13:23:49.993626 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:49.993586    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Sep 11 13:23:49.993752 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:49.993586    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Sep 11 13:23:49.993867 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:49.993796    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-7vxbm\""
Sep 11 13:23:49.994064 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:49.994049    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Sep 11 13:23:50.001373 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:50.001342    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-jnwg7"]
Sep 11 13:23:50.007085 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:50.007049    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-ldgml"]
Sep 11 13:23:50.016073 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:50.016041    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-gscfs"]
Sep 11 13:23:50.019378 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:50.019358    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-gscfs"
Sep 11 13:23:50.021469 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:50.021433    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 11 13:23:50.021612 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:50.021501    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 11 13:23:50.021612 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:50.021592    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 11 13:23:50.021919 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:50.021905    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-pmtd5\""
Sep 11 13:23:50.135770 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:50.135735    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7b15b37c-f15c-454a-a7d6-871e4b808b66-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gscfs\" (UID: \"7b15b37c-f15c-454a-a7d6-871e4b808b66\") " pod="openshift-monitoring/node-exporter-gscfs"
Sep 11 13:23:50.135770 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:50.135774    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/eb050379-a425-4c99-81d2-4f07256eb6dc-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-ldgml\" (UID: \"eb050379-a425-4c99-81d2-4f07256eb6dc\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-ldgml"
Sep 11 13:23:50.136062 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:50.135801    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4f231d67-fe86-4633-b1d9-050a1b9935d9-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-jnwg7\" (UID: \"4f231d67-fe86-4633-b1d9-050a1b9935d9\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-jnwg7"
Sep 11 13:23:50.136062 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:50.135847    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7b15b37c-f15c-454a-a7d6-871e4b808b66-sys\") pod \"node-exporter-gscfs\" (UID: \"7b15b37c-f15c-454a-a7d6-871e4b808b66\") " pod="openshift-monitoring/node-exporter-gscfs"
Sep 11 13:23:50.136062 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:50.135868    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7b15b37c-f15c-454a-a7d6-871e4b808b66-node-exporter-textfile\") pod \"node-exporter-gscfs\" (UID: \"7b15b37c-f15c-454a-a7d6-871e4b808b66\") " pod="openshift-monitoring/node-exporter-gscfs"
Sep 11 13:23:50.136062 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:50.135886    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7b15b37c-f15c-454a-a7d6-871e4b808b66-node-exporter-accelerators-collector-config\") pod \"node-exporter-gscfs\" (UID: \"7b15b37c-f15c-454a-a7d6-871e4b808b66\") " pod="openshift-monitoring/node-exporter-gscfs"
Sep 11 13:23:50.136062 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:50.135908    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7b15b37c-f15c-454a-a7d6-871e4b808b66-node-exporter-wtmp\") pod \"node-exporter-gscfs\" (UID: \"7b15b37c-f15c-454a-a7d6-871e4b808b66\") " pod="openshift-monitoring/node-exporter-gscfs"
Sep 11 13:23:50.136062 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:50.135967    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7b15b37c-f15c-454a-a7d6-871e4b808b66-root\") pod \"node-exporter-gscfs\" (UID: \"7b15b37c-f15c-454a-a7d6-871e4b808b66\") " pod="openshift-monitoring/node-exporter-gscfs"
Sep 11 13:23:50.136062 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:50.136031    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4f231d67-fe86-4633-b1d9-050a1b9935d9-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-jnwg7\" (UID: \"4f231d67-fe86-4633-b1d9-050a1b9935d9\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-jnwg7"
Sep 11 13:23:50.136359 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:50.136066    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/eb050379-a425-4c99-81d2-4f07256eb6dc-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-ldgml\" (UID: \"eb050379-a425-4c99-81d2-4f07256eb6dc\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-ldgml"
Sep 11 13:23:50.136359 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:50.136124    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpx9d\" (UniqueName: \"kubernetes.io/projected/7b15b37c-f15c-454a-a7d6-871e4b808b66-kube-api-access-tpx9d\") pod \"node-exporter-gscfs\" (UID: \"7b15b37c-f15c-454a-a7d6-871e4b808b66\") " pod="openshift-monitoring/node-exporter-gscfs"
Sep 11 13:23:50.136359 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:50.136162    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/eb050379-a425-4c99-81d2-4f07256eb6dc-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-ldgml\" (UID: \"eb050379-a425-4c99-81d2-4f07256eb6dc\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-ldgml"
Sep 11 13:23:50.136359 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:50.136199    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/eb050379-a425-4c99-81d2-4f07256eb6dc-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-ldgml\" (UID: \"eb050379-a425-4c99-81d2-4f07256eb6dc\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-ldgml"
Sep 11 13:23:50.136359 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:50.136270    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/eb050379-a425-4c99-81d2-4f07256eb6dc-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-ldgml\" (UID: \"eb050379-a425-4c99-81d2-4f07256eb6dc\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-ldgml"
Sep 11 13:23:50.136359 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:50.136324    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7b15b37c-f15c-454a-a7d6-871e4b808b66-metrics-client-ca\") pod \"node-exporter-gscfs\" (UID: \"7b15b37c-f15c-454a-a7d6-871e4b808b66\") " pod="openshift-monitoring/node-exporter-gscfs"
Sep 11 13:23:50.136615 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:50.136360    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/4f231d67-fe86-4633-b1d9-050a1b9935d9-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-jnwg7\" (UID: \"4f231d67-fe86-4633-b1d9-050a1b9935d9\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-jnwg7"
Sep 11 13:23:50.136615 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:50.136387    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vffsd\" (UniqueName: \"kubernetes.io/projected/eb050379-a425-4c99-81d2-4f07256eb6dc-kube-api-access-vffsd\") pod \"kube-state-metrics-59fbdd77d-ldgml\" (UID: \"eb050379-a425-4c99-81d2-4f07256eb6dc\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-ldgml"
Sep 11 13:23:50.136615 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:50.136449    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rr2dc\" (UniqueName: \"kubernetes.io/projected/4f231d67-fe86-4633-b1d9-050a1b9935d9-kube-api-access-rr2dc\") pod \"openshift-state-metrics-67dcf8c979-jnwg7\" (UID: \"4f231d67-fe86-4633-b1d9-050a1b9935d9\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-jnwg7"
Sep 11 13:23:50.136615 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:50.136483    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7b15b37c-f15c-454a-a7d6-871e4b808b66-node-exporter-tls\") pod \"node-exporter-gscfs\" (UID: \"7b15b37c-f15c-454a-a7d6-871e4b808b66\") " pod="openshift-monitoring/node-exporter-gscfs"
Sep 11 13:23:50.237864 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:50.237722    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4f231d67-fe86-4633-b1d9-050a1b9935d9-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-jnwg7\" (UID: \"4f231d67-fe86-4633-b1d9-050a1b9935d9\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-jnwg7"
Sep 11 13:23:50.237864 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:50.237777    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/eb050379-a425-4c99-81d2-4f07256eb6dc-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-ldgml\" (UID: \"eb050379-a425-4c99-81d2-4f07256eb6dc\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-ldgml"
Sep 11 13:23:50.237864 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:50.237817    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpx9d\" (UniqueName: \"kubernetes.io/projected/7b15b37c-f15c-454a-a7d6-871e4b808b66-kube-api-access-tpx9d\") pod \"node-exporter-gscfs\" (UID: \"7b15b37c-f15c-454a-a7d6-871e4b808b66\") " pod="openshift-monitoring/node-exporter-gscfs"
Sep 11 13:23:50.237864 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:50.237868    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/eb050379-a425-4c99-81d2-4f07256eb6dc-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-ldgml\" (UID: \"eb050379-a425-4c99-81d2-4f07256eb6dc\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-ldgml"
Sep 11 13:23:50.240168 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:50.237898    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/eb050379-a425-4c99-81d2-4f07256eb6dc-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-ldgml\" (UID: \"eb050379-a425-4c99-81d2-4f07256eb6dc\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-ldgml"
Sep 11 13:23:50.240168 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:50.237926    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/eb050379-a425-4c99-81d2-4f07256eb6dc-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-ldgml\" (UID: \"eb050379-a425-4c99-81d2-4f07256eb6dc\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-ldgml"
Sep 11 13:23:50.240168 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:50.237952    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7b15b37c-f15c-454a-a7d6-871e4b808b66-metrics-client-ca\") pod \"node-exporter-gscfs\" (UID: \"7b15b37c-f15c-454a-a7d6-871e4b808b66\") " pod="openshift-monitoring/node-exporter-gscfs"
Sep 11 13:23:50.240168 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:50.237981    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/4f231d67-fe86-4633-b1d9-050a1b9935d9-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-jnwg7\" (UID: \"4f231d67-fe86-4633-b1d9-050a1b9935d9\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-jnwg7"
Sep 11 13:23:50.240168 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:50.238003    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vffsd\" (UniqueName: \"kubernetes.io/projected/eb050379-a425-4c99-81d2-4f07256eb6dc-kube-api-access-vffsd\") pod \"kube-state-metrics-59fbdd77d-ldgml\" (UID: \"eb050379-a425-4c99-81d2-4f07256eb6dc\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-ldgml"
Sep 11 13:23:50.240168 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:50.238430    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rr2dc\" (UniqueName: \"kubernetes.io/projected/4f231d67-fe86-4633-b1d9-050a1b9935d9-kube-api-access-rr2dc\") pod \"openshift-state-metrics-67dcf8c979-jnwg7\" (UID: \"4f231d67-fe86-4633-b1d9-050a1b9935d9\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-jnwg7"
Sep 11 13:23:50.240168 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:50.238477    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7b15b37c-f15c-454a-a7d6-871e4b808b66-node-exporter-tls\") pod \"node-exporter-gscfs\" (UID: \"7b15b37c-f15c-454a-a7d6-871e4b808b66\") " pod="openshift-monitoring/node-exporter-gscfs"
Sep 11 13:23:50.240168 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:50.238769    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7b15b37c-f15c-454a-a7d6-871e4b808b66-metrics-client-ca\") pod \"node-exporter-gscfs\" (UID: \"7b15b37c-f15c-454a-a7d6-871e4b808b66\") " pod="openshift-monitoring/node-exporter-gscfs"
Sep 11 13:23:50.240168 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:50.238779    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/eb050379-a425-4c99-81d2-4f07256eb6dc-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-ldgml\" (UID: \"eb050379-a425-4c99-81d2-4f07256eb6dc\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-ldgml"
Sep 11 13:23:50.240168 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:50.238808    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7b15b37c-f15c-454a-a7d6-871e4b808b66-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gscfs\" (UID: \"7b15b37c-f15c-454a-a7d6-871e4b808b66\") " pod="openshift-monitoring/node-exporter-gscfs"
Sep 11 13:23:50.240168 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:50.238861    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/eb050379-a425-4c99-81d2-4f07256eb6dc-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-ldgml\" (UID: \"eb050379-a425-4c99-81d2-4f07256eb6dc\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-ldgml"
Sep 11 13:23:50.240168 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:50.238899    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4f231d67-fe86-4633-b1d9-050a1b9935d9-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-jnwg7\" (UID: \"4f231d67-fe86-4633-b1d9-050a1b9935d9\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-jnwg7"
Sep 11 13:23:50.240168 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:50.238931    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7b15b37c-f15c-454a-a7d6-871e4b808b66-sys\") pod \"node-exporter-gscfs\" (UID: \"7b15b37c-f15c-454a-a7d6-871e4b808b66\") " pod="openshift-monitoring/node-exporter-gscfs"
Sep 11 13:23:50.240168 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:50.238959    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7b15b37c-f15c-454a-a7d6-871e4b808b66-node-exporter-textfile\") pod \"node-exporter-gscfs\" (UID: \"7b15b37c-f15c-454a-a7d6-871e4b808b66\") " pod="openshift-monitoring/node-exporter-gscfs"
Sep 11 13:23:50.240168 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:50.238987    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7b15b37c-f15c-454a-a7d6-871e4b808b66-node-exporter-accelerators-collector-config\") pod \"node-exporter-gscfs\" (UID: \"7b15b37c-f15c-454a-a7d6-871e4b808b66\") " pod="openshift-monitoring/node-exporter-gscfs"
Sep 11 13:23:50.240168 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:50.239033    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7b15b37c-f15c-454a-a7d6-871e4b808b66-node-exporter-wtmp\") pod \"node-exporter-gscfs\" (UID: \"7b15b37c-f15c-454a-a7d6-871e4b808b66\") " pod="openshift-monitoring/node-exporter-gscfs"
Sep 11 13:23:50.241005 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:50.239035    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/eb050379-a425-4c99-81d2-4f07256eb6dc-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-ldgml\" (UID: \"eb050379-a425-4c99-81d2-4f07256eb6dc\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-ldgml"
Sep 11 13:23:50.241005 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:50.239065    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7b15b37c-f15c-454a-a7d6-871e4b808b66-root\") pod \"node-exporter-gscfs\" (UID: \"7b15b37c-f15c-454a-a7d6-871e4b808b66\") " pod="openshift-monitoring/node-exporter-gscfs"
Sep 11 13:23:50.241005 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:50.239115    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7b15b37c-f15c-454a-a7d6-871e4b808b66-sys\") pod \"node-exporter-gscfs\" (UID: \"7b15b37c-f15c-454a-a7d6-871e4b808b66\") " pod="openshift-monitoring/node-exporter-gscfs"
Sep 11 13:23:50.241005 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:50.239184    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7b15b37c-f15c-454a-a7d6-871e4b808b66-root\") pod \"node-exporter-gscfs\" (UID: \"7b15b37c-f15c-454a-a7d6-871e4b808b66\") " pod="openshift-monitoring/node-exporter-gscfs"
Sep 11 13:23:50.241005 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:50.239200    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/eb050379-a425-4c99-81d2-4f07256eb6dc-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-ldgml\" (UID: \"eb050379-a425-4c99-81d2-4f07256eb6dc\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-ldgml"
Sep 11 13:23:50.241005 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:50.239604    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7b15b37c-f15c-454a-a7d6-871e4b808b66-node-exporter-textfile\") pod \"node-exporter-gscfs\" (UID: \"7b15b37c-f15c-454a-a7d6-871e4b808b66\") " pod="openshift-monitoring/node-exporter-gscfs"
Sep 11 13:23:50.241617 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:50.241587    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4f231d67-fe86-4633-b1d9-050a1b9935d9-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-jnwg7\" (UID: \"4f231d67-fe86-4633-b1d9-050a1b9935d9\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-jnwg7"
Sep 11 13:23:50.241755 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:50.241706    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7b15b37c-f15c-454a-a7d6-871e4b808b66-node-exporter-accelerators-collector-config\") pod \"node-exporter-gscfs\" (UID: \"7b15b37c-f15c-454a-a7d6-871e4b808b66\") " pod="openshift-monitoring/node-exporter-gscfs"
Sep 11 13:23:50.242092 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:50.242068    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7b15b37c-f15c-454a-a7d6-871e4b808b66-node-exporter-tls\") pod \"node-exporter-gscfs\" (UID: \"7b15b37c-f15c-454a-a7d6-871e4b808b66\") " pod="openshift-monitoring/node-exporter-gscfs"
Sep 11 13:23:50.242330 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:50.242157    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/eb050379-a425-4c99-81d2-4f07256eb6dc-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-ldgml\" (UID: \"eb050379-a425-4c99-81d2-4f07256eb6dc\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-ldgml"
Sep 11 13:23:50.242403 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:50.242161    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/4f231d67-fe86-4633-b1d9-050a1b9935d9-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-jnwg7\" (UID: \"4f231d67-fe86-4633-b1d9-050a1b9935d9\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-jnwg7"
Sep 11 13:23:50.242525 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:50.242503    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7b15b37c-f15c-454a-a7d6-871e4b808b66-node-exporter-wtmp\") pod \"node-exporter-gscfs\" (UID: \"7b15b37c-f15c-454a-a7d6-871e4b808b66\") " pod="openshift-monitoring/node-exporter-gscfs"
Sep 11 13:23:50.247764 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:50.242603    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4f231d67-fe86-4633-b1d9-050a1b9935d9-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-jnwg7\" (UID: \"4f231d67-fe86-4633-b1d9-050a1b9935d9\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-jnwg7"
Sep 11 13:23:50.247764 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:50.243044    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7b15b37c-f15c-454a-a7d6-871e4b808b66-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gscfs\" (UID: \"7b15b37c-f15c-454a-a7d6-871e4b808b66\") " pod="openshift-monitoring/node-exporter-gscfs"
Sep 11 13:23:50.247764 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:50.243089    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/eb050379-a425-4c99-81d2-4f07256eb6dc-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-ldgml\" (UID: \"eb050379-a425-4c99-81d2-4f07256eb6dc\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-ldgml"
Sep 11 13:23:50.248316 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:50.248288    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpx9d\" (UniqueName: \"kubernetes.io/projected/7b15b37c-f15c-454a-a7d6-871e4b808b66-kube-api-access-tpx9d\") pod \"node-exporter-gscfs\" (UID: \"7b15b37c-f15c-454a-a7d6-871e4b808b66\") " pod="openshift-monitoring/node-exporter-gscfs"
Sep 11 13:23:50.249016 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:50.248991    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vffsd\" (UniqueName: \"kubernetes.io/projected/eb050379-a425-4c99-81d2-4f07256eb6dc-kube-api-access-vffsd\") pod \"kube-state-metrics-59fbdd77d-ldgml\" (UID: \"eb050379-a425-4c99-81d2-4f07256eb6dc\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-ldgml"
Sep 11 13:23:50.250108 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:50.250084    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rr2dc\" (UniqueName: \"kubernetes.io/projected/4f231d67-fe86-4633-b1d9-050a1b9935d9-kube-api-access-rr2dc\") pod \"openshift-state-metrics-67dcf8c979-jnwg7\" (UID: \"4f231d67-fe86-4633-b1d9-050a1b9935d9\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-jnwg7"
Sep 11 13:23:50.297029 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:50.296972    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-jnwg7"
Sep 11 13:23:50.302440 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:50.302406    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-ldgml"
Sep 11 13:23:50.329698 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:50.329658    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-gscfs"
Sep 11 13:23:50.346803 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:23:50.346485    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b15b37c_f15c_454a_a7d6_871e4b808b66.slice/crio-afdce5d8804ec7971a56ccf0666f1fce74e7995f6dc9c8f267ab742702ee0416 WatchSource:0}: Error finding container afdce5d8804ec7971a56ccf0666f1fce74e7995f6dc9c8f267ab742702ee0416: Status 404 returned error can't find the container with id afdce5d8804ec7971a56ccf0666f1fce74e7995f6dc9c8f267ab742702ee0416
Sep 11 13:23:50.586668 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:50.586571    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-jnwg7"]
Sep 11 13:23:50.589953 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:50.589916    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-ldgml"]
Sep 11 13:23:50.591072 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:23:50.591038    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f231d67_fe86_4633_b1d9_050a1b9935d9.slice/crio-7768ad4406112505be8a10e19f3b3192e0faa2624a8066d9b8323882d2c0a8b8 WatchSource:0}: Error finding container 7768ad4406112505be8a10e19f3b3192e0faa2624a8066d9b8323882d2c0a8b8: Status 404 returned error can't find the container with id 7768ad4406112505be8a10e19f3b3192e0faa2624a8066d9b8323882d2c0a8b8
Sep 11 13:23:50.593700 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:23:50.593668    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb050379_a425_4c99_81d2_4f07256eb6dc.slice/crio-d9285182502a921b419edb469b7c3fda3da52e96e18f69bfa4b559b793e6eb43 WatchSource:0}: Error finding container d9285182502a921b419edb469b7c3fda3da52e96e18f69bfa4b559b793e6eb43: Status 404 returned error can't find the container with id d9285182502a921b419edb469b7c3fda3da52e96e18f69bfa4b559b793e6eb43
Sep 11 13:23:50.793384 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:50.793336    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-ldgml" event={"ID":"eb050379-a425-4c99-81d2-4f07256eb6dc","Type":"ContainerStarted","Data":"d9285182502a921b419edb469b7c3fda3da52e96e18f69bfa4b559b793e6eb43"}
Sep 11 13:23:50.796009 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:50.795962    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-jnwg7" event={"ID":"4f231d67-fe86-4633-b1d9-050a1b9935d9","Type":"ContainerStarted","Data":"3348420fc46892d16ed80c4ae4b2442e680000eb1505ab8bab8f293f423b22ca"}
Sep 11 13:23:50.796224 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:50.796053    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-jnwg7" event={"ID":"4f231d67-fe86-4633-b1d9-050a1b9935d9","Type":"ContainerStarted","Data":"bee7684b5ae579a7f9e10b3374295d793163db3ee5a5e0eb8336cbc9fe842544"}
Sep 11 13:23:50.796224 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:50.796086    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-jnwg7" event={"ID":"4f231d67-fe86-4633-b1d9-050a1b9935d9","Type":"ContainerStarted","Data":"7768ad4406112505be8a10e19f3b3192e0faa2624a8066d9b8323882d2c0a8b8"}
Sep 11 13:23:50.797506 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:50.797477    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gscfs" event={"ID":"7b15b37c-f15c-454a-a7d6-871e4b808b66","Type":"ContainerStarted","Data":"afdce5d8804ec7971a56ccf0666f1fce74e7995f6dc9c8f267ab742702ee0416"}
Sep 11 13:23:50.995583 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:50.995498    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 11 13:23:51.002034 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:51.001992    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 11 13:23:51.004368 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:51.004337    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 11 13:23:51.004368 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:51.004360    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 11 13:23:51.005139 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:51.005102    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 11 13:23:51.005424 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:51.005166    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 11 13:23:51.005424 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:51.005165    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 11 13:23:51.005571 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:51.005503    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 11 13:23:51.005571 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:51.005511    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-m4n9v\""
Sep 11 13:23:51.005685 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:51.005599    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 11 13:23:51.005811 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:51.005797    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 11 13:23:51.005894 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:51.005813    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 11 13:23:51.015945 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:51.015911    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 11 13:23:51.149427 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:51.149389    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 13:23:51.149634 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:51.149452    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 13:23:51.149634 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:51.149523    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5kw6\" (UniqueName: \"kubernetes.io/projected/2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14-kube-api-access-z5kw6\") pod \"alertmanager-main-0\" (UID: \"2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 13:23:51.149634 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:51.149566    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14-tls-assets\") pod \"alertmanager-main-0\" (UID: \"2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 13:23:51.149634 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:51.149600    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 13:23:51.149851 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:51.149677    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14-web-config\") pod \"alertmanager-main-0\" (UID: \"2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 13:23:51.149851 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:51.149803    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 13:23:51.149970 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:51.149887    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14-config-volume\") pod \"alertmanager-main-0\" (UID: \"2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 13:23:51.149970 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:51.149917    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 13:23:51.150065 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:51.149968    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 13:23:51.150065 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:51.150009    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14-config-out\") pod \"alertmanager-main-0\" (UID: \"2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 13:23:51.150065 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:51.150047    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 13:23:51.150231 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:51.150087    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 13:23:51.251234 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:51.251140    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14-tls-assets\") pod \"alertmanager-main-0\" (UID: \"2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 13:23:51.251234 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:51.251201    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 13:23:51.251510 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:51.251240    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14-web-config\") pod \"alertmanager-main-0\" (UID: \"2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 13:23:51.251510 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:51.251358    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 13:23:51.251510 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:51.251410    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14-config-volume\") pod \"alertmanager-main-0\" (UID: \"2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 13:23:51.251510 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:51.251460    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 13:23:51.251779 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:51.251528    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 13:23:51.251779 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:51.251569    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14-config-out\") pod \"alertmanager-main-0\" (UID: \"2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 13:23:51.251779 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:51.251612    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 13:23:51.251779 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:51.251641    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 13:23:51.251779 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:51.251714    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 13:23:51.251779 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:51.251759    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 13:23:51.252916 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:51.252673    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5kw6\" (UniqueName: \"kubernetes.io/projected/2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14-kube-api-access-z5kw6\") pod \"alertmanager-main-0\" (UID: \"2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 13:23:51.253274 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:51.253246    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 13:23:51.254197 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:51.254068    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 13:23:51.255493 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:51.254732    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 13:23:51.257046 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:51.256440    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14-tls-assets\") pod \"alertmanager-main-0\" (UID: \"2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 13:23:51.257046 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:51.256985    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14-web-config\") pod \"alertmanager-main-0\" (UID: \"2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 13:23:51.257299 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:51.257277    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14-config-volume\") pod \"alertmanager-main-0\" (UID: \"2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 13:23:51.257360 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:51.257295    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 13:23:51.257863 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:51.257794    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 13:23:51.257983 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:51.257958    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14-config-out\") pod \"alertmanager-main-0\" (UID: \"2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 13:23:51.258104 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:51.258075    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 13:23:51.258531 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:51.258510    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 13:23:51.259478 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:51.259440    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 13:23:51.264504 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:51.264469    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5kw6\" (UniqueName: \"kubernetes.io/projected/2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14-kube-api-access-z5kw6\") pod \"alertmanager-main-0\" (UID: \"2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 13:23:51.313985 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:51.313888    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 11 13:23:51.621017 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:51.620874    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 11 13:23:51.803982 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:51.803873    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gscfs" event={"ID":"7b15b37c-f15c-454a-a7d6-871e4b808b66","Type":"ContainerStarted","Data":"68bd7071a88f204bae305435c262d3d672665ddbfe50db42904094ca0c8212ac"}
Sep 11 13:23:52.019934 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:23:52.019889    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ba2e3c9_64fc_4aa2_8f43_d21a8dba2c14.slice/crio-e2a36213848626d8a830b56bbbdb56267047e7b4c6ef74a93138c4c623351586 WatchSource:0}: Error finding container e2a36213848626d8a830b56bbbdb56267047e7b4c6ef74a93138c4c623351586: Status 404 returned error can't find the container with id e2a36213848626d8a830b56bbbdb56267047e7b4c6ef74a93138c4c623351586
Sep 11 13:23:52.809431 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:52.809387    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14","Type":"ContainerStarted","Data":"e2a36213848626d8a830b56bbbdb56267047e7b4c6ef74a93138c4c623351586"}
Sep 11 13:23:52.811616 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:52.811577    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-ldgml" event={"ID":"eb050379-a425-4c99-81d2-4f07256eb6dc","Type":"ContainerStarted","Data":"649c03cb1d6319a4b8f90c31ff107b9158460cc5d6cdf98ef6ca83ddb97499e5"}
Sep 11 13:23:52.811871 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:52.811652    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-ldgml" event={"ID":"eb050379-a425-4c99-81d2-4f07256eb6dc","Type":"ContainerStarted","Data":"cb52223a00dcc1ee0523b9228d341536406ed15ec5265477898eb98ab591e971"}
Sep 11 13:23:52.811871 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:52.811685    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-ldgml" event={"ID":"eb050379-a425-4c99-81d2-4f07256eb6dc","Type":"ContainerStarted","Data":"5c7bc2812962d98365feb5b7b3b7c62907d598f7058625f69c9abf4622bdf8d9"}
Sep 11 13:23:52.814096 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:52.814060    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-jnwg7" event={"ID":"4f231d67-fe86-4633-b1d9-050a1b9935d9","Type":"ContainerStarted","Data":"7674a0b7ca841f5c245d7555e4f7df43c8367a6c1b67bb37050e41662af65d55"}
Sep 11 13:23:52.815775 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:52.815745    2578 generic.go:356] "Generic (PLEG): container finished" podID="7b15b37c-f15c-454a-a7d6-871e4b808b66" containerID="68bd7071a88f204bae305435c262d3d672665ddbfe50db42904094ca0c8212ac" exitCode=0
Sep 11 13:23:52.815914 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:52.815784    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gscfs" event={"ID":"7b15b37c-f15c-454a-a7d6-871e4b808b66","Type":"ContainerDied","Data":"68bd7071a88f204bae305435c262d3d672665ddbfe50db42904094ca0c8212ac"}
Sep 11 13:23:52.835536 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:52.835482    2578 patch_prober.go:28] interesting pod/image-registry-75fbbd45d6-tcd9m container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 11 13:23:52.835793 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:52.835557    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-75fbbd45d6-tcd9m" podUID="c2d12c02-c63a-4cfe-a074-2d48bc8f3b3a" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 13:23:52.838928 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:52.836995    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-ldgml" podStartSLOduration=2.361143872 podStartE2EDuration="3.836974426s" totalImagesPullingTime="1.475830554s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:23:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:23:52.835349004 +0000 UTC m=+140.221452211" watchObservedRunningTime="2026-09-11 13:23:52.836974426 +0000 UTC m=+140.223077590"
Sep 11 13:23:52.856558 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:52.856485    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-jnwg7" podStartSLOduration=2.518291885 podStartE2EDuration="3.856466067s" totalImagesPullingTime="1.338174182s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:23:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:23:52.856017537 +0000 UTC m=+140.242120728" watchObservedRunningTime="2026-09-11 13:23:52.856466067 +0000 UTC m=+140.242569231"
Sep 11 13:23:53.094028 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:53.093927    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-79dbffb978-qgmlk"]
Sep 11 13:23:53.098412 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:53.098380    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-79dbffb978-qgmlk"
Sep 11 13:23:53.102481 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:53.102443    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Sep 11 13:23:53.102679 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:53.102594    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-fbfb6\""
Sep 11 13:23:53.102734 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:53.102712    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Sep 11 13:23:53.102780 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:53.102740    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-eu3rtdk11i6p6\""
Sep 11 13:23:53.102849 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:53.102818    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Sep 11 13:23:53.102972 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:53.102953    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Sep 11 13:23:53.103044 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:53.103028    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Sep 11 13:23:53.115626 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:53.115581    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-79dbffb978-qgmlk"]
Sep 11 13:23:53.174931 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:53.174871    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7p6h\" (UniqueName: \"kubernetes.io/projected/b36694c5-a445-43e4-b3e1-16c5fa7a1694-kube-api-access-w7p6h\") pod \"thanos-querier-79dbffb978-qgmlk\" (UID: \"b36694c5-a445-43e4-b3e1-16c5fa7a1694\") " pod="openshift-monitoring/thanos-querier-79dbffb978-qgmlk"
Sep 11 13:23:53.174931 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:53.174929    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b36694c5-a445-43e4-b3e1-16c5fa7a1694-metrics-client-ca\") pod \"thanos-querier-79dbffb978-qgmlk\" (UID: \"b36694c5-a445-43e4-b3e1-16c5fa7a1694\") " pod="openshift-monitoring/thanos-querier-79dbffb978-qgmlk"
Sep 11 13:23:53.175257 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:53.175046    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/b36694c5-a445-43e4-b3e1-16c5fa7a1694-secret-thanos-querier-tls\") pod \"thanos-querier-79dbffb978-qgmlk\" (UID: \"b36694c5-a445-43e4-b3e1-16c5fa7a1694\") " pod="openshift-monitoring/thanos-querier-79dbffb978-qgmlk"
Sep 11 13:23:53.175257 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:53.175090    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b36694c5-a445-43e4-b3e1-16c5fa7a1694-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-79dbffb978-qgmlk\" (UID: \"b36694c5-a445-43e4-b3e1-16c5fa7a1694\") " pod="openshift-monitoring/thanos-querier-79dbffb978-qgmlk"
Sep 11 13:23:53.175257 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:53.175132    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b36694c5-a445-43e4-b3e1-16c5fa7a1694-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-79dbffb978-qgmlk\" (UID: \"b36694c5-a445-43e4-b3e1-16c5fa7a1694\") " pod="openshift-monitoring/thanos-querier-79dbffb978-qgmlk"
Sep 11 13:23:53.175257 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:53.175190    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/b36694c5-a445-43e4-b3e1-16c5fa7a1694-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-79dbffb978-qgmlk\" (UID: \"b36694c5-a445-43e4-b3e1-16c5fa7a1694\") " pod="openshift-monitoring/thanos-querier-79dbffb978-qgmlk"
Sep 11 13:23:53.175257 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:53.175222    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/b36694c5-a445-43e4-b3e1-16c5fa7a1694-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-79dbffb978-qgmlk\" (UID: \"b36694c5-a445-43e4-b3e1-16c5fa7a1694\") " pod="openshift-monitoring/thanos-querier-79dbffb978-qgmlk"
Sep 11 13:23:53.175257 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:53.175249    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b36694c5-a445-43e4-b3e1-16c5fa7a1694-secret-grpc-tls\") pod \"thanos-querier-79dbffb978-qgmlk\" (UID: \"b36694c5-a445-43e4-b3e1-16c5fa7a1694\") " pod="openshift-monitoring/thanos-querier-79dbffb978-qgmlk"
Sep 11 13:23:53.276025 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:53.275894    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7p6h\" (UniqueName: \"kubernetes.io/projected/b36694c5-a445-43e4-b3e1-16c5fa7a1694-kube-api-access-w7p6h\") pod \"thanos-querier-79dbffb978-qgmlk\" (UID: \"b36694c5-a445-43e4-b3e1-16c5fa7a1694\") " pod="openshift-monitoring/thanos-querier-79dbffb978-qgmlk"
Sep 11 13:23:53.276025 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:53.275939    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b36694c5-a445-43e4-b3e1-16c5fa7a1694-metrics-client-ca\") pod \"thanos-querier-79dbffb978-qgmlk\" (UID: \"b36694c5-a445-43e4-b3e1-16c5fa7a1694\") " pod="openshift-monitoring/thanos-querier-79dbffb978-qgmlk"
Sep 11 13:23:53.276025 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:53.275990    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/b36694c5-a445-43e4-b3e1-16c5fa7a1694-secret-thanos-querier-tls\") pod \"thanos-querier-79dbffb978-qgmlk\" (UID: \"b36694c5-a445-43e4-b3e1-16c5fa7a1694\") " pod="openshift-monitoring/thanos-querier-79dbffb978-qgmlk"
Sep 11 13:23:53.276192 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:53.276029    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b36694c5-a445-43e4-b3e1-16c5fa7a1694-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-79dbffb978-qgmlk\" (UID: \"b36694c5-a445-43e4-b3e1-16c5fa7a1694\") " pod="openshift-monitoring/thanos-querier-79dbffb978-qgmlk"
Sep 11 13:23:53.276192 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:53.276073    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b36694c5-a445-43e4-b3e1-16c5fa7a1694-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-79dbffb978-qgmlk\" (UID: \"b36694c5-a445-43e4-b3e1-16c5fa7a1694\") " pod="openshift-monitoring/thanos-querier-79dbffb978-qgmlk"
Sep 11 13:23:53.276192 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:53.276121    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/b36694c5-a445-43e4-b3e1-16c5fa7a1694-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-79dbffb978-qgmlk\" (UID: \"b36694c5-a445-43e4-b3e1-16c5fa7a1694\") " pod="openshift-monitoring/thanos-querier-79dbffb978-qgmlk"
Sep 11 13:23:53.276192 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:53.276163    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/b36694c5-a445-43e4-b3e1-16c5fa7a1694-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-79dbffb978-qgmlk\" (UID: \"b36694c5-a445-43e4-b3e1-16c5fa7a1694\") " pod="openshift-monitoring/thanos-querier-79dbffb978-qgmlk"
Sep 11 13:23:53.276396 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:53.276200    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b36694c5-a445-43e4-b3e1-16c5fa7a1694-secret-grpc-tls\") pod \"thanos-querier-79dbffb978-qgmlk\" (UID: \"b36694c5-a445-43e4-b3e1-16c5fa7a1694\") " pod="openshift-monitoring/thanos-querier-79dbffb978-qgmlk"
Sep 11 13:23:53.276952 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:53.276921    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b36694c5-a445-43e4-b3e1-16c5fa7a1694-metrics-client-ca\") pod \"thanos-querier-79dbffb978-qgmlk\" (UID: \"b36694c5-a445-43e4-b3e1-16c5fa7a1694\") " pod="openshift-monitoring/thanos-querier-79dbffb978-qgmlk"
Sep 11 13:23:53.279874 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:53.279800    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/b36694c5-a445-43e4-b3e1-16c5fa7a1694-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-79dbffb978-qgmlk\" (UID: \"b36694c5-a445-43e4-b3e1-16c5fa7a1694\") " pod="openshift-monitoring/thanos-querier-79dbffb978-qgmlk"
Sep 11 13:23:53.279874 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:53.279814    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/b36694c5-a445-43e4-b3e1-16c5fa7a1694-secret-thanos-querier-tls\") pod \"thanos-querier-79dbffb978-qgmlk\" (UID: \"b36694c5-a445-43e4-b3e1-16c5fa7a1694\") " pod="openshift-monitoring/thanos-querier-79dbffb978-qgmlk"
Sep 11 13:23:53.280081 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:53.280020    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/b36694c5-a445-43e4-b3e1-16c5fa7a1694-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-79dbffb978-qgmlk\" (UID: \"b36694c5-a445-43e4-b3e1-16c5fa7a1694\") " pod="openshift-monitoring/thanos-querier-79dbffb978-qgmlk"
Sep 11 13:23:53.280081 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:53.280040    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b36694c5-a445-43e4-b3e1-16c5fa7a1694-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-79dbffb978-qgmlk\" (UID: \"b36694c5-a445-43e4-b3e1-16c5fa7a1694\") " pod="openshift-monitoring/thanos-querier-79dbffb978-qgmlk"
Sep 11 13:23:53.280081 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:53.280043    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b36694c5-a445-43e4-b3e1-16c5fa7a1694-secret-grpc-tls\") pod \"thanos-querier-79dbffb978-qgmlk\" (UID: \"b36694c5-a445-43e4-b3e1-16c5fa7a1694\") " pod="openshift-monitoring/thanos-querier-79dbffb978-qgmlk"
Sep 11 13:23:53.280561 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:53.280524    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b36694c5-a445-43e4-b3e1-16c5fa7a1694-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-79dbffb978-qgmlk\" (UID: \"b36694c5-a445-43e4-b3e1-16c5fa7a1694\") " pod="openshift-monitoring/thanos-querier-79dbffb978-qgmlk"
Sep 11 13:23:53.285809 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:53.285715    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7p6h\" (UniqueName: \"kubernetes.io/projected/b36694c5-a445-43e4-b3e1-16c5fa7a1694-kube-api-access-w7p6h\") pod \"thanos-querier-79dbffb978-qgmlk\" (UID: \"b36694c5-a445-43e4-b3e1-16c5fa7a1694\") " pod="openshift-monitoring/thanos-querier-79dbffb978-qgmlk"
Sep 11 13:23:53.394910 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:23:53.394845    2578 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ba2e3c9_64fc_4aa2_8f43_d21a8dba2c14.slice/crio-a540a924c6a2fd11bf96acb06d013e3fe5be80c1f329b83b7bda82923ebacf79.scope\": RecentStats: unable to find data in memory cache]"
Sep 11 13:23:53.411947 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:53.411906    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-79dbffb978-qgmlk"
Sep 11 13:23:53.702659 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:53.702557    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-79dbffb978-qgmlk"]
Sep 11 13:23:53.706748 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:23:53.706702    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb36694c5_a445_43e4_b3e1_16c5fa7a1694.slice/crio-21b18f682148b2742bd162fd33700504e5901246e3615abc631dc77e19b1fc00 WatchSource:0}: Error finding container 21b18f682148b2742bd162fd33700504e5901246e3615abc631dc77e19b1fc00: Status 404 returned error can't find the container with id 21b18f682148b2742bd162fd33700504e5901246e3615abc631dc77e19b1fc00
Sep 11 13:23:53.820692 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:53.820644    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-79dbffb978-qgmlk" event={"ID":"b36694c5-a445-43e4-b3e1-16c5fa7a1694","Type":"ContainerStarted","Data":"21b18f682148b2742bd162fd33700504e5901246e3615abc631dc77e19b1fc00"}
Sep 11 13:23:53.822978 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:53.822946    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gscfs" event={"ID":"7b15b37c-f15c-454a-a7d6-871e4b808b66","Type":"ContainerStarted","Data":"fa3c36fbe7993e65076825605c809ea1f7d73babfbb39dbcdb200b6e6242227b"}
Sep 11 13:23:53.823130 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:53.822999    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gscfs" event={"ID":"7b15b37c-f15c-454a-a7d6-871e4b808b66","Type":"ContainerStarted","Data":"48bcaee9227f9ed7fa34ed16456a3432d9aa1b244acc3e1d93fa7e2597815697"}
Sep 11 13:23:53.824715 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:53.824681    2578 generic.go:356] "Generic (PLEG): container finished" podID="2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14" containerID="a540a924c6a2fd11bf96acb06d013e3fe5be80c1f329b83b7bda82923ebacf79" exitCode=0
Sep 11 13:23:53.824872 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:53.824759    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14","Type":"ContainerDied","Data":"a540a924c6a2fd11bf96acb06d013e3fe5be80c1f329b83b7bda82923ebacf79"}
Sep 11 13:23:53.842620 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:53.842553    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-gscfs" podStartSLOduration=3.803403572 podStartE2EDuration="4.842529749s" totalImagesPullingTime="944.882486ms" totalInitContainerRuntime="94.243691ms" isStatefulPod=true podCreationTimestamp="2026-09-11 13:23:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:23:53.840984618 +0000 UTC m=+141.227087802" watchObservedRunningTime="2026-09-11 13:23:53.842529749 +0000 UTC m=+141.228632913"
Sep 11 13:23:54.261477 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:54.261430    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-6c44ff6d57-l66j4"]
Sep 11 13:23:54.265359 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:54.265322    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6c44ff6d57-l66j4"
Sep 11 13:23:54.268064 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:54.267816    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-5tznc\""
Sep 11 13:23:54.268379 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:54.268360    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 11 13:23:54.269115 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:54.268785    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Sep 11 13:23:54.269115 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:54.268803    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Sep 11 13:23:54.269115 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:54.268804    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-c8rbdi489f46q\""
Sep 11 13:23:54.269115 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:54.268936    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Sep 11 13:23:54.277962 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:54.277923    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6c44ff6d57-l66j4"]
Sep 11 13:23:54.388858 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:54.388792    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/8e49bb1c-288a-429e-8210-3f38ec6f9b43-metrics-server-audit-profiles\") pod \"metrics-server-6c44ff6d57-l66j4\" (UID: \"8e49bb1c-288a-429e-8210-3f38ec6f9b43\") " pod="openshift-monitoring/metrics-server-6c44ff6d57-l66j4"
Sep 11 13:23:54.389084 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:54.388871    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/8e49bb1c-288a-429e-8210-3f38ec6f9b43-client-ca-bundle\") pod \"metrics-server-6c44ff6d57-l66j4\" (UID: \"8e49bb1c-288a-429e-8210-3f38ec6f9b43\") " pod="openshift-monitoring/metrics-server-6c44ff6d57-l66j4"
Sep 11 13:23:54.389084 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:54.388910    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e49bb1c-288a-429e-8210-3f38ec6f9b43-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6c44ff6d57-l66j4\" (UID: \"8e49bb1c-288a-429e-8210-3f38ec6f9b43\") " pod="openshift-monitoring/metrics-server-6c44ff6d57-l66j4"
Sep 11 13:23:54.389084 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:54.388939    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/8e49bb1c-288a-429e-8210-3f38ec6f9b43-audit-log\") pod \"metrics-server-6c44ff6d57-l66j4\" (UID: \"8e49bb1c-288a-429e-8210-3f38ec6f9b43\") " pod="openshift-monitoring/metrics-server-6c44ff6d57-l66j4"
Sep 11 13:23:54.389084 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:54.389016    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/8e49bb1c-288a-429e-8210-3f38ec6f9b43-secret-metrics-server-client-certs\") pod \"metrics-server-6c44ff6d57-l66j4\" (UID: \"8e49bb1c-288a-429e-8210-3f38ec6f9b43\") " pod="openshift-monitoring/metrics-server-6c44ff6d57-l66j4"
Sep 11 13:23:54.389084 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:54.389055    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47v9t\" (UniqueName: \"kubernetes.io/projected/8e49bb1c-288a-429e-8210-3f38ec6f9b43-kube-api-access-47v9t\") pod \"metrics-server-6c44ff6d57-l66j4\" (UID: \"8e49bb1c-288a-429e-8210-3f38ec6f9b43\") " pod="openshift-monitoring/metrics-server-6c44ff6d57-l66j4"
Sep 11 13:23:54.389084 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:54.389110    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/8e49bb1c-288a-429e-8210-3f38ec6f9b43-secret-metrics-server-tls\") pod \"metrics-server-6c44ff6d57-l66j4\" (UID: \"8e49bb1c-288a-429e-8210-3f38ec6f9b43\") " pod="openshift-monitoring/metrics-server-6c44ff6d57-l66j4"
Sep 11 13:23:54.490615 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:54.490550    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/8e49bb1c-288a-429e-8210-3f38ec6f9b43-metrics-server-audit-profiles\") pod \"metrics-server-6c44ff6d57-l66j4\" (UID: \"8e49bb1c-288a-429e-8210-3f38ec6f9b43\") " pod="openshift-monitoring/metrics-server-6c44ff6d57-l66j4"
Sep 11 13:23:54.490615 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:54.490617    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/8e49bb1c-288a-429e-8210-3f38ec6f9b43-client-ca-bundle\") pod \"metrics-server-6c44ff6d57-l66j4\" (UID: \"8e49bb1c-288a-429e-8210-3f38ec6f9b43\") " pod="openshift-monitoring/metrics-server-6c44ff6d57-l66j4"
Sep 11 13:23:54.491005 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:54.490649    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e49bb1c-288a-429e-8210-3f38ec6f9b43-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6c44ff6d57-l66j4\" (UID: \"8e49bb1c-288a-429e-8210-3f38ec6f9b43\") " pod="openshift-monitoring/metrics-server-6c44ff6d57-l66j4"
Sep 11 13:23:54.491005 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:54.490679    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/8e49bb1c-288a-429e-8210-3f38ec6f9b43-audit-log\") pod \"metrics-server-6c44ff6d57-l66j4\" (UID: \"8e49bb1c-288a-429e-8210-3f38ec6f9b43\") " pod="openshift-monitoring/metrics-server-6c44ff6d57-l66j4"
Sep 11 13:23:54.491005 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:54.490741    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/8e49bb1c-288a-429e-8210-3f38ec6f9b43-secret-metrics-server-client-certs\") pod \"metrics-server-6c44ff6d57-l66j4\" (UID: \"8e49bb1c-288a-429e-8210-3f38ec6f9b43\") " pod="openshift-monitoring/metrics-server-6c44ff6d57-l66j4"
Sep 11 13:23:54.491005 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:54.490782    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47v9t\" (UniqueName: \"kubernetes.io/projected/8e49bb1c-288a-429e-8210-3f38ec6f9b43-kube-api-access-47v9t\") pod \"metrics-server-6c44ff6d57-l66j4\" (UID: \"8e49bb1c-288a-429e-8210-3f38ec6f9b43\") " pod="openshift-monitoring/metrics-server-6c44ff6d57-l66j4"
Sep 11 13:23:54.491005 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:54.490898    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/8e49bb1c-288a-429e-8210-3f38ec6f9b43-secret-metrics-server-tls\") pod \"metrics-server-6c44ff6d57-l66j4\" (UID: \"8e49bb1c-288a-429e-8210-3f38ec6f9b43\") " pod="openshift-monitoring/metrics-server-6c44ff6d57-l66j4"
Sep 11 13:23:54.491416 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:54.491361    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/8e49bb1c-288a-429e-8210-3f38ec6f9b43-audit-log\") pod \"metrics-server-6c44ff6d57-l66j4\" (UID: \"8e49bb1c-288a-429e-8210-3f38ec6f9b43\") " pod="openshift-monitoring/metrics-server-6c44ff6d57-l66j4"
Sep 11 13:23:54.491600 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:54.491541    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e49bb1c-288a-429e-8210-3f38ec6f9b43-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6c44ff6d57-l66j4\" (UID: \"8e49bb1c-288a-429e-8210-3f38ec6f9b43\") " pod="openshift-monitoring/metrics-server-6c44ff6d57-l66j4"
Sep 11 13:23:54.491816 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:54.491793    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/8e49bb1c-288a-429e-8210-3f38ec6f9b43-metrics-server-audit-profiles\") pod \"metrics-server-6c44ff6d57-l66j4\" (UID: \"8e49bb1c-288a-429e-8210-3f38ec6f9b43\") " pod="openshift-monitoring/metrics-server-6c44ff6d57-l66j4"
Sep 11 13:23:54.494697 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:54.494666    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/8e49bb1c-288a-429e-8210-3f38ec6f9b43-secret-metrics-server-client-certs\") pod \"metrics-server-6c44ff6d57-l66j4\" (UID: \"8e49bb1c-288a-429e-8210-3f38ec6f9b43\") " pod="openshift-monitoring/metrics-server-6c44ff6d57-l66j4"
Sep 11 13:23:54.494799 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:54.494726    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/8e49bb1c-288a-429e-8210-3f38ec6f9b43-client-ca-bundle\") pod \"metrics-server-6c44ff6d57-l66j4\" (UID: \"8e49bb1c-288a-429e-8210-3f38ec6f9b43\") " pod="openshift-monitoring/metrics-server-6c44ff6d57-l66j4"
Sep 11 13:23:54.494898 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:54.494875    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/8e49bb1c-288a-429e-8210-3f38ec6f9b43-secret-metrics-server-tls\") pod \"metrics-server-6c44ff6d57-l66j4\" (UID: \"8e49bb1c-288a-429e-8210-3f38ec6f9b43\") " pod="openshift-monitoring/metrics-server-6c44ff6d57-l66j4"
Sep 11 13:23:54.502804 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:54.502767    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-47v9t\" (UniqueName: \"kubernetes.io/projected/8e49bb1c-288a-429e-8210-3f38ec6f9b43-kube-api-access-47v9t\") pod \"metrics-server-6c44ff6d57-l66j4\" (UID: \"8e49bb1c-288a-429e-8210-3f38ec6f9b43\") " pod="openshift-monitoring/metrics-server-6c44ff6d57-l66j4"
Sep 11 13:23:54.594065 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:54.593931    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6c44ff6d57-l66j4"
Sep 11 13:23:54.899791 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:54.899449    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6c44ff6d57-l66j4"]
Sep 11 13:23:55.257307 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:23:55.257219    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e49bb1c_288a_429e_8210_3f38ec6f9b43.slice/crio-5951f5b4f0100223260a4e496976c6eb2bc33592d60744afd9b9752a223167f9 WatchSource:0}: Error finding container 5951f5b4f0100223260a4e496976c6eb2bc33592d60744afd9b9752a223167f9: Status 404 returned error can't find the container with id 5951f5b4f0100223260a4e496976c6eb2bc33592d60744afd9b9752a223167f9
Sep 11 13:23:55.837314 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:55.837261    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6c44ff6d57-l66j4" event={"ID":"8e49bb1c-288a-429e-8210-3f38ec6f9b43","Type":"ContainerStarted","Data":"5951f5b4f0100223260a4e496976c6eb2bc33592d60744afd9b9752a223167f9"}
Sep 11 13:23:56.444567 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:56.444427    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 11 13:23:56.450651 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:56.450605    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:23:56.457246 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:56.455177    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 11 13:23:56.457246 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:56.455694    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 11 13:23:56.457246 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:56.456136    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 11 13:23:56.457246 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:56.456348    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-jj7m5\""
Sep 11 13:23:56.457246 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:56.456674    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 11 13:23:56.457246 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:56.456890    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 11 13:23:56.457818 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:56.457593    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 11 13:23:56.457818 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:56.457671    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 11 13:23:56.457818 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:56.457670    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 11 13:23:56.457818 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:56.457732    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 11 13:23:56.459081 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:56.458997    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 11 13:23:56.459247 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:56.459195    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 11 13:23:56.459722 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:56.459534    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-iuoqnmid4ilk\""
Sep 11 13:23:56.459873 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:56.459787    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 11 13:23:56.460825 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:56.460802    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 11 13:23:56.467629 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:56.465899    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 11 13:23:56.468929 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:56.468895    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 11 13:23:56.618815 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:56.618774    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d48ed82-7fb0-40aa-93de-332e58a6f316-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9d48ed82-7fb0-40aa-93de-332e58a6f316\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:23:56.618815 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:56.618816    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9d48ed82-7fb0-40aa-93de-332e58a6f316-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"9d48ed82-7fb0-40aa-93de-332e58a6f316\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:23:56.619087 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:56.618876    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9d48ed82-7fb0-40aa-93de-332e58a6f316-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"9d48ed82-7fb0-40aa-93de-332e58a6f316\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:23:56.619087 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:56.618901    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/9d48ed82-7fb0-40aa-93de-332e58a6f316-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"9d48ed82-7fb0-40aa-93de-332e58a6f316\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:23:56.619087 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:56.618920    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/9d48ed82-7fb0-40aa-93de-332e58a6f316-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"9d48ed82-7fb0-40aa-93de-332e58a6f316\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:23:56.619087 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:56.618987    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9d48ed82-7fb0-40aa-93de-332e58a6f316-config-out\") pod \"prometheus-k8s-0\" (UID: \"9d48ed82-7fb0-40aa-93de-332e58a6f316\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:23:56.619087 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:56.619043    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d48ed82-7fb0-40aa-93de-332e58a6f316-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9d48ed82-7fb0-40aa-93de-332e58a6f316\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:23:56.619087 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:56.619075    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/9d48ed82-7fb0-40aa-93de-332e58a6f316-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"9d48ed82-7fb0-40aa-93de-332e58a6f316\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:23:56.619298 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:56.619115    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/9d48ed82-7fb0-40aa-93de-332e58a6f316-config\") pod \"prometheus-k8s-0\" (UID: \"9d48ed82-7fb0-40aa-93de-332e58a6f316\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:23:56.619298 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:56.619143    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9d48ed82-7fb0-40aa-93de-332e58a6f316-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"9d48ed82-7fb0-40aa-93de-332e58a6f316\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:23:56.619298 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:56.619180    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/9d48ed82-7fb0-40aa-93de-332e58a6f316-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"9d48ed82-7fb0-40aa-93de-332e58a6f316\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:23:56.619298 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:56.619227    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/9d48ed82-7fb0-40aa-93de-332e58a6f316-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"9d48ed82-7fb0-40aa-93de-332e58a6f316\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:23:56.619298 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:56.619253    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9d48ed82-7fb0-40aa-93de-332e58a6f316-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"9d48ed82-7fb0-40aa-93de-332e58a6f316\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:23:56.619298 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:56.619288    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/9d48ed82-7fb0-40aa-93de-332e58a6f316-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"9d48ed82-7fb0-40aa-93de-332e58a6f316\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:23:56.619485 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:56.619321    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9d48ed82-7fb0-40aa-93de-332e58a6f316-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"9d48ed82-7fb0-40aa-93de-332e58a6f316\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:23:56.619485 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:56.619356    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvzdf\" (UniqueName: \"kubernetes.io/projected/9d48ed82-7fb0-40aa-93de-332e58a6f316-kube-api-access-dvzdf\") pod \"prometheus-k8s-0\" (UID: \"9d48ed82-7fb0-40aa-93de-332e58a6f316\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:23:56.619485 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:56.619395    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d48ed82-7fb0-40aa-93de-332e58a6f316-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9d48ed82-7fb0-40aa-93de-332e58a6f316\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:23:56.619573 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:56.619479    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9d48ed82-7fb0-40aa-93de-332e58a6f316-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"9d48ed82-7fb0-40aa-93de-332e58a6f316\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:23:56.619573 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:56.619505    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9d48ed82-7fb0-40aa-93de-332e58a6f316-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"9d48ed82-7fb0-40aa-93de-332e58a6f316\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:23:56.619573 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:56.619537    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9d48ed82-7fb0-40aa-93de-332e58a6f316-web-config\") pod \"prometheus-k8s-0\" (UID: \"9d48ed82-7fb0-40aa-93de-332e58a6f316\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:23:56.720211 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:56.720153    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d48ed82-7fb0-40aa-93de-332e58a6f316-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9d48ed82-7fb0-40aa-93de-332e58a6f316\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:23:56.720448 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:56.720246    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9d48ed82-7fb0-40aa-93de-332e58a6f316-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"9d48ed82-7fb0-40aa-93de-332e58a6f316\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:23:56.720448 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:56.720274    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9d48ed82-7fb0-40aa-93de-332e58a6f316-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"9d48ed82-7fb0-40aa-93de-332e58a6f316\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:23:56.720448 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:56.720317    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9d48ed82-7fb0-40aa-93de-332e58a6f316-web-config\") pod \"prometheus-k8s-0\" (UID: \"9d48ed82-7fb0-40aa-93de-332e58a6f316\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:23:56.720448 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:56.720380    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d48ed82-7fb0-40aa-93de-332e58a6f316-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9d48ed82-7fb0-40aa-93de-332e58a6f316\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:23:56.720448 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:56.720409    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9d48ed82-7fb0-40aa-93de-332e58a6f316-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"9d48ed82-7fb0-40aa-93de-332e58a6f316\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:23:56.720717 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:56.720456    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9d48ed82-7fb0-40aa-93de-332e58a6f316-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"9d48ed82-7fb0-40aa-93de-332e58a6f316\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:23:56.720717 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:56.720495    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/9d48ed82-7fb0-40aa-93de-332e58a6f316-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"9d48ed82-7fb0-40aa-93de-332e58a6f316\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:23:56.720717 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:56.720533    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/9d48ed82-7fb0-40aa-93de-332e58a6f316-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"9d48ed82-7fb0-40aa-93de-332e58a6f316\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:23:56.720717 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:56.720563    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9d48ed82-7fb0-40aa-93de-332e58a6f316-config-out\") pod \"prometheus-k8s-0\" (UID: \"9d48ed82-7fb0-40aa-93de-332e58a6f316\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:23:56.720717 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:56.720593    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d48ed82-7fb0-40aa-93de-332e58a6f316-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9d48ed82-7fb0-40aa-93de-332e58a6f316\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:23:56.720717 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:56.720617    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/9d48ed82-7fb0-40aa-93de-332e58a6f316-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"9d48ed82-7fb0-40aa-93de-332e58a6f316\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:23:56.720717 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:56.720648    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/9d48ed82-7fb0-40aa-93de-332e58a6f316-config\") pod \"prometheus-k8s-0\" (UID: \"9d48ed82-7fb0-40aa-93de-332e58a6f316\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:23:56.720717 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:56.720673    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9d48ed82-7fb0-40aa-93de-332e58a6f316-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"9d48ed82-7fb0-40aa-93de-332e58a6f316\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:23:56.720717 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:56.720700    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/9d48ed82-7fb0-40aa-93de-332e58a6f316-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"9d48ed82-7fb0-40aa-93de-332e58a6f316\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:23:56.721155 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:56.720751    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/9d48ed82-7fb0-40aa-93de-332e58a6f316-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"9d48ed82-7fb0-40aa-93de-332e58a6f316\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:23:56.721155 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:56.720779    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9d48ed82-7fb0-40aa-93de-332e58a6f316-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"9d48ed82-7fb0-40aa-93de-332e58a6f316\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:23:56.721155 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:56.720814    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/9d48ed82-7fb0-40aa-93de-332e58a6f316-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"9d48ed82-7fb0-40aa-93de-332e58a6f316\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:23:56.721155 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:56.720876    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9d48ed82-7fb0-40aa-93de-332e58a6f316-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"9d48ed82-7fb0-40aa-93de-332e58a6f316\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:23:56.721155 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:56.720916    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvzdf\" (UniqueName: \"kubernetes.io/projected/9d48ed82-7fb0-40aa-93de-332e58a6f316-kube-api-access-dvzdf\") pod \"prometheus-k8s-0\" (UID: \"9d48ed82-7fb0-40aa-93de-332e58a6f316\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:23:56.721386 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:56.721309    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/9d48ed82-7fb0-40aa-93de-332e58a6f316-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"9d48ed82-7fb0-40aa-93de-332e58a6f316\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:23:56.721802 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:56.721709    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d48ed82-7fb0-40aa-93de-332e58a6f316-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9d48ed82-7fb0-40aa-93de-332e58a6f316\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:23:56.722084 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:56.722058    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d48ed82-7fb0-40aa-93de-332e58a6f316-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9d48ed82-7fb0-40aa-93de-332e58a6f316\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:23:56.722510 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:56.722479    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9d48ed82-7fb0-40aa-93de-332e58a6f316-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"9d48ed82-7fb0-40aa-93de-332e58a6f316\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:23:56.723493 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:56.723462    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/9d48ed82-7fb0-40aa-93de-332e58a6f316-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"9d48ed82-7fb0-40aa-93de-332e58a6f316\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:23:56.724071 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:56.724040    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/9d48ed82-7fb0-40aa-93de-332e58a6f316-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"9d48ed82-7fb0-40aa-93de-332e58a6f316\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:23:56.724288 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:56.724259    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d48ed82-7fb0-40aa-93de-332e58a6f316-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9d48ed82-7fb0-40aa-93de-332e58a6f316\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:23:56.729064 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:56.727783    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9d48ed82-7fb0-40aa-93de-332e58a6f316-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"9d48ed82-7fb0-40aa-93de-332e58a6f316\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:23:56.729064 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:56.728392    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9d48ed82-7fb0-40aa-93de-332e58a6f316-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"9d48ed82-7fb0-40aa-93de-332e58a6f316\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:23:56.730351 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:56.730287    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/9d48ed82-7fb0-40aa-93de-332e58a6f316-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"9d48ed82-7fb0-40aa-93de-332e58a6f316\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:23:56.732786 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:56.732749    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9d48ed82-7fb0-40aa-93de-332e58a6f316-config-out\") pod \"prometheus-k8s-0\" (UID: \"9d48ed82-7fb0-40aa-93de-332e58a6f316\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:23:56.733236 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:56.733189    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/9d48ed82-7fb0-40aa-93de-332e58a6f316-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"9d48ed82-7fb0-40aa-93de-332e58a6f316\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:23:56.733588 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:56.733562    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9d48ed82-7fb0-40aa-93de-332e58a6f316-web-config\") pod \"prometheus-k8s-0\" (UID: \"9d48ed82-7fb0-40aa-93de-332e58a6f316\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:23:56.734180 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:56.734150    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9d48ed82-7fb0-40aa-93de-332e58a6f316-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"9d48ed82-7fb0-40aa-93de-332e58a6f316\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:23:56.734310 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:56.734252    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/9d48ed82-7fb0-40aa-93de-332e58a6f316-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"9d48ed82-7fb0-40aa-93de-332e58a6f316\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:23:56.734727 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:56.734679    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9d48ed82-7fb0-40aa-93de-332e58a6f316-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"9d48ed82-7fb0-40aa-93de-332e58a6f316\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:23:56.734977 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:56.734823    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/9d48ed82-7fb0-40aa-93de-332e58a6f316-config\") pod \"prometheus-k8s-0\" (UID: \"9d48ed82-7fb0-40aa-93de-332e58a6f316\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:23:56.735206 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:56.735188    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9d48ed82-7fb0-40aa-93de-332e58a6f316-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"9d48ed82-7fb0-40aa-93de-332e58a6f316\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:23:56.735533 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:56.735500    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9d48ed82-7fb0-40aa-93de-332e58a6f316-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"9d48ed82-7fb0-40aa-93de-332e58a6f316\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:23:56.737183 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:56.737157    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvzdf\" (UniqueName: \"kubernetes.io/projected/9d48ed82-7fb0-40aa-93de-332e58a6f316-kube-api-access-dvzdf\") pod \"prometheus-k8s-0\" (UID: \"9d48ed82-7fb0-40aa-93de-332e58a6f316\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:23:56.773245 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:56.772745    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:23:56.846223 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:56.846161    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-79dbffb978-qgmlk" event={"ID":"b36694c5-a445-43e4-b3e1-16c5fa7a1694","Type":"ContainerStarted","Data":"694558a096b0c38025600a9a7939406c18e1246dfcd0c740b76ced5a628b9664"}
Sep 11 13:23:56.846719 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:56.846259    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-79dbffb978-qgmlk" event={"ID":"b36694c5-a445-43e4-b3e1-16c5fa7a1694","Type":"ContainerStarted","Data":"fc055969cf7df7d6c25f30df143edb5e37cb85c9c09ef939677dc686438173d3"}
Sep 11 13:23:56.846719 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:56.846309    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-79dbffb978-qgmlk" event={"ID":"b36694c5-a445-43e4-b3e1-16c5fa7a1694","Type":"ContainerStarted","Data":"d64f458f74c3535b5492148947a8c9f9ab551243a96e20ad04b2c1b007ca6d08"}
Sep 11 13:23:56.850184 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:56.850144    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14","Type":"ContainerStarted","Data":"22fc6b7c1d3c28be7845ea9f707cfd1c22eb0a3879655bd622dcc3bb9f45f471"}
Sep 11 13:23:56.850350 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:56.850235    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14","Type":"ContainerStarted","Data":"5bfb1479f2966d47bcca384ba4aa2e49ed5fc039cc6e7e4974cf92e8bd6489b2"}
Sep 11 13:23:56.850350 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:56.850270    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14","Type":"ContainerStarted","Data":"e9c8fac09357d085da9630755dee520fe35de8705cbb8ebeeb00f907c20f604e"}
Sep 11 13:23:56.850350 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:56.850298    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14","Type":"ContainerStarted","Data":"658e939ac9e6ccc0a86772de275134ddc80fbc6f865ef7261324c49de258fceb"}
Sep 11 13:23:56.850350 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:56.850338    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14","Type":"ContainerStarted","Data":"c221d02c24a8eac26413c1aba7ed6487aebc4a0be126e0fc569d877914b2915b"}
Sep 11 13:23:57.541393 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:57.541342    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 11 13:23:57.544701 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:23:57.544657    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d48ed82_7fb0_40aa_93de_332e58a6f316.slice/crio-77280823e2a42c65995982d15a83f2bd3125736c8c3ba3968dc74a839a1da94d WatchSource:0}: Error finding container 77280823e2a42c65995982d15a83f2bd3125736c8c3ba3968dc74a839a1da94d: Status 404 returned error can't find the container with id 77280823e2a42c65995982d15a83f2bd3125736c8c3ba3968dc74a839a1da94d
Sep 11 13:23:57.865867 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:57.865708    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14","Type":"ContainerStarted","Data":"ee4768a8e20cac1c19a36c54b7371a0e16f2340db32e96e9df7bf056460909ee"}
Sep 11 13:23:57.867800 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:57.867755    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6c44ff6d57-l66j4" event={"ID":"8e49bb1c-288a-429e-8210-3f38ec6f9b43","Type":"ContainerStarted","Data":"8e4d68e36f9f32313d79d49de39a4f5cb88b8a47b4ccfbc5fc92200377f95b34"}
Sep 11 13:23:57.869581 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:57.869541    2578 generic.go:356] "Generic (PLEG): container finished" podID="9d48ed82-7fb0-40aa-93de-332e58a6f316" containerID="fa724f0373acd6ed1c40dbccc8cdc9c5cd019c03181442172953c11045820322" exitCode=0
Sep 11 13:23:57.869755 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:57.869631    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9d48ed82-7fb0-40aa-93de-332e58a6f316","Type":"ContainerDied","Data":"fa724f0373acd6ed1c40dbccc8cdc9c5cd019c03181442172953c11045820322"}
Sep 11 13:23:57.869755 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:57.869702    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9d48ed82-7fb0-40aa-93de-332e58a6f316","Type":"ContainerStarted","Data":"77280823e2a42c65995982d15a83f2bd3125736c8c3ba3968dc74a839a1da94d"}
Sep 11 13:23:57.874012 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:57.873977    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-79dbffb978-qgmlk" event={"ID":"b36694c5-a445-43e4-b3e1-16c5fa7a1694","Type":"ContainerStarted","Data":"ff4303c610b30939ecd12e84ded78a7c69f128eb26e05429f45341588860ad7a"}
Sep 11 13:23:57.874125 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:57.874078    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-79dbffb978-qgmlk"
Sep 11 13:23:57.874211 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:57.874121    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-79dbffb978-qgmlk" event={"ID":"b36694c5-a445-43e4-b3e1-16c5fa7a1694","Type":"ContainerStarted","Data":"72f6ea158c0b92b4a3b0d52580953f09f2a29f441c73e02e436affd260edb595"}
Sep 11 13:23:57.874211 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:57.874166    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-79dbffb978-qgmlk" event={"ID":"b36694c5-a445-43e4-b3e1-16c5fa7a1694","Type":"ContainerStarted","Data":"0363cd70592cceaedbd90bf60ac38b8f79389082ca65c9a6888e32b7165810e0"}
Sep 11 13:23:57.897401 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:57.896663    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=4.233640696 podStartE2EDuration="7.896630722s" totalImagesPullingTime="3.645230641s" totalInitContainerRuntime="17.759385ms" isStatefulPod=false podCreationTimestamp="2026-09-11 13:23:50 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:23:57.8948067 +0000 UTC m=+145.280909864" watchObservedRunningTime="2026-09-11 13:23:57.896630722 +0000 UTC m=+145.282733887"
Sep 11 13:23:57.919090 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:57.919011    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-79dbffb978-qgmlk" podStartSLOduration=1.6759190259999999 podStartE2EDuration="4.918980406s" totalImagesPullingTime="3.24306138s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:23:53 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:23:57.916329747 +0000 UTC m=+145.302432926" watchObservedRunningTime="2026-09-11 13:23:57.918980406 +0000 UTC m=+145.305083571"
Sep 11 13:23:57.935521 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:23:57.935449    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-6c44ff6d57-l66j4" podStartSLOduration=1.945161122 podStartE2EDuration="3.935423824s" totalImagesPullingTime="1.990262702s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:23:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:23:57.934880166 +0000 UTC m=+145.320983362" watchObservedRunningTime="2026-09-11 13:23:57.935423824 +0000 UTC m=+145.321526989"
Sep 11 13:24:01.897811 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:24:01.897693    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9d48ed82-7fb0-40aa-93de-332e58a6f316","Type":"ContainerStarted","Data":"f1eff727d760318e7e6e70e4d4f33f72c8dbf595b92309bbb4c676b20693e840"}
Sep 11 13:24:01.897811 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:24:01.897802    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9d48ed82-7fb0-40aa-93de-332e58a6f316","Type":"ContainerStarted","Data":"9345ca19b3e6053d8dd6509541647281234bec2ad08f9bde59e68105bd8c8de9"}
Sep 11 13:24:01.898321 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:24:01.897870    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9d48ed82-7fb0-40aa-93de-332e58a6f316","Type":"ContainerStarted","Data":"95f2d5ecfced25cf854ca5f30fc92e9a1b395531eaf8fce98cd6e282287addda"}
Sep 11 13:24:01.898321 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:24:01.897898    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9d48ed82-7fb0-40aa-93de-332e58a6f316","Type":"ContainerStarted","Data":"6ede8f2c12fd7b67e056a1df15a43707e94db17818ecc2e23763fb8e9be47b5d"}
Sep 11 13:24:01.898321 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:24:01.897924    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9d48ed82-7fb0-40aa-93de-332e58a6f316","Type":"ContainerStarted","Data":"a25bd3976df0831c59a095681bd064c2f412e2de45db616678e232de0fd8ae50"}
Sep 11 13:24:01.898321 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:24:01.897949    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9d48ed82-7fb0-40aa-93de-332e58a6f316","Type":"ContainerStarted","Data":"2e2270017b25701463c432676a33ce4b9b3fe4e8ffd66c8a97ae6391812c463e"}
Sep 11 13:24:01.943241 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:24:01.943144    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.731304544 podStartE2EDuration="5.94311905s" totalImagesPullingTime="3.189354903s" totalInitContainerRuntime="22.459603ms" isStatefulPod=false podCreationTimestamp="2026-09-11 13:23:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:24:01.941233333 +0000 UTC m=+149.327336554" watchObservedRunningTime="2026-09-11 13:24:01.94311905 +0000 UTC m=+149.329222214"
Sep 11 13:24:02.833212 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:24:02.833167    2578 patch_prober.go:28] interesting pod/image-registry-75fbbd45d6-tcd9m container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 11 13:24:02.833428 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:24:02.833260    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-75fbbd45d6-tcd9m" podUID="c2d12c02-c63a-4cfe-a074-2d48bc8f3b3a" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 13:24:03.885591 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:24:03.885553    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-79dbffb978-qgmlk"
Sep 11 13:24:06.774028 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:24:06.773979    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:24:07.849088 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:24:07.849001    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-75fbbd45d6-tcd9m" podUID="c2d12c02-c63a-4cfe-a074-2d48bc8f3b3a" containerName="registry" containerID="cri-o://d8cd060cec2e03e880091016b056b46962e0ada9f7703ec1a3099d11a57cea8a" gracePeriod=30
Sep 11 13:24:08.112827 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:24:08.112727    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-75fbbd45d6-tcd9m"
Sep 11 13:24:08.147254 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:24:08.147206    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c2d12c02-c63a-4cfe-a074-2d48bc8f3b3a-bound-sa-token\") pod \"c2d12c02-c63a-4cfe-a074-2d48bc8f3b3a\" (UID: \"c2d12c02-c63a-4cfe-a074-2d48bc8f3b3a\") "
Sep 11 13:24:08.147254 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:24:08.147263    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c2d12c02-c63a-4cfe-a074-2d48bc8f3b3a-registry-certificates\") pod \"c2d12c02-c63a-4cfe-a074-2d48bc8f3b3a\" (UID: \"c2d12c02-c63a-4cfe-a074-2d48bc8f3b3a\") "
Sep 11 13:24:08.147545 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:24:08.147315    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c2d12c02-c63a-4cfe-a074-2d48bc8f3b3a-installation-pull-secrets\") pod \"c2d12c02-c63a-4cfe-a074-2d48bc8f3b3a\" (UID: \"c2d12c02-c63a-4cfe-a074-2d48bc8f3b3a\") "
Sep 11 13:24:08.147545 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:24:08.147357    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c2d12c02-c63a-4cfe-a074-2d48bc8f3b3a-ca-trust-extracted\") pod \"c2d12c02-c63a-4cfe-a074-2d48bc8f3b3a\" (UID: \"c2d12c02-c63a-4cfe-a074-2d48bc8f3b3a\") "
Sep 11 13:24:08.147545 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:24:08.147382    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c2d12c02-c63a-4cfe-a074-2d48bc8f3b3a-registry-tls\") pod \"c2d12c02-c63a-4cfe-a074-2d48bc8f3b3a\" (UID: \"c2d12c02-c63a-4cfe-a074-2d48bc8f3b3a\") "
Sep 11 13:24:08.147545 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:24:08.147404    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jmnb2\" (UniqueName: \"kubernetes.io/projected/c2d12c02-c63a-4cfe-a074-2d48bc8f3b3a-kube-api-access-jmnb2\") pod \"c2d12c02-c63a-4cfe-a074-2d48bc8f3b3a\" (UID: \"c2d12c02-c63a-4cfe-a074-2d48bc8f3b3a\") "
Sep 11 13:24:08.147545 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:24:08.147440    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c2d12c02-c63a-4cfe-a074-2d48bc8f3b3a-trusted-ca\") pod \"c2d12c02-c63a-4cfe-a074-2d48bc8f3b3a\" (UID: \"c2d12c02-c63a-4cfe-a074-2d48bc8f3b3a\") "
Sep 11 13:24:08.147545 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:24:08.147476    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c2d12c02-c63a-4cfe-a074-2d48bc8f3b3a-image-registry-private-configuration\") pod \"c2d12c02-c63a-4cfe-a074-2d48bc8f3b3a\" (UID: \"c2d12c02-c63a-4cfe-a074-2d48bc8f3b3a\") "
Sep 11 13:24:08.149809 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:24:08.147852    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2d12c02-c63a-4cfe-a074-2d48bc8f3b3a-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "c2d12c02-c63a-4cfe-a074-2d48bc8f3b3a" (UID: "c2d12c02-c63a-4cfe-a074-2d48bc8f3b3a"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 13:24:08.149809 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:24:08.148048    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2d12c02-c63a-4cfe-a074-2d48bc8f3b3a-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "c2d12c02-c63a-4cfe-a074-2d48bc8f3b3a" (UID: "c2d12c02-c63a-4cfe-a074-2d48bc8f3b3a"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 13:24:08.151196 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:24:08.151099    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2d12c02-c63a-4cfe-a074-2d48bc8f3b3a-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "c2d12c02-c63a-4cfe-a074-2d48bc8f3b3a" (UID: "c2d12c02-c63a-4cfe-a074-2d48bc8f3b3a"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 13:24:08.151196 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:24:08.151122    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2d12c02-c63a-4cfe-a074-2d48bc8f3b3a-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "c2d12c02-c63a-4cfe-a074-2d48bc8f3b3a" (UID: "c2d12c02-c63a-4cfe-a074-2d48bc8f3b3a"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 13:24:08.151765 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:24:08.151742    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2d12c02-c63a-4cfe-a074-2d48bc8f3b3a-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "c2d12c02-c63a-4cfe-a074-2d48bc8f3b3a" (UID: "c2d12c02-c63a-4cfe-a074-2d48bc8f3b3a"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 13:24:08.151898 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:24:08.151763    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2d12c02-c63a-4cfe-a074-2d48bc8f3b3a-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "c2d12c02-c63a-4cfe-a074-2d48bc8f3b3a" (UID: "c2d12c02-c63a-4cfe-a074-2d48bc8f3b3a"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 13:24:08.151898 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:24:08.151883    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2d12c02-c63a-4cfe-a074-2d48bc8f3b3a-kube-api-access-jmnb2" (OuterVolumeSpecName: "kube-api-access-jmnb2") pod "c2d12c02-c63a-4cfe-a074-2d48bc8f3b3a" (UID: "c2d12c02-c63a-4cfe-a074-2d48bc8f3b3a"). InnerVolumeSpecName "kube-api-access-jmnb2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 13:24:08.157292 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:24:08.157235    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2d12c02-c63a-4cfe-a074-2d48bc8f3b3a-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "c2d12c02-c63a-4cfe-a074-2d48bc8f3b3a" (UID: "c2d12c02-c63a-4cfe-a074-2d48bc8f3b3a"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 13:24:08.249004 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:24:08.248945    2578 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c2d12c02-c63a-4cfe-a074-2d48bc8f3b3a-installation-pull-secrets\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:24:08.249004 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:24:08.248992    2578 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c2d12c02-c63a-4cfe-a074-2d48bc8f3b3a-ca-trust-extracted\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:24:08.249004 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:24:08.249003    2578 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c2d12c02-c63a-4cfe-a074-2d48bc8f3b3a-registry-tls\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:24:08.249004 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:24:08.249015    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmnb2\" (UniqueName: \"kubernetes.io/projected/c2d12c02-c63a-4cfe-a074-2d48bc8f3b3a-kube-api-access-jmnb2\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:24:08.249004 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:24:08.249025    2578 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c2d12c02-c63a-4cfe-a074-2d48bc8f3b3a-trusted-ca\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:24:08.249373 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:24:08.249038    2578 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c2d12c02-c63a-4cfe-a074-2d48bc8f3b3a-image-registry-private-configuration\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:24:08.249373 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:24:08.249048    2578 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c2d12c02-c63a-4cfe-a074-2d48bc8f3b3a-bound-sa-token\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:24:08.249373 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:24:08.249057    2578 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c2d12c02-c63a-4cfe-a074-2d48bc8f3b3a-registry-certificates\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:24:08.943512 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:24:08.943467    2578 generic.go:356] "Generic (PLEG): container finished" podID="c2d12c02-c63a-4cfe-a074-2d48bc8f3b3a" containerID="d8cd060cec2e03e880091016b056b46962e0ada9f7703ec1a3099d11a57cea8a" exitCode=0
Sep 11 13:24:08.944026 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:24:08.943534    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-75fbbd45d6-tcd9m"
Sep 11 13:24:08.944026 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:24:08.943556    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-75fbbd45d6-tcd9m" event={"ID":"c2d12c02-c63a-4cfe-a074-2d48bc8f3b3a","Type":"ContainerDied","Data":"d8cd060cec2e03e880091016b056b46962e0ada9f7703ec1a3099d11a57cea8a"}
Sep 11 13:24:08.944026 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:24:08.943677    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-75fbbd45d6-tcd9m" event={"ID":"c2d12c02-c63a-4cfe-a074-2d48bc8f3b3a","Type":"ContainerDied","Data":"0c20f6d9ea532f1a883ad1e489f7ade214d4ba7c32fc6d356970e4d2a5404c66"}
Sep 11 13:24:08.944026 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:24:08.943726    2578 scope.go:117] "RemoveContainer" containerID="d8cd060cec2e03e880091016b056b46962e0ada9f7703ec1a3099d11a57cea8a"
Sep 11 13:24:08.954406 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:24:08.954387    2578 scope.go:117] "RemoveContainer" containerID="d8cd060cec2e03e880091016b056b46962e0ada9f7703ec1a3099d11a57cea8a"
Sep 11 13:24:08.954773 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:24:08.954754    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8cd060cec2e03e880091016b056b46962e0ada9f7703ec1a3099d11a57cea8a\": container with ID starting with d8cd060cec2e03e880091016b056b46962e0ada9f7703ec1a3099d11a57cea8a not found: ID does not exist" containerID="d8cd060cec2e03e880091016b056b46962e0ada9f7703ec1a3099d11a57cea8a"
Sep 11 13:24:08.954824 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:24:08.954786    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8cd060cec2e03e880091016b056b46962e0ada9f7703ec1a3099d11a57cea8a"} err="failed to get container status \"d8cd060cec2e03e880091016b056b46962e0ada9f7703ec1a3099d11a57cea8a\": rpc error: code = NotFound desc = could not find container \"d8cd060cec2e03e880091016b056b46962e0ada9f7703ec1a3099d11a57cea8a\": container with ID starting with d8cd060cec2e03e880091016b056b46962e0ada9f7703ec1a3099d11a57cea8a not found: ID does not exist"
Sep 11 13:24:08.966198 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:24:08.966161    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-75fbbd45d6-tcd9m"]
Sep 11 13:24:08.969585 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:24:08.969552    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-75fbbd45d6-tcd9m"]
Sep 11 13:24:09.106261 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:24:09.106215    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c2d12c02-c63a-4cfe-a074-2d48bc8f3b3a" path="/var/lib/kubelet/pods/c2d12c02-c63a-4cfe-a074-2d48bc8f3b3a/volumes"
Sep 11 13:24:09.950261 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:24:09.950226    2578 generic.go:356] "Generic (PLEG): container finished" podID="f833a58e-8d54-42fc-848c-bea07b176b0b" containerID="cc8ed32e78e27a2c821e92dbfd2970302c108db7cbb3cfb5d523948014426323" exitCode=0
Sep 11 13:24:09.950759 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:24:09.950307    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-m686n" event={"ID":"f833a58e-8d54-42fc-848c-bea07b176b0b","Type":"ContainerDied","Data":"cc8ed32e78e27a2c821e92dbfd2970302c108db7cbb3cfb5d523948014426323"}
Sep 11 13:24:09.950810 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:24:09.950756    2578 scope.go:117] "RemoveContainer" containerID="cc8ed32e78e27a2c821e92dbfd2970302c108db7cbb3cfb5d523948014426323"
Sep 11 13:24:10.959488 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:24:10.959446    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-m686n" event={"ID":"f833a58e-8d54-42fc-848c-bea07b176b0b","Type":"ContainerStarted","Data":"1fc5558adb582c5527af42d36d7a443a969433ac81cf28cef797448d8ffab2c5"}
Sep 11 13:24:12.025973 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:24:12.025916    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-pfffb" podUID="5ab38ce4-40c2-4630-98f3-62d1b84532a5"
Sep 11 13:24:12.860910 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:24:12.860868    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-z8p25_f7ab88b5-7073-460e-b3d4-a9841f50b185/dns-node-resolver/0.log"
Sep 11 13:24:12.968980 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:24:12.968940    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-pfffb"
Sep 11 13:24:13.464626 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:24:13.464589    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-8jj8l_d9bf058c-d89b-4236-86ce-3dee6f389711/node-ca/0.log"
Sep 11 13:24:14.062330 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:24:14.062290    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7798cf8dd4-5hzkk_a93d3879-2973-4fa1-bd7d-bc855a257ddf/router/0.log"
Sep 11 13:24:14.461702 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:24:14.461602    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-dcgnn_42df0901-f4f9-4a7e-85da-4551acd80be7/serve-healthcheck-canary/0.log"
Sep 11 13:24:14.594650 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:24:14.594593    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-6c44ff6d57-l66j4"
Sep 11 13:24:14.595159 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:24:14.594689    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-6c44ff6d57-l66j4"
Sep 11 13:24:16.937617 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:24:16.937547    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5ab38ce4-40c2-4630-98f3-62d1b84532a5-metrics-tls\") pod \"dns-default-pfffb\" (UID: \"5ab38ce4-40c2-4630-98f3-62d1b84532a5\") " pod="openshift-dns/dns-default-pfffb"
Sep 11 13:24:16.940497 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:24:16.940465    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5ab38ce4-40c2-4630-98f3-62d1b84532a5-metrics-tls\") pod \"dns-default-pfffb\" (UID: \"5ab38ce4-40c2-4630-98f3-62d1b84532a5\") " pod="openshift-dns/dns-default-pfffb"
Sep 11 13:24:17.180071 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:24:17.180025    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-vqfbt\""
Sep 11 13:24:17.180071 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:24:17.180055    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-pfffb"
Sep 11 13:24:17.476601 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:24:17.476539    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-pfffb"]
Sep 11 13:24:17.479265 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:24:17.479222    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ab38ce4_40c2_4630_98f3_62d1b84532a5.slice/crio-b7af970015c6800eddd688c45a02940d9149a8314465a7558a8e9e88d3706626 WatchSource:0}: Error finding container b7af970015c6800eddd688c45a02940d9149a8314465a7558a8e9e88d3706626: Status 404 returned error can't find the container with id b7af970015c6800eddd688c45a02940d9149a8314465a7558a8e9e88d3706626
Sep 11 13:24:17.994263 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:24:17.994193    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pfffb" event={"ID":"5ab38ce4-40c2-4630-98f3-62d1b84532a5","Type":"ContainerStarted","Data":"b7af970015c6800eddd688c45a02940d9149a8314465a7558a8e9e88d3706626"}
Sep 11 13:24:20.006574 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:24:20.006523    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pfffb" event={"ID":"5ab38ce4-40c2-4630-98f3-62d1b84532a5","Type":"ContainerStarted","Data":"9c988427b95339eb0c86aace1e6b3a68eb23904fbb7627bcf417bd6d05dafdba"}
Sep 11 13:24:20.007150 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:24:20.006599    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pfffb" event={"ID":"5ab38ce4-40c2-4630-98f3-62d1b84532a5","Type":"ContainerStarted","Data":"c2f91e009760d6567039695a41b6ef2057a21c20b53711222cb4504c430288c1"}
Sep 11 13:24:20.007150 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:24:20.006712    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-pfffb"
Sep 11 13:24:20.028295 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:24:20.028208    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-pfffb" podStartSLOduration=130.27134653 podStartE2EDuration="2m12.028187511s" totalImagesPullingTime="1.756840981s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:22:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:24:20.026986718 +0000 UTC m=+167.413089894" watchObservedRunningTime="2026-09-11 13:24:20.028187511 +0000 UTC m=+167.414290674"
Sep 11 13:24:30.014016 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:24:30.013913    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-pfffb"
Sep 11 13:24:34.600652 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:24:34.600606    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-6c44ff6d57-l66j4"
Sep 11 13:24:34.605053 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:24:34.605022    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-6c44ff6d57-l66j4"
Sep 11 13:24:56.774177 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:24:56.774100    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:24:56.793237 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:24:56.793199    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:24:57.198459 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:24:57.198370    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:25:10.497566 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:10.497517    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 11 13:25:10.498308 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:10.497968    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14" containerName="alertmanager" containerID="cri-o://c221d02c24a8eac26413c1aba7ed6487aebc4a0be126e0fc569d877914b2915b" gracePeriod=120
Sep 11 13:25:10.498308 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:10.498025    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14" containerName="kube-rbac-proxy-metric" containerID="cri-o://22fc6b7c1d3c28be7845ea9f707cfd1c22eb0a3879655bd622dcc3bb9f45f471" gracePeriod=120
Sep 11 13:25:10.498308 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:10.498071    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14" containerName="kube-rbac-proxy-web" containerID="cri-o://e9c8fac09357d085da9630755dee520fe35de8705cbb8ebeeb00f907c20f604e" gracePeriod=120
Sep 11 13:25:10.498308 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:10.498107    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14" containerName="config-reloader" containerID="cri-o://658e939ac9e6ccc0a86772de275134ddc80fbc6f865ef7261324c49de258fceb" gracePeriod=120
Sep 11 13:25:10.498308 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:10.498163    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14" containerName="prom-label-proxy" containerID="cri-o://ee4768a8e20cac1c19a36c54b7371a0e16f2340db32e96e9df7bf056460909ee" gracePeriod=120
Sep 11 13:25:10.498308 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:10.498103    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14" containerName="kube-rbac-proxy" containerID="cri-o://5bfb1479f2966d47bcca384ba4aa2e49ed5fc039cc6e7e4974cf92e8bd6489b2" gracePeriod=120
Sep 11 13:25:11.246438 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:11.246385    2578 generic.go:356] "Generic (PLEG): container finished" podID="2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14" containerID="ee4768a8e20cac1c19a36c54b7371a0e16f2340db32e96e9df7bf056460909ee" exitCode=0
Sep 11 13:25:11.246438 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:11.246425    2578 generic.go:356] "Generic (PLEG): container finished" podID="2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14" containerID="5bfb1479f2966d47bcca384ba4aa2e49ed5fc039cc6e7e4974cf92e8bd6489b2" exitCode=0
Sep 11 13:25:11.246438 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:11.246438    2578 generic.go:356] "Generic (PLEG): container finished" podID="2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14" containerID="e9c8fac09357d085da9630755dee520fe35de8705cbb8ebeeb00f907c20f604e" exitCode=0
Sep 11 13:25:11.246438 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:11.246445    2578 generic.go:356] "Generic (PLEG): container finished" podID="2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14" containerID="658e939ac9e6ccc0a86772de275134ddc80fbc6f865ef7261324c49de258fceb" exitCode=0
Sep 11 13:25:11.246438 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:11.246450    2578 generic.go:356] "Generic (PLEG): container finished" podID="2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14" containerID="c221d02c24a8eac26413c1aba7ed6487aebc4a0be126e0fc569d877914b2915b" exitCode=0
Sep 11 13:25:11.246818 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:11.246451    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14","Type":"ContainerDied","Data":"ee4768a8e20cac1c19a36c54b7371a0e16f2340db32e96e9df7bf056460909ee"}
Sep 11 13:25:11.246818 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:11.246552    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14","Type":"ContainerDied","Data":"5bfb1479f2966d47bcca384ba4aa2e49ed5fc039cc6e7e4974cf92e8bd6489b2"}
Sep 11 13:25:11.246818 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:11.246582    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14","Type":"ContainerDied","Data":"e9c8fac09357d085da9630755dee520fe35de8705cbb8ebeeb00f907c20f604e"}
Sep 11 13:25:11.246818 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:11.246610    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14","Type":"ContainerDied","Data":"658e939ac9e6ccc0a86772de275134ddc80fbc6f865ef7261324c49de258fceb"}
Sep 11 13:25:11.246818 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:11.246636    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14","Type":"ContainerDied","Data":"c221d02c24a8eac26413c1aba7ed6487aebc4a0be126e0fc569d877914b2915b"}
Sep 11 13:25:11.756636 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:11.756605    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 11 13:25:11.806903 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:11.806737    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14-secret-alertmanager-kube-rbac-proxy-metric\") pod \"2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14\" (UID: \"2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14\") "
Sep 11 13:25:11.806903 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:11.806810    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14-config-volume\") pod \"2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14\" (UID: \"2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14\") "
Sep 11 13:25:11.806903 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:11.806881    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14-secret-alertmanager-kube-rbac-proxy-web\") pod \"2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14\" (UID: \"2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14\") "
Sep 11 13:25:11.807228 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:11.806931    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14-secret-alertmanager-kube-rbac-proxy\") pod \"2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14\" (UID: \"2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14\") "
Sep 11 13:25:11.807228 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:11.806964    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14-cluster-tls-config\") pod \"2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14\" (UID: \"2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14\") "
Sep 11 13:25:11.807228 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:11.806996    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z5kw6\" (UniqueName: \"kubernetes.io/projected/2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14-kube-api-access-z5kw6\") pod \"2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14\" (UID: \"2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14\") "
Sep 11 13:25:11.807228 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:11.807021    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14-secret-alertmanager-main-tls\") pod \"2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14\" (UID: \"2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14\") "
Sep 11 13:25:11.807228 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:11.807079    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14-alertmanager-main-db\") pod \"2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14\" (UID: \"2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14\") "
Sep 11 13:25:11.807228 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:11.807142    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14-alertmanager-trusted-ca-bundle\") pod \"2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14\" (UID: \"2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14\") "
Sep 11 13:25:11.807228 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:11.807199    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14-metrics-client-ca\") pod \"2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14\" (UID: \"2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14\") "
Sep 11 13:25:11.807525 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:11.807254    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14-tls-assets\") pod \"2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14\" (UID: \"2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14\") "
Sep 11 13:25:11.807525 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:11.807299    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14-web-config\") pod \"2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14\" (UID: \"2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14\") "
Sep 11 13:25:11.807525 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:11.807344    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14-config-out\") pod \"2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14\" (UID: \"2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14\") "
Sep 11 13:25:11.808759 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:11.808456    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14" (UID: "2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 13:25:11.811035 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:11.810980    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14" (UID: "2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 13:25:11.811212 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:11.811082    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14" (UID: "2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 13:25:11.812012 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:11.811974    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14" (UID: "2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 13:25:11.812131 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:11.812068    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14-config-out" (OuterVolumeSpecName: "config-out") pod "2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14" (UID: "2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 13:25:11.813032 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:11.812996    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14-kube-api-access-z5kw6" (OuterVolumeSpecName: "kube-api-access-z5kw6") pod "2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14" (UID: "2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14"). InnerVolumeSpecName "kube-api-access-z5kw6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 13:25:11.814083 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:11.814033    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14" (UID: "2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 13:25:11.814531 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:11.814334    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14" (UID: "2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 13:25:11.814999 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:11.814952    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14" (UID: "2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 13:25:11.815532 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:11.815481    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14" (UID: "2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 13:25:11.816302 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:11.816276    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14-config-volume" (OuterVolumeSpecName: "config-volume") pod "2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14" (UID: "2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 13:25:11.818683 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:11.818645    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14" (UID: "2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 13:25:11.829162 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:11.829104    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14-web-config" (OuterVolumeSpecName: "web-config") pod "2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14" (UID: "2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 13:25:11.908773 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:11.908724    2578 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14-web-config\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:25:11.908773 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:11.908768    2578 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14-config-out\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:25:11.908773 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:11.908779    2578 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:25:11.908773 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:11.908790    2578 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14-config-volume\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:25:11.909110 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:11.908800    2578 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:25:11.909110 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:11.908810    2578 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:25:11.909110 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:11.908820    2578 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14-cluster-tls-config\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:25:11.909110 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:11.908853    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5kw6\" (UniqueName: \"kubernetes.io/projected/2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14-kube-api-access-z5kw6\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:25:11.909110 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:11.908863    2578 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14-secret-alertmanager-main-tls\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:25:11.909110 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:11.908873    2578 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14-alertmanager-main-db\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:25:11.909110 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:11.908882    2578 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:25:11.909110 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:11.908889    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14-metrics-client-ca\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:25:11.909110 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:11.908899    2578 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14-tls-assets\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:25:12.254348 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:12.254315    2578 generic.go:356] "Generic (PLEG): container finished" podID="2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14" containerID="22fc6b7c1d3c28be7845ea9f707cfd1c22eb0a3879655bd622dcc3bb9f45f471" exitCode=0
Sep 11 13:25:12.254557 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:12.254414    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 11 13:25:12.254557 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:12.254400    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14","Type":"ContainerDied","Data":"22fc6b7c1d3c28be7845ea9f707cfd1c22eb0a3879655bd622dcc3bb9f45f471"}
Sep 11 13:25:12.254635 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:12.254563    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14","Type":"ContainerDied","Data":"e2a36213848626d8a830b56bbbdb56267047e7b4c6ef74a93138c4c623351586"}
Sep 11 13:25:12.254635 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:12.254602    2578 scope.go:117] "RemoveContainer" containerID="ee4768a8e20cac1c19a36c54b7371a0e16f2340db32e96e9df7bf056460909ee"
Sep 11 13:25:12.264413 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:12.264387    2578 scope.go:117] "RemoveContainer" containerID="22fc6b7c1d3c28be7845ea9f707cfd1c22eb0a3879655bd622dcc3bb9f45f471"
Sep 11 13:25:12.273774 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:12.273747    2578 scope.go:117] "RemoveContainer" containerID="5bfb1479f2966d47bcca384ba4aa2e49ed5fc039cc6e7e4974cf92e8bd6489b2"
Sep 11 13:25:12.282285 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:12.282244    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 11 13:25:12.284455 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:12.284434    2578 scope.go:117] "RemoveContainer" containerID="e9c8fac09357d085da9630755dee520fe35de8705cbb8ebeeb00f907c20f604e"
Sep 11 13:25:12.290984 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:12.290948    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 11 13:25:12.294405 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:12.294378    2578 scope.go:117] "RemoveContainer" containerID="658e939ac9e6ccc0a86772de275134ddc80fbc6f865ef7261324c49de258fceb"
Sep 11 13:25:12.303878 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:12.303854    2578 scope.go:117] "RemoveContainer" containerID="c221d02c24a8eac26413c1aba7ed6487aebc4a0be126e0fc569d877914b2915b"
Sep 11 13:25:12.313458 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:12.313424    2578 scope.go:117] "RemoveContainer" containerID="a540a924c6a2fd11bf96acb06d013e3fe5be80c1f329b83b7bda82923ebacf79"
Sep 11 13:25:12.324171 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:12.324143    2578 scope.go:117] "RemoveContainer" containerID="ee4768a8e20cac1c19a36c54b7371a0e16f2340db32e96e9df7bf056460909ee"
Sep 11 13:25:12.324616 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:25:12.324594    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee4768a8e20cac1c19a36c54b7371a0e16f2340db32e96e9df7bf056460909ee\": container with ID starting with ee4768a8e20cac1c19a36c54b7371a0e16f2340db32e96e9df7bf056460909ee not found: ID does not exist" containerID="ee4768a8e20cac1c19a36c54b7371a0e16f2340db32e96e9df7bf056460909ee"
Sep 11 13:25:12.324691 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:12.324628    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee4768a8e20cac1c19a36c54b7371a0e16f2340db32e96e9df7bf056460909ee"} err="failed to get container status \"ee4768a8e20cac1c19a36c54b7371a0e16f2340db32e96e9df7bf056460909ee\": rpc error: code = NotFound desc = could not find container \"ee4768a8e20cac1c19a36c54b7371a0e16f2340db32e96e9df7bf056460909ee\": container with ID starting with ee4768a8e20cac1c19a36c54b7371a0e16f2340db32e96e9df7bf056460909ee not found: ID does not exist"
Sep 11 13:25:12.324691 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:12.324653    2578 scope.go:117] "RemoveContainer" containerID="22fc6b7c1d3c28be7845ea9f707cfd1c22eb0a3879655bd622dcc3bb9f45f471"
Sep 11 13:25:12.325008 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:25:12.324980    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22fc6b7c1d3c28be7845ea9f707cfd1c22eb0a3879655bd622dcc3bb9f45f471\": container with ID starting with 22fc6b7c1d3c28be7845ea9f707cfd1c22eb0a3879655bd622dcc3bb9f45f471 not found: ID does not exist" containerID="22fc6b7c1d3c28be7845ea9f707cfd1c22eb0a3879655bd622dcc3bb9f45f471"
Sep 11 13:25:12.325146 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:12.325015    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22fc6b7c1d3c28be7845ea9f707cfd1c22eb0a3879655bd622dcc3bb9f45f471"} err="failed to get container status \"22fc6b7c1d3c28be7845ea9f707cfd1c22eb0a3879655bd622dcc3bb9f45f471\": rpc error: code = NotFound desc = could not find container \"22fc6b7c1d3c28be7845ea9f707cfd1c22eb0a3879655bd622dcc3bb9f45f471\": container with ID starting with 22fc6b7c1d3c28be7845ea9f707cfd1c22eb0a3879655bd622dcc3bb9f45f471 not found: ID does not exist"
Sep 11 13:25:12.325146 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:12.325038    2578 scope.go:117] "RemoveContainer" containerID="5bfb1479f2966d47bcca384ba4aa2e49ed5fc039cc6e7e4974cf92e8bd6489b2"
Sep 11 13:25:12.325326 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:25:12.325306    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5bfb1479f2966d47bcca384ba4aa2e49ed5fc039cc6e7e4974cf92e8bd6489b2\": container with ID starting with 5bfb1479f2966d47bcca384ba4aa2e49ed5fc039cc6e7e4974cf92e8bd6489b2 not found: ID does not exist" containerID="5bfb1479f2966d47bcca384ba4aa2e49ed5fc039cc6e7e4974cf92e8bd6489b2"
Sep 11 13:25:12.325369 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:12.325334    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5bfb1479f2966d47bcca384ba4aa2e49ed5fc039cc6e7e4974cf92e8bd6489b2"} err="failed to get container status \"5bfb1479f2966d47bcca384ba4aa2e49ed5fc039cc6e7e4974cf92e8bd6489b2\": rpc error: code = NotFound desc = could not find container \"5bfb1479f2966d47bcca384ba4aa2e49ed5fc039cc6e7e4974cf92e8bd6489b2\": container with ID starting with 5bfb1479f2966d47bcca384ba4aa2e49ed5fc039cc6e7e4974cf92e8bd6489b2 not found: ID does not exist"
Sep 11 13:25:12.325369 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:12.325352    2578 scope.go:117] "RemoveContainer" containerID="e9c8fac09357d085da9630755dee520fe35de8705cbb8ebeeb00f907c20f604e"
Sep 11 13:25:12.325650 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:25:12.325633    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9c8fac09357d085da9630755dee520fe35de8705cbb8ebeeb00f907c20f604e\": container with ID starting with e9c8fac09357d085da9630755dee520fe35de8705cbb8ebeeb00f907c20f604e not found: ID does not exist" containerID="e9c8fac09357d085da9630755dee520fe35de8705cbb8ebeeb00f907c20f604e"
Sep 11 13:25:12.325697 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:12.325654    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9c8fac09357d085da9630755dee520fe35de8705cbb8ebeeb00f907c20f604e"} err="failed to get container status \"e9c8fac09357d085da9630755dee520fe35de8705cbb8ebeeb00f907c20f604e\": rpc error: code = NotFound desc = could not find container \"e9c8fac09357d085da9630755dee520fe35de8705cbb8ebeeb00f907c20f604e\": container with ID starting with e9c8fac09357d085da9630755dee520fe35de8705cbb8ebeeb00f907c20f604e not found: ID does not exist"
Sep 11 13:25:12.325697 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:12.325671    2578 scope.go:117] "RemoveContainer" containerID="658e939ac9e6ccc0a86772de275134ddc80fbc6f865ef7261324c49de258fceb"
Sep 11 13:25:12.325916 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:25:12.325899    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"658e939ac9e6ccc0a86772de275134ddc80fbc6f865ef7261324c49de258fceb\": container with ID starting with 658e939ac9e6ccc0a86772de275134ddc80fbc6f865ef7261324c49de258fceb not found: ID does not exist" containerID="658e939ac9e6ccc0a86772de275134ddc80fbc6f865ef7261324c49de258fceb"
Sep 11 13:25:12.325974 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:12.325921    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"658e939ac9e6ccc0a86772de275134ddc80fbc6f865ef7261324c49de258fceb"} err="failed to get container status \"658e939ac9e6ccc0a86772de275134ddc80fbc6f865ef7261324c49de258fceb\": rpc error: code = NotFound desc = could not find container \"658e939ac9e6ccc0a86772de275134ddc80fbc6f865ef7261324c49de258fceb\": container with ID starting with 658e939ac9e6ccc0a86772de275134ddc80fbc6f865ef7261324c49de258fceb not found: ID does not exist"
Sep 11 13:25:12.325974 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:12.325935    2578 scope.go:117] "RemoveContainer" containerID="c221d02c24a8eac26413c1aba7ed6487aebc4a0be126e0fc569d877914b2915b"
Sep 11 13:25:12.326180 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:25:12.326161    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c221d02c24a8eac26413c1aba7ed6487aebc4a0be126e0fc569d877914b2915b\": container with ID starting with c221d02c24a8eac26413c1aba7ed6487aebc4a0be126e0fc569d877914b2915b not found: ID does not exist" containerID="c221d02c24a8eac26413c1aba7ed6487aebc4a0be126e0fc569d877914b2915b"
Sep 11 13:25:12.326224 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:12.326184    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c221d02c24a8eac26413c1aba7ed6487aebc4a0be126e0fc569d877914b2915b"} err="failed to get container status \"c221d02c24a8eac26413c1aba7ed6487aebc4a0be126e0fc569d877914b2915b\": rpc error: code = NotFound desc = could not find container \"c221d02c24a8eac26413c1aba7ed6487aebc4a0be126e0fc569d877914b2915b\": container with ID starting with c221d02c24a8eac26413c1aba7ed6487aebc4a0be126e0fc569d877914b2915b not found: ID does not exist"
Sep 11 13:25:12.326224 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:12.326199    2578 scope.go:117] "RemoveContainer" containerID="a540a924c6a2fd11bf96acb06d013e3fe5be80c1f329b83b7bda82923ebacf79"
Sep 11 13:25:12.326419 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:25:12.326388    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a540a924c6a2fd11bf96acb06d013e3fe5be80c1f329b83b7bda82923ebacf79\": container with ID starting with a540a924c6a2fd11bf96acb06d013e3fe5be80c1f329b83b7bda82923ebacf79 not found: ID does not exist" containerID="a540a924c6a2fd11bf96acb06d013e3fe5be80c1f329b83b7bda82923ebacf79"
Sep 11 13:25:12.326468 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:12.326422    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a540a924c6a2fd11bf96acb06d013e3fe5be80c1f329b83b7bda82923ebacf79"} err="failed to get container status \"a540a924c6a2fd11bf96acb06d013e3fe5be80c1f329b83b7bda82923ebacf79\": rpc error: code = NotFound desc = could not find container \"a540a924c6a2fd11bf96acb06d013e3fe5be80c1f329b83b7bda82923ebacf79\": container with ID starting with a540a924c6a2fd11bf96acb06d013e3fe5be80c1f329b83b7bda82923ebacf79 not found: ID does not exist"
Sep 11 13:25:12.331041 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:12.331004    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 11 13:25:12.331370 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:12.331358    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14" containerName="alertmanager"
Sep 11 13:25:12.331428 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:12.331372    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14" containerName="alertmanager"
Sep 11 13:25:12.331428 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:12.331383    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14" containerName="kube-rbac-proxy-metric"
Sep 11 13:25:12.331428 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:12.331389    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14" containerName="kube-rbac-proxy-metric"
Sep 11 13:25:12.331428 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:12.331397    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14" containerName="kube-rbac-proxy-web"
Sep 11 13:25:12.331428 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:12.331403    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14" containerName="kube-rbac-proxy-web"
Sep 11 13:25:12.331428 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:12.331419    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14" containerName="config-reloader"
Sep 11 13:25:12.331428 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:12.331425    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14" containerName="config-reloader"
Sep 11 13:25:12.331428 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:12.331431    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14" containerName="kube-rbac-proxy"
Sep 11 13:25:12.331671 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:12.331437    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14" containerName="kube-rbac-proxy"
Sep 11 13:25:12.331671 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:12.331443    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14" containerName="prom-label-proxy"
Sep 11 13:25:12.331671 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:12.331448    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14" containerName="prom-label-proxy"
Sep 11 13:25:12.331671 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:12.331459    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14" containerName="init-config-reloader"
Sep 11 13:25:12.331671 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:12.331466    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14" containerName="init-config-reloader"
Sep 11 13:25:12.331671 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:12.331478    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2d12c02-c63a-4cfe-a074-2d48bc8f3b3a" containerName="registry"
Sep 11 13:25:12.331671 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:12.331483    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2d12c02-c63a-4cfe-a074-2d48bc8f3b3a" containerName="registry"
Sep 11 13:25:12.331671 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:12.331527    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14" containerName="config-reloader"
Sep 11 13:25:12.331671 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:12.331536    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14" containerName="kube-rbac-proxy"
Sep 11 13:25:12.331671 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:12.331542    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14" containerName="prom-label-proxy"
Sep 11 13:25:12.331671 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:12.331547    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="c2d12c02-c63a-4cfe-a074-2d48bc8f3b3a" containerName="registry"
Sep 11 13:25:12.331671 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:12.331555    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14" containerName="alertmanager"
Sep 11 13:25:12.331671 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:12.331560    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14" containerName="kube-rbac-proxy-web"
Sep 11 13:25:12.331671 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:12.331566    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14" containerName="kube-rbac-proxy-metric"
Sep 11 13:25:12.336150 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:12.336114    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 11 13:25:12.339121 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:12.339093    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 11 13:25:12.339256 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:12.339151    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 11 13:25:12.339359 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:12.339340    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 11 13:25:12.339709 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:12.339688    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 11 13:25:12.339877 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:12.339691    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 11 13:25:12.339877 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:12.339818    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 11 13:25:12.340003 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:12.339973    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-m4n9v\""
Sep 11 13:25:12.340341 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:12.340319    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 11 13:25:12.340486 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:12.340357    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 11 13:25:12.347561 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:12.347535    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 11 13:25:12.353196 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:12.353149    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 11 13:25:12.414490 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:12.414434    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2ee7ae7e-c851-4356-899c-b7ad526d2c1f-web-config\") pod \"alertmanager-main-0\" (UID: \"2ee7ae7e-c851-4356-899c-b7ad526d2c1f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 13:25:12.414490 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:12.414487    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2ee7ae7e-c851-4356-899c-b7ad526d2c1f-config-out\") pod \"alertmanager-main-0\" (UID: \"2ee7ae7e-c851-4356-899c-b7ad526d2c1f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 13:25:12.414895 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:12.414564    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2ee7ae7e-c851-4356-899c-b7ad526d2c1f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"2ee7ae7e-c851-4356-899c-b7ad526d2c1f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 13:25:12.414895 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:12.414630    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/2ee7ae7e-c851-4356-899c-b7ad526d2c1f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"2ee7ae7e-c851-4356-899c-b7ad526d2c1f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 13:25:12.414895 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:12.414677    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2ee7ae7e-c851-4356-899c-b7ad526d2c1f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"2ee7ae7e-c851-4356-899c-b7ad526d2c1f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 13:25:12.414895 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:12.414710    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7p4zk\" (UniqueName: \"kubernetes.io/projected/2ee7ae7e-c851-4356-899c-b7ad526d2c1f-kube-api-access-7p4zk\") pod \"alertmanager-main-0\" (UID: \"2ee7ae7e-c851-4356-899c-b7ad526d2c1f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 13:25:12.414895 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:12.414819    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/2ee7ae7e-c851-4356-899c-b7ad526d2c1f-config-volume\") pod \"alertmanager-main-0\" (UID: \"2ee7ae7e-c851-4356-899c-b7ad526d2c1f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 13:25:12.414895 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:12.414887    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2ee7ae7e-c851-4356-899c-b7ad526d2c1f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"2ee7ae7e-c851-4356-899c-b7ad526d2c1f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 13:25:12.415190 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:12.414921    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2ee7ae7e-c851-4356-899c-b7ad526d2c1f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"2ee7ae7e-c851-4356-899c-b7ad526d2c1f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 13:25:12.415190 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:12.414953    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/2ee7ae7e-c851-4356-899c-b7ad526d2c1f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"2ee7ae7e-c851-4356-899c-b7ad526d2c1f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 13:25:12.415190 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:12.414988    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/2ee7ae7e-c851-4356-899c-b7ad526d2c1f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"2ee7ae7e-c851-4356-899c-b7ad526d2c1f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 13:25:12.415190 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:12.415050    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/2ee7ae7e-c851-4356-899c-b7ad526d2c1f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"2ee7ae7e-c851-4356-899c-b7ad526d2c1f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 13:25:12.415190 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:12.415106    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2ee7ae7e-c851-4356-899c-b7ad526d2c1f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"2ee7ae7e-c851-4356-899c-b7ad526d2c1f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 13:25:12.516247 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:12.516114    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2ee7ae7e-c851-4356-899c-b7ad526d2c1f-web-config\") pod \"alertmanager-main-0\" (UID: \"2ee7ae7e-c851-4356-899c-b7ad526d2c1f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 13:25:12.516247 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:12.516159    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2ee7ae7e-c851-4356-899c-b7ad526d2c1f-config-out\") pod \"alertmanager-main-0\" (UID: \"2ee7ae7e-c851-4356-899c-b7ad526d2c1f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 13:25:12.516549 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:12.516295    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2ee7ae7e-c851-4356-899c-b7ad526d2c1f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"2ee7ae7e-c851-4356-899c-b7ad526d2c1f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 13:25:12.516549 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:12.516343    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/2ee7ae7e-c851-4356-899c-b7ad526d2c1f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"2ee7ae7e-c851-4356-899c-b7ad526d2c1f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 13:25:12.516549 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:12.516370    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2ee7ae7e-c851-4356-899c-b7ad526d2c1f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"2ee7ae7e-c851-4356-899c-b7ad526d2c1f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 13:25:12.516549 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:12.516397    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7p4zk\" (UniqueName: \"kubernetes.io/projected/2ee7ae7e-c851-4356-899c-b7ad526d2c1f-kube-api-access-7p4zk\") pod \"alertmanager-main-0\" (UID: \"2ee7ae7e-c851-4356-899c-b7ad526d2c1f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 13:25:12.516549 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:12.516476    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/2ee7ae7e-c851-4356-899c-b7ad526d2c1f-config-volume\") pod \"alertmanager-main-0\" (UID: \"2ee7ae7e-c851-4356-899c-b7ad526d2c1f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 13:25:12.516549 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:12.516524    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2ee7ae7e-c851-4356-899c-b7ad526d2c1f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"2ee7ae7e-c851-4356-899c-b7ad526d2c1f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 13:25:12.516867 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:12.516575    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2ee7ae7e-c851-4356-899c-b7ad526d2c1f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"2ee7ae7e-c851-4356-899c-b7ad526d2c1f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 13:25:12.516867 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:12.516620    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/2ee7ae7e-c851-4356-899c-b7ad526d2c1f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"2ee7ae7e-c851-4356-899c-b7ad526d2c1f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 13:25:12.516867 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:12.516648    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/2ee7ae7e-c851-4356-899c-b7ad526d2c1f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"2ee7ae7e-c851-4356-899c-b7ad526d2c1f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 13:25:12.516867 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:12.516686    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/2ee7ae7e-c851-4356-899c-b7ad526d2c1f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"2ee7ae7e-c851-4356-899c-b7ad526d2c1f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 13:25:12.516867 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:12.516738    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2ee7ae7e-c851-4356-899c-b7ad526d2c1f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"2ee7ae7e-c851-4356-899c-b7ad526d2c1f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 13:25:12.518367 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:12.517496    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2ee7ae7e-c851-4356-899c-b7ad526d2c1f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"2ee7ae7e-c851-4356-899c-b7ad526d2c1f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 13:25:12.518367 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:12.517801    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/2ee7ae7e-c851-4356-899c-b7ad526d2c1f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"2ee7ae7e-c851-4356-899c-b7ad526d2c1f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 13:25:12.518367 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:12.518259    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2ee7ae7e-c851-4356-899c-b7ad526d2c1f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"2ee7ae7e-c851-4356-899c-b7ad526d2c1f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 13:25:12.520023 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:12.519991    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2ee7ae7e-c851-4356-899c-b7ad526d2c1f-config-out\") pod \"alertmanager-main-0\" (UID: \"2ee7ae7e-c851-4356-899c-b7ad526d2c1f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 13:25:12.520147 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:12.520069    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2ee7ae7e-c851-4356-899c-b7ad526d2c1f-web-config\") pod \"alertmanager-main-0\" (UID: \"2ee7ae7e-c851-4356-899c-b7ad526d2c1f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 13:25:12.520343 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:12.520315    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/2ee7ae7e-c851-4356-899c-b7ad526d2c1f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"2ee7ae7e-c851-4356-899c-b7ad526d2c1f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 13:25:12.520525 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:12.520502    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2ee7ae7e-c851-4356-899c-b7ad526d2c1f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"2ee7ae7e-c851-4356-899c-b7ad526d2c1f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 13:25:12.520707 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:12.520682    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2ee7ae7e-c851-4356-899c-b7ad526d2c1f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"2ee7ae7e-c851-4356-899c-b7ad526d2c1f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 13:25:12.520785 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:12.520741    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/2ee7ae7e-c851-4356-899c-b7ad526d2c1f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"2ee7ae7e-c851-4356-899c-b7ad526d2c1f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 13:25:12.521138 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:12.521116    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2ee7ae7e-c851-4356-899c-b7ad526d2c1f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"2ee7ae7e-c851-4356-899c-b7ad526d2c1f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 13:25:12.521629 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:12.521608    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/2ee7ae7e-c851-4356-899c-b7ad526d2c1f-config-volume\") pod \"alertmanager-main-0\" (UID: \"2ee7ae7e-c851-4356-899c-b7ad526d2c1f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 13:25:12.522341 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:12.522318    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/2ee7ae7e-c851-4356-899c-b7ad526d2c1f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"2ee7ae7e-c851-4356-899c-b7ad526d2c1f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 13:25:12.525795 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:12.525762    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7p4zk\" (UniqueName: \"kubernetes.io/projected/2ee7ae7e-c851-4356-899c-b7ad526d2c1f-kube-api-access-7p4zk\") pod \"alertmanager-main-0\" (UID: \"2ee7ae7e-c851-4356-899c-b7ad526d2c1f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 13:25:12.647614 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:12.647564    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 11 13:25:12.952777 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:12.952742    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 11 13:25:12.955491 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:25:12.955465    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ee7ae7e_c851_4356_899c_b7ad526d2c1f.slice/crio-99c37848fa80038e5e4872380b16a1bff8606fa28c0a50dce46d9ecf20c368ef WatchSource:0}: Error finding container 99c37848fa80038e5e4872380b16a1bff8606fa28c0a50dce46d9ecf20c368ef: Status 404 returned error can't find the container with id 99c37848fa80038e5e4872380b16a1bff8606fa28c0a50dce46d9ecf20c368ef
Sep 11 13:25:13.111009 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:13.110967    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14" path="/var/lib/kubelet/pods/2ba2e3c9-64fc-4aa2-8f43-d21a8dba2c14/volumes"
Sep 11 13:25:13.262053 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:13.262002    2578 generic.go:356] "Generic (PLEG): container finished" podID="2ee7ae7e-c851-4356-899c-b7ad526d2c1f" containerID="40a924476bc2b2b7395a0259e5ae8bba56ae4d9efabb03ed762f3fae54bf2a6b" exitCode=0
Sep 11 13:25:13.262327 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:13.262092    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2ee7ae7e-c851-4356-899c-b7ad526d2c1f","Type":"ContainerDied","Data":"40a924476bc2b2b7395a0259e5ae8bba56ae4d9efabb03ed762f3fae54bf2a6b"}
Sep 11 13:25:13.262327 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:13.262172    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2ee7ae7e-c851-4356-899c-b7ad526d2c1f","Type":"ContainerStarted","Data":"99c37848fa80038e5e4872380b16a1bff8606fa28c0a50dce46d9ecf20c368ef"}
Sep 11 13:25:14.274002 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:14.273957    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2ee7ae7e-c851-4356-899c-b7ad526d2c1f","Type":"ContainerStarted","Data":"f535bdf036d6a51d2a0a0fde4c3dcc4691c2b773a4e10082d87dc3478d0885f2"}
Sep 11 13:25:14.274460 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:14.274030    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2ee7ae7e-c851-4356-899c-b7ad526d2c1f","Type":"ContainerStarted","Data":"a49318d82a89118dd8b82c4a473763d32973419305d2ad2b8bb6e787d380749f"}
Sep 11 13:25:14.274460 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:14.274059    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2ee7ae7e-c851-4356-899c-b7ad526d2c1f","Type":"ContainerStarted","Data":"77cbacce57757933ff7480d023db2245bd1b5072794c8fd65611b4a1b8623d0a"}
Sep 11 13:25:14.274460 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:14.274085    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2ee7ae7e-c851-4356-899c-b7ad526d2c1f","Type":"ContainerStarted","Data":"63ddf8bc22255d77edaa4713cee353fe8e87ac06148de2a2f7e2d61bc1ebfb1f"}
Sep 11 13:25:14.274460 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:14.274112    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2ee7ae7e-c851-4356-899c-b7ad526d2c1f","Type":"ContainerStarted","Data":"e44c15a33ecf92fc15c5be31079858b91aa46b2b1747b6d991f1b3f0db05d082"}
Sep 11 13:25:14.274460 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:14.274137    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2ee7ae7e-c851-4356-899c-b7ad526d2c1f","Type":"ContainerStarted","Data":"4849a4871ed31bf6e8a54a17f3864980dc576e1efa19a0f9c7a8ca2e0fbea887"}
Sep 11 13:25:14.302117 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:14.302024    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.284614189 podStartE2EDuration="2.301992058s" totalImagesPullingTime="0s" totalInitContainerRuntime="17.377869ms" isStatefulPod=false podCreationTimestamp="2026-09-11 13:25:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:25:14.300171562 +0000 UTC m=+221.686274749" watchObservedRunningTime="2026-09-11 13:25:14.301992058 +0000 UTC m=+221.688095221"
Sep 11 13:25:14.814172 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:14.814121    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 11 13:25:14.815052 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:14.814788    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="9d48ed82-7fb0-40aa-93de-332e58a6f316" containerName="prometheus" containerID="cri-o://2e2270017b25701463c432676a33ce4b9b3fe4e8ffd66c8a97ae6391812c463e" gracePeriod=600
Sep 11 13:25:14.815052 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:14.814860    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="9d48ed82-7fb0-40aa-93de-332e58a6f316" containerName="config-reloader" containerID="cri-o://a25bd3976df0831c59a095681bd064c2f412e2de45db616678e232de0fd8ae50" gracePeriod=600
Sep 11 13:25:14.815052 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:14.814872    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="9d48ed82-7fb0-40aa-93de-332e58a6f316" containerName="thanos-sidecar" containerID="cri-o://6ede8f2c12fd7b67e056a1df15a43707e94db17818ecc2e23763fb8e9be47b5d" gracePeriod=600
Sep 11 13:25:14.815052 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:14.814902    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="9d48ed82-7fb0-40aa-93de-332e58a6f316" containerName="kube-rbac-proxy-web" containerID="cri-o://95f2d5ecfced25cf854ca5f30fc92e9a1b395531eaf8fce98cd6e282287addda" gracePeriod=600
Sep 11 13:25:14.815052 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:14.814805    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="9d48ed82-7fb0-40aa-93de-332e58a6f316" containerName="kube-rbac-proxy" containerID="cri-o://9345ca19b3e6053d8dd6509541647281234bec2ad08f9bde59e68105bd8c8de9" gracePeriod=600
Sep 11 13:25:14.815052 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:14.814877    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="9d48ed82-7fb0-40aa-93de-332e58a6f316" containerName="kube-rbac-proxy-thanos" containerID="cri-o://f1eff727d760318e7e6e70e4d4f33f72c8dbf595b92309bbb4c676b20693e840" gracePeriod=600
Sep 11 13:25:15.075599 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.075561    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:25:15.147642 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.147607    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9d48ed82-7fb0-40aa-93de-332e58a6f316-configmap-metrics-client-ca\") pod \"9d48ed82-7fb0-40aa-93de-332e58a6f316\" (UID: \"9d48ed82-7fb0-40aa-93de-332e58a6f316\") "
Sep 11 13:25:15.147642 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.147646    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9d48ed82-7fb0-40aa-93de-332e58a6f316-web-config\") pod \"9d48ed82-7fb0-40aa-93de-332e58a6f316\" (UID: \"9d48ed82-7fb0-40aa-93de-332e58a6f316\") "
Sep 11 13:25:15.147954 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.147690    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/9d48ed82-7fb0-40aa-93de-332e58a6f316-prometheus-k8s-db\") pod \"9d48ed82-7fb0-40aa-93de-332e58a6f316\" (UID: \"9d48ed82-7fb0-40aa-93de-332e58a6f316\") "
Sep 11 13:25:15.147954 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.147720    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dvzdf\" (UniqueName: \"kubernetes.io/projected/9d48ed82-7fb0-40aa-93de-332e58a6f316-kube-api-access-dvzdf\") pod \"9d48ed82-7fb0-40aa-93de-332e58a6f316\" (UID: \"9d48ed82-7fb0-40aa-93de-332e58a6f316\") "
Sep 11 13:25:15.147954 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.147761    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9d48ed82-7fb0-40aa-93de-332e58a6f316-prometheus-k8s-rulefiles-0\") pod \"9d48ed82-7fb0-40aa-93de-332e58a6f316\" (UID: \"9d48ed82-7fb0-40aa-93de-332e58a6f316\") "
Sep 11 13:25:15.147954 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.147796    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/9d48ed82-7fb0-40aa-93de-332e58a6f316-prometheus-k8s-rulefiles-1\") pod \"9d48ed82-7fb0-40aa-93de-332e58a6f316\" (UID: \"9d48ed82-7fb0-40aa-93de-332e58a6f316\") "
Sep 11 13:25:15.147954 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.147821    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/9d48ed82-7fb0-40aa-93de-332e58a6f316-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"9d48ed82-7fb0-40aa-93de-332e58a6f316\" (UID: \"9d48ed82-7fb0-40aa-93de-332e58a6f316\") "
Sep 11 13:25:15.147954 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.147878    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/9d48ed82-7fb0-40aa-93de-332e58a6f316-config\") pod \"9d48ed82-7fb0-40aa-93de-332e58a6f316\" (UID: \"9d48ed82-7fb0-40aa-93de-332e58a6f316\") "
Sep 11 13:25:15.147954 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.147911    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/9d48ed82-7fb0-40aa-93de-332e58a6f316-prometheus-k8s-rulefiles-2\") pod \"9d48ed82-7fb0-40aa-93de-332e58a6f316\" (UID: \"9d48ed82-7fb0-40aa-93de-332e58a6f316\") "
Sep 11 13:25:15.147954 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.147935    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9d48ed82-7fb0-40aa-93de-332e58a6f316-secret-kube-rbac-proxy\") pod \"9d48ed82-7fb0-40aa-93de-332e58a6f316\" (UID: \"9d48ed82-7fb0-40aa-93de-332e58a6f316\") "
Sep 11 13:25:15.148336 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.147970    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d48ed82-7fb0-40aa-93de-332e58a6f316-configmap-kubelet-serving-ca-bundle\") pod \"9d48ed82-7fb0-40aa-93de-332e58a6f316\" (UID: \"9d48ed82-7fb0-40aa-93de-332e58a6f316\") "
Sep 11 13:25:15.148336 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.148011    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/9d48ed82-7fb0-40aa-93de-332e58a6f316-secret-metrics-client-certs\") pod \"9d48ed82-7fb0-40aa-93de-332e58a6f316\" (UID: \"9d48ed82-7fb0-40aa-93de-332e58a6f316\") "
Sep 11 13:25:15.148336 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.148032    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/9d48ed82-7fb0-40aa-93de-332e58a6f316-secret-prometheus-k8s-tls\") pod \"9d48ed82-7fb0-40aa-93de-332e58a6f316\" (UID: \"9d48ed82-7fb0-40aa-93de-332e58a6f316\") "
Sep 11 13:25:15.148336 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.148056    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d48ed82-7fb0-40aa-93de-332e58a6f316-prometheus-trusted-ca-bundle\") pod \"9d48ed82-7fb0-40aa-93de-332e58a6f316\" (UID: \"9d48ed82-7fb0-40aa-93de-332e58a6f316\") "
Sep 11 13:25:15.148336 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.148090    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d48ed82-7fb0-40aa-93de-332e58a6f316-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "9d48ed82-7fb0-40aa-93de-332e58a6f316" (UID: "9d48ed82-7fb0-40aa-93de-332e58a6f316"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 13:25:15.148336 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.148106    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9d48ed82-7fb0-40aa-93de-332e58a6f316-tls-assets\") pod \"9d48ed82-7fb0-40aa-93de-332e58a6f316\" (UID: \"9d48ed82-7fb0-40aa-93de-332e58a6f316\") "
Sep 11 13:25:15.148336 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.148219    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d48ed82-7fb0-40aa-93de-332e58a6f316-configmap-serving-certs-ca-bundle\") pod \"9d48ed82-7fb0-40aa-93de-332e58a6f316\" (UID: \"9d48ed82-7fb0-40aa-93de-332e58a6f316\") "
Sep 11 13:25:15.148336 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.148274    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9d48ed82-7fb0-40aa-93de-332e58a6f316-config-out\") pod \"9d48ed82-7fb0-40aa-93de-332e58a6f316\" (UID: \"9d48ed82-7fb0-40aa-93de-332e58a6f316\") "
Sep 11 13:25:15.148336 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.148314    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9d48ed82-7fb0-40aa-93de-332e58a6f316-secret-grpc-tls\") pod \"9d48ed82-7fb0-40aa-93de-332e58a6f316\" (UID: \"9d48ed82-7fb0-40aa-93de-332e58a6f316\") "
Sep 11 13:25:15.148742 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.148343    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9d48ed82-7fb0-40aa-93de-332e58a6f316-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"9d48ed82-7fb0-40aa-93de-332e58a6f316\" (UID: \"9d48ed82-7fb0-40aa-93de-332e58a6f316\") "
Sep 11 13:25:15.148742 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.148368    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9d48ed82-7fb0-40aa-93de-332e58a6f316-thanos-prometheus-http-client-file\") pod \"9d48ed82-7fb0-40aa-93de-332e58a6f316\" (UID: \"9d48ed82-7fb0-40aa-93de-332e58a6f316\") "
Sep 11 13:25:15.148742 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.148624    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d48ed82-7fb0-40aa-93de-332e58a6f316-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "9d48ed82-7fb0-40aa-93de-332e58a6f316" (UID: "9d48ed82-7fb0-40aa-93de-332e58a6f316"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 13:25:15.148742 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.148707    2578 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/9d48ed82-7fb0-40aa-93de-332e58a6f316-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:25:15.148742 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.148726    2578 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9d48ed82-7fb0-40aa-93de-332e58a6f316-configmap-metrics-client-ca\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:25:15.149006 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.148796    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d48ed82-7fb0-40aa-93de-332e58a6f316-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "9d48ed82-7fb0-40aa-93de-332e58a6f316" (UID: "9d48ed82-7fb0-40aa-93de-332e58a6f316"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 13:25:15.150989 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.150228    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d48ed82-7fb0-40aa-93de-332e58a6f316-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "9d48ed82-7fb0-40aa-93de-332e58a6f316" (UID: "9d48ed82-7fb0-40aa-93de-332e58a6f316"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 13:25:15.150989 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.150255    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d48ed82-7fb0-40aa-93de-332e58a6f316-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "9d48ed82-7fb0-40aa-93de-332e58a6f316" (UID: "9d48ed82-7fb0-40aa-93de-332e58a6f316"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 13:25:15.150989 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.150647    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d48ed82-7fb0-40aa-93de-332e58a6f316-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "9d48ed82-7fb0-40aa-93de-332e58a6f316" (UID: "9d48ed82-7fb0-40aa-93de-332e58a6f316"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 13:25:15.151280 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.151051    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d48ed82-7fb0-40aa-93de-332e58a6f316-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "9d48ed82-7fb0-40aa-93de-332e58a6f316" (UID: "9d48ed82-7fb0-40aa-93de-332e58a6f316"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 13:25:15.151280 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.151133    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d48ed82-7fb0-40aa-93de-332e58a6f316-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "9d48ed82-7fb0-40aa-93de-332e58a6f316" (UID: "9d48ed82-7fb0-40aa-93de-332e58a6f316"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 13:25:15.151820 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.151777    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d48ed82-7fb0-40aa-93de-332e58a6f316-kube-api-access-dvzdf" (OuterVolumeSpecName: "kube-api-access-dvzdf") pod "9d48ed82-7fb0-40aa-93de-332e58a6f316" (UID: "9d48ed82-7fb0-40aa-93de-332e58a6f316"). InnerVolumeSpecName "kube-api-access-dvzdf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 13:25:15.152103 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.152082    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d48ed82-7fb0-40aa-93de-332e58a6f316-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "9d48ed82-7fb0-40aa-93de-332e58a6f316" (UID: "9d48ed82-7fb0-40aa-93de-332e58a6f316"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 13:25:15.154056 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.153785    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d48ed82-7fb0-40aa-93de-332e58a6f316-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "9d48ed82-7fb0-40aa-93de-332e58a6f316" (UID: "9d48ed82-7fb0-40aa-93de-332e58a6f316"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 13:25:15.154724 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.154680    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d48ed82-7fb0-40aa-93de-332e58a6f316-config-out" (OuterVolumeSpecName: "config-out") pod "9d48ed82-7fb0-40aa-93de-332e58a6f316" (UID: "9d48ed82-7fb0-40aa-93de-332e58a6f316"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 13:25:15.155085 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.155042    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d48ed82-7fb0-40aa-93de-332e58a6f316-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "9d48ed82-7fb0-40aa-93de-332e58a6f316" (UID: "9d48ed82-7fb0-40aa-93de-332e58a6f316"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 13:25:15.155304 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.155267    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d48ed82-7fb0-40aa-93de-332e58a6f316-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "9d48ed82-7fb0-40aa-93de-332e58a6f316" (UID: "9d48ed82-7fb0-40aa-93de-332e58a6f316"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 13:25:15.155498 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.155391    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d48ed82-7fb0-40aa-93de-332e58a6f316-config" (OuterVolumeSpecName: "config") pod "9d48ed82-7fb0-40aa-93de-332e58a6f316" (UID: "9d48ed82-7fb0-40aa-93de-332e58a6f316"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 13:25:15.155498 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.155417    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d48ed82-7fb0-40aa-93de-332e58a6f316-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "9d48ed82-7fb0-40aa-93de-332e58a6f316" (UID: "9d48ed82-7fb0-40aa-93de-332e58a6f316"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 13:25:15.155498 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.155449    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d48ed82-7fb0-40aa-93de-332e58a6f316-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "9d48ed82-7fb0-40aa-93de-332e58a6f316" (UID: "9d48ed82-7fb0-40aa-93de-332e58a6f316"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 13:25:15.155498 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.155465    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d48ed82-7fb0-40aa-93de-332e58a6f316-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "9d48ed82-7fb0-40aa-93de-332e58a6f316" (UID: "9d48ed82-7fb0-40aa-93de-332e58a6f316"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 13:25:15.158191 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.158156    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d48ed82-7fb0-40aa-93de-332e58a6f316-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "9d48ed82-7fb0-40aa-93de-332e58a6f316" (UID: "9d48ed82-7fb0-40aa-93de-332e58a6f316"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 13:25:15.171059 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.171006    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d48ed82-7fb0-40aa-93de-332e58a6f316-web-config" (OuterVolumeSpecName: "web-config") pod "9d48ed82-7fb0-40aa-93de-332e58a6f316" (UID: "9d48ed82-7fb0-40aa-93de-332e58a6f316"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 13:25:15.250023 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.249967    2578 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d48ed82-7fb0-40aa-93de-332e58a6f316-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:25:15.250023 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.250018    2578 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9d48ed82-7fb0-40aa-93de-332e58a6f316-config-out\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:25:15.250023 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.250031    2578 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9d48ed82-7fb0-40aa-93de-332e58a6f316-secret-grpc-tls\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:25:15.250391 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.250041    2578 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9d48ed82-7fb0-40aa-93de-332e58a6f316-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:25:15.250391 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.250053    2578 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9d48ed82-7fb0-40aa-93de-332e58a6f316-thanos-prometheus-http-client-file\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:25:15.250391 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.250100    2578 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9d48ed82-7fb0-40aa-93de-332e58a6f316-web-config\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:25:15.250391 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.250114    2578 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/9d48ed82-7fb0-40aa-93de-332e58a6f316-prometheus-k8s-db\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:25:15.250391 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.250128    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvzdf\" (UniqueName: \"kubernetes.io/projected/9d48ed82-7fb0-40aa-93de-332e58a6f316-kube-api-access-dvzdf\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:25:15.250391 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.250141    2578 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9d48ed82-7fb0-40aa-93de-332e58a6f316-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:25:15.250391 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.250154    2578 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/9d48ed82-7fb0-40aa-93de-332e58a6f316-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:25:15.250391 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.250171    2578 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/9d48ed82-7fb0-40aa-93de-332e58a6f316-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:25:15.250391 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.250184    2578 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/9d48ed82-7fb0-40aa-93de-332e58a6f316-config\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:25:15.250391 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.250195    2578 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9d48ed82-7fb0-40aa-93de-332e58a6f316-secret-kube-rbac-proxy\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:25:15.250391 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.250205    2578 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d48ed82-7fb0-40aa-93de-332e58a6f316-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:25:15.250391 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.250215    2578 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/9d48ed82-7fb0-40aa-93de-332e58a6f316-secret-metrics-client-certs\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:25:15.250391 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.250229    2578 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/9d48ed82-7fb0-40aa-93de-332e58a6f316-secret-prometheus-k8s-tls\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:25:15.250391 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.250243    2578 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d48ed82-7fb0-40aa-93de-332e58a6f316-prometheus-trusted-ca-bundle\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:25:15.250391 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.250258    2578 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9d48ed82-7fb0-40aa-93de-332e58a6f316-tls-assets\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:25:15.283056 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.283011    2578 generic.go:356] "Generic (PLEG): container finished" podID="9d48ed82-7fb0-40aa-93de-332e58a6f316" containerID="f1eff727d760318e7e6e70e4d4f33f72c8dbf595b92309bbb4c676b20693e840" exitCode=0
Sep 11 13:25:15.283056 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.283044    2578 generic.go:356] "Generic (PLEG): container finished" podID="9d48ed82-7fb0-40aa-93de-332e58a6f316" containerID="9345ca19b3e6053d8dd6509541647281234bec2ad08f9bde59e68105bd8c8de9" exitCode=0
Sep 11 13:25:15.283056 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.283052    2578 generic.go:356] "Generic (PLEG): container finished" podID="9d48ed82-7fb0-40aa-93de-332e58a6f316" containerID="95f2d5ecfced25cf854ca5f30fc92e9a1b395531eaf8fce98cd6e282287addda" exitCode=0
Sep 11 13:25:15.283056 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.283062    2578 generic.go:356] "Generic (PLEG): container finished" podID="9d48ed82-7fb0-40aa-93de-332e58a6f316" containerID="6ede8f2c12fd7b67e056a1df15a43707e94db17818ecc2e23763fb8e9be47b5d" exitCode=0
Sep 11 13:25:15.283056 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.283068    2578 generic.go:356] "Generic (PLEG): container finished" podID="9d48ed82-7fb0-40aa-93de-332e58a6f316" containerID="a25bd3976df0831c59a095681bd064c2f412e2de45db616678e232de0fd8ae50" exitCode=0
Sep 11 13:25:15.283718 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.283074    2578 generic.go:356] "Generic (PLEG): container finished" podID="9d48ed82-7fb0-40aa-93de-332e58a6f316" containerID="2e2270017b25701463c432676a33ce4b9b3fe4e8ffd66c8a97ae6391812c463e" exitCode=0
Sep 11 13:25:15.283718 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.283153    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9d48ed82-7fb0-40aa-93de-332e58a6f316","Type":"ContainerDied","Data":"f1eff727d760318e7e6e70e4d4f33f72c8dbf595b92309bbb4c676b20693e840"}
Sep 11 13:25:15.283718 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.283209    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:25:15.283718 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.283272    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9d48ed82-7fb0-40aa-93de-332e58a6f316","Type":"ContainerDied","Data":"9345ca19b3e6053d8dd6509541647281234bec2ad08f9bde59e68105bd8c8de9"}
Sep 11 13:25:15.283718 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.283317    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9d48ed82-7fb0-40aa-93de-332e58a6f316","Type":"ContainerDied","Data":"95f2d5ecfced25cf854ca5f30fc92e9a1b395531eaf8fce98cd6e282287addda"}
Sep 11 13:25:15.283718 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.283356    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9d48ed82-7fb0-40aa-93de-332e58a6f316","Type":"ContainerDied","Data":"6ede8f2c12fd7b67e056a1df15a43707e94db17818ecc2e23763fb8e9be47b5d"}
Sep 11 13:25:15.283718 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.283340    2578 scope.go:117] "RemoveContainer" containerID="f1eff727d760318e7e6e70e4d4f33f72c8dbf595b92309bbb4c676b20693e840"
Sep 11 13:25:15.283718 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.283399    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9d48ed82-7fb0-40aa-93de-332e58a6f316","Type":"ContainerDied","Data":"a25bd3976df0831c59a095681bd064c2f412e2de45db616678e232de0fd8ae50"}
Sep 11 13:25:15.283718 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.283446    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9d48ed82-7fb0-40aa-93de-332e58a6f316","Type":"ContainerDied","Data":"2e2270017b25701463c432676a33ce4b9b3fe4e8ffd66c8a97ae6391812c463e"}
Sep 11 13:25:15.283718 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.283490    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9d48ed82-7fb0-40aa-93de-332e58a6f316","Type":"ContainerDied","Data":"77280823e2a42c65995982d15a83f2bd3125736c8c3ba3968dc74a839a1da94d"}
Sep 11 13:25:15.293753 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.293728    2578 scope.go:117] "RemoveContainer" containerID="9345ca19b3e6053d8dd6509541647281234bec2ad08f9bde59e68105bd8c8de9"
Sep 11 13:25:15.303270 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.303245    2578 scope.go:117] "RemoveContainer" containerID="95f2d5ecfced25cf854ca5f30fc92e9a1b395531eaf8fce98cd6e282287addda"
Sep 11 13:25:15.310088 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.310055    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 11 13:25:15.312585 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.312564    2578 scope.go:117] "RemoveContainer" containerID="6ede8f2c12fd7b67e056a1df15a43707e94db17818ecc2e23763fb8e9be47b5d"
Sep 11 13:25:15.318133 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.318071    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 11 13:25:15.321967 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.321943    2578 scope.go:117] "RemoveContainer" containerID="a25bd3976df0831c59a095681bd064c2f412e2de45db616678e232de0fd8ae50"
Sep 11 13:25:15.330888 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.330857    2578 scope.go:117] "RemoveContainer" containerID="2e2270017b25701463c432676a33ce4b9b3fe4e8ffd66c8a97ae6391812c463e"
Sep 11 13:25:15.340173 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.340146    2578 scope.go:117] "RemoveContainer" containerID="fa724f0373acd6ed1c40dbccc8cdc9c5cd019c03181442172953c11045820322"
Sep 11 13:25:15.349136 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.349101    2578 scope.go:117] "RemoveContainer" containerID="f1eff727d760318e7e6e70e4d4f33f72c8dbf595b92309bbb4c676b20693e840"
Sep 11 13:25:15.349561 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:25:15.349531    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1eff727d760318e7e6e70e4d4f33f72c8dbf595b92309bbb4c676b20693e840\": container with ID starting with f1eff727d760318e7e6e70e4d4f33f72c8dbf595b92309bbb4c676b20693e840 not found: ID does not exist" containerID="f1eff727d760318e7e6e70e4d4f33f72c8dbf595b92309bbb4c676b20693e840"
Sep 11 13:25:15.349697 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.349580    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1eff727d760318e7e6e70e4d4f33f72c8dbf595b92309bbb4c676b20693e840"} err="failed to get container status \"f1eff727d760318e7e6e70e4d4f33f72c8dbf595b92309bbb4c676b20693e840\": rpc error: code = NotFound desc = could not find container \"f1eff727d760318e7e6e70e4d4f33f72c8dbf595b92309bbb4c676b20693e840\": container with ID starting with f1eff727d760318e7e6e70e4d4f33f72c8dbf595b92309bbb4c676b20693e840 not found: ID does not exist"
Sep 11 13:25:15.349697 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.349624    2578 scope.go:117] "RemoveContainer" containerID="9345ca19b3e6053d8dd6509541647281234bec2ad08f9bde59e68105bd8c8de9"
Sep 11 13:25:15.350123 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:25:15.350095    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9345ca19b3e6053d8dd6509541647281234bec2ad08f9bde59e68105bd8c8de9\": container with ID starting with 9345ca19b3e6053d8dd6509541647281234bec2ad08f9bde59e68105bd8c8de9 not found: ID does not exist" containerID="9345ca19b3e6053d8dd6509541647281234bec2ad08f9bde59e68105bd8c8de9"
Sep 11 13:25:15.350286 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.350130    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9345ca19b3e6053d8dd6509541647281234bec2ad08f9bde59e68105bd8c8de9"} err="failed to get container status \"9345ca19b3e6053d8dd6509541647281234bec2ad08f9bde59e68105bd8c8de9\": rpc error: code = NotFound desc = could not find container \"9345ca19b3e6053d8dd6509541647281234bec2ad08f9bde59e68105bd8c8de9\": container with ID starting with 9345ca19b3e6053d8dd6509541647281234bec2ad08f9bde59e68105bd8c8de9 not found: ID does not exist"
Sep 11 13:25:15.350286 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.350153    2578 scope.go:117] "RemoveContainer" containerID="95f2d5ecfced25cf854ca5f30fc92e9a1b395531eaf8fce98cd6e282287addda"
Sep 11 13:25:15.350577 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:25:15.350537    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95f2d5ecfced25cf854ca5f30fc92e9a1b395531eaf8fce98cd6e282287addda\": container with ID starting with 95f2d5ecfced25cf854ca5f30fc92e9a1b395531eaf8fce98cd6e282287addda not found: ID does not exist" containerID="95f2d5ecfced25cf854ca5f30fc92e9a1b395531eaf8fce98cd6e282287addda"
Sep 11 13:25:15.350626 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.350582    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95f2d5ecfced25cf854ca5f30fc92e9a1b395531eaf8fce98cd6e282287addda"} err="failed to get container status \"95f2d5ecfced25cf854ca5f30fc92e9a1b395531eaf8fce98cd6e282287addda\": rpc error: code = NotFound desc = could not find container \"95f2d5ecfced25cf854ca5f30fc92e9a1b395531eaf8fce98cd6e282287addda\": container with ID starting with 95f2d5ecfced25cf854ca5f30fc92e9a1b395531eaf8fce98cd6e282287addda not found: ID does not exist"
Sep 11 13:25:15.350626 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.350598    2578 scope.go:117] "RemoveContainer" containerID="6ede8f2c12fd7b67e056a1df15a43707e94db17818ecc2e23763fb8e9be47b5d"
Sep 11 13:25:15.350943 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:25:15.350922    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ede8f2c12fd7b67e056a1df15a43707e94db17818ecc2e23763fb8e9be47b5d\": container with ID starting with 6ede8f2c12fd7b67e056a1df15a43707e94db17818ecc2e23763fb8e9be47b5d not found: ID does not exist" containerID="6ede8f2c12fd7b67e056a1df15a43707e94db17818ecc2e23763fb8e9be47b5d"
Sep 11 13:25:15.351042 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.350945    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ede8f2c12fd7b67e056a1df15a43707e94db17818ecc2e23763fb8e9be47b5d"} err="failed to get container status \"6ede8f2c12fd7b67e056a1df15a43707e94db17818ecc2e23763fb8e9be47b5d\": rpc error: code = NotFound desc = could not find container \"6ede8f2c12fd7b67e056a1df15a43707e94db17818ecc2e23763fb8e9be47b5d\": container with ID starting with 6ede8f2c12fd7b67e056a1df15a43707e94db17818ecc2e23763fb8e9be47b5d not found: ID does not exist"
Sep 11 13:25:15.351042 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.350964    2578 scope.go:117] "RemoveContainer" containerID="a25bd3976df0831c59a095681bd064c2f412e2de45db616678e232de0fd8ae50"
Sep 11 13:25:15.351509 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:25:15.351483    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a25bd3976df0831c59a095681bd064c2f412e2de45db616678e232de0fd8ae50\": container with ID starting with a25bd3976df0831c59a095681bd064c2f412e2de45db616678e232de0fd8ae50 not found: ID does not exist" containerID="a25bd3976df0831c59a095681bd064c2f412e2de45db616678e232de0fd8ae50"
Sep 11 13:25:15.351577 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.351517    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a25bd3976df0831c59a095681bd064c2f412e2de45db616678e232de0fd8ae50"} err="failed to get container status \"a25bd3976df0831c59a095681bd064c2f412e2de45db616678e232de0fd8ae50\": rpc error: code = NotFound desc = could not find container \"a25bd3976df0831c59a095681bd064c2f412e2de45db616678e232de0fd8ae50\": container with ID starting with a25bd3976df0831c59a095681bd064c2f412e2de45db616678e232de0fd8ae50 not found: ID does not exist"
Sep 11 13:25:15.351577 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.351533    2578 scope.go:117] "RemoveContainer" containerID="2e2270017b25701463c432676a33ce4b9b3fe4e8ffd66c8a97ae6391812c463e"
Sep 11 13:25:15.351876 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:25:15.351857    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e2270017b25701463c432676a33ce4b9b3fe4e8ffd66c8a97ae6391812c463e\": container with ID starting with 2e2270017b25701463c432676a33ce4b9b3fe4e8ffd66c8a97ae6391812c463e not found: ID does not exist" containerID="2e2270017b25701463c432676a33ce4b9b3fe4e8ffd66c8a97ae6391812c463e"
Sep 11 13:25:15.351960 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.351884    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e2270017b25701463c432676a33ce4b9b3fe4e8ffd66c8a97ae6391812c463e"} err="failed to get container status \"2e2270017b25701463c432676a33ce4b9b3fe4e8ffd66c8a97ae6391812c463e\": rpc error: code = NotFound desc = could not find container \"2e2270017b25701463c432676a33ce4b9b3fe4e8ffd66c8a97ae6391812c463e\": container with ID starting with 2e2270017b25701463c432676a33ce4b9b3fe4e8ffd66c8a97ae6391812c463e not found: ID does not exist"
Sep 11 13:25:15.351960 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.351906    2578 scope.go:117] "RemoveContainer" containerID="fa724f0373acd6ed1c40dbccc8cdc9c5cd019c03181442172953c11045820322"
Sep 11 13:25:15.352220 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:25:15.352199    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa724f0373acd6ed1c40dbccc8cdc9c5cd019c03181442172953c11045820322\": container with ID starting with fa724f0373acd6ed1c40dbccc8cdc9c5cd019c03181442172953c11045820322 not found: ID does not exist" containerID="fa724f0373acd6ed1c40dbccc8cdc9c5cd019c03181442172953c11045820322"
Sep 11 13:25:15.352255 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.352226    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa724f0373acd6ed1c40dbccc8cdc9c5cd019c03181442172953c11045820322"} err="failed to get container status \"fa724f0373acd6ed1c40dbccc8cdc9c5cd019c03181442172953c11045820322\": rpc error: code = NotFound desc = could not find container \"fa724f0373acd6ed1c40dbccc8cdc9c5cd019c03181442172953c11045820322\": container with ID starting with fa724f0373acd6ed1c40dbccc8cdc9c5cd019c03181442172953c11045820322 not found: ID does not exist"
Sep 11 13:25:15.352255 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.352241    2578 scope.go:117] "RemoveContainer" containerID="f1eff727d760318e7e6e70e4d4f33f72c8dbf595b92309bbb4c676b20693e840"
Sep 11 13:25:15.352514 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.352493    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1eff727d760318e7e6e70e4d4f33f72c8dbf595b92309bbb4c676b20693e840"} err="failed to get container status \"f1eff727d760318e7e6e70e4d4f33f72c8dbf595b92309bbb4c676b20693e840\": rpc error: code = NotFound desc = could not find container \"f1eff727d760318e7e6e70e4d4f33f72c8dbf595b92309bbb4c676b20693e840\": container with ID starting with f1eff727d760318e7e6e70e4d4f33f72c8dbf595b92309bbb4c676b20693e840 not found: ID does not exist"
Sep 11 13:25:15.352578 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.352517    2578 scope.go:117] "RemoveContainer" containerID="9345ca19b3e6053d8dd6509541647281234bec2ad08f9bde59e68105bd8c8de9"
Sep 11 13:25:15.352797 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.352765    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9345ca19b3e6053d8dd6509541647281234bec2ad08f9bde59e68105bd8c8de9"} err="failed to get container status \"9345ca19b3e6053d8dd6509541647281234bec2ad08f9bde59e68105bd8c8de9\": rpc error: code = NotFound desc = could not find container \"9345ca19b3e6053d8dd6509541647281234bec2ad08f9bde59e68105bd8c8de9\": container with ID starting with 9345ca19b3e6053d8dd6509541647281234bec2ad08f9bde59e68105bd8c8de9 not found: ID does not exist"
Sep 11 13:25:15.352861 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.352799    2578 scope.go:117] "RemoveContainer" containerID="95f2d5ecfced25cf854ca5f30fc92e9a1b395531eaf8fce98cd6e282287addda"
Sep 11 13:25:15.353059 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.353042    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95f2d5ecfced25cf854ca5f30fc92e9a1b395531eaf8fce98cd6e282287addda"} err="failed to get container status \"95f2d5ecfced25cf854ca5f30fc92e9a1b395531eaf8fce98cd6e282287addda\": rpc error: code = NotFound desc = could not find container \"95f2d5ecfced25cf854ca5f30fc92e9a1b395531eaf8fce98cd6e282287addda\": container with ID starting with 95f2d5ecfced25cf854ca5f30fc92e9a1b395531eaf8fce98cd6e282287addda not found: ID does not exist"
Sep 11 13:25:15.353123 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.353059    2578 scope.go:117] "RemoveContainer" containerID="6ede8f2c12fd7b67e056a1df15a43707e94db17818ecc2e23763fb8e9be47b5d"
Sep 11 13:25:15.353326 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.353306    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ede8f2c12fd7b67e056a1df15a43707e94db17818ecc2e23763fb8e9be47b5d"} err="failed to get container status \"6ede8f2c12fd7b67e056a1df15a43707e94db17818ecc2e23763fb8e9be47b5d\": rpc error: code = NotFound desc = could not find container \"6ede8f2c12fd7b67e056a1df15a43707e94db17818ecc2e23763fb8e9be47b5d\": container with ID starting with 6ede8f2c12fd7b67e056a1df15a43707e94db17818ecc2e23763fb8e9be47b5d not found: ID does not exist"
Sep 11 13:25:15.353390 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.353328    2578 scope.go:117] "RemoveContainer" containerID="a25bd3976df0831c59a095681bd064c2f412e2de45db616678e232de0fd8ae50"
Sep 11 13:25:15.353614 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.353595    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a25bd3976df0831c59a095681bd064c2f412e2de45db616678e232de0fd8ae50"} err="failed to get container status \"a25bd3976df0831c59a095681bd064c2f412e2de45db616678e232de0fd8ae50\": rpc error: code = NotFound desc = could not find container \"a25bd3976df0831c59a095681bd064c2f412e2de45db616678e232de0fd8ae50\": container with ID starting with a25bd3976df0831c59a095681bd064c2f412e2de45db616678e232de0fd8ae50 not found: ID does not exist"
Sep 11 13:25:15.353665 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.353615    2578 scope.go:117] "RemoveContainer" containerID="2e2270017b25701463c432676a33ce4b9b3fe4e8ffd66c8a97ae6391812c463e"
Sep 11 13:25:15.353868 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.353849    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e2270017b25701463c432676a33ce4b9b3fe4e8ffd66c8a97ae6391812c463e"} err="failed to get container status \"2e2270017b25701463c432676a33ce4b9b3fe4e8ffd66c8a97ae6391812c463e\": rpc error: code = NotFound desc = could not find container \"2e2270017b25701463c432676a33ce4b9b3fe4e8ffd66c8a97ae6391812c463e\": container with ID starting with 2e2270017b25701463c432676a33ce4b9b3fe4e8ffd66c8a97ae6391812c463e not found: ID does not exist"
Sep 11 13:25:15.353868 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.353868    2578 scope.go:117] "RemoveContainer" containerID="fa724f0373acd6ed1c40dbccc8cdc9c5cd019c03181442172953c11045820322"
Sep 11 13:25:15.354107 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.354087    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa724f0373acd6ed1c40dbccc8cdc9c5cd019c03181442172953c11045820322"} err="failed to get container status \"fa724f0373acd6ed1c40dbccc8cdc9c5cd019c03181442172953c11045820322\": rpc error: code = NotFound desc = could not find container \"fa724f0373acd6ed1c40dbccc8cdc9c5cd019c03181442172953c11045820322\": container with ID starting with fa724f0373acd6ed1c40dbccc8cdc9c5cd019c03181442172953c11045820322 not found: ID does not exist"
Sep 11 13:25:15.354175 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.354110    2578 scope.go:117] "RemoveContainer" containerID="f1eff727d760318e7e6e70e4d4f33f72c8dbf595b92309bbb4c676b20693e840"
Sep 11 13:25:15.354362 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.354334    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1eff727d760318e7e6e70e4d4f33f72c8dbf595b92309bbb4c676b20693e840"} err="failed to get container status \"f1eff727d760318e7e6e70e4d4f33f72c8dbf595b92309bbb4c676b20693e840\": rpc error: code = NotFound desc = could not find container \"f1eff727d760318e7e6e70e4d4f33f72c8dbf595b92309bbb4c676b20693e840\": container with ID starting with f1eff727d760318e7e6e70e4d4f33f72c8dbf595b92309bbb4c676b20693e840 not found: ID does not exist"
Sep 11 13:25:15.354413 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.354363    2578 scope.go:117] "RemoveContainer" containerID="9345ca19b3e6053d8dd6509541647281234bec2ad08f9bde59e68105bd8c8de9"
Sep 11 13:25:15.354587 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.354571    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9345ca19b3e6053d8dd6509541647281234bec2ad08f9bde59e68105bd8c8de9"} err="failed to get container status \"9345ca19b3e6053d8dd6509541647281234bec2ad08f9bde59e68105bd8c8de9\": rpc error: code = NotFound desc = could not find container \"9345ca19b3e6053d8dd6509541647281234bec2ad08f9bde59e68105bd8c8de9\": container with ID starting with 9345ca19b3e6053d8dd6509541647281234bec2ad08f9bde59e68105bd8c8de9 not found: ID does not exist"
Sep 11 13:25:15.354635 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.354587    2578 scope.go:117] "RemoveContainer" containerID="95f2d5ecfced25cf854ca5f30fc92e9a1b395531eaf8fce98cd6e282287addda"
Sep 11 13:25:15.354824 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.354805    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95f2d5ecfced25cf854ca5f30fc92e9a1b395531eaf8fce98cd6e282287addda"} err="failed to get container status \"95f2d5ecfced25cf854ca5f30fc92e9a1b395531eaf8fce98cd6e282287addda\": rpc error: code = NotFound desc = could not find container \"95f2d5ecfced25cf854ca5f30fc92e9a1b395531eaf8fce98cd6e282287addda\": container with ID starting with 95f2d5ecfced25cf854ca5f30fc92e9a1b395531eaf8fce98cd6e282287addda not found: ID does not exist"
Sep 11 13:25:15.354893 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.354825    2578 scope.go:117] "RemoveContainer" containerID="6ede8f2c12fd7b67e056a1df15a43707e94db17818ecc2e23763fb8e9be47b5d"
Sep 11 13:25:15.355116 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.355098    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ede8f2c12fd7b67e056a1df15a43707e94db17818ecc2e23763fb8e9be47b5d"} err="failed to get container status \"6ede8f2c12fd7b67e056a1df15a43707e94db17818ecc2e23763fb8e9be47b5d\": rpc error: code = NotFound desc = could not find container \"6ede8f2c12fd7b67e056a1df15a43707e94db17818ecc2e23763fb8e9be47b5d\": container with ID starting with 6ede8f2c12fd7b67e056a1df15a43707e94db17818ecc2e23763fb8e9be47b5d not found: ID does not exist"
Sep 11 13:25:15.355116 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.355115    2578 scope.go:117] "RemoveContainer" containerID="a25bd3976df0831c59a095681bd064c2f412e2de45db616678e232de0fd8ae50"
Sep 11 13:25:15.355354 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.355334    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a25bd3976df0831c59a095681bd064c2f412e2de45db616678e232de0fd8ae50"} err="failed to get container status \"a25bd3976df0831c59a095681bd064c2f412e2de45db616678e232de0fd8ae50\": rpc error: code = NotFound desc = could not find container \"a25bd3976df0831c59a095681bd064c2f412e2de45db616678e232de0fd8ae50\": container with ID starting with a25bd3976df0831c59a095681bd064c2f412e2de45db616678e232de0fd8ae50 not found: ID does not exist"
Sep 11 13:25:15.355402 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.355355    2578 scope.go:117] "RemoveContainer" containerID="2e2270017b25701463c432676a33ce4b9b3fe4e8ffd66c8a97ae6391812c463e"
Sep 11 13:25:15.355627 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.355608    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e2270017b25701463c432676a33ce4b9b3fe4e8ffd66c8a97ae6391812c463e"} err="failed to get container status \"2e2270017b25701463c432676a33ce4b9b3fe4e8ffd66c8a97ae6391812c463e\": rpc error: code = NotFound desc = could not find container \"2e2270017b25701463c432676a33ce4b9b3fe4e8ffd66c8a97ae6391812c463e\": container with ID starting with 2e2270017b25701463c432676a33ce4b9b3fe4e8ffd66c8a97ae6391812c463e not found: ID does not exist"
Sep 11 13:25:15.355699 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.355628    2578 scope.go:117] "RemoveContainer" containerID="fa724f0373acd6ed1c40dbccc8cdc9c5cd019c03181442172953c11045820322"
Sep 11 13:25:15.355920 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.355903    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa724f0373acd6ed1c40dbccc8cdc9c5cd019c03181442172953c11045820322"} err="failed to get container status \"fa724f0373acd6ed1c40dbccc8cdc9c5cd019c03181442172953c11045820322\": rpc error: code = NotFound desc = could not find container \"fa724f0373acd6ed1c40dbccc8cdc9c5cd019c03181442172953c11045820322\": container with ID starting with fa724f0373acd6ed1c40dbccc8cdc9c5cd019c03181442172953c11045820322 not found: ID does not exist"
Sep 11 13:25:15.355979 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.355923    2578 scope.go:117] "RemoveContainer" containerID="f1eff727d760318e7e6e70e4d4f33f72c8dbf595b92309bbb4c676b20693e840"
Sep 11 13:25:15.356181 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.356163    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1eff727d760318e7e6e70e4d4f33f72c8dbf595b92309bbb4c676b20693e840"} err="failed to get container status \"f1eff727d760318e7e6e70e4d4f33f72c8dbf595b92309bbb4c676b20693e840\": rpc error: code = NotFound desc = could not find container \"f1eff727d760318e7e6e70e4d4f33f72c8dbf595b92309bbb4c676b20693e840\": container with ID starting with f1eff727d760318e7e6e70e4d4f33f72c8dbf595b92309bbb4c676b20693e840 not found: ID does not exist"
Sep 11 13:25:15.356250 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.356183    2578 scope.go:117] "RemoveContainer" containerID="9345ca19b3e6053d8dd6509541647281234bec2ad08f9bde59e68105bd8c8de9"
Sep 11 13:25:15.356432 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.356412    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9345ca19b3e6053d8dd6509541647281234bec2ad08f9bde59e68105bd8c8de9"} err="failed to get container status \"9345ca19b3e6053d8dd6509541647281234bec2ad08f9bde59e68105bd8c8de9\": rpc error: code = NotFound desc = could not find container \"9345ca19b3e6053d8dd6509541647281234bec2ad08f9bde59e68105bd8c8de9\": container with ID starting with 9345ca19b3e6053d8dd6509541647281234bec2ad08f9bde59e68105bd8c8de9 not found: ID does not exist"
Sep 11 13:25:15.356475 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.356432    2578 scope.go:117] "RemoveContainer" containerID="95f2d5ecfced25cf854ca5f30fc92e9a1b395531eaf8fce98cd6e282287addda"
Sep 11 13:25:15.356705 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.356683    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95f2d5ecfced25cf854ca5f30fc92e9a1b395531eaf8fce98cd6e282287addda"} err="failed to get container status \"95f2d5ecfced25cf854ca5f30fc92e9a1b395531eaf8fce98cd6e282287addda\": rpc error: code = NotFound desc = could not find container \"95f2d5ecfced25cf854ca5f30fc92e9a1b395531eaf8fce98cd6e282287addda\": container with ID starting with 95f2d5ecfced25cf854ca5f30fc92e9a1b395531eaf8fce98cd6e282287addda not found: ID does not exist"
Sep 11 13:25:15.356759 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.356705    2578 scope.go:117] "RemoveContainer" containerID="6ede8f2c12fd7b67e056a1df15a43707e94db17818ecc2e23763fb8e9be47b5d"
Sep 11 13:25:15.357135 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.357105    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ede8f2c12fd7b67e056a1df15a43707e94db17818ecc2e23763fb8e9be47b5d"} err="failed to get container status \"6ede8f2c12fd7b67e056a1df15a43707e94db17818ecc2e23763fb8e9be47b5d\": rpc error: code = NotFound desc = could not find container \"6ede8f2c12fd7b67e056a1df15a43707e94db17818ecc2e23763fb8e9be47b5d\": container with ID starting with 6ede8f2c12fd7b67e056a1df15a43707e94db17818ecc2e23763fb8e9be47b5d not found: ID does not exist"
Sep 11 13:25:15.357198 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.357135    2578 scope.go:117] "RemoveContainer" containerID="a25bd3976df0831c59a095681bd064c2f412e2de45db616678e232de0fd8ae50"
Sep 11 13:25:15.357415 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.357398    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a25bd3976df0831c59a095681bd064c2f412e2de45db616678e232de0fd8ae50"} err="failed to get container status \"a25bd3976df0831c59a095681bd064c2f412e2de45db616678e232de0fd8ae50\": rpc error: code = NotFound desc = could not find container \"a25bd3976df0831c59a095681bd064c2f412e2de45db616678e232de0fd8ae50\": container with ID starting with a25bd3976df0831c59a095681bd064c2f412e2de45db616678e232de0fd8ae50 not found: ID does not exist"
Sep 11 13:25:15.357475 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.357416    2578 scope.go:117] "RemoveContainer" containerID="2e2270017b25701463c432676a33ce4b9b3fe4e8ffd66c8a97ae6391812c463e"
Sep 11 13:25:15.357676 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.357650    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e2270017b25701463c432676a33ce4b9b3fe4e8ffd66c8a97ae6391812c463e"} err="failed to get container status \"2e2270017b25701463c432676a33ce4b9b3fe4e8ffd66c8a97ae6391812c463e\": rpc error: code = NotFound desc = could not find container \"2e2270017b25701463c432676a33ce4b9b3fe4e8ffd66c8a97ae6391812c463e\": container with ID starting with 2e2270017b25701463c432676a33ce4b9b3fe4e8ffd66c8a97ae6391812c463e not found: ID does not exist"
Sep 11 13:25:15.357756 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.357680    2578 scope.go:117] "RemoveContainer" containerID="fa724f0373acd6ed1c40dbccc8cdc9c5cd019c03181442172953c11045820322"
Sep 11 13:25:15.357978 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.357961    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa724f0373acd6ed1c40dbccc8cdc9c5cd019c03181442172953c11045820322"} err="failed to get container status \"fa724f0373acd6ed1c40dbccc8cdc9c5cd019c03181442172953c11045820322\": rpc error: code = NotFound desc = could not find container \"fa724f0373acd6ed1c40dbccc8cdc9c5cd019c03181442172953c11045820322\": container with ID starting with fa724f0373acd6ed1c40dbccc8cdc9c5cd019c03181442172953c11045820322 not found: ID does not exist"
Sep 11 13:25:15.358047 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.357979    2578 scope.go:117] "RemoveContainer" containerID="f1eff727d760318e7e6e70e4d4f33f72c8dbf595b92309bbb4c676b20693e840"
Sep 11 13:25:15.358225 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.358207    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1eff727d760318e7e6e70e4d4f33f72c8dbf595b92309bbb4c676b20693e840"} err="failed to get container status \"f1eff727d760318e7e6e70e4d4f33f72c8dbf595b92309bbb4c676b20693e840\": rpc error: code = NotFound desc = could not find container \"f1eff727d760318e7e6e70e4d4f33f72c8dbf595b92309bbb4c676b20693e840\": container with ID starting with f1eff727d760318e7e6e70e4d4f33f72c8dbf595b92309bbb4c676b20693e840 not found: ID does not exist"
Sep 11 13:25:15.358266 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.358225    2578 scope.go:117] "RemoveContainer" containerID="9345ca19b3e6053d8dd6509541647281234bec2ad08f9bde59e68105bd8c8de9"
Sep 11 13:25:15.358469 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.358449    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9345ca19b3e6053d8dd6509541647281234bec2ad08f9bde59e68105bd8c8de9"} err="failed to get container status \"9345ca19b3e6053d8dd6509541647281234bec2ad08f9bde59e68105bd8c8de9\": rpc error: code = NotFound desc = could not find container \"9345ca19b3e6053d8dd6509541647281234bec2ad08f9bde59e68105bd8c8de9\": container with ID starting with 9345ca19b3e6053d8dd6509541647281234bec2ad08f9bde59e68105bd8c8de9 not found: ID does not exist"
Sep 11 13:25:15.358513 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.358470    2578 scope.go:117] "RemoveContainer" containerID="95f2d5ecfced25cf854ca5f30fc92e9a1b395531eaf8fce98cd6e282287addda"
Sep 11 13:25:15.358717 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.358696    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95f2d5ecfced25cf854ca5f30fc92e9a1b395531eaf8fce98cd6e282287addda"} err="failed to get container status \"95f2d5ecfced25cf854ca5f30fc92e9a1b395531eaf8fce98cd6e282287addda\": rpc error: code = NotFound desc = could not find container \"95f2d5ecfced25cf854ca5f30fc92e9a1b395531eaf8fce98cd6e282287addda\": container with ID starting with 95f2d5ecfced25cf854ca5f30fc92e9a1b395531eaf8fce98cd6e282287addda not found: ID does not exist"
Sep 11 13:25:15.358764 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.358721    2578 scope.go:117] "RemoveContainer" containerID="6ede8f2c12fd7b67e056a1df15a43707e94db17818ecc2e23763fb8e9be47b5d"
Sep 11 13:25:15.359032 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.359011    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ede8f2c12fd7b67e056a1df15a43707e94db17818ecc2e23763fb8e9be47b5d"} err="failed to get container status \"6ede8f2c12fd7b67e056a1df15a43707e94db17818ecc2e23763fb8e9be47b5d\": rpc error: code = NotFound desc = could not find container \"6ede8f2c12fd7b67e056a1df15a43707e94db17818ecc2e23763fb8e9be47b5d\": container with ID starting with 6ede8f2c12fd7b67e056a1df15a43707e94db17818ecc2e23763fb8e9be47b5d not found: ID does not exist"
Sep 11 13:25:15.359075 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.359032    2578 scope.go:117] "RemoveContainer" containerID="a25bd3976df0831c59a095681bd064c2f412e2de45db616678e232de0fd8ae50"
Sep 11 13:25:15.359264 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.359248    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a25bd3976df0831c59a095681bd064c2f412e2de45db616678e232de0fd8ae50"} err="failed to get container status \"a25bd3976df0831c59a095681bd064c2f412e2de45db616678e232de0fd8ae50\": rpc error: code = NotFound desc = could not find container \"a25bd3976df0831c59a095681bd064c2f412e2de45db616678e232de0fd8ae50\": container with ID starting with a25bd3976df0831c59a095681bd064c2f412e2de45db616678e232de0fd8ae50 not found: ID does not exist"
Sep 11 13:25:15.359311 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.359265    2578 scope.go:117] "RemoveContainer" containerID="2e2270017b25701463c432676a33ce4b9b3fe4e8ffd66c8a97ae6391812c463e"
Sep 11 13:25:15.359489 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.359470    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e2270017b25701463c432676a33ce4b9b3fe4e8ffd66c8a97ae6391812c463e"} err="failed to get container status \"2e2270017b25701463c432676a33ce4b9b3fe4e8ffd66c8a97ae6391812c463e\": rpc error: code = NotFound desc = could not find container \"2e2270017b25701463c432676a33ce4b9b3fe4e8ffd66c8a97ae6391812c463e\": container with ID starting with 2e2270017b25701463c432676a33ce4b9b3fe4e8ffd66c8a97ae6391812c463e not found: ID does not exist"
Sep 11 13:25:15.359551 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.359489    2578 scope.go:117] "RemoveContainer" containerID="fa724f0373acd6ed1c40dbccc8cdc9c5cd019c03181442172953c11045820322"
Sep 11 13:25:15.359690 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.359674    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa724f0373acd6ed1c40dbccc8cdc9c5cd019c03181442172953c11045820322"} err="failed to get container status \"fa724f0373acd6ed1c40dbccc8cdc9c5cd019c03181442172953c11045820322\": rpc error: code = NotFound desc = could not find container \"fa724f0373acd6ed1c40dbccc8cdc9c5cd019c03181442172953c11045820322\": container with ID starting with fa724f0373acd6ed1c40dbccc8cdc9c5cd019c03181442172953c11045820322 not found: ID does not exist"
Sep 11 13:25:15.359732 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.359689    2578 scope.go:117] "RemoveContainer" containerID="f1eff727d760318e7e6e70e4d4f33f72c8dbf595b92309bbb4c676b20693e840"
Sep 11 13:25:15.359924 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.359907    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1eff727d760318e7e6e70e4d4f33f72c8dbf595b92309bbb4c676b20693e840"} err="failed to get container status \"f1eff727d760318e7e6e70e4d4f33f72c8dbf595b92309bbb4c676b20693e840\": rpc error: code = NotFound desc = could not find container \"f1eff727d760318e7e6e70e4d4f33f72c8dbf595b92309bbb4c676b20693e840\": container with ID starting with f1eff727d760318e7e6e70e4d4f33f72c8dbf595b92309bbb4c676b20693e840 not found: ID does not exist"
Sep 11 13:25:15.359977 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.359925    2578 scope.go:117] "RemoveContainer" containerID="9345ca19b3e6053d8dd6509541647281234bec2ad08f9bde59e68105bd8c8de9"
Sep 11 13:25:15.360250 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.360233    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9345ca19b3e6053d8dd6509541647281234bec2ad08f9bde59e68105bd8c8de9"} err="failed to get container status \"9345ca19b3e6053d8dd6509541647281234bec2ad08f9bde59e68105bd8c8de9\": rpc error: code = NotFound desc = could not find container \"9345ca19b3e6053d8dd6509541647281234bec2ad08f9bde59e68105bd8c8de9\": container with ID starting with 9345ca19b3e6053d8dd6509541647281234bec2ad08f9bde59e68105bd8c8de9 not found: ID does not exist"
Sep 11 13:25:15.360310 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.360251    2578 scope.go:117] "RemoveContainer" containerID="95f2d5ecfced25cf854ca5f30fc92e9a1b395531eaf8fce98cd6e282287addda"
Sep 11 13:25:15.360507 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.360486    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95f2d5ecfced25cf854ca5f30fc92e9a1b395531eaf8fce98cd6e282287addda"} err="failed to get container status \"95f2d5ecfced25cf854ca5f30fc92e9a1b395531eaf8fce98cd6e282287addda\": rpc error: code = NotFound desc = could not find container \"95f2d5ecfced25cf854ca5f30fc92e9a1b395531eaf8fce98cd6e282287addda\": container with ID starting with 95f2d5ecfced25cf854ca5f30fc92e9a1b395531eaf8fce98cd6e282287addda not found: ID does not exist"
Sep 11 13:25:15.360686 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.360512    2578 scope.go:117] "RemoveContainer" containerID="6ede8f2c12fd7b67e056a1df15a43707e94db17818ecc2e23763fb8e9be47b5d"
Sep 11 13:25:15.360859 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.360805    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ede8f2c12fd7b67e056a1df15a43707e94db17818ecc2e23763fb8e9be47b5d"} err="failed to get container status \"6ede8f2c12fd7b67e056a1df15a43707e94db17818ecc2e23763fb8e9be47b5d\": rpc error: code = NotFound desc = could not find container \"6ede8f2c12fd7b67e056a1df15a43707e94db17818ecc2e23763fb8e9be47b5d\": container with ID starting with 6ede8f2c12fd7b67e056a1df15a43707e94db17818ecc2e23763fb8e9be47b5d not found: ID does not exist"
Sep 11 13:25:15.360859 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.360857    2578 scope.go:117] "RemoveContainer" containerID="a25bd3976df0831c59a095681bd064c2f412e2de45db616678e232de0fd8ae50"
Sep 11 13:25:15.361192 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.361169    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a25bd3976df0831c59a095681bd064c2f412e2de45db616678e232de0fd8ae50"} err="failed to get container status \"a25bd3976df0831c59a095681bd064c2f412e2de45db616678e232de0fd8ae50\": rpc error: code = NotFound desc = could not find container \"a25bd3976df0831c59a095681bd064c2f412e2de45db616678e232de0fd8ae50\": container with ID starting with a25bd3976df0831c59a095681bd064c2f412e2de45db616678e232de0fd8ae50 not found: ID does not exist"
Sep 11 13:25:15.361287 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.361194    2578 scope.go:117] "RemoveContainer" containerID="2e2270017b25701463c432676a33ce4b9b3fe4e8ffd66c8a97ae6391812c463e"
Sep 11 13:25:15.361583 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.361543    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e2270017b25701463c432676a33ce4b9b3fe4e8ffd66c8a97ae6391812c463e"} err="failed to get container status \"2e2270017b25701463c432676a33ce4b9b3fe4e8ffd66c8a97ae6391812c463e\": rpc error: code = NotFound desc = could not find container \"2e2270017b25701463c432676a33ce4b9b3fe4e8ffd66c8a97ae6391812c463e\": container with ID starting with 2e2270017b25701463c432676a33ce4b9b3fe4e8ffd66c8a97ae6391812c463e not found: ID does not exist"
Sep 11 13:25:15.361733 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.361584    2578 scope.go:117] "RemoveContainer" containerID="fa724f0373acd6ed1c40dbccc8cdc9c5cd019c03181442172953c11045820322"
Sep 11 13:25:15.362010 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.361973    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa724f0373acd6ed1c40dbccc8cdc9c5cd019c03181442172953c11045820322"} err="failed to get container status \"fa724f0373acd6ed1c40dbccc8cdc9c5cd019c03181442172953c11045820322\": rpc error: code = NotFound desc = could not find container \"fa724f0373acd6ed1c40dbccc8cdc9c5cd019c03181442172953c11045820322\": container with ID starting with fa724f0373acd6ed1c40dbccc8cdc9c5cd019c03181442172953c11045820322 not found: ID does not exist"
Sep 11 13:25:15.363123 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.363104    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 11 13:25:15.363445 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.363434    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d48ed82-7fb0-40aa-93de-332e58a6f316" containerName="prometheus"
Sep 11 13:25:15.363490 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.363448    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d48ed82-7fb0-40aa-93de-332e58a6f316" containerName="prometheus"
Sep 11 13:25:15.363490 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.363456    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d48ed82-7fb0-40aa-93de-332e58a6f316" containerName="thanos-sidecar"
Sep 11 13:25:15.363490 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.363462    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d48ed82-7fb0-40aa-93de-332e58a6f316" containerName="thanos-sidecar"
Sep 11 13:25:15.363490 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.363474    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d48ed82-7fb0-40aa-93de-332e58a6f316" containerName="config-reloader"
Sep 11 13:25:15.363490 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.363479    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d48ed82-7fb0-40aa-93de-332e58a6f316" containerName="config-reloader"
Sep 11 13:25:15.363490 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.363484    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d48ed82-7fb0-40aa-93de-332e58a6f316" containerName="kube-rbac-proxy"
Sep 11 13:25:15.363490 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.363489    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d48ed82-7fb0-40aa-93de-332e58a6f316" containerName="kube-rbac-proxy"
Sep 11 13:25:15.363683 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.363499    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d48ed82-7fb0-40aa-93de-332e58a6f316" containerName="kube-rbac-proxy-web"
Sep 11 13:25:15.363683 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.363506    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d48ed82-7fb0-40aa-93de-332e58a6f316" containerName="kube-rbac-proxy-web"
Sep 11 13:25:15.363683 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.363512    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d48ed82-7fb0-40aa-93de-332e58a6f316" containerName="init-config-reloader"
Sep 11 13:25:15.363683 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.363517    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d48ed82-7fb0-40aa-93de-332e58a6f316" containerName="init-config-reloader"
Sep 11 13:25:15.363683 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.363529    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d48ed82-7fb0-40aa-93de-332e58a6f316" containerName="kube-rbac-proxy-thanos"
Sep 11 13:25:15.363683 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.363533    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d48ed82-7fb0-40aa-93de-332e58a6f316" containerName="kube-rbac-proxy-thanos"
Sep 11 13:25:15.363683 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.363583    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d48ed82-7fb0-40aa-93de-332e58a6f316" containerName="kube-rbac-proxy"
Sep 11 13:25:15.363683 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.363592    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d48ed82-7fb0-40aa-93de-332e58a6f316" containerName="kube-rbac-proxy-web"
Sep 11 13:25:15.363683 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.363599    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d48ed82-7fb0-40aa-93de-332e58a6f316" containerName="prometheus"
Sep 11 13:25:15.363683 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.363605    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d48ed82-7fb0-40aa-93de-332e58a6f316" containerName="kube-rbac-proxy-thanos"
Sep 11 13:25:15.363683 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.363611    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d48ed82-7fb0-40aa-93de-332e58a6f316" containerName="thanos-sidecar"
Sep 11 13:25:15.363683 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.363616    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d48ed82-7fb0-40aa-93de-332e58a6f316" containerName="config-reloader"
Sep 11 13:25:15.368161 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.368136    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:25:15.371644 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.371424    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 11 13:25:15.371870 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.371788    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 11 13:25:15.371870 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.371862    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 11 13:25:15.372007 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.371969    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-jj7m5\""
Sep 11 13:25:15.372088 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.371980    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 11 13:25:15.372142 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.372115    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 11 13:25:15.372190 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.372123    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 11 13:25:15.372418 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.372401    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 11 13:25:15.372479 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.372449    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-iuoqnmid4ilk\""
Sep 11 13:25:15.372890 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.372870    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 11 13:25:15.372979 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.372903    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 11 13:25:15.373151 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.373132    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 11 13:25:15.373288 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.373269    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 11 13:25:15.373346 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.373324    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 11 13:25:15.374677 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.374657    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 11 13:25:15.376961 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.376941    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 11 13:25:15.384394 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.384369    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 11 13:25:15.452368 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.452323    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8a7d435a-339c-4f49-9218-dc6812e9479e-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"8a7d435a-339c-4f49-9218-dc6812e9479e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:25:15.452368 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.452365    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8a7d435a-339c-4f49-9218-dc6812e9479e-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"8a7d435a-339c-4f49-9218-dc6812e9479e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:25:15.452664 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.452386    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8a7d435a-339c-4f49-9218-dc6812e9479e-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"8a7d435a-339c-4f49-9218-dc6812e9479e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:25:15.452664 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.452460    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8a7d435a-339c-4f49-9218-dc6812e9479e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"8a7d435a-339c-4f49-9218-dc6812e9479e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:25:15.452664 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.452500    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8a7d435a-339c-4f49-9218-dc6812e9479e-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"8a7d435a-339c-4f49-9218-dc6812e9479e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:25:15.452664 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.452531    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8a7d435a-339c-4f49-9218-dc6812e9479e-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"8a7d435a-339c-4f49-9218-dc6812e9479e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:25:15.452664 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.452561    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8a7d435a-339c-4f49-9218-dc6812e9479e-config-out\") pod \"prometheus-k8s-0\" (UID: \"8a7d435a-339c-4f49-9218-dc6812e9479e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:25:15.452664 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.452578    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8a7d435a-339c-4f49-9218-dc6812e9479e-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"8a7d435a-339c-4f49-9218-dc6812e9479e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:25:15.452664 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.452607    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8a7d435a-339c-4f49-9218-dc6812e9479e-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"8a7d435a-339c-4f49-9218-dc6812e9479e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:25:15.452664 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.452653    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8a7d435a-339c-4f49-9218-dc6812e9479e-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8a7d435a-339c-4f49-9218-dc6812e9479e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:25:15.453077 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.452715    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8a7d435a-339c-4f49-9218-dc6812e9479e-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"8a7d435a-339c-4f49-9218-dc6812e9479e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:25:15.453077 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.452743    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8a7d435a-339c-4f49-9218-dc6812e9479e-web-config\") pod \"prometheus-k8s-0\" (UID: \"8a7d435a-339c-4f49-9218-dc6812e9479e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:25:15.453077 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.452777    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/8a7d435a-339c-4f49-9218-dc6812e9479e-config\") pod \"prometheus-k8s-0\" (UID: \"8a7d435a-339c-4f49-9218-dc6812e9479e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:25:15.453077 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.452805    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8a7d435a-339c-4f49-9218-dc6812e9479e-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"8a7d435a-339c-4f49-9218-dc6812e9479e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:25:15.453077 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.452869    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8a7d435a-339c-4f49-9218-dc6812e9479e-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"8a7d435a-339c-4f49-9218-dc6812e9479e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:25:15.453077 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.452901    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8a7d435a-339c-4f49-9218-dc6812e9479e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"8a7d435a-339c-4f49-9218-dc6812e9479e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:25:15.453077 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.452941    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vv9q7\" (UniqueName: \"kubernetes.io/projected/8a7d435a-339c-4f49-9218-dc6812e9479e-kube-api-access-vv9q7\") pod \"prometheus-k8s-0\" (UID: \"8a7d435a-339c-4f49-9218-dc6812e9479e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:25:15.453077 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.452977    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8a7d435a-339c-4f49-9218-dc6812e9479e-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"8a7d435a-339c-4f49-9218-dc6812e9479e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:25:15.453077 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.453007    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8a7d435a-339c-4f49-9218-dc6812e9479e-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8a7d435a-339c-4f49-9218-dc6812e9479e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:25:15.453077 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.453035    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8a7d435a-339c-4f49-9218-dc6812e9479e-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8a7d435a-339c-4f49-9218-dc6812e9479e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:25:15.554550 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.554490    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8a7d435a-339c-4f49-9218-dc6812e9479e-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"8a7d435a-339c-4f49-9218-dc6812e9479e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:25:15.554550 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.554557    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8a7d435a-339c-4f49-9218-dc6812e9479e-config-out\") pod \"prometheus-k8s-0\" (UID: \"8a7d435a-339c-4f49-9218-dc6812e9479e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:25:15.554844 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.554579    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8a7d435a-339c-4f49-9218-dc6812e9479e-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"8a7d435a-339c-4f49-9218-dc6812e9479e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:25:15.554844 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.554601    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8a7d435a-339c-4f49-9218-dc6812e9479e-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"8a7d435a-339c-4f49-9218-dc6812e9479e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:25:15.554844 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.554628    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8a7d435a-339c-4f49-9218-dc6812e9479e-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8a7d435a-339c-4f49-9218-dc6812e9479e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:25:15.554844 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.554661    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8a7d435a-339c-4f49-9218-dc6812e9479e-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"8a7d435a-339c-4f49-9218-dc6812e9479e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:25:15.554844 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.554678    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8a7d435a-339c-4f49-9218-dc6812e9479e-web-config\") pod \"prometheus-k8s-0\" (UID: \"8a7d435a-339c-4f49-9218-dc6812e9479e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:25:15.555099 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.554704    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/8a7d435a-339c-4f49-9218-dc6812e9479e-config\") pod \"prometheus-k8s-0\" (UID: \"8a7d435a-339c-4f49-9218-dc6812e9479e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:25:15.555099 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.554881    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8a7d435a-339c-4f49-9218-dc6812e9479e-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"8a7d435a-339c-4f49-9218-dc6812e9479e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:25:15.555099 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.554908    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8a7d435a-339c-4f49-9218-dc6812e9479e-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"8a7d435a-339c-4f49-9218-dc6812e9479e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:25:15.555099 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.554937    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8a7d435a-339c-4f49-9218-dc6812e9479e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"8a7d435a-339c-4f49-9218-dc6812e9479e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:25:15.555099 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.554965    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vv9q7\" (UniqueName: \"kubernetes.io/projected/8a7d435a-339c-4f49-9218-dc6812e9479e-kube-api-access-vv9q7\") pod \"prometheus-k8s-0\" (UID: \"8a7d435a-339c-4f49-9218-dc6812e9479e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:25:15.555099 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.555004    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8a7d435a-339c-4f49-9218-dc6812e9479e-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"8a7d435a-339c-4f49-9218-dc6812e9479e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:25:15.555099 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.555039    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8a7d435a-339c-4f49-9218-dc6812e9479e-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8a7d435a-339c-4f49-9218-dc6812e9479e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:25:15.555099 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.555068    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8a7d435a-339c-4f49-9218-dc6812e9479e-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8a7d435a-339c-4f49-9218-dc6812e9479e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:25:15.555453 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.555107    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8a7d435a-339c-4f49-9218-dc6812e9479e-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"8a7d435a-339c-4f49-9218-dc6812e9479e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:25:15.555453 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.555135    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8a7d435a-339c-4f49-9218-dc6812e9479e-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"8a7d435a-339c-4f49-9218-dc6812e9479e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:25:15.555453 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.555165    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8a7d435a-339c-4f49-9218-dc6812e9479e-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"8a7d435a-339c-4f49-9218-dc6812e9479e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:25:15.555453 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.555207    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8a7d435a-339c-4f49-9218-dc6812e9479e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"8a7d435a-339c-4f49-9218-dc6812e9479e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:25:15.555453 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.555263    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8a7d435a-339c-4f49-9218-dc6812e9479e-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"8a7d435a-339c-4f49-9218-dc6812e9479e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:25:15.555453 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.555316    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8a7d435a-339c-4f49-9218-dc6812e9479e-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"8a7d435a-339c-4f49-9218-dc6812e9479e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:25:15.555958 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.555717    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8a7d435a-339c-4f49-9218-dc6812e9479e-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"8a7d435a-339c-4f49-9218-dc6812e9479e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:25:15.556765 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.556430    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8a7d435a-339c-4f49-9218-dc6812e9479e-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8a7d435a-339c-4f49-9218-dc6812e9479e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:25:15.556765 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.556537    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8a7d435a-339c-4f49-9218-dc6812e9479e-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8a7d435a-339c-4f49-9218-dc6812e9479e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:25:15.558390 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.558192    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8a7d435a-339c-4f49-9218-dc6812e9479e-config-out\") pod \"prometheus-k8s-0\" (UID: \"8a7d435a-339c-4f49-9218-dc6812e9479e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:25:15.558390 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.558205    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8a7d435a-339c-4f49-9218-dc6812e9479e-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"8a7d435a-339c-4f49-9218-dc6812e9479e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:25:15.558390 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.558258    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8a7d435a-339c-4f49-9218-dc6812e9479e-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"8a7d435a-339c-4f49-9218-dc6812e9479e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:25:15.558390 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.558280    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8a7d435a-339c-4f49-9218-dc6812e9479e-web-config\") pod \"prometheus-k8s-0\" (UID: \"8a7d435a-339c-4f49-9218-dc6812e9479e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:25:15.558735 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.558629    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/8a7d435a-339c-4f49-9218-dc6812e9479e-config\") pod \"prometheus-k8s-0\" (UID: \"8a7d435a-339c-4f49-9218-dc6812e9479e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:25:15.558735 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.558652    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8a7d435a-339c-4f49-9218-dc6812e9479e-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"8a7d435a-339c-4f49-9218-dc6812e9479e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:25:15.559167 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.558984    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8a7d435a-339c-4f49-9218-dc6812e9479e-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"8a7d435a-339c-4f49-9218-dc6812e9479e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:25:15.559263 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.559163    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8a7d435a-339c-4f49-9218-dc6812e9479e-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"8a7d435a-339c-4f49-9218-dc6812e9479e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:25:15.559344 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.559322    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8a7d435a-339c-4f49-9218-dc6812e9479e-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8a7d435a-339c-4f49-9218-dc6812e9479e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:25:15.559752 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.559724    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8a7d435a-339c-4f49-9218-dc6812e9479e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"8a7d435a-339c-4f49-9218-dc6812e9479e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:25:15.560580 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.560552    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8a7d435a-339c-4f49-9218-dc6812e9479e-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"8a7d435a-339c-4f49-9218-dc6812e9479e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:25:15.560788 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.560768    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8a7d435a-339c-4f49-9218-dc6812e9479e-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"8a7d435a-339c-4f49-9218-dc6812e9479e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:25:15.561227 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.561207    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8a7d435a-339c-4f49-9218-dc6812e9479e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"8a7d435a-339c-4f49-9218-dc6812e9479e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:25:15.561497 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.561475    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8a7d435a-339c-4f49-9218-dc6812e9479e-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"8a7d435a-339c-4f49-9218-dc6812e9479e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:25:15.562141 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.562123    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8a7d435a-339c-4f49-9218-dc6812e9479e-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"8a7d435a-339c-4f49-9218-dc6812e9479e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:25:15.572764 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.572691    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vv9q7\" (UniqueName: \"kubernetes.io/projected/8a7d435a-339c-4f49-9218-dc6812e9479e-kube-api-access-vv9q7\") pod \"prometheus-k8s-0\" (UID: \"8a7d435a-339c-4f49-9218-dc6812e9479e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:25:15.682355 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.682300    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:25:15.973500 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:15.973449    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 11 13:25:15.976930 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:25:15.976897    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a7d435a_339c_4f49_9218_dc6812e9479e.slice/crio-c76e0b64dcc3d01b5878bbb8762d43e4316f50c756cb8b50f658e38c7fcc17f8 WatchSource:0}: Error finding container c76e0b64dcc3d01b5878bbb8762d43e4316f50c756cb8b50f658e38c7fcc17f8: Status 404 returned error can't find the container with id c76e0b64dcc3d01b5878bbb8762d43e4316f50c756cb8b50f658e38c7fcc17f8
Sep 11 13:25:16.290331 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:16.290278    2578 generic.go:356] "Generic (PLEG): container finished" podID="8a7d435a-339c-4f49-9218-dc6812e9479e" containerID="7b3b22227f95baebd6917ceb9d2baff09bd6e37d27f497bd67c8d17c937195b7" exitCode=0
Sep 11 13:25:16.290793 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:16.290365    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8a7d435a-339c-4f49-9218-dc6812e9479e","Type":"ContainerDied","Data":"7b3b22227f95baebd6917ceb9d2baff09bd6e37d27f497bd67c8d17c937195b7"}
Sep 11 13:25:16.290793 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:16.290444    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8a7d435a-339c-4f49-9218-dc6812e9479e","Type":"ContainerStarted","Data":"c76e0b64dcc3d01b5878bbb8762d43e4316f50c756cb8b50f658e38c7fcc17f8"}
Sep 11 13:25:17.106009 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:17.105898    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9d48ed82-7fb0-40aa-93de-332e58a6f316" path="/var/lib/kubelet/pods/9d48ed82-7fb0-40aa-93de-332e58a6f316/volumes"
Sep 11 13:25:17.301186 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:17.301135    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8a7d435a-339c-4f49-9218-dc6812e9479e","Type":"ContainerStarted","Data":"ff8f68c33df29c7d302638dd7fd26bd8c9509c67480ac3aeb7bb5ff45c9fbba0"}
Sep 11 13:25:17.301644 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:17.301228    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8a7d435a-339c-4f49-9218-dc6812e9479e","Type":"ContainerStarted","Data":"2e63f00551f1dd718e49aeab97664b8773858a82b5e5258a97b761ebe55c3ad5"}
Sep 11 13:25:17.301644 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:17.301260    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8a7d435a-339c-4f49-9218-dc6812e9479e","Type":"ContainerStarted","Data":"505e61e7c4fa815a2ed8f9b8da145875c418fdd1897fb09b587103d2fd507586"}
Sep 11 13:25:17.301644 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:17.301286    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8a7d435a-339c-4f49-9218-dc6812e9479e","Type":"ContainerStarted","Data":"9ba7ce53ca6b4d758cc08f12d0e9af09a1ba52ff5d4e3ce27825621e8073e67a"}
Sep 11 13:25:17.301644 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:17.301313    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8a7d435a-339c-4f49-9218-dc6812e9479e","Type":"ContainerStarted","Data":"bac9bd31e52d577c3b328b8ad97c7b5c8ca6b2eb736343ba0d4739e9e71fb7f8"}
Sep 11 13:25:17.301644 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:17.301339    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8a7d435a-339c-4f49-9218-dc6812e9479e","Type":"ContainerStarted","Data":"0abbf81a0fbb6db780339a554bafc1dafd2094633d377e12e223a68ef3404f34"}
Sep 11 13:25:20.682728 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:25:20.682678    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:26:15.682984 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:26:15.682911    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:26:15.701787 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:26:15.701744    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:26:15.731780 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:26:15.731330    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=60.711165161 podStartE2EDuration="1m0.73130473s" totalImagesPullingTime="0s" totalInitContainerRuntime="20.139569ms" isStatefulPod=false podCreationTimestamp="2026-09-11 13:25:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:25:17.360744098 +0000 UTC m=+224.746847283" watchObservedRunningTime="2026-09-11 13:26:15.73130473 +0000 UTC m=+283.117407894"
Sep 11 13:26:16.600962 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:26:16.600927    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 13:26:33.016654 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:26:33.016614    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6gltj_e384caae-3cb1-4264-8a25-55b40ac8ac45/ovn-acl-logging/0.log"
Sep 11 13:26:33.020202 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:26:33.020174    2578 kubelet.go:1648] "Image garbage collection succeeded"
Sep 11 13:26:33.022923 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:26:33.022893    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6gltj_e384caae-3cb1-4264-8a25-55b40ac8ac45/ovn-acl-logging/0.log"
Sep 11 13:27:07.599641 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:27:07.599598    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-ngcxg"]
Sep 11 13:27:07.603052 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:27:07.603029    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-ngcxg"
Sep 11 13:27:07.610011 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:27:07.609976    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 11 13:27:07.610223 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:27:07.609982    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 11 13:27:07.610223 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:27:07.610113    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-9prvn\""
Sep 11 13:27:07.613885 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:27:07.613823    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-ngcxg"]
Sep 11 13:27:07.738660 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:27:07.738614    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6gx6\" (UniqueName: \"kubernetes.io/projected/e1218d19-1c75-4a99-ba56-961d57155117-kube-api-access-j6gx6\") pod \"cert-manager-55b4d954d5-ngcxg\" (UID: \"e1218d19-1c75-4a99-ba56-961d57155117\") " pod="cert-manager/cert-manager-55b4d954d5-ngcxg"
Sep 11 13:27:07.738921 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:27:07.738756    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e1218d19-1c75-4a99-ba56-961d57155117-bound-sa-token\") pod \"cert-manager-55b4d954d5-ngcxg\" (UID: \"e1218d19-1c75-4a99-ba56-961d57155117\") " pod="cert-manager/cert-manager-55b4d954d5-ngcxg"
Sep 11 13:27:07.839718 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:27:07.839662    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e1218d19-1c75-4a99-ba56-961d57155117-bound-sa-token\") pod \"cert-manager-55b4d954d5-ngcxg\" (UID: \"e1218d19-1c75-4a99-ba56-961d57155117\") " pod="cert-manager/cert-manager-55b4d954d5-ngcxg"
Sep 11 13:27:07.839718 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:27:07.839730    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6gx6\" (UniqueName: \"kubernetes.io/projected/e1218d19-1c75-4a99-ba56-961d57155117-kube-api-access-j6gx6\") pod \"cert-manager-55b4d954d5-ngcxg\" (UID: \"e1218d19-1c75-4a99-ba56-961d57155117\") " pod="cert-manager/cert-manager-55b4d954d5-ngcxg"
Sep 11 13:27:07.854662 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:27:07.854576    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e1218d19-1c75-4a99-ba56-961d57155117-bound-sa-token\") pod \"cert-manager-55b4d954d5-ngcxg\" (UID: \"e1218d19-1c75-4a99-ba56-961d57155117\") " pod="cert-manager/cert-manager-55b4d954d5-ngcxg"
Sep 11 13:27:07.855771 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:27:07.855747    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6gx6\" (UniqueName: \"kubernetes.io/projected/e1218d19-1c75-4a99-ba56-961d57155117-kube-api-access-j6gx6\") pod \"cert-manager-55b4d954d5-ngcxg\" (UID: \"e1218d19-1c75-4a99-ba56-961d57155117\") " pod="cert-manager/cert-manager-55b4d954d5-ngcxg"
Sep 11 13:27:07.929219 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:27:07.929163    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-ngcxg"
Sep 11 13:27:08.209909 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:27:08.209747    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-ngcxg"]
Sep 11 13:27:08.214492 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:27:08.214439    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode1218d19_1c75_4a99_ba56_961d57155117.slice/crio-f282f5d73054790221d4229ac886fe95e8bc41476a116982f7dbf5a0082fa8ff WatchSource:0}: Error finding container f282f5d73054790221d4229ac886fe95e8bc41476a116982f7dbf5a0082fa8ff: Status 404 returned error can't find the container with id f282f5d73054790221d4229ac886fe95e8bc41476a116982f7dbf5a0082fa8ff
Sep 11 13:27:08.216665 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:27:08.216644    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 13:27:08.817743 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:27:08.817692    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-ngcxg" event={"ID":"e1218d19-1c75-4a99-ba56-961d57155117","Type":"ContainerStarted","Data":"f282f5d73054790221d4229ac886fe95e8bc41476a116982f7dbf5a0082fa8ff"}
Sep 11 13:27:10.999802 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:27:10.999750    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-7pj9t"]
Sep 11 13:27:11.004023 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:27:11.003991    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-7pj9t"
Sep 11 13:27:11.006878 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:27:11.006852    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 11 13:27:11.007033 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:27:11.006852    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 11 13:27:11.007494 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:27:11.007444    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-whkk4\""
Sep 11 13:27:11.014416 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:27:11.014112    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-7pj9t"]
Sep 11 13:27:11.076759 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:27:11.076716    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqxnk\" (UniqueName: \"kubernetes.io/projected/c08d81cf-415c-4971-b610-c4e94047eceb-kube-api-access-tqxnk\") pod \"openshift-lws-operator-fd8ccff4c-7pj9t\" (UID: \"c08d81cf-415c-4971-b610-c4e94047eceb\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-7pj9t"
Sep 11 13:27:11.077019 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:27:11.076795    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c08d81cf-415c-4971-b610-c4e94047eceb-tmp\") pod \"openshift-lws-operator-fd8ccff4c-7pj9t\" (UID: \"c08d81cf-415c-4971-b610-c4e94047eceb\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-7pj9t"
Sep 11 13:27:11.178307 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:27:11.178244    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqxnk\" (UniqueName: \"kubernetes.io/projected/c08d81cf-415c-4971-b610-c4e94047eceb-kube-api-access-tqxnk\") pod \"openshift-lws-operator-fd8ccff4c-7pj9t\" (UID: \"c08d81cf-415c-4971-b610-c4e94047eceb\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-7pj9t"
Sep 11 13:27:11.178598 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:27:11.178344    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c08d81cf-415c-4971-b610-c4e94047eceb-tmp\") pod \"openshift-lws-operator-fd8ccff4c-7pj9t\" (UID: \"c08d81cf-415c-4971-b610-c4e94047eceb\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-7pj9t"
Sep 11 13:27:11.178952 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:27:11.178916    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c08d81cf-415c-4971-b610-c4e94047eceb-tmp\") pod \"openshift-lws-operator-fd8ccff4c-7pj9t\" (UID: \"c08d81cf-415c-4971-b610-c4e94047eceb\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-7pj9t"
Sep 11 13:27:11.189634 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:27:11.189597    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqxnk\" (UniqueName: \"kubernetes.io/projected/c08d81cf-415c-4971-b610-c4e94047eceb-kube-api-access-tqxnk\") pod \"openshift-lws-operator-fd8ccff4c-7pj9t\" (UID: \"c08d81cf-415c-4971-b610-c4e94047eceb\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-7pj9t"
Sep 11 13:27:11.318027 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:27:11.317921    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-7pj9t"
Sep 11 13:27:11.646098 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:27:11.646064    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-7pj9t"]
Sep 11 13:27:11.649001 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:27:11.648958    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc08d81cf_415c_4971_b610_c4e94047eceb.slice/crio-7a304401e84808da7ecd78e8fc278504ba0ca89cd5d87fb3dcd315221e037785 WatchSource:0}: Error finding container 7a304401e84808da7ecd78e8fc278504ba0ca89cd5d87fb3dcd315221e037785: Status 404 returned error can't find the container with id 7a304401e84808da7ecd78e8fc278504ba0ca89cd5d87fb3dcd315221e037785
Sep 11 13:27:11.837551 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:27:11.837504    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-7pj9t" event={"ID":"c08d81cf-415c-4971-b610-c4e94047eceb","Type":"ContainerStarted","Data":"7a304401e84808da7ecd78e8fc278504ba0ca89cd5d87fb3dcd315221e037785"}
Sep 11 13:27:11.839400 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:27:11.839359    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-ngcxg" event={"ID":"e1218d19-1c75-4a99-ba56-961d57155117","Type":"ContainerStarted","Data":"bf8d1c48abd36371619c19a257a9926ded62c9d770f6e68dc43b45d241942aa7"}
Sep 11 13:27:14.858318 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:27:14.858254    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-7pj9t" event={"ID":"c08d81cf-415c-4971-b610-c4e94047eceb","Type":"ContainerStarted","Data":"d86bd2e7326e94ddc583347b54031aaa95df918c657521de03cdfa5cc3ed8f68"}
Sep 11 13:27:14.876298 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:27:14.876005    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-ngcxg" podStartSLOduration=4.7161893070000005 podStartE2EDuration="7.875981178s" totalImagesPullingTime="3.159791871s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:27:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:27:11.872105761 +0000 UTC m=+339.258208925" watchObservedRunningTime="2026-09-11 13:27:14.875981178 +0000 UTC m=+342.262084343"
Sep 11 13:27:14.877336 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:27:14.877297    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-7pj9t" podStartSLOduration=2.54694605 podStartE2EDuration="4.877283072s" totalImagesPullingTime="2.330337022s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:27:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:27:14.875398932 +0000 UTC m=+342.261502097" watchObservedRunningTime="2026-09-11 13:27:14.877283072 +0000 UTC m=+342.263386236"
Sep 11 13:27:30.558879 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:27:30.558769    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-9576ccb85-jz224"]
Sep 11 13:27:30.565241 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:27:30.565215    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-9576ccb85-jz224"
Sep 11 13:27:30.567580 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:27:30.567544    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 11 13:27:30.567764 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:27:30.567688    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 11 13:27:30.567764 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:27:30.567688    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 11 13:27:30.567911 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:27:30.567794    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-2ksm4\""
Sep 11 13:27:30.567911 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:27:30.567882    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 11 13:27:30.584991 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:27:30.584935    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-9576ccb85-jz224"]
Sep 11 13:27:30.671218 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:27:30.671177    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bthwm\" (UniqueName: \"kubernetes.io/projected/5b7f1cc4-b656-46e4-bfa0-2248d0e988be-kube-api-access-bthwm\") pod \"opendatahub-operator-controller-manager-9576ccb85-jz224\" (UID: \"5b7f1cc4-b656-46e4-bfa0-2248d0e988be\") " pod="opendatahub/opendatahub-operator-controller-manager-9576ccb85-jz224"
Sep 11 13:27:30.671218 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:27:30.671227    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5b7f1cc4-b656-46e4-bfa0-2248d0e988be-tmp\") pod \"opendatahub-operator-controller-manager-9576ccb85-jz224\" (UID: \"5b7f1cc4-b656-46e4-bfa0-2248d0e988be\") " pod="opendatahub/opendatahub-operator-controller-manager-9576ccb85-jz224"
Sep 11 13:27:30.671466 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:27:30.671261    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/5b7f1cc4-b656-46e4-bfa0-2248d0e988be-manifests\") pod \"opendatahub-operator-controller-manager-9576ccb85-jz224\" (UID: \"5b7f1cc4-b656-46e4-bfa0-2248d0e988be\") " pod="opendatahub/opendatahub-operator-controller-manager-9576ccb85-jz224"
Sep 11 13:27:30.671466 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:27:30.671281    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5b7f1cc4-b656-46e4-bfa0-2248d0e988be-webhook-cert\") pod \"opendatahub-operator-controller-manager-9576ccb85-jz224\" (UID: \"5b7f1cc4-b656-46e4-bfa0-2248d0e988be\") " pod="opendatahub/opendatahub-operator-controller-manager-9576ccb85-jz224"
Sep 11 13:27:30.671466 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:27:30.671325    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/5b7f1cc4-b656-46e4-bfa0-2248d0e988be-apiservice-cert\") pod \"opendatahub-operator-controller-manager-9576ccb85-jz224\" (UID: \"5b7f1cc4-b656-46e4-bfa0-2248d0e988be\") " pod="opendatahub/opendatahub-operator-controller-manager-9576ccb85-jz224"
Sep 11 13:27:30.772057 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:27:30.772007    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bthwm\" (UniqueName: \"kubernetes.io/projected/5b7f1cc4-b656-46e4-bfa0-2248d0e988be-kube-api-access-bthwm\") pod \"opendatahub-operator-controller-manager-9576ccb85-jz224\" (UID: \"5b7f1cc4-b656-46e4-bfa0-2248d0e988be\") " pod="opendatahub/opendatahub-operator-controller-manager-9576ccb85-jz224"
Sep 11 13:27:30.772315 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:27:30.772096    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5b7f1cc4-b656-46e4-bfa0-2248d0e988be-tmp\") pod \"opendatahub-operator-controller-manager-9576ccb85-jz224\" (UID: \"5b7f1cc4-b656-46e4-bfa0-2248d0e988be\") " pod="opendatahub/opendatahub-operator-controller-manager-9576ccb85-jz224"
Sep 11 13:27:30.772315 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:27:30.772154    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/5b7f1cc4-b656-46e4-bfa0-2248d0e988be-manifests\") pod \"opendatahub-operator-controller-manager-9576ccb85-jz224\" (UID: \"5b7f1cc4-b656-46e4-bfa0-2248d0e988be\") " pod="opendatahub/opendatahub-operator-controller-manager-9576ccb85-jz224"
Sep 11 13:27:30.772315 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:27:30.772183    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5b7f1cc4-b656-46e4-bfa0-2248d0e988be-webhook-cert\") pod \"opendatahub-operator-controller-manager-9576ccb85-jz224\" (UID: \"5b7f1cc4-b656-46e4-bfa0-2248d0e988be\") " pod="opendatahub/opendatahub-operator-controller-manager-9576ccb85-jz224"
Sep 11 13:27:30.772315 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:27:30.772254    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/5b7f1cc4-b656-46e4-bfa0-2248d0e988be-apiservice-cert\") pod \"opendatahub-operator-controller-manager-9576ccb85-jz224\" (UID: \"5b7f1cc4-b656-46e4-bfa0-2248d0e988be\") " pod="opendatahub/opendatahub-operator-controller-manager-9576ccb85-jz224"
Sep 11 13:27:30.772690 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:27:30.772654    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/5b7f1cc4-b656-46e4-bfa0-2248d0e988be-manifests\") pod \"opendatahub-operator-controller-manager-9576ccb85-jz224\" (UID: \"5b7f1cc4-b656-46e4-bfa0-2248d0e988be\") " pod="opendatahub/opendatahub-operator-controller-manager-9576ccb85-jz224"
Sep 11 13:27:30.775032 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:27:30.775002    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5b7f1cc4-b656-46e4-bfa0-2248d0e988be-tmp\") pod \"opendatahub-operator-controller-manager-9576ccb85-jz224\" (UID: \"5b7f1cc4-b656-46e4-bfa0-2248d0e988be\") " pod="opendatahub/opendatahub-operator-controller-manager-9576ccb85-jz224"
Sep 11 13:27:30.775213 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:27:30.775135    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/5b7f1cc4-b656-46e4-bfa0-2248d0e988be-apiservice-cert\") pod \"opendatahub-operator-controller-manager-9576ccb85-jz224\" (UID: \"5b7f1cc4-b656-46e4-bfa0-2248d0e988be\") " pod="opendatahub/opendatahub-operator-controller-manager-9576ccb85-jz224"
Sep 11 13:27:30.775254 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:27:30.775213    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5b7f1cc4-b656-46e4-bfa0-2248d0e988be-webhook-cert\") pod \"opendatahub-operator-controller-manager-9576ccb85-jz224\" (UID: \"5b7f1cc4-b656-46e4-bfa0-2248d0e988be\") " pod="opendatahub/opendatahub-operator-controller-manager-9576ccb85-jz224"
Sep 11 13:27:30.784375 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:27:30.784338    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bthwm\" (UniqueName: \"kubernetes.io/projected/5b7f1cc4-b656-46e4-bfa0-2248d0e988be-kube-api-access-bthwm\") pod \"opendatahub-operator-controller-manager-9576ccb85-jz224\" (UID: \"5b7f1cc4-b656-46e4-bfa0-2248d0e988be\") " pod="opendatahub/opendatahub-operator-controller-manager-9576ccb85-jz224"
Sep 11 13:27:30.877940 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:27:30.877769    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-9576ccb85-jz224"
Sep 11 13:27:31.178919 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:27:31.178884    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-9576ccb85-jz224"]
Sep 11 13:27:31.182426 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:27:31.182390    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b7f1cc4_b656_46e4_bfa0_2248d0e988be.slice/crio-0ddadc5ce6e4f4ad259e3bb1622126122142d063925d5989ff1380aa2adadca5 WatchSource:0}: Error finding container 0ddadc5ce6e4f4ad259e3bb1622126122142d063925d5989ff1380aa2adadca5: Status 404 returned error can't find the container with id 0ddadc5ce6e4f4ad259e3bb1622126122142d063925d5989ff1380aa2adadca5
Sep 11 13:27:31.948875 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:27:31.948787    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-9576ccb85-jz224" event={"ID":"5b7f1cc4-b656-46e4-bfa0-2248d0e988be","Type":"ContainerStarted","Data":"0ddadc5ce6e4f4ad259e3bb1622126122142d063925d5989ff1380aa2adadca5"}
Sep 11 13:27:34.970121 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:27:34.970071    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-9576ccb85-jz224" event={"ID":"5b7f1cc4-b656-46e4-bfa0-2248d0e988be","Type":"ContainerStarted","Data":"4b9b0789ddf37e1d07eac700046c6abde746fafaf7c8ed6c4cbef1e48086a340"}
Sep 11 13:27:35.977684 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:27:35.977627    2578 generic.go:356] "Generic (PLEG): container finished" podID="5b7f1cc4-b656-46e4-bfa0-2248d0e988be" containerID="4b9b0789ddf37e1d07eac700046c6abde746fafaf7c8ed6c4cbef1e48086a340" exitCode=0
Sep 11 13:27:35.978262 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:27:35.977719    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-9576ccb85-jz224" event={"ID":"5b7f1cc4-b656-46e4-bfa0-2248d0e988be","Type":"ContainerDied","Data":"4b9b0789ddf37e1d07eac700046c6abde746fafaf7c8ed6c4cbef1e48086a340"}
Sep 11 13:27:36.985809 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:27:36.985755    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-9576ccb85-jz224" event={"ID":"5b7f1cc4-b656-46e4-bfa0-2248d0e988be","Type":"ContainerStarted","Data":"3a3449d83de2a7b6b9b2a9356b751b04e78b19dfdebf4ce353dddace8abf7974"}
Sep 11 13:27:36.986339 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:27:36.986007    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-9576ccb85-jz224"
Sep 11 13:27:37.013172 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:27:37.013096    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-9576ccb85-jz224" podStartSLOduration=2.376646192 podStartE2EDuration="7.013069377s" totalImagesPullingTime="3.42815665s" totalInitContainerRuntime="1.208266535s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:27:30 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:27:37.011392115 +0000 UTC m=+364.397495296" watchObservedRunningTime="2026-09-11 13:27:37.013069377 +0000 UTC m=+364.399172541"
Sep 11 13:27:47.994184 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:27:47.994152    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-9576ccb85-jz224"
Sep 11 13:27:50.321164 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:27:50.321116    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-7c9bdb7954-fwnlt"]
Sep 11 13:27:50.328073 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:27:50.328028    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7c9bdb7954-fwnlt"
Sep 11 13:27:50.330401 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:27:50.330368    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 11 13:27:50.330577 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:27:50.330373    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 11 13:27:50.342916 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:27:50.342869    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7c9bdb7954-fwnlt"]
Sep 11 13:27:50.475444 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:27:50.475383    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbrmj\" (UniqueName: \"kubernetes.io/projected/d1d8740e-7e95-42db-9247-3613ec035a2d-kube-api-access-kbrmj\") pod \"kube-auth-proxy-7c9bdb7954-fwnlt\" (UID: \"d1d8740e-7e95-42db-9247-3613ec035a2d\") " pod="openshift-ingress/kube-auth-proxy-7c9bdb7954-fwnlt"
Sep 11 13:27:50.475672 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:27:50.475527    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1d8740e-7e95-42db-9247-3613ec035a2d-tls-certs\") pod \"kube-auth-proxy-7c9bdb7954-fwnlt\" (UID: \"d1d8740e-7e95-42db-9247-3613ec035a2d\") " pod="openshift-ingress/kube-auth-proxy-7c9bdb7954-fwnlt"
Sep 11 13:27:50.475672 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:27:50.475552    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d1d8740e-7e95-42db-9247-3613ec035a2d-tmp\") pod \"kube-auth-proxy-7c9bdb7954-fwnlt\" (UID: \"d1d8740e-7e95-42db-9247-3613ec035a2d\") " pod="openshift-ingress/kube-auth-proxy-7c9bdb7954-fwnlt"
Sep 11 13:27:50.576270 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:27:50.576131    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbrmj\" (UniqueName: \"kubernetes.io/projected/d1d8740e-7e95-42db-9247-3613ec035a2d-kube-api-access-kbrmj\") pod \"kube-auth-proxy-7c9bdb7954-fwnlt\" (UID: \"d1d8740e-7e95-42db-9247-3613ec035a2d\") " pod="openshift-ingress/kube-auth-proxy-7c9bdb7954-fwnlt"
Sep 11 13:27:50.576270 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:27:50.576248    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1d8740e-7e95-42db-9247-3613ec035a2d-tls-certs\") pod \"kube-auth-proxy-7c9bdb7954-fwnlt\" (UID: \"d1d8740e-7e95-42db-9247-3613ec035a2d\") " pod="openshift-ingress/kube-auth-proxy-7c9bdb7954-fwnlt"
Sep 11 13:27:50.576270 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:27:50.576268    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d1d8740e-7e95-42db-9247-3613ec035a2d-tmp\") pod \"kube-auth-proxy-7c9bdb7954-fwnlt\" (UID: \"d1d8740e-7e95-42db-9247-3613ec035a2d\") " pod="openshift-ingress/kube-auth-proxy-7c9bdb7954-fwnlt"
Sep 11 13:27:50.579105 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:27:50.579068    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d1d8740e-7e95-42db-9247-3613ec035a2d-tmp\") pod \"kube-auth-proxy-7c9bdb7954-fwnlt\" (UID: \"d1d8740e-7e95-42db-9247-3613ec035a2d\") " pod="openshift-ingress/kube-auth-proxy-7c9bdb7954-fwnlt"
Sep 11 13:27:50.579284 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:27:50.579245    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1d8740e-7e95-42db-9247-3613ec035a2d-tls-certs\") pod \"kube-auth-proxy-7c9bdb7954-fwnlt\" (UID: \"d1d8740e-7e95-42db-9247-3613ec035a2d\") " pod="openshift-ingress/kube-auth-proxy-7c9bdb7954-fwnlt"
Sep 11 13:27:50.594742 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:27:50.594711    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbrmj\" (UniqueName: \"kubernetes.io/projected/d1d8740e-7e95-42db-9247-3613ec035a2d-kube-api-access-kbrmj\") pod \"kube-auth-proxy-7c9bdb7954-fwnlt\" (UID: \"d1d8740e-7e95-42db-9247-3613ec035a2d\") " pod="openshift-ingress/kube-auth-proxy-7c9bdb7954-fwnlt"
Sep 11 13:27:50.640915 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:27:50.640867    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7c9bdb7954-fwnlt"
Sep 11 13:27:50.934102 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:27:50.934008    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7c9bdb7954-fwnlt"]
Sep 11 13:27:50.942194 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:27:50.942154    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1d8740e_7e95_42db_9247_3613ec035a2d.slice/crio-c8c0b8823673024bc417a96ab4978d7af73ba574c03e933af6b1f436d10ba4dd WatchSource:0}: Error finding container c8c0b8823673024bc417a96ab4978d7af73ba574c03e933af6b1f436d10ba4dd: Status 404 returned error can't find the container with id c8c0b8823673024bc417a96ab4978d7af73ba574c03e933af6b1f436d10ba4dd
Sep 11 13:27:51.063712 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:27:51.063678    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7c9bdb7954-fwnlt" event={"ID":"d1d8740e-7e95-42db-9247-3613ec035a2d","Type":"ContainerStarted","Data":"c8c0b8823673024bc417a96ab4978d7af73ba574c03e933af6b1f436d10ba4dd"}
Sep 11 13:27:51.799392 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:27:51.799336    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-fnsw4"]
Sep 11 13:27:51.804045 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:27:51.804005    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-fnsw4"
Sep 11 13:27:51.810123 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:27:51.810083    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 11 13:27:51.810361 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:27:51.810275    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Sep 11 13:27:51.810561 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:27:51.810506    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Sep 11 13:27:51.810762 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:27:51.810647    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Sep 11 13:27:51.811206 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:27:51.811060    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-c5xh9\""
Sep 11 13:27:51.823336 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:27:51.823303    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-fnsw4"]
Sep 11 13:27:51.890985 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:27:51.890892    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b72e3683-af0e-4c04-841f-bac4a0e8be0a-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-fnsw4\" (UID: \"b72e3683-af0e-4c04-841f-bac4a0e8be0a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-fnsw4"
Sep 11 13:27:51.890985 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:27:51.890943    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b72e3683-af0e-4c04-841f-bac4a0e8be0a-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-fnsw4\" (UID: \"b72e3683-af0e-4c04-841f-bac4a0e8be0a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-fnsw4"
Sep 11 13:27:51.891293 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:27:51.891047    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b72e3683-af0e-4c04-841f-bac4a0e8be0a-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-fnsw4\" (UID: \"b72e3683-af0e-4c04-841f-bac4a0e8be0a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-fnsw4"
Sep 11 13:27:51.891293 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:27:51.891106    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b72e3683-af0e-4c04-841f-bac4a0e8be0a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-fnsw4\" (UID: \"b72e3683-af0e-4c04-841f-bac4a0e8be0a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-fnsw4"
Sep 11 13:27:51.891293 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:27:51.891177    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kr6xz\" (UniqueName: \"kubernetes.io/projected/b72e3683-af0e-4c04-841f-bac4a0e8be0a-kube-api-access-kr6xz\") pod \"istiod-openshift-gateway-7b7995d5cd-fnsw4\" (UID: \"b72e3683-af0e-4c04-841f-bac4a0e8be0a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-fnsw4"
Sep 11 13:27:51.891460 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:27:51.891309    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b72e3683-af0e-4c04-841f-bac4a0e8be0a-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-fnsw4\" (UID: \"b72e3683-af0e-4c04-841f-bac4a0e8be0a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-fnsw4"
Sep 11 13:27:51.891460 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:27:51.891342    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b72e3683-af0e-4c04-841f-bac4a0e8be0a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-fnsw4\" (UID: \"b72e3683-af0e-4c04-841f-bac4a0e8be0a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-fnsw4"
Sep 11 13:27:51.992710 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:27:51.992649    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kr6xz\" (UniqueName: \"kubernetes.io/projected/b72e3683-af0e-4c04-841f-bac4a0e8be0a-kube-api-access-kr6xz\") pod \"istiod-openshift-gateway-7b7995d5cd-fnsw4\" (UID: \"b72e3683-af0e-4c04-841f-bac4a0e8be0a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-fnsw4"
Sep 11 13:27:51.993001 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:27:51.992807    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b72e3683-af0e-4c04-841f-bac4a0e8be0a-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-fnsw4\" (UID: \"b72e3683-af0e-4c04-841f-bac4a0e8be0a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-fnsw4"
Sep 11 13:27:51.993001 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:27:51.992864    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b72e3683-af0e-4c04-841f-bac4a0e8be0a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-fnsw4\" (UID: \"b72e3683-af0e-4c04-841f-bac4a0e8be0a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-fnsw4"
Sep 11 13:27:51.993001 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:27:51.992926    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b72e3683-af0e-4c04-841f-bac4a0e8be0a-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-fnsw4\" (UID: \"b72e3683-af0e-4c04-841f-bac4a0e8be0a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-fnsw4"
Sep 11 13:27:51.993001 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:27:51.992948    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b72e3683-af0e-4c04-841f-bac4a0e8be0a-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-fnsw4\" (UID: \"b72e3683-af0e-4c04-841f-bac4a0e8be0a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-fnsw4"
Sep 11 13:27:51.993001 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:27:51.992973    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b72e3683-af0e-4c04-841f-bac4a0e8be0a-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-fnsw4\" (UID: \"b72e3683-af0e-4c04-841f-bac4a0e8be0a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-fnsw4"
Sep 11 13:27:51.993260 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:27:51.993011    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b72e3683-af0e-4c04-841f-bac4a0e8be0a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-fnsw4\" (UID: \"b72e3683-af0e-4c04-841f-bac4a0e8be0a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-fnsw4"
Sep 11 13:27:51.993794 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:27:51.993753    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b72e3683-af0e-4c04-841f-bac4a0e8be0a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-fnsw4\" (UID: \"b72e3683-af0e-4c04-841f-bac4a0e8be0a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-fnsw4"
Sep 11 13:27:51.997196 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:27:51.997130    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b72e3683-af0e-4c04-841f-bac4a0e8be0a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-fnsw4\" (UID: \"b72e3683-af0e-4c04-841f-bac4a0e8be0a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-fnsw4"
Sep 11 13:27:51.997196 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:27:51.997163    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b72e3683-af0e-4c04-841f-bac4a0e8be0a-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-fnsw4\" (UID: \"b72e3683-af0e-4c04-841f-bac4a0e8be0a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-fnsw4"
Sep 11 13:27:51.997421 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:27:51.997301    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b72e3683-af0e-4c04-841f-bac4a0e8be0a-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-fnsw4\" (UID: \"b72e3683-af0e-4c04-841f-bac4a0e8be0a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-fnsw4"
Sep 11 13:27:51.997605 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:27:51.997579    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b72e3683-af0e-4c04-841f-bac4a0e8be0a-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-fnsw4\" (UID: \"b72e3683-af0e-4c04-841f-bac4a0e8be0a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-fnsw4"
Sep 11 13:27:52.002212 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:27:52.002179    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b72e3683-af0e-4c04-841f-bac4a0e8be0a-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-fnsw4\" (UID: \"b72e3683-af0e-4c04-841f-bac4a0e8be0a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-fnsw4"
Sep 11 13:27:52.002506 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:27:52.002479    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kr6xz\" (UniqueName: \"kubernetes.io/projected/b72e3683-af0e-4c04-841f-bac4a0e8be0a-kube-api-access-kr6xz\") pod \"istiod-openshift-gateway-7b7995d5cd-fnsw4\" (UID: \"b72e3683-af0e-4c04-841f-bac4a0e8be0a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-fnsw4"
Sep 11 13:27:52.123294 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:27:52.122956    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-fnsw4"
Sep 11 13:27:53.764612 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:27:53.764513    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-fnsw4"]
Sep 11 13:27:53.768749 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:27:53.768708    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb72e3683_af0e_4c04_841f_bac4a0e8be0a.slice/crio-d0f4e8aed6cc134bbeb76e334ef7dfba6ef7ff2b8d7ad7637d4df47e52af82f9 WatchSource:0}: Error finding container d0f4e8aed6cc134bbeb76e334ef7dfba6ef7ff2b8d7ad7637d4df47e52af82f9: Status 404 returned error can't find the container with id d0f4e8aed6cc134bbeb76e334ef7dfba6ef7ff2b8d7ad7637d4df47e52af82f9
Sep 11 13:27:54.090380 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:27:54.090243    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-fnsw4" event={"ID":"b72e3683-af0e-4c04-841f-bac4a0e8be0a","Type":"ContainerStarted","Data":"d0f4e8aed6cc134bbeb76e334ef7dfba6ef7ff2b8d7ad7637d4df47e52af82f9"}
Sep 11 13:27:54.092082 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:27:54.092044    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7c9bdb7954-fwnlt" event={"ID":"d1d8740e-7e95-42db-9247-3613ec035a2d","Type":"ContainerStarted","Data":"ca2fd44d7a8bd38cf64555430768eea3c6e1e704f3d846a48b414485c3d5a2d1"}
Sep 11 13:27:54.111121 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:27:54.111061    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-7c9bdb7954-fwnlt" podStartSLOduration=1.541689106 podStartE2EDuration="4.111042309s" totalImagesPullingTime="2.569353203s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:27:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:27:54.10884437 +0000 UTC m=+381.494947548" watchObservedRunningTime="2026-09-11 13:27:54.111042309 +0000 UTC m=+381.497145518"
Sep 11 13:27:55.962796 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:27:55.962748    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 11 13:27:55.963230 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:27:55.962894    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 11 13:27:56.105602 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:27:56.105552    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-fnsw4" event={"ID":"b72e3683-af0e-4c04-841f-bac4a0e8be0a","Type":"ContainerStarted","Data":"2b94a3d63cfc29bdda5a2c5d736c301a8211e02e32d14554a5fb5be093506b3f"}
Sep 11 13:27:56.105889 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:27:56.105738    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-fnsw4"
Sep 11 13:27:56.140951 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:27:56.140880    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-fnsw4" podStartSLOduration=2.949781444 podStartE2EDuration="5.140854205s" totalImagesPullingTime="2.191072761s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:27:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:27:56.138947794 +0000 UTC m=+383.525050977" watchObservedRunningTime="2026-09-11 13:27:56.140854205 +0000 UTC m=+383.526957359"
Sep 11 13:27:57.114112 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:27:57.114061    2578 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-fnsw4 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 11 13:27:57.114626 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:27:57.114137    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-fnsw4" podUID="b72e3683-af0e-4c04-841f-bac4a0e8be0a" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 13:28:00.113130 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:28:00.113076    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-fnsw4"
Sep 11 13:28:12.888591 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:28:12.888551    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-fnsw4"]
Sep 11 13:28:12.889190 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:28:12.888896    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-fnsw4" podUID="b72e3683-af0e-4c04-841f-bac4a0e8be0a" containerName="discovery" containerID="cri-o://2b94a3d63cfc29bdda5a2c5d736c301a8211e02e32d14554a5fb5be093506b3f" gracePeriod=30
Sep 11 13:28:13.149035 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:28:13.148987    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-5clkn"]
Sep 11 13:28:13.153099 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:28:13.153060    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-5clkn"
Sep 11 13:28:13.158368 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:28:13.158335    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Sep 11 13:28:13.158768 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:28:13.158746    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-fnsw4"
Sep 11 13:28:13.159282 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:28:13.159262    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-gsd7h\""
Sep 11 13:28:13.174856 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:28:13.174799    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-42rpr"]
Sep 11 13:28:13.175339 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:28:13.175320    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b72e3683-af0e-4c04-841f-bac4a0e8be0a" containerName="discovery"
Sep 11 13:28:13.175339 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:28:13.175342    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b72e3683-af0e-4c04-841f-bac4a0e8be0a" containerName="discovery"
Sep 11 13:28:13.175623 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:28:13.175472    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="b72e3683-af0e-4c04-841f-bac4a0e8be0a" containerName="discovery"
Sep 11 13:28:13.179097 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:28:13.179067    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-42rpr"
Sep 11 13:28:13.195552 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:28:13.195403    2578 generic.go:356] "Generic (PLEG): container finished" podID="b72e3683-af0e-4c04-841f-bac4a0e8be0a" containerID="2b94a3d63cfc29bdda5a2c5d736c301a8211e02e32d14554a5fb5be093506b3f" exitCode=0
Sep 11 13:28:13.195552 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:28:13.195448    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-fnsw4" event={"ID":"b72e3683-af0e-4c04-841f-bac4a0e8be0a","Type":"ContainerDied","Data":"2b94a3d63cfc29bdda5a2c5d736c301a8211e02e32d14554a5fb5be093506b3f"}
Sep 11 13:28:13.195552 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:28:13.195459    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-fnsw4"
Sep 11 13:28:13.195552 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:28:13.195493    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-fnsw4" event={"ID":"b72e3683-af0e-4c04-841f-bac4a0e8be0a","Type":"ContainerDied","Data":"d0f4e8aed6cc134bbeb76e334ef7dfba6ef7ff2b8d7ad7637d4df47e52af82f9"}
Sep 11 13:28:13.195552 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:28:13.195521    2578 scope.go:117] "RemoveContainer" containerID="2b94a3d63cfc29bdda5a2c5d736c301a8211e02e32d14554a5fb5be093506b3f"
Sep 11 13:28:13.204473 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:28:13.204437    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b72e3683-af0e-4c04-841f-bac4a0e8be0a-cacerts\") pod \"b72e3683-af0e-4c04-841f-bac4a0e8be0a\" (UID: \"b72e3683-af0e-4c04-841f-bac4a0e8be0a\") "
Sep 11 13:28:13.204892 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:28:13.204560    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b72e3683-af0e-4c04-841f-bac4a0e8be0a-istio-kubeconfig\") pod \"b72e3683-af0e-4c04-841f-bac4a0e8be0a\" (UID: \"b72e3683-af0e-4c04-841f-bac4a0e8be0a\") "
Sep 11 13:28:13.204892 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:28:13.204595    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b72e3683-af0e-4c04-841f-bac4a0e8be0a-istio-csr-ca-configmap\") pod \"b72e3683-af0e-4c04-841f-bac4a0e8be0a\" (UID: \"b72e3683-af0e-4c04-841f-bac4a0e8be0a\") "
Sep 11 13:28:13.204892 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:28:13.204633    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b72e3683-af0e-4c04-841f-bac4a0e8be0a-istio-csr-dns-cert\") pod \"b72e3683-af0e-4c04-841f-bac4a0e8be0a\" (UID: \"b72e3683-af0e-4c04-841f-bac4a0e8be0a\") "
Sep 11 13:28:13.204892 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:28:13.204677    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kr6xz\" (UniqueName: \"kubernetes.io/projected/b72e3683-af0e-4c04-841f-bac4a0e8be0a-kube-api-access-kr6xz\") pod \"b72e3683-af0e-4c04-841f-bac4a0e8be0a\" (UID: \"b72e3683-af0e-4c04-841f-bac4a0e8be0a\") "
Sep 11 13:28:13.204892 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:28:13.204704    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b72e3683-af0e-4c04-841f-bac4a0e8be0a-istio-token\") pod \"b72e3683-af0e-4c04-841f-bac4a0e8be0a\" (UID: \"b72e3683-af0e-4c04-841f-bac4a0e8be0a\") "
Sep 11 13:28:13.204892 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:28:13.204739    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b72e3683-af0e-4c04-841f-bac4a0e8be0a-local-certs\") pod \"b72e3683-af0e-4c04-841f-bac4a0e8be0a\" (UID: \"b72e3683-af0e-4c04-841f-bac4a0e8be0a\") "
Sep 11 13:28:13.205246 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:28:13.204956    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d97fd737-e7b3-4ffa-a705-a800fa9090fb-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-42rpr\" (UID: \"d97fd737-e7b3-4ffa-a705-a800fa9090fb\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-42rpr"
Sep 11 13:28:13.205246 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:28:13.205098    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjt55\" (UniqueName: \"kubernetes.io/projected/d97fd737-e7b3-4ffa-a705-a800fa9090fb-kube-api-access-bjt55\") pod \"llmisvc-controller-manager-777c6bd7d5-42rpr\" (UID: \"d97fd737-e7b3-4ffa-a705-a800fa9090fb\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-42rpr"
Sep 11 13:28:13.205246 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:28:13.205134    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/723fc010-ed76-4913-8dcc-61e0a16e063a-cert\") pod \"kserve-controller-manager-d6f4b65b8-5clkn\" (UID: \"723fc010-ed76-4913-8dcc-61e0a16e063a\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-5clkn"
Sep 11 13:28:13.205246 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:28:13.205159    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8sd58\" (UniqueName: \"kubernetes.io/projected/723fc010-ed76-4913-8dcc-61e0a16e063a-kube-api-access-8sd58\") pod \"kserve-controller-manager-d6f4b65b8-5clkn\" (UID: \"723fc010-ed76-4913-8dcc-61e0a16e063a\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-5clkn"
Sep 11 13:28:13.205627 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:28:13.205330    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b72e3683-af0e-4c04-841f-bac4a0e8be0a-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "b72e3683-af0e-4c04-841f-bac4a0e8be0a" (UID: "b72e3683-af0e-4c04-841f-bac4a0e8be0a"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 13:28:13.208286 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:28:13.208235    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b72e3683-af0e-4c04-841f-bac4a0e8be0a-cacerts" (OuterVolumeSpecName: "cacerts") pod "b72e3683-af0e-4c04-841f-bac4a0e8be0a" (UID: "b72e3683-af0e-4c04-841f-bac4a0e8be0a"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 13:28:13.208471 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:28:13.208277    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b72e3683-af0e-4c04-841f-bac4a0e8be0a-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "b72e3683-af0e-4c04-841f-bac4a0e8be0a" (UID: "b72e3683-af0e-4c04-841f-bac4a0e8be0a"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 13:28:13.209146 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:28:13.209085    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b72e3683-af0e-4c04-841f-bac4a0e8be0a-local-certs" (OuterVolumeSpecName: "local-certs") pod "b72e3683-af0e-4c04-841f-bac4a0e8be0a" (UID: "b72e3683-af0e-4c04-841f-bac4a0e8be0a"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 13:28:13.209309 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:28:13.208822    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b72e3683-af0e-4c04-841f-bac4a0e8be0a-istio-token" (OuterVolumeSpecName: "istio-token") pod "b72e3683-af0e-4c04-841f-bac4a0e8be0a" (UID: "b72e3683-af0e-4c04-841f-bac4a0e8be0a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 13:28:13.210392 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:28:13.210364    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b72e3683-af0e-4c04-841f-bac4a0e8be0a-kube-api-access-kr6xz" (OuterVolumeSpecName: "kube-api-access-kr6xz") pod "b72e3683-af0e-4c04-841f-bac4a0e8be0a" (UID: "b72e3683-af0e-4c04-841f-bac4a0e8be0a"). InnerVolumeSpecName "kube-api-access-kr6xz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 13:28:13.211711 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:28:13.211571    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b72e3683-af0e-4c04-841f-bac4a0e8be0a-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "b72e3683-af0e-4c04-841f-bac4a0e8be0a" (UID: "b72e3683-af0e-4c04-841f-bac4a0e8be0a"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 13:28:13.211711 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:28:13.211648    2578 scope.go:117] "RemoveContainer" containerID="2b94a3d63cfc29bdda5a2c5d736c301a8211e02e32d14554a5fb5be093506b3f"
Sep 11 13:28:13.212206 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:28:13.212112    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b94a3d63cfc29bdda5a2c5d736c301a8211e02e32d14554a5fb5be093506b3f\": container with ID starting with 2b94a3d63cfc29bdda5a2c5d736c301a8211e02e32d14554a5fb5be093506b3f not found: ID does not exist" containerID="2b94a3d63cfc29bdda5a2c5d736c301a8211e02e32d14554a5fb5be093506b3f"
Sep 11 13:28:13.212206 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:28:13.212153    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b94a3d63cfc29bdda5a2c5d736c301a8211e02e32d14554a5fb5be093506b3f"} err="failed to get container status \"2b94a3d63cfc29bdda5a2c5d736c301a8211e02e32d14554a5fb5be093506b3f\": rpc error: code = NotFound desc = could not find container \"2b94a3d63cfc29bdda5a2c5d736c301a8211e02e32d14554a5fb5be093506b3f\": container with ID starting with 2b94a3d63cfc29bdda5a2c5d736c301a8211e02e32d14554a5fb5be093506b3f not found: ID does not exist"
Sep 11 13:28:13.212206 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:28:13.212170    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Sep 11 13:28:13.215316 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:28:13.215286    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-5clkn"]
Sep 11 13:28:13.219424 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:28:13.219395    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-kbhkw\""
Sep 11 13:28:13.277320 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:28:13.277195    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-42rpr"]
Sep 11 13:28:13.306342 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:28:13.306278    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d97fd737-e7b3-4ffa-a705-a800fa9090fb-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-42rpr\" (UID: \"d97fd737-e7b3-4ffa-a705-a800fa9090fb\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-42rpr"
Sep 11 13:28:13.306645 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:28:13.306408    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjt55\" (UniqueName: \"kubernetes.io/projected/d97fd737-e7b3-4ffa-a705-a800fa9090fb-kube-api-access-bjt55\") pod \"llmisvc-controller-manager-777c6bd7d5-42rpr\" (UID: \"d97fd737-e7b3-4ffa-a705-a800fa9090fb\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-42rpr"
Sep 11 13:28:13.306645 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:28:13.306446    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/723fc010-ed76-4913-8dcc-61e0a16e063a-cert\") pod \"kserve-controller-manager-d6f4b65b8-5clkn\" (UID: \"723fc010-ed76-4913-8dcc-61e0a16e063a\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-5clkn"
Sep 11 13:28:13.306645 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:28:13.306469    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8sd58\" (UniqueName: \"kubernetes.io/projected/723fc010-ed76-4913-8dcc-61e0a16e063a-kube-api-access-8sd58\") pod \"kserve-controller-manager-d6f4b65b8-5clkn\" (UID: \"723fc010-ed76-4913-8dcc-61e0a16e063a\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-5clkn"
Sep 11 13:28:13.306645 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:28:13.306603    2578 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b72e3683-af0e-4c04-841f-bac4a0e8be0a-istio-kubeconfig\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:28:13.306645 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:28:13.306618    2578 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b72e3683-af0e-4c04-841f-bac4a0e8be0a-istio-csr-ca-configmap\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:28:13.306645 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:28:13.306634    2578 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b72e3683-af0e-4c04-841f-bac4a0e8be0a-istio-csr-dns-cert\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:28:13.306645 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:28:13.306649    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kr6xz\" (UniqueName: \"kubernetes.io/projected/b72e3683-af0e-4c04-841f-bac4a0e8be0a-kube-api-access-kr6xz\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:28:13.307053 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:28:13.306663    2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b72e3683-af0e-4c04-841f-bac4a0e8be0a-istio-token\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:28:13.307053 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:28:13.306687    2578 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b72e3683-af0e-4c04-841f-bac4a0e8be0a-local-certs\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:28:13.307053 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:28:13.306701    2578 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b72e3683-af0e-4c04-841f-bac4a0e8be0a-cacerts\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:28:13.307053 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:28:13.306967    2578 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found
Sep 11 13:28:13.307247 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:28:13.307058    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d97fd737-e7b3-4ffa-a705-a800fa9090fb-cert podName:d97fd737-e7b3-4ffa-a705-a800fa9090fb nodeName:}" failed. No retries permitted until 2026-09-11 13:28:13.807027914 +0000 UTC m=+401.193131077 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/d97fd737-e7b3-4ffa-a705-a800fa9090fb-cert") pod "llmisvc-controller-manager-777c6bd7d5-42rpr" (UID: "d97fd737-e7b3-4ffa-a705-a800fa9090fb") : secret "llmisvc-webhook-server-cert" not found
Sep 11 13:28:13.307247 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:28:13.306974    2578 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found
Sep 11 13:28:13.307247 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:28:13.307138    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/723fc010-ed76-4913-8dcc-61e0a16e063a-cert podName:723fc010-ed76-4913-8dcc-61e0a16e063a nodeName:}" failed. No retries permitted until 2026-09-11 13:28:13.807114452 +0000 UTC m=+401.193217609 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/723fc010-ed76-4913-8dcc-61e0a16e063a-cert") pod "kserve-controller-manager-d6f4b65b8-5clkn" (UID: "723fc010-ed76-4913-8dcc-61e0a16e063a") : secret "kserve-webhook-server-cert" not found
Sep 11 13:28:13.333009 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:28:13.332968    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjt55\" (UniqueName: \"kubernetes.io/projected/d97fd737-e7b3-4ffa-a705-a800fa9090fb-kube-api-access-bjt55\") pod \"llmisvc-controller-manager-777c6bd7d5-42rpr\" (UID: \"d97fd737-e7b3-4ffa-a705-a800fa9090fb\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-42rpr"
Sep 11 13:28:13.359568 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:28:13.359529    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8sd58\" (UniqueName: \"kubernetes.io/projected/723fc010-ed76-4913-8dcc-61e0a16e063a-kube-api-access-8sd58\") pod \"kserve-controller-manager-d6f4b65b8-5clkn\" (UID: \"723fc010-ed76-4913-8dcc-61e0a16e063a\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-5clkn"
Sep 11 13:28:13.564666 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:28:13.564614    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-fnsw4"]
Sep 11 13:28:13.604228 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:28:13.604182    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-fnsw4"]
Sep 11 13:28:13.811640 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:28:13.811519    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/723fc010-ed76-4913-8dcc-61e0a16e063a-cert\") pod \"kserve-controller-manager-d6f4b65b8-5clkn\" (UID: \"723fc010-ed76-4913-8dcc-61e0a16e063a\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-5clkn"
Sep 11 13:28:13.811640 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:28:13.811629    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d97fd737-e7b3-4ffa-a705-a800fa9090fb-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-42rpr\" (UID: \"d97fd737-e7b3-4ffa-a705-a800fa9090fb\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-42rpr"
Sep 11 13:28:13.814916 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:28:13.814870    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d97fd737-e7b3-4ffa-a705-a800fa9090fb-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-42rpr\" (UID: \"d97fd737-e7b3-4ffa-a705-a800fa9090fb\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-42rpr"
Sep 11 13:28:13.814916 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:28:13.814879    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/723fc010-ed76-4913-8dcc-61e0a16e063a-cert\") pod \"kserve-controller-manager-d6f4b65b8-5clkn\" (UID: \"723fc010-ed76-4913-8dcc-61e0a16e063a\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-5clkn"
Sep 11 13:28:14.065909 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:28:14.065725    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-5clkn"
Sep 11 13:28:14.091461 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:28:14.091415    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-42rpr"
Sep 11 13:28:14.362992 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:28:14.362957    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-5clkn"]
Sep 11 13:28:14.366575 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:28:14.366532    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod723fc010_ed76_4913_8dcc_61e0a16e063a.slice/crio-9e6442fc1d817d44e6356a615f088e477f08ae14cda0cb21e37b3a668940bef8 WatchSource:0}: Error finding container 9e6442fc1d817d44e6356a615f088e477f08ae14cda0cb21e37b3a668940bef8: Status 404 returned error can't find the container with id 9e6442fc1d817d44e6356a615f088e477f08ae14cda0cb21e37b3a668940bef8
Sep 11 13:28:14.394056 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:28:14.394025    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-42rpr"]
Sep 11 13:28:14.397052 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:28:14.397011    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podd97fd737_e7b3_4ffa_a705_a800fa9090fb.slice/crio-27ba84cea5509658aad948503a9ef68796397156c2ebc6975ca0b37ad00328d2 WatchSource:0}: Error finding container 27ba84cea5509658aad948503a9ef68796397156c2ebc6975ca0b37ad00328d2: Status 404 returned error can't find the container with id 27ba84cea5509658aad948503a9ef68796397156c2ebc6975ca0b37ad00328d2
Sep 11 13:28:15.109526 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:28:15.109407    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b72e3683-af0e-4c04-841f-bac4a0e8be0a" path="/var/lib/kubelet/pods/b72e3683-af0e-4c04-841f-bac4a0e8be0a/volumes"
Sep 11 13:28:15.212466 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:28:15.212407    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-5clkn" event={"ID":"723fc010-ed76-4913-8dcc-61e0a16e063a","Type":"ContainerStarted","Data":"9e6442fc1d817d44e6356a615f088e477f08ae14cda0cb21e37b3a668940bef8"}
Sep 11 13:28:15.214556 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:28:15.214509    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-42rpr" event={"ID":"d97fd737-e7b3-4ffa-a705-a800fa9090fb","Type":"ContainerStarted","Data":"27ba84cea5509658aad948503a9ef68796397156c2ebc6975ca0b37ad00328d2"}
Sep 11 13:28:18.242547 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:28:18.242509    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-42rpr" event={"ID":"d97fd737-e7b3-4ffa-a705-a800fa9090fb","Type":"ContainerStarted","Data":"e60b3e25e1b522871cd8c54e40358663117e0ad2dd231a7f0bc46de92112def9"}
Sep 11 13:28:18.243142 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:28:18.242589    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-42rpr"
Sep 11 13:28:18.244307 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:28:18.244280    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-5clkn" event={"ID":"723fc010-ed76-4913-8dcc-61e0a16e063a","Type":"ContainerStarted","Data":"9d729eea77d96551d81a7bfe677d8a07b825a3b271fdf7b50db1ae741a12a32f"}
Sep 11 13:28:18.244437 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:28:18.244348    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-5clkn"
Sep 11 13:28:18.279537 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:28:18.279468    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-42rpr" podStartSLOduration=2.472577422 podStartE2EDuration="5.279446862s" totalImagesPullingTime="2.80686944s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:28:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:28:18.278245544 +0000 UTC m=+405.664348722" watchObservedRunningTime="2026-09-11 13:28:18.279446862 +0000 UTC m=+405.665550117"
Sep 11 13:28:49.252926 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:28:49.252887    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-42rpr"
Sep 11 13:28:49.256127 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:28:49.256078    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-5clkn"
Sep 11 13:28:49.272262 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:28:49.272182    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-5clkn" podStartSLOduration=33.489599865 podStartE2EDuration="36.272160106s" totalImagesPullingTime="2.782560241s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:28:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:28:18.33516285 +0000 UTC m=+405.721266034" watchObservedRunningTime="2026-09-11 13:28:49.272160106 +0000 UTC m=+436.658263270"
Sep 11 13:29:19.568135 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:29:19.568087    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-8tfmj"]
Sep 11 13:29:19.571472 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:29:19.571444    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-8tfmj"
Sep 11 13:29:19.574988 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:29:19.574951    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 11 13:29:19.575214 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:29:19.575017    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-mmj2x\""
Sep 11 13:29:19.586161 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:29:19.586128    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxmsw\" (UniqueName: \"kubernetes.io/projected/ce37c0ef-3e90-415d-bf21-ec4eef68401e-kube-api-access-fxmsw\") pod \"authorino-operator-76b478bd78-8tfmj\" (UID: \"ce37c0ef-3e90-415d-bf21-ec4eef68401e\") " pod="kuadrant-system/authorino-operator-76b478bd78-8tfmj"
Sep 11 13:29:19.588547 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:29:19.588524    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 11 13:29:19.599536 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:29:19.599494    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-8tfmj"]
Sep 11 13:29:19.687771 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:29:19.687693    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxmsw\" (UniqueName: \"kubernetes.io/projected/ce37c0ef-3e90-415d-bf21-ec4eef68401e-kube-api-access-fxmsw\") pod \"authorino-operator-76b478bd78-8tfmj\" (UID: \"ce37c0ef-3e90-415d-bf21-ec4eef68401e\") " pod="kuadrant-system/authorino-operator-76b478bd78-8tfmj"
Sep 11 13:29:19.722117 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:29:19.722072    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxmsw\" (UniqueName: \"kubernetes.io/projected/ce37c0ef-3e90-415d-bf21-ec4eef68401e-kube-api-access-fxmsw\") pod \"authorino-operator-76b478bd78-8tfmj\" (UID: \"ce37c0ef-3e90-415d-bf21-ec4eef68401e\") " pod="kuadrant-system/authorino-operator-76b478bd78-8tfmj"
Sep 11 13:29:19.884518 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:29:19.884390    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-8tfmj"
Sep 11 13:29:20.188732 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:29:20.188701    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-8tfmj"]
Sep 11 13:29:20.191363 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:29:20.191320    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce37c0ef_3e90_415d_bf21_ec4eef68401e.slice/crio-e8b0679bf0aaa55f5aad40ee48d95e35e0aa1539b805de27259fdccb8df82a2a WatchSource:0}: Error finding container e8b0679bf0aaa55f5aad40ee48d95e35e0aa1539b805de27259fdccb8df82a2a: Status 404 returned error can't find the container with id e8b0679bf0aaa55f5aad40ee48d95e35e0aa1539b805de27259fdccb8df82a2a
Sep 11 13:29:20.583516 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:29:20.583473    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-8tfmj" event={"ID":"ce37c0ef-3e90-415d-bf21-ec4eef68401e","Type":"ContainerStarted","Data":"e8b0679bf0aaa55f5aad40ee48d95e35e0aa1539b805de27259fdccb8df82a2a"}
Sep 11 13:29:23.606243 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:29:23.606197    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-8tfmj" event={"ID":"ce37c0ef-3e90-415d-bf21-ec4eef68401e","Type":"ContainerStarted","Data":"792720f29f1d82f965c096a71f35c6f97ac2d22a40116ae8ba60a0469fecf412"}
Sep 11 13:29:23.606764 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:29:23.606289    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-76b478bd78-8tfmj"
Sep 11 13:29:23.631340 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:29:23.631267    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-76b478bd78-8tfmj" podStartSLOduration=2.118838418 podStartE2EDuration="4.631246607s" totalImagesPullingTime="2.512408189s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:29:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:29:23.629215435 +0000 UTC m=+471.015318614" watchObservedRunningTime="2026-09-11 13:29:23.631246607 +0000 UTC m=+471.017349770"
Sep 11 13:29:24.317140 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:29:24.317088    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-sh289"]
Sep 11 13:29:24.322631 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:29:24.322605    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-sh289"
Sep 11 13:29:24.325065 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:29:24.325030    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-tws25\""
Sep 11 13:29:24.332661 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:29:24.332632    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-sh289"]
Sep 11 13:29:24.436908 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:29:24.436790    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/da9b7760-8a3d-4094-baed-7d8993ccdaf4-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-sh289\" (UID: \"da9b7760-8a3d-4094-baed-7d8993ccdaf4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-sh289"
Sep 11 13:29:24.437136 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:29:24.436951    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8jb8\" (UniqueName: \"kubernetes.io/projected/da9b7760-8a3d-4094-baed-7d8993ccdaf4-kube-api-access-w8jb8\") pod \"kuadrant-operator-controller-manager-76f8f686bf-sh289\" (UID: \"da9b7760-8a3d-4094-baed-7d8993ccdaf4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-sh289"
Sep 11 13:29:24.537991 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:29:24.537935    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8jb8\" (UniqueName: \"kubernetes.io/projected/da9b7760-8a3d-4094-baed-7d8993ccdaf4-kube-api-access-w8jb8\") pod \"kuadrant-operator-controller-manager-76f8f686bf-sh289\" (UID: \"da9b7760-8a3d-4094-baed-7d8993ccdaf4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-sh289"
Sep 11 13:29:24.538258 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:29:24.538073    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/da9b7760-8a3d-4094-baed-7d8993ccdaf4-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-sh289\" (UID: \"da9b7760-8a3d-4094-baed-7d8993ccdaf4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-sh289"
Sep 11 13:29:24.538525 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:29:24.538493    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/da9b7760-8a3d-4094-baed-7d8993ccdaf4-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-sh289\" (UID: \"da9b7760-8a3d-4094-baed-7d8993ccdaf4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-sh289"
Sep 11 13:29:24.560494 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:29:24.560447    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8jb8\" (UniqueName: \"kubernetes.io/projected/da9b7760-8a3d-4094-baed-7d8993ccdaf4-kube-api-access-w8jb8\") pod \"kuadrant-operator-controller-manager-76f8f686bf-sh289\" (UID: \"da9b7760-8a3d-4094-baed-7d8993ccdaf4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-sh289"
Sep 11 13:29:24.636148 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:29:24.636010    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-sh289"
Sep 11 13:29:24.952946 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:29:24.952812    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-sh289"]
Sep 11 13:29:24.956582 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:29:24.956545    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda9b7760_8a3d_4094_baed_7d8993ccdaf4.slice/crio-db189d7f003402ee7ad7c8965748597c36c363aa21f4468a19be069afd556b80 WatchSource:0}: Error finding container db189d7f003402ee7ad7c8965748597c36c363aa21f4468a19be069afd556b80: Status 404 returned error can't find the container with id db189d7f003402ee7ad7c8965748597c36c363aa21f4468a19be069afd556b80
Sep 11 13:29:25.626129 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:29:25.626089    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-sh289" event={"ID":"da9b7760-8a3d-4094-baed-7d8993ccdaf4","Type":"ContainerStarted","Data":"db189d7f003402ee7ad7c8965748597c36c363aa21f4468a19be069afd556b80"}
Sep 11 13:29:26.489064 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:29:26.489023    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-g7qtw"]
Sep 11 13:29:26.493856 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:29:26.493802    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-g7qtw"
Sep 11 13:29:26.497509 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:29:26.497435    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Sep 11 13:29:26.497509 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:29:26.497456    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-97l9m\""
Sep 11 13:29:26.508476 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:29:26.508440    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-g7qtw"]
Sep 11 13:29:26.568471 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:29:26.568410    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m57q4\" (UniqueName: \"kubernetes.io/projected/adcd135f-bce8-410d-a231-0ae0660f8e32-kube-api-access-m57q4\") pod \"dns-operator-controller-manager-7b84cd7c9c-g7qtw\" (UID: \"adcd135f-bce8-410d-a231-0ae0660f8e32\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-g7qtw"
Sep 11 13:29:26.669779 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:29:26.669725    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m57q4\" (UniqueName: \"kubernetes.io/projected/adcd135f-bce8-410d-a231-0ae0660f8e32-kube-api-access-m57q4\") pod \"dns-operator-controller-manager-7b84cd7c9c-g7qtw\" (UID: \"adcd135f-bce8-410d-a231-0ae0660f8e32\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-g7qtw"
Sep 11 13:29:26.680337 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:29:26.680294    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m57q4\" (UniqueName: \"kubernetes.io/projected/adcd135f-bce8-410d-a231-0ae0660f8e32-kube-api-access-m57q4\") pod \"dns-operator-controller-manager-7b84cd7c9c-g7qtw\" (UID: \"adcd135f-bce8-410d-a231-0ae0660f8e32\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-g7qtw"
Sep 11 13:29:26.809805 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:29:26.809685    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-g7qtw"
Sep 11 13:29:27.119207 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:29:27.119168    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-g7qtw"]
Sep 11 13:29:27.122787 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:29:27.122747    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadcd135f_bce8_410d_a231_0ae0660f8e32.slice/crio-5098554b40363dab1a4470182ef783a161a0e7f9892eb2e4718c120f9ab7a9ed WatchSource:0}: Error finding container 5098554b40363dab1a4470182ef783a161a0e7f9892eb2e4718c120f9ab7a9ed: Status 404 returned error can't find the container with id 5098554b40363dab1a4470182ef783a161a0e7f9892eb2e4718c120f9ab7a9ed
Sep 11 13:29:27.643475 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:29:27.643431    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-g7qtw" event={"ID":"adcd135f-bce8-410d-a231-0ae0660f8e32","Type":"ContainerStarted","Data":"5098554b40363dab1a4470182ef783a161a0e7f9892eb2e4718c120f9ab7a9ed"}
Sep 11 13:29:28.662217 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:29:28.662143    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-xz6w7"]
Sep 11 13:29:28.669097 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:29:28.669059    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-xz6w7"
Sep 11 13:29:28.672164 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:29:28.672133    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-hdv9f\""
Sep 11 13:29:28.672801 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:29:28.672761    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-xz6w7"]
Sep 11 13:29:28.790381 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:29:28.790332    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ft6cd\" (UniqueName: \"kubernetes.io/projected/a1afae03-2bfc-4376-b11b-c47a690c130a-kube-api-access-ft6cd\") pod \"limitador-operator-controller-manager-74495c69f8-xz6w7\" (UID: \"a1afae03-2bfc-4376-b11b-c47a690c130a\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-xz6w7"
Sep 11 13:29:28.892057 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:29:28.891888    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ft6cd\" (UniqueName: \"kubernetes.io/projected/a1afae03-2bfc-4376-b11b-c47a690c130a-kube-api-access-ft6cd\") pod \"limitador-operator-controller-manager-74495c69f8-xz6w7\" (UID: \"a1afae03-2bfc-4376-b11b-c47a690c130a\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-xz6w7"
Sep 11 13:29:28.905089 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:29:28.905041    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ft6cd\" (UniqueName: \"kubernetes.io/projected/a1afae03-2bfc-4376-b11b-c47a690c130a-kube-api-access-ft6cd\") pod \"limitador-operator-controller-manager-74495c69f8-xz6w7\" (UID: \"a1afae03-2bfc-4376-b11b-c47a690c130a\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-xz6w7"
Sep 11 13:29:28.985288 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:29:28.985237    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-xz6w7"
Sep 11 13:29:29.296448 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:29:29.296380    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-xz6w7"]
Sep 11 13:29:29.440760 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:29:29.440713    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1afae03_2bfc_4376_b11b_c47a690c130a.slice/crio-f5e33616a5cf6d9d1d4496f701206862ce70d15a1392de22159095725007fff3 WatchSource:0}: Error finding container f5e33616a5cf6d9d1d4496f701206862ce70d15a1392de22159095725007fff3: Status 404 returned error can't find the container with id f5e33616a5cf6d9d1d4496f701206862ce70d15a1392de22159095725007fff3
Sep 11 13:29:29.659436 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:29:29.659389    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-sh289" event={"ID":"da9b7760-8a3d-4094-baed-7d8993ccdaf4","Type":"ContainerStarted","Data":"63051e2340663d9a1684571c8e016be5afde4216b763b73323daacaa0739ebd2"}
Sep 11 13:29:29.659694 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:29:29.659464    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-sh289"
Sep 11 13:29:29.661099 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:29:29.661063    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-xz6w7" event={"ID":"a1afae03-2bfc-4376-b11b-c47a690c130a","Type":"ContainerStarted","Data":"f5e33616a5cf6d9d1d4496f701206862ce70d15a1392de22159095725007fff3"}
Sep 11 13:29:29.662952 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:29:29.662921    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-g7qtw" event={"ID":"adcd135f-bce8-410d-a231-0ae0660f8e32","Type":"ContainerStarted","Data":"6731fa361ef46597f9213e359911bc4842a9001f9e605f1279444d73fe9a7faf"}
Sep 11 13:29:29.663350 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:29:29.663064    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-g7qtw"
Sep 11 13:29:29.690203 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:29:29.690126    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-sh289" podStartSLOduration=2.067419966 podStartE2EDuration="5.69009429s" totalImagesPullingTime="3.622674324s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:29:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:29:29.684527534 +0000 UTC m=+477.070630710" watchObservedRunningTime="2026-09-11 13:29:29.69009429 +0000 UTC m=+477.076197456"
Sep 11 13:29:29.711167 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:29:29.711099    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-g7qtw" podStartSLOduration=1.334468823 podStartE2EDuration="3.711074686s" totalImagesPullingTime="2.376605863s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:29:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:29:29.709507028 +0000 UTC m=+477.095610213" watchObservedRunningTime="2026-09-11 13:29:29.711074686 +0000 UTC m=+477.097177913"
Sep 11 13:29:31.679125 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:29:31.678995    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-xz6w7" event={"ID":"a1afae03-2bfc-4376-b11b-c47a690c130a","Type":"ContainerStarted","Data":"c49ce68f880a870c81ee00d8ebb406da3e87575984b46f7153aabaa0de7985ff"}
Sep 11 13:29:31.679125 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:29:31.679089    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-xz6w7"
Sep 11 13:29:31.702598 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:29:31.702500    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-xz6w7" podStartSLOduration=1.888993384 podStartE2EDuration="3.702473288s" totalImagesPullingTime="1.813479904s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:29:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:29:31.701467414 +0000 UTC m=+479.087570597" watchObservedRunningTime="2026-09-11 13:29:31.702473288 +0000 UTC m=+479.088576452"
Sep 11 13:29:34.622134 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:29:34.622090    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-76b478bd78-8tfmj"
Sep 11 13:29:40.673167 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:29:40.673122    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-g7qtw"
Sep 11 13:29:40.673659 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:29:40.673218    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-sh289"
Sep 11 13:29:42.687160 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:29:42.687118    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-xz6w7"
Sep 11 13:29:43.539601 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:29:43.539552    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-sh289"]
Sep 11 13:29:43.539902 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:29:43.539846    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-sh289" podUID="da9b7760-8a3d-4094-baed-7d8993ccdaf4" containerName="manager" containerID="cri-o://63051e2340663d9a1684571c8e016be5afde4216b763b73323daacaa0739ebd2" gracePeriod=2
Sep 11 13:29:43.570009 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:29:43.569963    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-sh289"]
Sep 11 13:29:43.575044 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:29:43.574989    2578 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"da9b7760-8a3d-4094-baed-7d8993ccdaf4\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-76f8f686bf-sh289\": pods \"kuadrant-operator-controller-manager-76f8f686bf-sh289\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-sh289"
Sep 11 13:29:43.594896 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:29:43.594823    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-xz6w7"]
Sep 11 13:29:43.595114 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:29:43.595088    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-xz6w7" podUID="a1afae03-2bfc-4376-b11b-c47a690c130a" containerName="manager" containerID="cri-o://c49ce68f880a870c81ee00d8ebb406da3e87575984b46f7153aabaa0de7985ff" gracePeriod=2
Sep 11 13:29:43.597203 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:29:43.597160    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-sh289\" is forbidden: User \"system:node:ip-10-0-131-86.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-86.ec2.internal' and this object" podUID="da9b7760-8a3d-4094-baed-7d8993ccdaf4" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-sh289"
Sep 11 13:29:43.612534 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:29:43.612490    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-xz6w7"]
Sep 11 13:29:43.623343 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:29:43.623281    2578 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"a1afae03-2bfc-4376-b11b-c47a690c130a\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"limitador-operator-controller-manager-74495c69f8-xz6w7\": pods \"limitador-operator-controller-manager-74495c69f8-xz6w7\" not found" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-xz6w7"
Sep 11 13:29:43.628940 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:29:43.628904    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-k9c7d"]
Sep 11 13:29:43.629352 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:29:43.629339    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da9b7760-8a3d-4094-baed-7d8993ccdaf4" containerName="manager"
Sep 11 13:29:43.629405 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:29:43.629355    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="da9b7760-8a3d-4094-baed-7d8993ccdaf4" containerName="manager"
Sep 11 13:29:43.629405 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:29:43.629383    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1afae03-2bfc-4376-b11b-c47a690c130a" containerName="manager"
Sep 11 13:29:43.629405 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:29:43.629390    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1afae03-2bfc-4376-b11b-c47a690c130a" containerName="manager"
Sep 11 13:29:43.629495 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:29:43.629461    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="a1afae03-2bfc-4376-b11b-c47a690c130a" containerName="manager"
Sep 11 13:29:43.629495 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:29:43.629470    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="da9b7760-8a3d-4094-baed-7d8993ccdaf4" containerName="manager"
Sep 11 13:29:43.632758 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:29:43.632732    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-k9c7d"
Sep 11 13:29:43.636081 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:29:43.636030    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-sh289\" is forbidden: User \"system:node:ip-10-0-131-86.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-86.ec2.internal' and this object" podUID="da9b7760-8a3d-4094-baed-7d8993ccdaf4" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-sh289"
Sep 11 13:29:43.640438 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:29:43.640399    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-xxs4x"]
Sep 11 13:29:43.640636 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:29:43.640436    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-xz6w7\" is forbidden: User \"system:node:ip-10-0-131-86.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-86.ec2.internal' and this object" podUID="a1afae03-2bfc-4376-b11b-c47a690c130a" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-xz6w7"
Sep 11 13:29:43.644129 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:29:43.644100    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-xxs4x"
Sep 11 13:29:43.649103 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:29:43.649057    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/61b9b730-2b90-4bda-976e-ac2c07e38f2b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-k9c7d\" (UID: \"61b9b730-2b90-4bda-976e-ac2c07e38f2b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-k9c7d"
Sep 11 13:29:43.649322 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:29:43.649142    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pllj\" (UniqueName: \"kubernetes.io/projected/61b9b730-2b90-4bda-976e-ac2c07e38f2b-kube-api-access-9pllj\") pod \"kuadrant-operator-controller-manager-76f8f686bf-k9c7d\" (UID: \"61b9b730-2b90-4bda-976e-ac2c07e38f2b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-k9c7d"
Sep 11 13:29:43.649322 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:29:43.649219    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgtq8\" (UniqueName: \"kubernetes.io/projected/a5d2f9a4-004d-4efe-b282-6ba0a4621aa8-kube-api-access-pgtq8\") pod \"limitador-operator-controller-manager-74495c69f8-xxs4x\" (UID: \"a5d2f9a4-004d-4efe-b282-6ba0a4621aa8\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-xxs4x"
Sep 11 13:29:43.658265 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:29:43.658222    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-k9c7d"]
Sep 11 13:29:43.668947 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:29:43.668914    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-xxs4x"]
Sep 11 13:29:43.749750 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:29:43.749704    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/61b9b730-2b90-4bda-976e-ac2c07e38f2b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-k9c7d\" (UID: \"61b9b730-2b90-4bda-976e-ac2c07e38f2b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-k9c7d"
Sep 11 13:29:43.749750 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:29:43.749762    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pllj\" (UniqueName: \"kubernetes.io/projected/61b9b730-2b90-4bda-976e-ac2c07e38f2b-kube-api-access-9pllj\") pod \"kuadrant-operator-controller-manager-76f8f686bf-k9c7d\" (UID: \"61b9b730-2b90-4bda-976e-ac2c07e38f2b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-k9c7d"
Sep 11 13:29:43.750391 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:29:43.749889    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgtq8\" (UniqueName: \"kubernetes.io/projected/a5d2f9a4-004d-4efe-b282-6ba0a4621aa8-kube-api-access-pgtq8\") pod \"limitador-operator-controller-manager-74495c69f8-xxs4x\" (UID: \"a5d2f9a4-004d-4efe-b282-6ba0a4621aa8\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-xxs4x"
Sep 11 13:29:43.750391 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:29:43.750254    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/61b9b730-2b90-4bda-976e-ac2c07e38f2b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-k9c7d\" (UID: \"61b9b730-2b90-4bda-976e-ac2c07e38f2b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-k9c7d"
Sep 11 13:29:43.753239 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:29:43.753201    2578 generic.go:356] "Generic (PLEG): container finished" podID="da9b7760-8a3d-4094-baed-7d8993ccdaf4" containerID="63051e2340663d9a1684571c8e016be5afde4216b763b73323daacaa0739ebd2" exitCode=0
Sep 11 13:29:43.756192 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:29:43.756157    2578 generic.go:356] "Generic (PLEG): container finished" podID="a1afae03-2bfc-4376-b11b-c47a690c130a" containerID="c49ce68f880a870c81ee00d8ebb406da3e87575984b46f7153aabaa0de7985ff" exitCode=0
Sep 11 13:29:43.758265 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:29:43.758232    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-sh289\" is forbidden: User \"system:node:ip-10-0-131-86.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-86.ec2.internal' and this object" podUID="da9b7760-8a3d-4094-baed-7d8993ccdaf4" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-sh289"
Sep 11 13:29:43.763045 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:29:43.763005    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-xz6w7\" is forbidden: User \"system:node:ip-10-0-131-86.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-86.ec2.internal' and this object" podUID="a1afae03-2bfc-4376-b11b-c47a690c130a" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-xz6w7"
Sep 11 13:29:43.775488 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:29:43.775427    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgtq8\" (UniqueName: \"kubernetes.io/projected/a5d2f9a4-004d-4efe-b282-6ba0a4621aa8-kube-api-access-pgtq8\") pod \"limitador-operator-controller-manager-74495c69f8-xxs4x\" (UID: \"a5d2f9a4-004d-4efe-b282-6ba0a4621aa8\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-xxs4x"
Sep 11 13:29:43.775488 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:29:43.775427    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pllj\" (UniqueName: \"kubernetes.io/projected/61b9b730-2b90-4bda-976e-ac2c07e38f2b-kube-api-access-9pllj\") pod \"kuadrant-operator-controller-manager-76f8f686bf-k9c7d\" (UID: \"61b9b730-2b90-4bda-976e-ac2c07e38f2b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-k9c7d"
Sep 11 13:29:43.872698 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:29:43.872655    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-sh289"
Sep 11 13:29:43.876002 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:29:43.875969    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-xz6w7"
Sep 11 13:29:43.876324 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:29:43.876295    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-sh289\" is forbidden: User \"system:node:ip-10-0-131-86.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-86.ec2.internal' and this object" podUID="da9b7760-8a3d-4094-baed-7d8993ccdaf4" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-sh289"
Sep 11 13:29:43.879917 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:29:43.879879    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-xz6w7\" is forbidden: User \"system:node:ip-10-0-131-86.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-86.ec2.internal' and this object" podUID="a1afae03-2bfc-4376-b11b-c47a690c130a" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-xz6w7"
Sep 11 13:29:43.882053 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:29:43.882019    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-sh289\" is forbidden: User \"system:node:ip-10-0-131-86.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-86.ec2.internal' and this object" podUID="da9b7760-8a3d-4094-baed-7d8993ccdaf4" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-sh289"
Sep 11 13:29:43.884265 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:29:43.884222    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-xz6w7\" is forbidden: User \"system:node:ip-10-0-131-86.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-86.ec2.internal' and this object" podUID="a1afae03-2bfc-4376-b11b-c47a690c130a" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-xz6w7"
Sep 11 13:29:43.951532 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:29:43.951495    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w8jb8\" (UniqueName: \"kubernetes.io/projected/da9b7760-8a3d-4094-baed-7d8993ccdaf4-kube-api-access-w8jb8\") pod \"da9b7760-8a3d-4094-baed-7d8993ccdaf4\" (UID: \"da9b7760-8a3d-4094-baed-7d8993ccdaf4\") "
Sep 11 13:29:43.951730 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:29:43.951549    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/da9b7760-8a3d-4094-baed-7d8993ccdaf4-extensions-socket-volume\") pod \"da9b7760-8a3d-4094-baed-7d8993ccdaf4\" (UID: \"da9b7760-8a3d-4094-baed-7d8993ccdaf4\") "
Sep 11 13:29:43.951730 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:29:43.951621    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ft6cd\" (UniqueName: \"kubernetes.io/projected/a1afae03-2bfc-4376-b11b-c47a690c130a-kube-api-access-ft6cd\") pod \"a1afae03-2bfc-4376-b11b-c47a690c130a\" (UID: \"a1afae03-2bfc-4376-b11b-c47a690c130a\") "
Sep 11 13:29:43.952065 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:29:43.952032    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da9b7760-8a3d-4094-baed-7d8993ccdaf4-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "da9b7760-8a3d-4094-baed-7d8993ccdaf4" (UID: "da9b7760-8a3d-4094-baed-7d8993ccdaf4"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 13:29:43.954269 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:29:43.954237    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1afae03-2bfc-4376-b11b-c47a690c130a-kube-api-access-ft6cd" (OuterVolumeSpecName: "kube-api-access-ft6cd") pod "a1afae03-2bfc-4376-b11b-c47a690c130a" (UID: "a1afae03-2bfc-4376-b11b-c47a690c130a"). InnerVolumeSpecName "kube-api-access-ft6cd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 13:29:43.954394 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:29:43.954281    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da9b7760-8a3d-4094-baed-7d8993ccdaf4-kube-api-access-w8jb8" (OuterVolumeSpecName: "kube-api-access-w8jb8") pod "da9b7760-8a3d-4094-baed-7d8993ccdaf4" (UID: "da9b7760-8a3d-4094-baed-7d8993ccdaf4"). InnerVolumeSpecName "kube-api-access-w8jb8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 13:29:43.977664 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:29:43.977618    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-k9c7d"
Sep 11 13:29:43.985194 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:29:43.985169    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-xxs4x"
Sep 11 13:29:44.052796 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:29:44.052747    2578 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/da9b7760-8a3d-4094-baed-7d8993ccdaf4-extensions-socket-volume\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:29:44.052997 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:29:44.052807    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ft6cd\" (UniqueName: \"kubernetes.io/projected/a1afae03-2bfc-4376-b11b-c47a690c130a-kube-api-access-ft6cd\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:29:44.053080 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:29:44.053051    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w8jb8\" (UniqueName: \"kubernetes.io/projected/da9b7760-8a3d-4094-baed-7d8993ccdaf4-kube-api-access-w8jb8\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:29:44.285556 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:29:44.285409    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-k9c7d"]
Sep 11 13:29:44.287592 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:29:44.287549    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-xxs4x"]
Sep 11 13:29:44.288254 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:29:44.288222    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61b9b730_2b90_4bda_976e_ac2c07e38f2b.slice/crio-44ffa3529edbf37903f3edc9f82ae89427c4346b94cc6e92407f8e2473a3f191 WatchSource:0}: Error finding container 44ffa3529edbf37903f3edc9f82ae89427c4346b94cc6e92407f8e2473a3f191: Status 404 returned error can't find the container with id 44ffa3529edbf37903f3edc9f82ae89427c4346b94cc6e92407f8e2473a3f191
Sep 11 13:29:44.288933 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:29:44.288903    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5d2f9a4_004d_4efe_b282_6ba0a4621aa8.slice/crio-ab655226e1698d50f37595c1b53a72964b7176a0759fadfe500c692848d12185 WatchSource:0}: Error finding container ab655226e1698d50f37595c1b53a72964b7176a0759fadfe500c692848d12185: Status 404 returned error can't find the container with id ab655226e1698d50f37595c1b53a72964b7176a0759fadfe500c692848d12185
Sep 11 13:29:44.765212 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:29:44.765155    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-xxs4x" event={"ID":"a5d2f9a4-004d-4efe-b282-6ba0a4621aa8","Type":"ContainerStarted","Data":"b75258c6236b4954985bc7eaec4a71879519975c879c8691bb96d8c5f3405fec"}
Sep 11 13:29:44.765725 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:29:44.765231    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-xxs4x" event={"ID":"a5d2f9a4-004d-4efe-b282-6ba0a4621aa8","Type":"ContainerStarted","Data":"ab655226e1698d50f37595c1b53a72964b7176a0759fadfe500c692848d12185"}
Sep 11 13:29:44.765725 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:29:44.765269    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-xxs4x"
Sep 11 13:29:44.767019 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:29:44.766987    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-sh289"
Sep 11 13:29:44.767213 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:29:44.766995    2578 scope.go:117] "RemoveContainer" containerID="63051e2340663d9a1684571c8e016be5afde4216b763b73323daacaa0739ebd2"
Sep 11 13:29:44.769474 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:29:44.769433    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-k9c7d" event={"ID":"61b9b730-2b90-4bda-976e-ac2c07e38f2b","Type":"ContainerStarted","Data":"63a8eacc15fa8dddacee7beea80f88b548c0f7a38fcd2f873285c56312ef9354"}
Sep 11 13:29:44.769621 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:29:44.769507    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-k9c7d"
Sep 11 13:29:44.769621 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:29:44.769532    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-k9c7d" event={"ID":"61b9b730-2b90-4bda-976e-ac2c07e38f2b","Type":"ContainerStarted","Data":"44ffa3529edbf37903f3edc9f82ae89427c4346b94cc6e92407f8e2473a3f191"}
Sep 11 13:29:44.771359 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:29:44.771337    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-xz6w7"
Sep 11 13:29:44.779447 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:29:44.779419    2578 scope.go:117] "RemoveContainer" containerID="c49ce68f880a870c81ee00d8ebb406da3e87575984b46f7153aabaa0de7985ff"
Sep 11 13:29:44.847668 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:29:44.847538    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-sh289\" is forbidden: User \"system:node:ip-10-0-131-86.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-86.ec2.internal' and this object" podUID="da9b7760-8a3d-4094-baed-7d8993ccdaf4" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-sh289"
Sep 11 13:29:44.859181 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:29:44.859133    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-xz6w7\" is forbidden: User \"system:node:ip-10-0-131-86.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-86.ec2.internal' and this object" podUID="a1afae03-2bfc-4376-b11b-c47a690c130a" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-xz6w7"
Sep 11 13:29:44.905575 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:29:44.905487    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-k9c7d" podStartSLOduration=1.905453199 podStartE2EDuration="1.905453199s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:29:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:29:44.900628339 +0000 UTC m=+492.286731503" watchObservedRunningTime="2026-09-11 13:29:44.905453199 +0000 UTC m=+492.291556364"
Sep 11 13:29:44.905941 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:29:44.905878    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-xxs4x" podStartSLOduration=1.905865843 podStartE2EDuration="1.905865843s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:29:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:29:44.833214372 +0000 UTC m=+492.219317537" watchObservedRunningTime="2026-09-11 13:29:44.905865843 +0000 UTC m=+492.291969011"
Sep 11 13:29:44.906853 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:29:44.906803    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-sh289\" is forbidden: User \"system:node:ip-10-0-131-86.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-86.ec2.internal' and this object" podUID="da9b7760-8a3d-4094-baed-7d8993ccdaf4" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-sh289"
Sep 11 13:29:44.908195 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:29:44.908176    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-d8lgv"]
Sep 11 13:29:44.910007 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:29:44.909985    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-xz6w7\" is forbidden: User \"system:node:ip-10-0-131-86.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-86.ec2.internal' and this object" podUID="a1afae03-2bfc-4376-b11b-c47a690c130a" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-xz6w7"
Sep 11 13:29:44.913296 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:29:44.913278    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-d8lgv"
Sep 11 13:29:44.917141 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:29:44.917101    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-sh289\" is forbidden: User \"system:node:ip-10-0-131-86.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-86.ec2.internal' and this object" podUID="da9b7760-8a3d-4094-baed-7d8993ccdaf4" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-sh289"
Sep 11 13:29:44.919115 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:29:44.919083    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-xz6w7\" is forbidden: User \"system:node:ip-10-0-131-86.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-86.ec2.internal' and this object" podUID="a1afae03-2bfc-4376-b11b-c47a690c130a" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-xz6w7"
Sep 11 13:29:44.953141 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:29:44.953082    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-d8lgv"]
Sep 11 13:29:44.960030 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:29:44.959991    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/944e63d2-014c-40e1-9b2c-caa5f1259b21-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-d8lgv\" (UID: \"944e63d2-014c-40e1-9b2c-caa5f1259b21\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-d8lgv"
Sep 11 13:29:44.960246 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:29:44.960088    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldj5v\" (UniqueName: \"kubernetes.io/projected/944e63d2-014c-40e1-9b2c-caa5f1259b21-kube-api-access-ldj5v\") pod \"kuadrant-operator-controller-manager-5769f99d9f-d8lgv\" (UID: \"944e63d2-014c-40e1-9b2c-caa5f1259b21\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-d8lgv"
Sep 11 13:29:45.060865 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:29:45.060781    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldj5v\" (UniqueName: \"kubernetes.io/projected/944e63d2-014c-40e1-9b2c-caa5f1259b21-kube-api-access-ldj5v\") pod \"kuadrant-operator-controller-manager-5769f99d9f-d8lgv\" (UID: \"944e63d2-014c-40e1-9b2c-caa5f1259b21\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-d8lgv"
Sep 11 13:29:45.061085 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:29:45.060970    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/944e63d2-014c-40e1-9b2c-caa5f1259b21-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-d8lgv\" (UID: \"944e63d2-014c-40e1-9b2c-caa5f1259b21\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-d8lgv"
Sep 11 13:29:45.061382 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:29:45.061359    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/944e63d2-014c-40e1-9b2c-caa5f1259b21-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-d8lgv\" (UID: \"944e63d2-014c-40e1-9b2c-caa5f1259b21\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-d8lgv"
Sep 11 13:29:45.075241 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:29:45.075194    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldj5v\" (UniqueName: \"kubernetes.io/projected/944e63d2-014c-40e1-9b2c-caa5f1259b21-kube-api-access-ldj5v\") pod \"kuadrant-operator-controller-manager-5769f99d9f-d8lgv\" (UID: \"944e63d2-014c-40e1-9b2c-caa5f1259b21\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-d8lgv"
Sep 11 13:29:45.106757 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:29:45.106666    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a1afae03-2bfc-4376-b11b-c47a690c130a" path="/var/lib/kubelet/pods/a1afae03-2bfc-4376-b11b-c47a690c130a/volumes"
Sep 11 13:29:45.107123 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:29:45.107107    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="da9b7760-8a3d-4094-baed-7d8993ccdaf4" path="/var/lib/kubelet/pods/da9b7760-8a3d-4094-baed-7d8993ccdaf4/volumes"
Sep 11 13:29:45.225797 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:29:45.225740    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-d8lgv"
Sep 11 13:29:45.539370 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:29:45.539336    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod944e63d2_014c_40e1_9b2c_caa5f1259b21.slice/crio-b419ae587d2f0e18d71c04d22123aa31b381bd23c5a48d4482f8ee9662432f9a WatchSource:0}: Error finding container b419ae587d2f0e18d71c04d22123aa31b381bd23c5a48d4482f8ee9662432f9a: Status 404 returned error can't find the container with id b419ae587d2f0e18d71c04d22123aa31b381bd23c5a48d4482f8ee9662432f9a
Sep 11 13:29:45.542748 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:29:45.542712    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-d8lgv"]
Sep 11 13:29:45.779647 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:29:45.779611    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-d8lgv" event={"ID":"944e63d2-014c-40e1-9b2c-caa5f1259b21","Type":"ContainerStarted","Data":"716f452cecdef11943e418b4748eec2e32cd57c43fc5161d7eff1d3f3683dd88"}
Sep 11 13:29:45.780174 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:29:45.779672    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-d8lgv"
Sep 11 13:29:45.780174 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:29:45.779695    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-d8lgv" event={"ID":"944e63d2-014c-40e1-9b2c-caa5f1259b21","Type":"ContainerStarted","Data":"b419ae587d2f0e18d71c04d22123aa31b381bd23c5a48d4482f8ee9662432f9a"}
Sep 11 13:29:45.887056 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:29:45.886896    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-d8lgv" podStartSLOduration=1.886872503 podStartE2EDuration="1.886872503s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:29:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:29:45.886089037 +0000 UTC m=+493.272192216" watchObservedRunningTime="2026-09-11 13:29:45.886872503 +0000 UTC m=+493.272975664"
Sep 11 13:29:55.785192 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:29:55.785144    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-xxs4x"
Sep 11 13:29:55.786030 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:29:55.785271    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-k9c7d"
Sep 11 13:29:56.790737 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:29:56.790682    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-d8lgv"
Sep 11 13:29:56.854740 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:29:56.854700    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-k9c7d"]
Sep 11 13:29:56.855001 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:29:56.854957    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-k9c7d" podUID="61b9b730-2b90-4bda-976e-ac2c07e38f2b" containerName="manager" containerID="cri-o://63a8eacc15fa8dddacee7beea80f88b548c0f7a38fcd2f873285c56312ef9354" gracePeriod=10
Sep 11 13:29:57.136622 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:29:57.136577    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-k9c7d"
Sep 11 13:29:57.177395 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:29:57.177357    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9pllj\" (UniqueName: \"kubernetes.io/projected/61b9b730-2b90-4bda-976e-ac2c07e38f2b-kube-api-access-9pllj\") pod \"61b9b730-2b90-4bda-976e-ac2c07e38f2b\" (UID: \"61b9b730-2b90-4bda-976e-ac2c07e38f2b\") "
Sep 11 13:29:57.177395 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:29:57.177416    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/61b9b730-2b90-4bda-976e-ac2c07e38f2b-extensions-socket-volume\") pod \"61b9b730-2b90-4bda-976e-ac2c07e38f2b\" (UID: \"61b9b730-2b90-4bda-976e-ac2c07e38f2b\") "
Sep 11 13:29:57.178014 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:29:57.177971    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61b9b730-2b90-4bda-976e-ac2c07e38f2b-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "61b9b730-2b90-4bda-976e-ac2c07e38f2b" (UID: "61b9b730-2b90-4bda-976e-ac2c07e38f2b"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 13:29:57.181000 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:29:57.180957    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61b9b730-2b90-4bda-976e-ac2c07e38f2b-kube-api-access-9pllj" (OuterVolumeSpecName: "kube-api-access-9pllj") pod "61b9b730-2b90-4bda-976e-ac2c07e38f2b" (UID: "61b9b730-2b90-4bda-976e-ac2c07e38f2b"). InnerVolumeSpecName "kube-api-access-9pllj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 13:29:57.279163 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:29:57.279110    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9pllj\" (UniqueName: \"kubernetes.io/projected/61b9b730-2b90-4bda-976e-ac2c07e38f2b-kube-api-access-9pllj\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:29:57.279163 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:29:57.279159    2578 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/61b9b730-2b90-4bda-976e-ac2c07e38f2b-extensions-socket-volume\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:29:57.863690 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:29:57.863632    2578 generic.go:356] "Generic (PLEG): container finished" podID="61b9b730-2b90-4bda-976e-ac2c07e38f2b" containerID="63a8eacc15fa8dddacee7beea80f88b548c0f7a38fcd2f873285c56312ef9354" exitCode=0
Sep 11 13:29:57.863690 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:29:57.863701    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-k9c7d"
Sep 11 13:29:57.864286 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:29:57.863714    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-k9c7d" event={"ID":"61b9b730-2b90-4bda-976e-ac2c07e38f2b","Type":"ContainerDied","Data":"63a8eacc15fa8dddacee7beea80f88b548c0f7a38fcd2f873285c56312ef9354"}
Sep 11 13:29:57.864286 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:29:57.863783    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-k9c7d" event={"ID":"61b9b730-2b90-4bda-976e-ac2c07e38f2b","Type":"ContainerDied","Data":"44ffa3529edbf37903f3edc9f82ae89427c4346b94cc6e92407f8e2473a3f191"}
Sep 11 13:29:57.864286 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:29:57.863810    2578 scope.go:117] "RemoveContainer" containerID="63a8eacc15fa8dddacee7beea80f88b548c0f7a38fcd2f873285c56312ef9354"
Sep 11 13:29:57.874736 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:29:57.874706    2578 scope.go:117] "RemoveContainer" containerID="63a8eacc15fa8dddacee7beea80f88b548c0f7a38fcd2f873285c56312ef9354"
Sep 11 13:29:57.875191 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:29:57.875166    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63a8eacc15fa8dddacee7beea80f88b548c0f7a38fcd2f873285c56312ef9354\": container with ID starting with 63a8eacc15fa8dddacee7beea80f88b548c0f7a38fcd2f873285c56312ef9354 not found: ID does not exist" containerID="63a8eacc15fa8dddacee7beea80f88b548c0f7a38fcd2f873285c56312ef9354"
Sep 11 13:29:57.875316 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:29:57.875202    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63a8eacc15fa8dddacee7beea80f88b548c0f7a38fcd2f873285c56312ef9354"} err="failed to get container status \"63a8eacc15fa8dddacee7beea80f88b548c0f7a38fcd2f873285c56312ef9354\": rpc error: code = NotFound desc = could not find container \"63a8eacc15fa8dddacee7beea80f88b548c0f7a38fcd2f873285c56312ef9354\": container with ID starting with 63a8eacc15fa8dddacee7beea80f88b548c0f7a38fcd2f873285c56312ef9354 not found: ID does not exist"
Sep 11 13:29:57.891066 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:29:57.891021    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-k9c7d"]
Sep 11 13:29:57.895599 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:29:57.895554    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-k9c7d"]
Sep 11 13:29:59.107217 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:29:59.107166    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="61b9b730-2b90-4bda-976e-ac2c07e38f2b" path="/var/lib/kubelet/pods/61b9b730-2b90-4bda-976e-ac2c07e38f2b/volumes"
Sep 11 13:30:13.300778 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:30:13.300739    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-6kwrp"]
Sep 11 13:30:13.301473 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:30:13.301450    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61b9b730-2b90-4bda-976e-ac2c07e38f2b" containerName="manager"
Sep 11 13:30:13.301520 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:30:13.301476    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="61b9b730-2b90-4bda-976e-ac2c07e38f2b" containerName="manager"
Sep 11 13:30:13.301577 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:30:13.301567    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="61b9b730-2b90-4bda-976e-ac2c07e38f2b" containerName="manager"
Sep 11 13:30:13.305767 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:30:13.305731    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-6kwrp"
Sep 11 13:30:13.309148 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:30:13.309118    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-mqx9m\""
Sep 11 13:30:13.309529 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:30:13.309505    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 11 13:30:13.322691 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:30:13.322641    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-6kwrp"]
Sep 11 13:30:13.457395 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:30:13.457342    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1f2ae7a-43d1-4043-a1fc-9d59b34d1b90-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5ccb968c66-6kwrp\" (UID: \"b1f2ae7a-43d1-4043-a1fc-9d59b34d1b90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-6kwrp"
Sep 11 13:30:13.457672 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:30:13.457433    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1f2ae7a-43d1-4043-a1fc-9d59b34d1b90-workload-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-6kwrp\" (UID: \"b1f2ae7a-43d1-4043-a1fc-9d59b34d1b90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-6kwrp"
Sep 11 13:30:13.457672 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:30:13.457455    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1f2ae7a-43d1-4043-a1fc-9d59b34d1b90-credential-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-6kwrp\" (UID: \"b1f2ae7a-43d1-4043-a1fc-9d59b34d1b90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-6kwrp"
Sep 11 13:30:13.457672 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:30:13.457489    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1f2ae7a-43d1-4043-a1fc-9d59b34d1b90-istio-envoy\") pod \"maas-default-gateway-openshift-default-5ccb968c66-6kwrp\" (UID: \"b1f2ae7a-43d1-4043-a1fc-9d59b34d1b90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-6kwrp"
Sep 11 13:30:13.457672 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:30:13.457515    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1f2ae7a-43d1-4043-a1fc-9d59b34d1b90-istio-data\") pod \"maas-default-gateway-openshift-default-5ccb968c66-6kwrp\" (UID: \"b1f2ae7a-43d1-4043-a1fc-9d59b34d1b90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-6kwrp"
Sep 11 13:30:13.457672 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:30:13.457543    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1f2ae7a-43d1-4043-a1fc-9d59b34d1b90-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5ccb968c66-6kwrp\" (UID: \"b1f2ae7a-43d1-4043-a1fc-9d59b34d1b90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-6kwrp"
Sep 11 13:30:13.457672 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:30:13.457563    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1f2ae7a-43d1-4043-a1fc-9d59b34d1b90-workload-certs\") pod \"maas-default-gateway-openshift-default-5ccb968c66-6kwrp\" (UID: \"b1f2ae7a-43d1-4043-a1fc-9d59b34d1b90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-6kwrp"
Sep 11 13:30:13.457672 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:30:13.457580    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6t6sk\" (UniqueName: \"kubernetes.io/projected/b1f2ae7a-43d1-4043-a1fc-9d59b34d1b90-kube-api-access-6t6sk\") pod \"maas-default-gateway-openshift-default-5ccb968c66-6kwrp\" (UID: \"b1f2ae7a-43d1-4043-a1fc-9d59b34d1b90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-6kwrp"
Sep 11 13:30:13.457992 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:30:13.457720    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1f2ae7a-43d1-4043-a1fc-9d59b34d1b90-istio-token\") pod \"maas-default-gateway-openshift-default-5ccb968c66-6kwrp\" (UID: \"b1f2ae7a-43d1-4043-a1fc-9d59b34d1b90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-6kwrp"
Sep 11 13:30:13.558511 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:30:13.558361    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1f2ae7a-43d1-4043-a1fc-9d59b34d1b90-workload-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-6kwrp\" (UID: \"b1f2ae7a-43d1-4043-a1fc-9d59b34d1b90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-6kwrp"
Sep 11 13:30:13.558511 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:30:13.558440    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1f2ae7a-43d1-4043-a1fc-9d59b34d1b90-credential-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-6kwrp\" (UID: \"b1f2ae7a-43d1-4043-a1fc-9d59b34d1b90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-6kwrp"
Sep 11 13:30:13.558511 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:30:13.558477    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1f2ae7a-43d1-4043-a1fc-9d59b34d1b90-istio-envoy\") pod \"maas-default-gateway-openshift-default-5ccb968c66-6kwrp\" (UID: \"b1f2ae7a-43d1-4043-a1fc-9d59b34d1b90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-6kwrp"
Sep 11 13:30:13.558511 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:30:13.558493    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1f2ae7a-43d1-4043-a1fc-9d59b34d1b90-istio-data\") pod \"maas-default-gateway-openshift-default-5ccb968c66-6kwrp\" (UID: \"b1f2ae7a-43d1-4043-a1fc-9d59b34d1b90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-6kwrp"
Sep 11 13:30:13.558511 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:30:13.558525    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1f2ae7a-43d1-4043-a1fc-9d59b34d1b90-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5ccb968c66-6kwrp\" (UID: \"b1f2ae7a-43d1-4043-a1fc-9d59b34d1b90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-6kwrp"
Sep 11 13:30:13.559056 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:30:13.558552    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1f2ae7a-43d1-4043-a1fc-9d59b34d1b90-workload-certs\") pod \"maas-default-gateway-openshift-default-5ccb968c66-6kwrp\" (UID: \"b1f2ae7a-43d1-4043-a1fc-9d59b34d1b90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-6kwrp"
Sep 11 13:30:13.559056 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:30:13.558579    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6t6sk\" (UniqueName: \"kubernetes.io/projected/b1f2ae7a-43d1-4043-a1fc-9d59b34d1b90-kube-api-access-6t6sk\") pod \"maas-default-gateway-openshift-default-5ccb968c66-6kwrp\" (UID: \"b1f2ae7a-43d1-4043-a1fc-9d59b34d1b90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-6kwrp"
Sep 11 13:30:13.559056 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:30:13.558685    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1f2ae7a-43d1-4043-a1fc-9d59b34d1b90-istio-token\") pod \"maas-default-gateway-openshift-default-5ccb968c66-6kwrp\" (UID: \"b1f2ae7a-43d1-4043-a1fc-9d59b34d1b90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-6kwrp"
Sep 11 13:30:13.559056 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:30:13.558715    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1f2ae7a-43d1-4043-a1fc-9d59b34d1b90-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5ccb968c66-6kwrp\" (UID: \"b1f2ae7a-43d1-4043-a1fc-9d59b34d1b90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-6kwrp"
Sep 11 13:30:13.559056 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:30:13.558942    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1f2ae7a-43d1-4043-a1fc-9d59b34d1b90-workload-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-6kwrp\" (UID: \"b1f2ae7a-43d1-4043-a1fc-9d59b34d1b90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-6kwrp"
Sep 11 13:30:13.559056 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:30:13.559002    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1f2ae7a-43d1-4043-a1fc-9d59b34d1b90-credential-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-6kwrp\" (UID: \"b1f2ae7a-43d1-4043-a1fc-9d59b34d1b90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-6kwrp"
Sep 11 13:30:13.559378 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:30:13.559107    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1f2ae7a-43d1-4043-a1fc-9d59b34d1b90-workload-certs\") pod \"maas-default-gateway-openshift-default-5ccb968c66-6kwrp\" (UID: \"b1f2ae7a-43d1-4043-a1fc-9d59b34d1b90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-6kwrp"
Sep 11 13:30:13.559378 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:30:13.559279    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1f2ae7a-43d1-4043-a1fc-9d59b34d1b90-istio-data\") pod \"maas-default-gateway-openshift-default-5ccb968c66-6kwrp\" (UID: \"b1f2ae7a-43d1-4043-a1fc-9d59b34d1b90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-6kwrp"
Sep 11 13:30:13.559500 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:30:13.559446    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1f2ae7a-43d1-4043-a1fc-9d59b34d1b90-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5ccb968c66-6kwrp\" (UID: \"b1f2ae7a-43d1-4043-a1fc-9d59b34d1b90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-6kwrp"
Sep 11 13:30:13.561635 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:30:13.561610    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1f2ae7a-43d1-4043-a1fc-9d59b34d1b90-istio-envoy\") pod \"maas-default-gateway-openshift-default-5ccb968c66-6kwrp\" (UID: \"b1f2ae7a-43d1-4043-a1fc-9d59b34d1b90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-6kwrp"
Sep 11 13:30:13.561769 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:30:13.561751    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1f2ae7a-43d1-4043-a1fc-9d59b34d1b90-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5ccb968c66-6kwrp\" (UID: \"b1f2ae7a-43d1-4043-a1fc-9d59b34d1b90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-6kwrp"
Sep 11 13:30:13.568471 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:30:13.568440    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1f2ae7a-43d1-4043-a1fc-9d59b34d1b90-istio-token\") pod \"maas-default-gateway-openshift-default-5ccb968c66-6kwrp\" (UID: \"b1f2ae7a-43d1-4043-a1fc-9d59b34d1b90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-6kwrp"
Sep 11 13:30:13.568571 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:30:13.568482    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6t6sk\" (UniqueName: \"kubernetes.io/projected/b1f2ae7a-43d1-4043-a1fc-9d59b34d1b90-kube-api-access-6t6sk\") pod \"maas-default-gateway-openshift-default-5ccb968c66-6kwrp\" (UID: \"b1f2ae7a-43d1-4043-a1fc-9d59b34d1b90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-6kwrp"
Sep 11 13:30:13.620622 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:30:13.620573    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-6kwrp"
Sep 11 13:30:13.924559 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:30:13.922348    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-6kwrp"]
Sep 11 13:30:13.925940 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:30:13.925909    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1f2ae7a_43d1_4043_a1fc_9d59b34d1b90.slice/crio-4cd1a343c587d059bcf9a3d0e5887740c6ea1fa57b0309a4cc3067619c9c0b69 WatchSource:0}: Error finding container 4cd1a343c587d059bcf9a3d0e5887740c6ea1fa57b0309a4cc3067619c9c0b69: Status 404 returned error can't find the container with id 4cd1a343c587d059bcf9a3d0e5887740c6ea1fa57b0309a4cc3067619c9c0b69
Sep 11 13:30:13.961578 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:30:13.961533    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-6kwrp" event={"ID":"b1f2ae7a-43d1-4043-a1fc-9d59b34d1b90","Type":"ContainerStarted","Data":"4cd1a343c587d059bcf9a3d0e5887740c6ea1fa57b0309a4cc3067619c9c0b69"}
Sep 11 13:30:24.871901 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:30:24.871864    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 11 13:30:24.872176 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:30:24.871944    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 11 13:30:24.872176 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:30:24.871973    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 11 13:30:25.031748 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:30:25.031697    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-6kwrp" event={"ID":"b1f2ae7a-43d1-4043-a1fc-9d59b34d1b90","Type":"ContainerStarted","Data":"410fe54b5a0db2b0306992367ab1a8afd8be4a1e0bb702b3a8ed7045db7e2b19"}
Sep 11 13:30:25.055716 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:30:25.055645    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-6kwrp" podStartSLOduration=1.11242132 podStartE2EDuration="12.055617676s" totalImagesPullingTime="10.943196356s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:30:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:30:25.052915779 +0000 UTC m=+532.439018964" watchObservedRunningTime="2026-09-11 13:30:25.055617676 +0000 UTC m=+532.441720839"
Sep 11 13:30:25.621213 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:30:25.621152    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-6kwrp"
Sep 11 13:30:25.627803 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:30:25.627764    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-6kwrp"
Sep 11 13:30:26.038271 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:30:26.038235    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-6kwrp"
Sep 11 13:30:26.039677 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:30:26.039655    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-6kwrp"
Sep 11 13:30:28.037382 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:30:28.037330    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-2zr7r"]
Sep 11 13:30:28.040940 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:30:28.040914    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-2zr7r"
Sep 11 13:30:28.043550 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:30:28.043524    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-whbpf\""
Sep 11 13:30:28.044196 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:30:28.044176    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 11 13:30:28.123394 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:30:28.123314    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thcfc\" (UniqueName: \"kubernetes.io/projected/13228530-04e6-422f-b369-d456f3f472ce-kube-api-access-thcfc\") pod \"limitador-limitador-8588749977-2zr7r\" (UID: \"13228530-04e6-422f-b369-d456f3f472ce\") " pod="kuadrant-system/limitador-limitador-8588749977-2zr7r"
Sep 11 13:30:28.123631 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:30:28.123545    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/13228530-04e6-422f-b369-d456f3f472ce-config-file\") pod \"limitador-limitador-8588749977-2zr7r\" (UID: \"13228530-04e6-422f-b369-d456f3f472ce\") " pod="kuadrant-system/limitador-limitador-8588749977-2zr7r"
Sep 11 13:30:28.224513 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:30:28.224470    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/13228530-04e6-422f-b369-d456f3f472ce-config-file\") pod \"limitador-limitador-8588749977-2zr7r\" (UID: \"13228530-04e6-422f-b369-d456f3f472ce\") " pod="kuadrant-system/limitador-limitador-8588749977-2zr7r"
Sep 11 13:30:28.224751 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:30:28.224539    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thcfc\" (UniqueName: \"kubernetes.io/projected/13228530-04e6-422f-b369-d456f3f472ce-kube-api-access-thcfc\") pod \"limitador-limitador-8588749977-2zr7r\" (UID: \"13228530-04e6-422f-b369-d456f3f472ce\") " pod="kuadrant-system/limitador-limitador-8588749977-2zr7r"
Sep 11 13:30:28.225233 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:30:28.225206    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/13228530-04e6-422f-b369-d456f3f472ce-config-file\") pod \"limitador-limitador-8588749977-2zr7r\" (UID: \"13228530-04e6-422f-b369-d456f3f472ce\") " pod="kuadrant-system/limitador-limitador-8588749977-2zr7r"
Sep 11 13:30:28.235187 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:30:28.235142    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-thcfc\" (UniqueName: \"kubernetes.io/projected/13228530-04e6-422f-b369-d456f3f472ce-kube-api-access-thcfc\") pod \"limitador-limitador-8588749977-2zr7r\" (UID: \"13228530-04e6-422f-b369-d456f3f472ce\") " pod="kuadrant-system/limitador-limitador-8588749977-2zr7r"
Sep 11 13:30:28.354099 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:30:28.353772    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-2zr7r"
Sep 11 13:30:28.357508 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:30:28.357016    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-2zr7r"]
Sep 11 13:30:28.512086 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:30:28.512039    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-2zr7r"]
Sep 11 13:30:28.659458 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:30:28.659422    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-2zr7r"]
Sep 11 13:30:28.659615 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:30:28.659482    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13228530_04e6_422f_b369_d456f3f472ce.slice/crio-ad3f3b17ae0810cf0dce0bc5f0ad4ad0b256fa781775ff4d676b0dbfbf49150e WatchSource:0}: Error finding container ad3f3b17ae0810cf0dce0bc5f0ad4ad0b256fa781775ff4d676b0dbfbf49150e: Status 404 returned error can't find the container with id ad3f3b17ae0810cf0dce0bc5f0ad4ad0b256fa781775ff4d676b0dbfbf49150e
Sep 11 13:30:29.059879 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:30:29.059799    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-2zr7r" event={"ID":"13228530-04e6-422f-b369-d456f3f472ce","Type":"ContainerStarted","Data":"ad3f3b17ae0810cf0dce0bc5f0ad4ad0b256fa781775ff4d676b0dbfbf49150e"}
Sep 11 13:30:29.285340 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:30:29.285282    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59c974bd48-c5hnl"]
Sep 11 13:30:29.289005 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:30:29.288966    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-c5hnl"
Sep 11 13:30:29.292197 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:30:29.292155    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-rq57j\""
Sep 11 13:30:29.304574 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:30:29.304526    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-c5hnl"]
Sep 11 13:30:29.336718 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:30:29.336612    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vh7hm\" (UniqueName: \"kubernetes.io/projected/9d73181f-4102-42ba-81ba-e0f96b7d4134-kube-api-access-vh7hm\") pod \"authorino-59c974bd48-c5hnl\" (UID: \"9d73181f-4102-42ba-81ba-e0f96b7d4134\") " pod="kuadrant-system/authorino-59c974bd48-c5hnl"
Sep 11 13:30:29.438448 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:30:29.438393    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vh7hm\" (UniqueName: \"kubernetes.io/projected/9d73181f-4102-42ba-81ba-e0f96b7d4134-kube-api-access-vh7hm\") pod \"authorino-59c974bd48-c5hnl\" (UID: \"9d73181f-4102-42ba-81ba-e0f96b7d4134\") " pod="kuadrant-system/authorino-59c974bd48-c5hnl"
Sep 11 13:30:29.449126 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:30:29.449081    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vh7hm\" (UniqueName: \"kubernetes.io/projected/9d73181f-4102-42ba-81ba-e0f96b7d4134-kube-api-access-vh7hm\") pod \"authorino-59c974bd48-c5hnl\" (UID: \"9d73181f-4102-42ba-81ba-e0f96b7d4134\") " pod="kuadrant-system/authorino-59c974bd48-c5hnl"
Sep 11 13:30:29.606278 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:30:29.606230    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-c5hnl"
Sep 11 13:30:29.889792 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:30:29.889755    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-c5hnl"]
Sep 11 13:30:29.892332 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:30:29.892286    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9d73181f_4102_42ba_81ba_e0f96b7d4134.slice/crio-87efbe2f069db4a7b0ac4e854c177355720968343c07f71197933d561df8e536 WatchSource:0}: Error finding container 87efbe2f069db4a7b0ac4e854c177355720968343c07f71197933d561df8e536: Status 404 returned error can't find the container with id 87efbe2f069db4a7b0ac4e854c177355720968343c07f71197933d561df8e536
Sep 11 13:30:30.067149 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:30:30.067101    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-2zr7r" event={"ID":"13228530-04e6-422f-b369-d456f3f472ce","Type":"ContainerStarted","Data":"24a7d32b8836ca95c5ed80d221e41b9e0af0c58f1fd7306b166cd9b690b91b3b"}
Sep 11 13:30:30.067692 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:30:30.067242    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-8588749977-2zr7r"
Sep 11 13:30:30.068653 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:30:30.068613    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-c5hnl" event={"ID":"9d73181f-4102-42ba-81ba-e0f96b7d4134","Type":"ContainerStarted","Data":"87efbe2f069db4a7b0ac4e854c177355720968343c07f71197933d561df8e536"}
Sep 11 13:30:30.086448 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:30:30.086374    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-8588749977-2zr7r" podStartSLOduration=1.202144732 podStartE2EDuration="2.08635579s" totalImagesPullingTime="884.211058ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:30:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:30:30.084622387 +0000 UTC m=+537.470725577" watchObservedRunningTime="2026-09-11 13:30:30.08635579 +0000 UTC m=+537.472458953"
Sep 11 13:30:32.084328 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:30:32.084276    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-c5hnl" event={"ID":"9d73181f-4102-42ba-81ba-e0f96b7d4134","Type":"ContainerStarted","Data":"cc9f756e6526fe60f3c8452fcdbf0f6a51bf779147fc44b76f1051589d38a752"}
Sep 11 13:30:32.101675 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:30:32.101585    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59c974bd48-c5hnl" podStartSLOduration=1.471670107 podStartE2EDuration="3.101554556s" totalImagesPullingTime="1.629884449s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:30:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:30:32.100409067 +0000 UTC m=+539.486512231" watchObservedRunningTime="2026-09-11 13:30:32.101554556 +0000 UTC m=+539.487657720"
Sep 11 13:30:41.077485 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:30:41.077446    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-8588749977-2zr7r"
Sep 11 13:30:57.036918 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:30:57.036872    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-2zr7r"]
Sep 11 13:30:57.037582 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:30:57.037135    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-8588749977-2zr7r" podUID="13228530-04e6-422f-b369-d456f3f472ce" containerName="limitador" containerID="cri-o://24a7d32b8836ca95c5ed80d221e41b9e0af0c58f1fd7306b166cd9b690b91b3b" gracePeriod=30
Sep 11 13:30:57.601995 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:30:57.601897    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-2zr7r"
Sep 11 13:30:57.631209 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:30:57.631173    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-thcfc\" (UniqueName: \"kubernetes.io/projected/13228530-04e6-422f-b369-d456f3f472ce-kube-api-access-thcfc\") pod \"13228530-04e6-422f-b369-d456f3f472ce\" (UID: \"13228530-04e6-422f-b369-d456f3f472ce\") "
Sep 11 13:30:57.631434 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:30:57.631228    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/13228530-04e6-422f-b369-d456f3f472ce-config-file\") pod \"13228530-04e6-422f-b369-d456f3f472ce\" (UID: \"13228530-04e6-422f-b369-d456f3f472ce\") "
Sep 11 13:30:57.631633 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:30:57.631608    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13228530-04e6-422f-b369-d456f3f472ce-config-file" (OuterVolumeSpecName: "config-file") pod "13228530-04e6-422f-b369-d456f3f472ce" (UID: "13228530-04e6-422f-b369-d456f3f472ce"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 13:30:57.633748 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:30:57.633720    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13228530-04e6-422f-b369-d456f3f472ce-kube-api-access-thcfc" (OuterVolumeSpecName: "kube-api-access-thcfc") pod "13228530-04e6-422f-b369-d456f3f472ce" (UID: "13228530-04e6-422f-b369-d456f3f472ce"). InnerVolumeSpecName "kube-api-access-thcfc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 13:30:57.732415 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:30:57.732362    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-thcfc\" (UniqueName: \"kubernetes.io/projected/13228530-04e6-422f-b369-d456f3f472ce-kube-api-access-thcfc\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:30:57.732415 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:30:57.732407    2578 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/13228530-04e6-422f-b369-d456f3f472ce-config-file\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:30:58.245616 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:30:58.245570    2578 generic.go:356] "Generic (PLEG): container finished" podID="13228530-04e6-422f-b369-d456f3f472ce" containerID="24a7d32b8836ca95c5ed80d221e41b9e0af0c58f1fd7306b166cd9b690b91b3b" exitCode=0
Sep 11 13:30:58.246236 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:30:58.245640    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-2zr7r" event={"ID":"13228530-04e6-422f-b369-d456f3f472ce","Type":"ContainerDied","Data":"24a7d32b8836ca95c5ed80d221e41b9e0af0c58f1fd7306b166cd9b690b91b3b"}
Sep 11 13:30:58.246236 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:30:58.245651    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-2zr7r"
Sep 11 13:30:58.246236 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:30:58.245743    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-2zr7r" event={"ID":"13228530-04e6-422f-b369-d456f3f472ce","Type":"ContainerDied","Data":"ad3f3b17ae0810cf0dce0bc5f0ad4ad0b256fa781775ff4d676b0dbfbf49150e"}
Sep 11 13:30:58.246236 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:30:58.245781    2578 scope.go:117] "RemoveContainer" containerID="24a7d32b8836ca95c5ed80d221e41b9e0af0c58f1fd7306b166cd9b690b91b3b"
Sep 11 13:30:58.257561 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:30:58.257541    2578 scope.go:117] "RemoveContainer" containerID="24a7d32b8836ca95c5ed80d221e41b9e0af0c58f1fd7306b166cd9b690b91b3b"
Sep 11 13:30:58.257964 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:30:58.257938    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24a7d32b8836ca95c5ed80d221e41b9e0af0c58f1fd7306b166cd9b690b91b3b\": container with ID starting with 24a7d32b8836ca95c5ed80d221e41b9e0af0c58f1fd7306b166cd9b690b91b3b not found: ID does not exist" containerID="24a7d32b8836ca95c5ed80d221e41b9e0af0c58f1fd7306b166cd9b690b91b3b"
Sep 11 13:30:58.258084 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:30:58.257974    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24a7d32b8836ca95c5ed80d221e41b9e0af0c58f1fd7306b166cd9b690b91b3b"} err="failed to get container status \"24a7d32b8836ca95c5ed80d221e41b9e0af0c58f1fd7306b166cd9b690b91b3b\": rpc error: code = NotFound desc = could not find container \"24a7d32b8836ca95c5ed80d221e41b9e0af0c58f1fd7306b166cd9b690b91b3b\": container with ID starting with 24a7d32b8836ca95c5ed80d221e41b9e0af0c58f1fd7306b166cd9b690b91b3b not found: ID does not exist"
Sep 11 13:30:58.271057 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:30:58.271010    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-2zr7r"]
Sep 11 13:30:58.275969 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:30:58.275911    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-2zr7r"]
Sep 11 13:30:59.107935 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:30:59.107876    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="13228530-04e6-422f-b369-d456f3f472ce" path="/var/lib/kubelet/pods/13228530-04e6-422f-b369-d456f3f472ce/volumes"
Sep 11 13:30:59.740789 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:30:59.740732    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-jscrc"]
Sep 11 13:30:59.741356 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:30:59.741336    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13228530-04e6-422f-b369-d456f3f472ce" containerName="limitador"
Sep 11 13:30:59.741422 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:30:59.741359    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="13228530-04e6-422f-b369-d456f3f472ce" containerName="limitador"
Sep 11 13:30:59.741460 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:30:59.741449    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="13228530-04e6-422f-b369-d456f3f472ce" containerName="limitador"
Sep 11 13:30:59.746936 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:30:59.746900    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-jscrc"
Sep 11 13:30:59.750643 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:30:59.750596    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-szmf2\""
Sep 11 13:30:59.750943 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:30:59.750750    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 13:30:59.750943 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:30:59.750639    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Sep 11 13:30:59.751097 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:30:59.750599    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 11 13:30:59.752786 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:30:59.752751    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-jscrc"]
Sep 11 13:30:59.852669 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:30:59.852625    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/0220ee9d-6757-41c7-a654-bb7f56272bcc-data\") pod \"postgres-69fb4594cf-jscrc\" (UID: \"0220ee9d-6757-41c7-a654-bb7f56272bcc\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-jscrc"
Sep 11 13:30:59.852929 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:30:59.852874    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlfcm\" (UniqueName: \"kubernetes.io/projected/0220ee9d-6757-41c7-a654-bb7f56272bcc-kube-api-access-zlfcm\") pod \"postgres-69fb4594cf-jscrc\" (UID: \"0220ee9d-6757-41c7-a654-bb7f56272bcc\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-jscrc"
Sep 11 13:30:59.954132 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:30:59.954071    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlfcm\" (UniqueName: \"kubernetes.io/projected/0220ee9d-6757-41c7-a654-bb7f56272bcc-kube-api-access-zlfcm\") pod \"postgres-69fb4594cf-jscrc\" (UID: \"0220ee9d-6757-41c7-a654-bb7f56272bcc\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-jscrc"
Sep 11 13:30:59.954132 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:30:59.954146    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/0220ee9d-6757-41c7-a654-bb7f56272bcc-data\") pod \"postgres-69fb4594cf-jscrc\" (UID: \"0220ee9d-6757-41c7-a654-bb7f56272bcc\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-jscrc"
Sep 11 13:30:59.954578 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:30:59.954554    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/0220ee9d-6757-41c7-a654-bb7f56272bcc-data\") pod \"postgres-69fb4594cf-jscrc\" (UID: \"0220ee9d-6757-41c7-a654-bb7f56272bcc\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-jscrc"
Sep 11 13:30:59.963481 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:30:59.963439    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlfcm\" (UniqueName: \"kubernetes.io/projected/0220ee9d-6757-41c7-a654-bb7f56272bcc-kube-api-access-zlfcm\") pod \"postgres-69fb4594cf-jscrc\" (UID: \"0220ee9d-6757-41c7-a654-bb7f56272bcc\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-jscrc"
Sep 11 13:31:00.063503 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:31:00.063383    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-jscrc"
Sep 11 13:31:00.358890 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:31:00.358849    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-jscrc"]
Sep 11 13:31:00.361601 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:31:00.361564    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0220ee9d_6757_41c7_a654_bb7f56272bcc.slice/crio-2e8238e885aadbef6fabebf37fd9290e46b738e5d7c3fae11156b7d6a914ba0b WatchSource:0}: Error finding container 2e8238e885aadbef6fabebf37fd9290e46b738e5d7c3fae11156b7d6a914ba0b: Status 404 returned error can't find the container with id 2e8238e885aadbef6fabebf37fd9290e46b738e5d7c3fae11156b7d6a914ba0b
Sep 11 13:31:01.273131 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:31:01.273073    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-jscrc" event={"ID":"0220ee9d-6757-41c7-a654-bb7f56272bcc","Type":"ContainerStarted","Data":"2e8238e885aadbef6fabebf37fd9290e46b738e5d7c3fae11156b7d6a914ba0b"}
Sep 11 13:31:06.308868 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:31:06.308798    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-jscrc" event={"ID":"0220ee9d-6757-41c7-a654-bb7f56272bcc","Type":"ContainerStarted","Data":"d2cab6603ad66b76a9815d4b743c22d445435ae8aa10953d8838f8df4a60bf81"}
Sep 11 13:31:06.309551 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:31:06.309020    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-jscrc"
Sep 11 13:31:06.328488 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:31:06.328407    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-jscrc" podStartSLOduration=2.379797405 podStartE2EDuration="7.328377969s" totalImagesPullingTime="4.948580564s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:30:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:31:06.326199232 +0000 UTC m=+573.712302410" watchObservedRunningTime="2026-09-11 13:31:06.328377969 +0000 UTC m=+573.714481132"
Sep 11 13:31:12.348031 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:31:12.347989    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-jscrc"
Sep 11 13:31:13.188125 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:31:13.188008    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-b9b6744ff-wcl9n"]
Sep 11 13:31:13.203529 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:31:13.203481    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-b9b6744ff-wcl9n"]
Sep 11 13:31:13.203774 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:31:13.203650    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b9b6744ff-wcl9n"
Sep 11 13:31:13.290014 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:31:13.289957    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-b9b6744ff-wcl9n"]
Sep 11 13:31:13.290427 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:31:13.290399    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-d75td], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-b9b6744ff-wcl9n" podUID="114572d7-ef03-49ad-886e-f2c598473b18"
Sep 11 13:31:13.302363 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:31:13.302308    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d75td\" (UniqueName: \"kubernetes.io/projected/114572d7-ef03-49ad-886e-f2c598473b18-kube-api-access-d75td\") pod \"authorino-b9b6744ff-wcl9n\" (UID: \"114572d7-ef03-49ad-886e-f2c598473b18\") " pod="kuadrant-system/authorino-b9b6744ff-wcl9n"
Sep 11 13:31:13.352062 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:31:13.352023    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b9b6744ff-wcl9n"
Sep 11 13:31:13.366958 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:31:13.366923    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b9b6744ff-wcl9n"
Sep 11 13:31:13.403228 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:31:13.403188    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d75td\" (UniqueName: \"kubernetes.io/projected/114572d7-ef03-49ad-886e-f2c598473b18-kube-api-access-d75td\") pod \"authorino-b9b6744ff-wcl9n\" (UID: \"114572d7-ef03-49ad-886e-f2c598473b18\") " pod="kuadrant-system/authorino-b9b6744ff-wcl9n"
Sep 11 13:31:13.416823 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:31:13.416772    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d75td\" (UniqueName: \"kubernetes.io/projected/114572d7-ef03-49ad-886e-f2c598473b18-kube-api-access-d75td\") pod \"authorino-b9b6744ff-wcl9n\" (UID: \"114572d7-ef03-49ad-886e-f2c598473b18\") " pod="kuadrant-system/authorino-b9b6744ff-wcl9n"
Sep 11 13:31:13.503998 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:31:13.503961    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d75td\" (UniqueName: \"kubernetes.io/projected/114572d7-ef03-49ad-886e-f2c598473b18-kube-api-access-d75td\") pod \"114572d7-ef03-49ad-886e-f2c598473b18\" (UID: \"114572d7-ef03-49ad-886e-f2c598473b18\") "
Sep 11 13:31:13.506330 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:31:13.506289    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/114572d7-ef03-49ad-886e-f2c598473b18-kube-api-access-d75td" (OuterVolumeSpecName: "kube-api-access-d75td") pod "114572d7-ef03-49ad-886e-f2c598473b18" (UID: "114572d7-ef03-49ad-886e-f2c598473b18"). InnerVolumeSpecName "kube-api-access-d75td". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 13:31:13.605233 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:31:13.605185    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d75td\" (UniqueName: \"kubernetes.io/projected/114572d7-ef03-49ad-886e-f2c598473b18-kube-api-access-d75td\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:31:14.359359 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:31:14.359308    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b9b6744ff-wcl9n"
Sep 11 13:31:14.390634 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:31:14.390584    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-b9b6744ff-wcl9n"]
Sep 11 13:31:14.396477 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:31:14.396435    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-b9b6744ff-wcl9n"]
Sep 11 13:31:15.108989 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:31:15.108944    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="114572d7-ef03-49ad-886e-f2c598473b18" path="/var/lib/kubelet/pods/114572d7-ef03-49ad-886e-f2c598473b18/volumes"
Sep 11 13:31:15.191293 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:31:15.191244    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-c5hnl"]
Sep 11 13:31:15.191535 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:31:15.191505    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59c974bd48-c5hnl" podUID="9d73181f-4102-42ba-81ba-e0f96b7d4134" containerName="authorino" containerID="cri-o://cc9f756e6526fe60f3c8452fcdbf0f6a51bf779147fc44b76f1051589d38a752" gracePeriod=30
Sep 11 13:31:15.368260 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:31:15.368127    2578 generic.go:356] "Generic (PLEG): container finished" podID="9d73181f-4102-42ba-81ba-e0f96b7d4134" containerID="cc9f756e6526fe60f3c8452fcdbf0f6a51bf779147fc44b76f1051589d38a752" exitCode=0
Sep 11 13:31:15.368260 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:31:15.368213    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-c5hnl" event={"ID":"9d73181f-4102-42ba-81ba-e0f96b7d4134","Type":"ContainerDied","Data":"cc9f756e6526fe60f3c8452fcdbf0f6a51bf779147fc44b76f1051589d38a752"}
Sep 11 13:31:15.469574 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:31:15.469541    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-c5hnl"
Sep 11 13:31:15.627561 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:31:15.627447    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vh7hm\" (UniqueName: \"kubernetes.io/projected/9d73181f-4102-42ba-81ba-e0f96b7d4134-kube-api-access-vh7hm\") pod \"9d73181f-4102-42ba-81ba-e0f96b7d4134\" (UID: \"9d73181f-4102-42ba-81ba-e0f96b7d4134\") "
Sep 11 13:31:15.629869 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:31:15.629823    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d73181f-4102-42ba-81ba-e0f96b7d4134-kube-api-access-vh7hm" (OuterVolumeSpecName: "kube-api-access-vh7hm") pod "9d73181f-4102-42ba-81ba-e0f96b7d4134" (UID: "9d73181f-4102-42ba-81ba-e0f96b7d4134"). InnerVolumeSpecName "kube-api-access-vh7hm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 13:31:15.729292 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:31:15.729242    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vh7hm\" (UniqueName: \"kubernetes.io/projected/9d73181f-4102-42ba-81ba-e0f96b7d4134-kube-api-access-vh7hm\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:31:16.377398 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:31:16.377365    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-c5hnl"
Sep 11 13:31:16.377398 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:31:16.377371    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-c5hnl" event={"ID":"9d73181f-4102-42ba-81ba-e0f96b7d4134","Type":"ContainerDied","Data":"87efbe2f069db4a7b0ac4e854c177355720968343c07f71197933d561df8e536"}
Sep 11 13:31:16.378095 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:31:16.377434    2578 scope.go:117] "RemoveContainer" containerID="cc9f756e6526fe60f3c8452fcdbf0f6a51bf779147fc44b76f1051589d38a752"
Sep 11 13:31:16.404770 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:31:16.404731    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-c5hnl"]
Sep 11 13:31:16.409779 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:31:16.409735    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59c974bd48-c5hnl"]
Sep 11 13:31:17.107826 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:31:17.107783    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9d73181f-4102-42ba-81ba-e0f96b7d4134" path="/var/lib/kubelet/pods/9d73181f-4102-42ba-81ba-e0f96b7d4134/volumes"
Sep 11 13:31:20.023274 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:31:20.023218    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7c9d797ff5-cmld7"]
Sep 11 13:31:20.023780 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:31:20.023744    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d73181f-4102-42ba-81ba-e0f96b7d4134" containerName="authorino"
Sep 11 13:31:20.023780 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:31:20.023757    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d73181f-4102-42ba-81ba-e0f96b7d4134" containerName="authorino"
Sep 11 13:31:20.023913 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:31:20.023879    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d73181f-4102-42ba-81ba-e0f96b7d4134" containerName="authorino"
Sep 11 13:31:20.026956 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:31:20.026933    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7c9d797ff5-cmld7"
Sep 11 13:31:20.029393 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:31:20.029356    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 11 13:31:20.029393 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:31:20.029377    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-dpcsj\""
Sep 11 13:31:20.029637 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:31:20.029356    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 11 13:31:20.029637 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:31:20.029412    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 11 13:31:20.044503 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:31:20.044471    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7c9d797ff5-cmld7"]
Sep 11 13:31:20.178466 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:31:20.178406    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fad8b11d-4aba-4fe0-8f25-5a592bf0da60-webhook-cert\") pod \"maas-controller-7c9d797ff5-cmld7\" (UID: \"fad8b11d-4aba-4fe0-8f25-5a592bf0da60\") " pod="opendatahub/maas-controller-7c9d797ff5-cmld7"
Sep 11 13:31:20.178701 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:31:20.178482    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bjkx\" (UniqueName: \"kubernetes.io/projected/fad8b11d-4aba-4fe0-8f25-5a592bf0da60-kube-api-access-8bjkx\") pod \"maas-controller-7c9d797ff5-cmld7\" (UID: \"fad8b11d-4aba-4fe0-8f25-5a592bf0da60\") " pod="opendatahub/maas-controller-7c9d797ff5-cmld7"
Sep 11 13:31:20.178746 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:31:20.178686    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fad8b11d-4aba-4fe0-8f25-5a592bf0da60-metrics-certs\") pod \"maas-controller-7c9d797ff5-cmld7\" (UID: \"fad8b11d-4aba-4fe0-8f25-5a592bf0da60\") " pod="opendatahub/maas-controller-7c9d797ff5-cmld7"
Sep 11 13:31:20.185526 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:31:20.185486    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6b447f7bf9-7cs9k"]
Sep 11 13:31:20.192896 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:31:20.192809    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6b447f7bf9-7cs9k"
Sep 11 13:31:20.201472 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:31:20.201435    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6b447f7bf9-7cs9k"]
Sep 11 13:31:20.280044 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:31:20.279926    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fad8b11d-4aba-4fe0-8f25-5a592bf0da60-metrics-certs\") pod \"maas-controller-7c9d797ff5-cmld7\" (UID: \"fad8b11d-4aba-4fe0-8f25-5a592bf0da60\") " pod="opendatahub/maas-controller-7c9d797ff5-cmld7"
Sep 11 13:31:20.280268 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:31:20.280068    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fad8b11d-4aba-4fe0-8f25-5a592bf0da60-webhook-cert\") pod \"maas-controller-7c9d797ff5-cmld7\" (UID: \"fad8b11d-4aba-4fe0-8f25-5a592bf0da60\") " pod="opendatahub/maas-controller-7c9d797ff5-cmld7"
Sep 11 13:31:20.280343 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:31:20.280258    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bjkx\" (UniqueName: \"kubernetes.io/projected/fad8b11d-4aba-4fe0-8f25-5a592bf0da60-kube-api-access-8bjkx\") pod \"maas-controller-7c9d797ff5-cmld7\" (UID: \"fad8b11d-4aba-4fe0-8f25-5a592bf0da60\") " pod="opendatahub/maas-controller-7c9d797ff5-cmld7"
Sep 11 13:31:20.283121 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:31:20.283088    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fad8b11d-4aba-4fe0-8f25-5a592bf0da60-webhook-cert\") pod \"maas-controller-7c9d797ff5-cmld7\" (UID: \"fad8b11d-4aba-4fe0-8f25-5a592bf0da60\") " pod="opendatahub/maas-controller-7c9d797ff5-cmld7"
Sep 11 13:31:20.283308 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:31:20.283091    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fad8b11d-4aba-4fe0-8f25-5a592bf0da60-metrics-certs\") pod \"maas-controller-7c9d797ff5-cmld7\" (UID: \"fad8b11d-4aba-4fe0-8f25-5a592bf0da60\") " pod="opendatahub/maas-controller-7c9d797ff5-cmld7"
Sep 11 13:31:20.293109 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:31:20.293078    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bjkx\" (UniqueName: \"kubernetes.io/projected/fad8b11d-4aba-4fe0-8f25-5a592bf0da60-kube-api-access-8bjkx\") pod \"maas-controller-7c9d797ff5-cmld7\" (UID: \"fad8b11d-4aba-4fe0-8f25-5a592bf0da60\") " pod="opendatahub/maas-controller-7c9d797ff5-cmld7"
Sep 11 13:31:20.339000 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:31:20.338950    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7c9d797ff5-cmld7"
Sep 11 13:31:20.382185 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:31:20.382131    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/babc7d58-1418-46b3-ace0-4341db9351ee-webhook-cert\") pod \"maas-controller-6b447f7bf9-7cs9k\" (UID: \"babc7d58-1418-46b3-ace0-4341db9351ee\") " pod="opendatahub/maas-controller-6b447f7bf9-7cs9k"
Sep 11 13:31:20.382381 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:31:20.382276    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cz7lb\" (UniqueName: \"kubernetes.io/projected/babc7d58-1418-46b3-ace0-4341db9351ee-kube-api-access-cz7lb\") pod \"maas-controller-6b447f7bf9-7cs9k\" (UID: \"babc7d58-1418-46b3-ace0-4341db9351ee\") " pod="opendatahub/maas-controller-6b447f7bf9-7cs9k"
Sep 11 13:31:20.382428 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:31:20.382408    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/babc7d58-1418-46b3-ace0-4341db9351ee-metrics-certs\") pod \"maas-controller-6b447f7bf9-7cs9k\" (UID: \"babc7d58-1418-46b3-ace0-4341db9351ee\") " pod="opendatahub/maas-controller-6b447f7bf9-7cs9k"
Sep 11 13:31:20.483552 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:31:20.483502    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/babc7d58-1418-46b3-ace0-4341db9351ee-metrics-certs\") pod \"maas-controller-6b447f7bf9-7cs9k\" (UID: \"babc7d58-1418-46b3-ace0-4341db9351ee\") " pod="opendatahub/maas-controller-6b447f7bf9-7cs9k"
Sep 11 13:31:20.483783 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:31:20.483587    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/babc7d58-1418-46b3-ace0-4341db9351ee-webhook-cert\") pod \"maas-controller-6b447f7bf9-7cs9k\" (UID: \"babc7d58-1418-46b3-ace0-4341db9351ee\") " pod="opendatahub/maas-controller-6b447f7bf9-7cs9k"
Sep 11 13:31:20.483783 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:31:20.483639    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cz7lb\" (UniqueName: \"kubernetes.io/projected/babc7d58-1418-46b3-ace0-4341db9351ee-kube-api-access-cz7lb\") pod \"maas-controller-6b447f7bf9-7cs9k\" (UID: \"babc7d58-1418-46b3-ace0-4341db9351ee\") " pod="opendatahub/maas-controller-6b447f7bf9-7cs9k"
Sep 11 13:31:20.486719 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:31:20.486694    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/babc7d58-1418-46b3-ace0-4341db9351ee-metrics-certs\") pod \"maas-controller-6b447f7bf9-7cs9k\" (UID: \"babc7d58-1418-46b3-ace0-4341db9351ee\") " pod="opendatahub/maas-controller-6b447f7bf9-7cs9k"
Sep 11 13:31:20.486782 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:31:20.486716    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/babc7d58-1418-46b3-ace0-4341db9351ee-webhook-cert\") pod \"maas-controller-6b447f7bf9-7cs9k\" (UID: \"babc7d58-1418-46b3-ace0-4341db9351ee\") " pod="opendatahub/maas-controller-6b447f7bf9-7cs9k"
Sep 11 13:31:20.493483 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:31:20.493456    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cz7lb\" (UniqueName: \"kubernetes.io/projected/babc7d58-1418-46b3-ace0-4341db9351ee-kube-api-access-cz7lb\") pod \"maas-controller-6b447f7bf9-7cs9k\" (UID: \"babc7d58-1418-46b3-ace0-4341db9351ee\") " pod="opendatahub/maas-controller-6b447f7bf9-7cs9k"
Sep 11 13:31:20.508363 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:31:20.508326    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6b447f7bf9-7cs9k"
Sep 11 13:31:20.630065 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:31:20.630028    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7c9d797ff5-cmld7"]
Sep 11 13:31:20.632957 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:31:20.632903    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfad8b11d_4aba_4fe0_8f25_5a592bf0da60.slice/crio-66c749c6ac7dfbe0f67ef525f6287c6b9c5b46efe707042c2ea653077633fede WatchSource:0}: Error finding container 66c749c6ac7dfbe0f67ef525f6287c6b9c5b46efe707042c2ea653077633fede: Status 404 returned error can't find the container with id 66c749c6ac7dfbe0f67ef525f6287c6b9c5b46efe707042c2ea653077633fede
Sep 11 13:31:20.791379 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:31:20.791225    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6b447f7bf9-7cs9k"]
Sep 11 13:31:20.793938 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:31:20.793905    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbabc7d58_1418_46b3_ace0_4341db9351ee.slice/crio-64319b088b7822816db1604278e5d71314a074de25fd091afa4c3c4a76e45633 WatchSource:0}: Error finding container 64319b088b7822816db1604278e5d71314a074de25fd091afa4c3c4a76e45633: Status 404 returned error can't find the container with id 64319b088b7822816db1604278e5d71314a074de25fd091afa4c3c4a76e45633
Sep 11 13:31:21.419490 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:31:21.419422    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7c9d797ff5-cmld7" event={"ID":"fad8b11d-4aba-4fe0-8f25-5a592bf0da60","Type":"ContainerStarted","Data":"66c749c6ac7dfbe0f67ef525f6287c6b9c5b46efe707042c2ea653077633fede"}
Sep 11 13:31:21.422001 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:31:21.421873    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6b447f7bf9-7cs9k" event={"ID":"babc7d58-1418-46b3-ace0-4341db9351ee","Type":"ContainerStarted","Data":"64319b088b7822816db1604278e5d71314a074de25fd091afa4c3c4a76e45633"}
Sep 11 13:31:24.443749 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:31:24.443678    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7c9d797ff5-cmld7" event={"ID":"fad8b11d-4aba-4fe0-8f25-5a592bf0da60","Type":"ContainerStarted","Data":"ae9442d1556821bf61854c66e8ae819133e7f3578843ece8efc4174fd738bfca"}
Sep 11 13:31:24.444355 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:31:24.443771    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7c9d797ff5-cmld7"
Sep 11 13:31:24.445885 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:31:24.445860    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6b447f7bf9-7cs9k" event={"ID":"babc7d58-1418-46b3-ace0-4341db9351ee","Type":"ContainerStarted","Data":"bb9eb29982700e680ff48ba65ba83420267edcc244437a847e429ec93b034979"}
Sep 11 13:31:24.446037 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:31:24.445960    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6b447f7bf9-7cs9k"
Sep 11 13:31:24.468494 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:31:24.468415    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-7c9d797ff5-cmld7" podStartSLOduration=1.644617848 podStartE2EDuration="4.468393552s" totalImagesPullingTime="2.823775704s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:31:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:31:24.465475162 +0000 UTC m=+591.851578354" watchObservedRunningTime="2026-09-11 13:31:24.468393552 +0000 UTC m=+591.854496717"
Sep 11 13:31:24.486571 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:31:24.486485    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-6b447f7bf9-7cs9k" podStartSLOduration=1.8162268849999998 podStartE2EDuration="4.486451573s" totalImagesPullingTime="2.670224688s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:31:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:31:24.484372377 +0000 UTC m=+591.870475561" watchObservedRunningTime="2026-09-11 13:31:24.486451573 +0000 UTC m=+591.872554789"
Sep 11 13:31:33.060907 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:31:33.060865    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6gltj_e384caae-3cb1-4264-8a25-55b40ac8ac45/ovn-acl-logging/0.log"
Sep 11 13:31:33.062730 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:31:33.062703    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6gltj_e384caae-3cb1-4264-8a25-55b40ac8ac45/ovn-acl-logging/0.log"
Sep 11 13:31:35.454778 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:31:35.454729    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6b447f7bf9-7cs9k"
Sep 11 13:31:35.455420 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:31:35.454856    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7c9d797ff5-cmld7"
Sep 11 13:31:35.530110 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:31:35.530061    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7c9d797ff5-cmld7"]
Sep 11 13:31:35.530374 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:31:35.530316    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-7c9d797ff5-cmld7" podUID="fad8b11d-4aba-4fe0-8f25-5a592bf0da60" containerName="manager" containerID="cri-o://ae9442d1556821bf61854c66e8ae819133e7f3578843ece8efc4174fd738bfca" gracePeriod=30
Sep 11 13:31:35.818457 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:31:35.818421    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7c9d797ff5-cmld7"
Sep 11 13:31:35.957202 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:31:35.957138    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fad8b11d-4aba-4fe0-8f25-5a592bf0da60-webhook-cert\") pod \"fad8b11d-4aba-4fe0-8f25-5a592bf0da60\" (UID: \"fad8b11d-4aba-4fe0-8f25-5a592bf0da60\") "
Sep 11 13:31:35.957202 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:31:35.957208    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8bjkx\" (UniqueName: \"kubernetes.io/projected/fad8b11d-4aba-4fe0-8f25-5a592bf0da60-kube-api-access-8bjkx\") pod \"fad8b11d-4aba-4fe0-8f25-5a592bf0da60\" (UID: \"fad8b11d-4aba-4fe0-8f25-5a592bf0da60\") "
Sep 11 13:31:35.957539 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:31:35.957311    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fad8b11d-4aba-4fe0-8f25-5a592bf0da60-metrics-certs\") pod \"fad8b11d-4aba-4fe0-8f25-5a592bf0da60\" (UID: \"fad8b11d-4aba-4fe0-8f25-5a592bf0da60\") "
Sep 11 13:31:35.960385 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:31:35.960297    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fad8b11d-4aba-4fe0-8f25-5a592bf0da60-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "fad8b11d-4aba-4fe0-8f25-5a592bf0da60" (UID: "fad8b11d-4aba-4fe0-8f25-5a592bf0da60"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 13:31:35.960385 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:31:35.960301    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fad8b11d-4aba-4fe0-8f25-5a592bf0da60-kube-api-access-8bjkx" (OuterVolumeSpecName: "kube-api-access-8bjkx") pod "fad8b11d-4aba-4fe0-8f25-5a592bf0da60" (UID: "fad8b11d-4aba-4fe0-8f25-5a592bf0da60"). InnerVolumeSpecName "kube-api-access-8bjkx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 13:31:35.960385 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:31:35.960311    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fad8b11d-4aba-4fe0-8f25-5a592bf0da60-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "fad8b11d-4aba-4fe0-8f25-5a592bf0da60" (UID: "fad8b11d-4aba-4fe0-8f25-5a592bf0da60"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 13:31:36.059212 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:31:36.059139    2578 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fad8b11d-4aba-4fe0-8f25-5a592bf0da60-webhook-cert\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:31:36.059212 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:31:36.059198    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8bjkx\" (UniqueName: \"kubernetes.io/projected/fad8b11d-4aba-4fe0-8f25-5a592bf0da60-kube-api-access-8bjkx\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:31:36.059212 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:31:36.059212    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fad8b11d-4aba-4fe0-8f25-5a592bf0da60-metrics-certs\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:31:36.520314 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:31:36.520272    2578 generic.go:356] "Generic (PLEG): container finished" podID="fad8b11d-4aba-4fe0-8f25-5a592bf0da60" containerID="ae9442d1556821bf61854c66e8ae819133e7f3578843ece8efc4174fd738bfca" exitCode=0
Sep 11 13:31:36.520759 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:31:36.520329    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7c9d797ff5-cmld7"
Sep 11 13:31:36.520759 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:31:36.520359    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7c9d797ff5-cmld7" event={"ID":"fad8b11d-4aba-4fe0-8f25-5a592bf0da60","Type":"ContainerDied","Data":"ae9442d1556821bf61854c66e8ae819133e7f3578843ece8efc4174fd738bfca"}
Sep 11 13:31:36.520759 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:31:36.520430    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7c9d797ff5-cmld7" event={"ID":"fad8b11d-4aba-4fe0-8f25-5a592bf0da60","Type":"ContainerDied","Data":"66c749c6ac7dfbe0f67ef525f6287c6b9c5b46efe707042c2ea653077633fede"}
Sep 11 13:31:36.520759 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:31:36.520456    2578 scope.go:117] "RemoveContainer" containerID="ae9442d1556821bf61854c66e8ae819133e7f3578843ece8efc4174fd738bfca"
Sep 11 13:31:36.531253 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:31:36.531092    2578 scope.go:117] "RemoveContainer" containerID="ae9442d1556821bf61854c66e8ae819133e7f3578843ece8efc4174fd738bfca"
Sep 11 13:31:36.531491 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:31:36.531471    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae9442d1556821bf61854c66e8ae819133e7f3578843ece8efc4174fd738bfca\": container with ID starting with ae9442d1556821bf61854c66e8ae819133e7f3578843ece8efc4174fd738bfca not found: ID does not exist" containerID="ae9442d1556821bf61854c66e8ae819133e7f3578843ece8efc4174fd738bfca"
Sep 11 13:31:36.531543 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:31:36.531503    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae9442d1556821bf61854c66e8ae819133e7f3578843ece8efc4174fd738bfca"} err="failed to get container status \"ae9442d1556821bf61854c66e8ae819133e7f3578843ece8efc4174fd738bfca\": rpc error: code = NotFound desc = could not find container \"ae9442d1556821bf61854c66e8ae819133e7f3578843ece8efc4174fd738bfca\": container with ID starting with ae9442d1556821bf61854c66e8ae819133e7f3578843ece8efc4174fd738bfca not found: ID does not exist"
Sep 11 13:31:36.544590 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:31:36.544542    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7c9d797ff5-cmld7"]
Sep 11 13:31:36.553782 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:31:36.553746    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-7c9d797ff5-cmld7"]
Sep 11 13:31:37.107448 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:31:37.107405    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fad8b11d-4aba-4fe0-8f25-5a592bf0da60" path="/var/lib/kubelet/pods/fad8b11d-4aba-4fe0-8f25-5a592bf0da60/volumes"
Sep 11 13:31:57.135079 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:31:57.134985    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5d987ff8b5-pvm5s"]
Sep 11 13:31:57.135553 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:31:57.135480    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fad8b11d-4aba-4fe0-8f25-5a592bf0da60" containerName="manager"
Sep 11 13:31:57.135553 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:31:57.135498    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="fad8b11d-4aba-4fe0-8f25-5a592bf0da60" containerName="manager"
Sep 11 13:31:57.135627 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:31:57.135611    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="fad8b11d-4aba-4fe0-8f25-5a592bf0da60" containerName="manager"
Sep 11 13:31:57.142034 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:31:57.142006    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5d987ff8b5-pvm5s"
Sep 11 13:31:57.146360 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:31:57.146324    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 11 13:31:57.146585 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:31:57.146324    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-6zc5p\""
Sep 11 13:31:57.146585 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:31:57.146324    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 11 13:31:57.152419 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:31:57.152379    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5d987ff8b5-pvm5s"]
Sep 11 13:31:57.183340 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:31:57.183295    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3015eac6-2320-4916-82ab-fd3b873bbeae-metrics-certs\") pod \"maas-api-5d987ff8b5-pvm5s\" (UID: \"3015eac6-2320-4916-82ab-fd3b873bbeae\") " pod="odh-ai-gateway-infra/maas-api-5d987ff8b5-pvm5s"
Sep 11 13:31:57.183605 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:31:57.183386    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6k6s\" (UniqueName: \"kubernetes.io/projected/3015eac6-2320-4916-82ab-fd3b873bbeae-kube-api-access-t6k6s\") pod \"maas-api-5d987ff8b5-pvm5s\" (UID: \"3015eac6-2320-4916-82ab-fd3b873bbeae\") " pod="odh-ai-gateway-infra/maas-api-5d987ff8b5-pvm5s"
Sep 11 13:31:57.183605 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:31:57.183446    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3015eac6-2320-4916-82ab-fd3b873bbeae-openshift-service-ca\") pod \"maas-api-5d987ff8b5-pvm5s\" (UID: \"3015eac6-2320-4916-82ab-fd3b873bbeae\") " pod="odh-ai-gateway-infra/maas-api-5d987ff8b5-pvm5s"
Sep 11 13:31:57.183605 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:31:57.183484    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3015eac6-2320-4916-82ab-fd3b873bbeae-maas-api-tls\") pod \"maas-api-5d987ff8b5-pvm5s\" (UID: \"3015eac6-2320-4916-82ab-fd3b873bbeae\") " pod="odh-ai-gateway-infra/maas-api-5d987ff8b5-pvm5s"
Sep 11 13:31:57.285125 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:31:57.285075    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3015eac6-2320-4916-82ab-fd3b873bbeae-openshift-service-ca\") pod \"maas-api-5d987ff8b5-pvm5s\" (UID: \"3015eac6-2320-4916-82ab-fd3b873bbeae\") " pod="odh-ai-gateway-infra/maas-api-5d987ff8b5-pvm5s"
Sep 11 13:31:57.285125 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:31:57.285134    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3015eac6-2320-4916-82ab-fd3b873bbeae-maas-api-tls\") pod \"maas-api-5d987ff8b5-pvm5s\" (UID: \"3015eac6-2320-4916-82ab-fd3b873bbeae\") " pod="odh-ai-gateway-infra/maas-api-5d987ff8b5-pvm5s"
Sep 11 13:31:57.285425 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:31:57.285267    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3015eac6-2320-4916-82ab-fd3b873bbeae-metrics-certs\") pod \"maas-api-5d987ff8b5-pvm5s\" (UID: \"3015eac6-2320-4916-82ab-fd3b873bbeae\") " pod="odh-ai-gateway-infra/maas-api-5d987ff8b5-pvm5s"
Sep 11 13:31:57.285425 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:31:57.285314    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6k6s\" (UniqueName: \"kubernetes.io/projected/3015eac6-2320-4916-82ab-fd3b873bbeae-kube-api-access-t6k6s\") pod \"maas-api-5d987ff8b5-pvm5s\" (UID: \"3015eac6-2320-4916-82ab-fd3b873bbeae\") " pod="odh-ai-gateway-infra/maas-api-5d987ff8b5-pvm5s"
Sep 11 13:31:57.285982 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:31:57.285947    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3015eac6-2320-4916-82ab-fd3b873bbeae-openshift-service-ca\") pod \"maas-api-5d987ff8b5-pvm5s\" (UID: \"3015eac6-2320-4916-82ab-fd3b873bbeae\") " pod="odh-ai-gateway-infra/maas-api-5d987ff8b5-pvm5s"
Sep 11 13:31:57.288330 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:31:57.288298    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3015eac6-2320-4916-82ab-fd3b873bbeae-maas-api-tls\") pod \"maas-api-5d987ff8b5-pvm5s\" (UID: \"3015eac6-2320-4916-82ab-fd3b873bbeae\") " pod="odh-ai-gateway-infra/maas-api-5d987ff8b5-pvm5s"
Sep 11 13:31:57.288475 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:31:57.288426    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3015eac6-2320-4916-82ab-fd3b873bbeae-metrics-certs\") pod \"maas-api-5d987ff8b5-pvm5s\" (UID: \"3015eac6-2320-4916-82ab-fd3b873bbeae\") " pod="odh-ai-gateway-infra/maas-api-5d987ff8b5-pvm5s"
Sep 11 13:31:57.294959 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:31:57.294927    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6k6s\" (UniqueName: \"kubernetes.io/projected/3015eac6-2320-4916-82ab-fd3b873bbeae-kube-api-access-t6k6s\") pod \"maas-api-5d987ff8b5-pvm5s\" (UID: \"3015eac6-2320-4916-82ab-fd3b873bbeae\") " pod="odh-ai-gateway-infra/maas-api-5d987ff8b5-pvm5s"
Sep 11 13:31:57.455153 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:31:57.455010    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5d987ff8b5-pvm5s"
Sep 11 13:31:57.705905 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:31:57.705782    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-4cgf5"]
Sep 11 13:31:57.711746 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:31:57.711702    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-4cgf5"
Sep 11 13:31:57.714414 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:31:57.714382    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-rt9df\""
Sep 11 13:31:57.714414 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:31:57.714401    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Sep 11 13:31:57.726709 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:31:57.726669    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-4cgf5"]
Sep 11 13:31:57.741123 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:31:57.741082    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-c8r4s"]
Sep 11 13:31:57.745565 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:31:57.745534    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-c8r4s"
Sep 11 13:31:57.759377 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:31:57.759350    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-c8r4s"]
Sep 11 13:31:57.762447 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:31:57.762410    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5d987ff8b5-pvm5s"]
Sep 11 13:31:57.768186 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:31:57.767909    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3015eac6_2320_4916_82ab_fd3b873bbeae.slice/crio-f5af346e061981dd656c4583684ac7eac6fdc27acd4088ad2116fe53993ff4df WatchSource:0}: Error finding container f5af346e061981dd656c4583684ac7eac6fdc27acd4088ad2116fe53993ff4df: Status 404 returned error can't find the container with id f5af346e061981dd656c4583684ac7eac6fdc27acd4088ad2116fe53993ff4df
Sep 11 13:31:57.791621 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:31:57.791570    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/773e0144-50b8-4d59-8fed-57d39bb74da2-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-4cgf5\" (UID: \"773e0144-50b8-4d59-8fed-57d39bb74da2\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-4cgf5"
Sep 11 13:31:57.791872 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:31:57.791635    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cfdc9b69-bbd2-4073-8010-782f82019e75-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-c8r4s\" (UID: \"cfdc9b69-bbd2-4073-8010-782f82019e75\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-c8r4s"
Sep 11 13:31:57.791872 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:31:57.791679    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rq9v\" (UniqueName: \"kubernetes.io/projected/773e0144-50b8-4d59-8fed-57d39bb74da2-kube-api-access-9rq9v\") pod \"payload-processing-5b54b7fbb7-4cgf5\" (UID: \"773e0144-50b8-4d59-8fed-57d39bb74da2\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-4cgf5"
Sep 11 13:31:57.791872 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:31:57.791824    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsjdp\" (UniqueName: \"kubernetes.io/projected/cfdc9b69-bbd2-4073-8010-782f82019e75-kube-api-access-tsjdp\") pod \"payload-pre-processing-59fbc47bf8-c8r4s\" (UID: \"cfdc9b69-bbd2-4073-8010-782f82019e75\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-c8r4s"
Sep 11 13:31:57.893022 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:31:57.892969    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsjdp\" (UniqueName: \"kubernetes.io/projected/cfdc9b69-bbd2-4073-8010-782f82019e75-kube-api-access-tsjdp\") pod \"payload-pre-processing-59fbc47bf8-c8r4s\" (UID: \"cfdc9b69-bbd2-4073-8010-782f82019e75\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-c8r4s"
Sep 11 13:31:57.893281 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:31:57.893079    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/773e0144-50b8-4d59-8fed-57d39bb74da2-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-4cgf5\" (UID: \"773e0144-50b8-4d59-8fed-57d39bb74da2\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-4cgf5"
Sep 11 13:31:57.893281 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:31:57.893133    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cfdc9b69-bbd2-4073-8010-782f82019e75-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-c8r4s\" (UID: \"cfdc9b69-bbd2-4073-8010-782f82019e75\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-c8r4s"
Sep 11 13:31:57.893281 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:31:57.893202    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rq9v\" (UniqueName: \"kubernetes.io/projected/773e0144-50b8-4d59-8fed-57d39bb74da2-kube-api-access-9rq9v\") pod \"payload-processing-5b54b7fbb7-4cgf5\" (UID: \"773e0144-50b8-4d59-8fed-57d39bb74da2\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-4cgf5"
Sep 11 13:31:57.893978 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:31:57.893948    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cfdc9b69-bbd2-4073-8010-782f82019e75-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-c8r4s\" (UID: \"cfdc9b69-bbd2-4073-8010-782f82019e75\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-c8r4s"
Sep 11 13:31:57.893978 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:31:57.893964    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/773e0144-50b8-4d59-8fed-57d39bb74da2-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-4cgf5\" (UID: \"773e0144-50b8-4d59-8fed-57d39bb74da2\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-4cgf5"
Sep 11 13:31:57.910748 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:31:57.910707    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsjdp\" (UniqueName: \"kubernetes.io/projected/cfdc9b69-bbd2-4073-8010-782f82019e75-kube-api-access-tsjdp\") pod \"payload-pre-processing-59fbc47bf8-c8r4s\" (UID: \"cfdc9b69-bbd2-4073-8010-782f82019e75\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-c8r4s"
Sep 11 13:31:57.911916 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:31:57.911875    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rq9v\" (UniqueName: \"kubernetes.io/projected/773e0144-50b8-4d59-8fed-57d39bb74da2-kube-api-access-9rq9v\") pod \"payload-processing-5b54b7fbb7-4cgf5\" (UID: \"773e0144-50b8-4d59-8fed-57d39bb74da2\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-4cgf5"
Sep 11 13:31:58.026750 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:31:58.026695    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-4cgf5"
Sep 11 13:31:58.059250 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:31:58.059202    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-c8r4s"
Sep 11 13:31:58.324663 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:31:58.324533    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-4cgf5"]
Sep 11 13:31:58.327605 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:31:58.327561    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod773e0144_50b8_4d59_8fed_57d39bb74da2.slice/crio-0866af90d7ababd55f5e9f4516fb28afd21185b4710d890ff24900f41f7bee30 WatchSource:0}: Error finding container 0866af90d7ababd55f5e9f4516fb28afd21185b4710d890ff24900f41f7bee30: Status 404 returned error can't find the container with id 0866af90d7ababd55f5e9f4516fb28afd21185b4710d890ff24900f41f7bee30
Sep 11 13:31:58.365761 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:31:58.365711    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-c8r4s"]
Sep 11 13:31:58.369132 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:31:58.369093    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfdc9b69_bbd2_4073_8010_782f82019e75.slice/crio-1316c50fd694d93a4e2c9f72c95f11b45fa3b92198ba4b298b53c59329fbbe4a WatchSource:0}: Error finding container 1316c50fd694d93a4e2c9f72c95f11b45fa3b92198ba4b298b53c59329fbbe4a: Status 404 returned error can't find the container with id 1316c50fd694d93a4e2c9f72c95f11b45fa3b92198ba4b298b53c59329fbbe4a
Sep 11 13:31:58.648898 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:31:58.648788    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-4cgf5" event={"ID":"773e0144-50b8-4d59-8fed-57d39bb74da2","Type":"ContainerStarted","Data":"0866af90d7ababd55f5e9f4516fb28afd21185b4710d890ff24900f41f7bee30"}
Sep 11 13:31:58.650403 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:31:58.650368    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5d987ff8b5-pvm5s" event={"ID":"3015eac6-2320-4916-82ab-fd3b873bbeae","Type":"ContainerStarted","Data":"f5af346e061981dd656c4583684ac7eac6fdc27acd4088ad2116fe53993ff4df"}
Sep 11 13:31:58.652339 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:31:58.652302    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-c8r4s" event={"ID":"cfdc9b69-bbd2-4073-8010-782f82019e75","Type":"ContainerStarted","Data":"1316c50fd694d93a4e2c9f72c95f11b45fa3b92198ba4b298b53c59329fbbe4a"}
Sep 11 13:32:01.677583 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:32:01.677074    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-4cgf5" event={"ID":"773e0144-50b8-4d59-8fed-57d39bb74da2","Type":"ContainerStarted","Data":"e95c84bc00d60be1c4c37ca5780fb1c4dd605a8813104311f4e0b12e76cf4799"}
Sep 11 13:32:01.677583 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:32:01.677319    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-4cgf5"
Sep 11 13:32:01.678974 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:32:01.678936    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5d987ff8b5-pvm5s" event={"ID":"3015eac6-2320-4916-82ab-fd3b873bbeae","Type":"ContainerStarted","Data":"05099fdfb20539272138a0265fcefcb57d969ada3d37272678d8ed236c3ceac5"}
Sep 11 13:32:01.679159 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:32:01.679019    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5d987ff8b5-pvm5s"
Sep 11 13:32:01.681187 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:32:01.681156    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-c8r4s" event={"ID":"cfdc9b69-bbd2-4073-8010-782f82019e75","Type":"ContainerStarted","Data":"e1700b8cc10e37548cec29084660b3afe8fda8dbb0fb50c8f4675f1c4c83998b"}
Sep 11 13:32:01.681367 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:32:01.681254    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-c8r4s"
Sep 11 13:32:01.726609 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:32:01.726515    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5b54b7fbb7-4cgf5" podStartSLOduration=2.074664609 podStartE2EDuration="4.726481935s" totalImagesPullingTime="2.651817326s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:31:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:32:01.69814827 +0000 UTC m=+629.084251435" watchObservedRunningTime="2026-09-11 13:32:01.726481935 +0000 UTC m=+629.112585077"
Sep 11 13:32:01.727183 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:32:01.727131    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-c8r4s" podStartSLOduration=2.082774585 podStartE2EDuration="4.727113439s" totalImagesPullingTime="2.644338854s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:31:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:32:01.716531172 +0000 UTC m=+629.102634361" watchObservedRunningTime="2026-09-11 13:32:01.727113439 +0000 UTC m=+629.113216603"
Sep 11 13:32:01.743693 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:32:01.743626    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5d987ff8b5-pvm5s" podStartSLOduration=1.531048217 podStartE2EDuration="4.743598671s" totalImagesPullingTime="3.212550454s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:31:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:32:01.740360687 +0000 UTC m=+629.126463875" watchObservedRunningTime="2026-09-11 13:32:01.743598671 +0000 UTC m=+629.129701837"
Sep 11 13:32:07.695518 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:32:07.695474    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5d987ff8b5-pvm5s"
Sep 11 13:32:08.735381 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:32:08.735333    2578 generic.go:356] "Generic (PLEG): container finished" podID="b1f2ae7a-43d1-4043-a1fc-9d59b34d1b90" containerID="410fe54b5a0db2b0306992367ab1a8afd8be4a1e0bb702b3a8ed7045db7e2b19" exitCode=0
Sep 11 13:32:08.735921 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:32:08.735412    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-6kwrp" event={"ID":"b1f2ae7a-43d1-4043-a1fc-9d59b34d1b90","Type":"ContainerDied","Data":"410fe54b5a0db2b0306992367ab1a8afd8be4a1e0bb702b3a8ed7045db7e2b19"}
Sep 11 13:32:08.736006 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:32:08.735989    2578 scope.go:117] "RemoveContainer" containerID="410fe54b5a0db2b0306992367ab1a8afd8be4a1e0bb702b3a8ed7045db7e2b19"
Sep 11 13:32:08.737256 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:32:08.737214    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 11 13:32:08.737405 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:32:08.737330    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 11 13:32:08.737405 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:32:08.737386    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 11 13:32:09.743759 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:32:09.743720    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-6kwrp" event={"ID":"b1f2ae7a-43d1-4043-a1fc-9d59b34d1b90","Type":"ContainerStarted","Data":"afc3c08088f3e9160a1a5c5a9d0004bfa45ce24533a6cbe02f9166c4e2cce424"}
Sep 11 13:32:10.621291 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:32:10.621229    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-6kwrp"
Sep 11 13:32:10.623143 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:32:10.623111    2578 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-6kwrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Sep 11 13:32:10.623361 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:32:10.623176    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-6kwrp" podUID="b1f2ae7a-43d1-4043-a1fc-9d59b34d1b90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Sep 11 13:32:11.038323 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:32:11.038278    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-6kwrp"
Sep 11 13:32:11.621819 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:32:11.621762    2578 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-6kwrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Sep 11 13:32:11.622139 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:32:11.621883    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-6kwrp" podUID="b1f2ae7a-43d1-4043-a1fc-9d59b34d1b90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Sep 11 13:32:12.622097 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:32:12.622048    2578 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-6kwrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Sep 11 13:32:12.622585 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:32:12.622126    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-6kwrp" podUID="b1f2ae7a-43d1-4043-a1fc-9d59b34d1b90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Sep 11 13:32:12.693133 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:32:12.693080    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-4cgf5"
Sep 11 13:32:12.693408 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:32:12.693203    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-c8r4s"
Sep 11 13:32:13.621809 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:32:13.621762    2578 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-6kwrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Sep 11 13:32:13.622095 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:32:13.621849    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-6kwrp" podUID="b1f2ae7a-43d1-4043-a1fc-9d59b34d1b90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Sep 11 13:32:14.621197 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:32:14.621155    2578 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-6kwrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Sep 11 13:32:14.621686 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:32:14.621220    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-6kwrp" podUID="b1f2ae7a-43d1-4043-a1fc-9d59b34d1b90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Sep 11 13:32:15.627007 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:32:15.626945    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-6kwrp"
Sep 11 13:32:15.628502 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:32:15.628470    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-6kwrp"
Sep 11 13:32:35.448183 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:32:35.448139    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-csbw2"]
Sep 11 13:32:35.454172 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:32:35.454132    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-csbw2"
Sep 11 13:32:35.458304 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:32:35.458270    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 11 13:32:35.458304 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:32:35.458306    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\""
Sep 11 13:32:35.458594 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:32:35.458271    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-6z2kw\""
Sep 11 13:32:35.458594 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:32:35.458271    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 11 13:32:35.462455 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:32:35.462419    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-csbw2"]
Sep 11 13:32:35.513675 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:32:35.513611    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/afbfd468-29d1-4f10-9703-52cfa77cf44d-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-csbw2\" (UID: \"afbfd468-29d1-4f10-9703-52cfa77cf44d\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-csbw2"
Sep 11 13:32:35.513675 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:32:35.513678    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/afbfd468-29d1-4f10-9703-52cfa77cf44d-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-csbw2\" (UID: \"afbfd468-29d1-4f10-9703-52cfa77cf44d\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-csbw2"
Sep 11 13:32:35.514047 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:32:35.513722    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmswz\" (UniqueName: \"kubernetes.io/projected/afbfd468-29d1-4f10-9703-52cfa77cf44d-kube-api-access-dmswz\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-csbw2\" (UID: \"afbfd468-29d1-4f10-9703-52cfa77cf44d\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-csbw2"
Sep 11 13:32:35.514047 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:32:35.513786    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/afbfd468-29d1-4f10-9703-52cfa77cf44d-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-csbw2\" (UID: \"afbfd468-29d1-4f10-9703-52cfa77cf44d\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-csbw2"
Sep 11 13:32:35.514047 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:32:35.513932    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/afbfd468-29d1-4f10-9703-52cfa77cf44d-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-csbw2\" (UID: \"afbfd468-29d1-4f10-9703-52cfa77cf44d\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-csbw2"
Sep 11 13:32:35.514047 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:32:35.513968    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/afbfd468-29d1-4f10-9703-52cfa77cf44d-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-csbw2\" (UID: \"afbfd468-29d1-4f10-9703-52cfa77cf44d\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-csbw2"
Sep 11 13:32:35.615316 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:32:35.615269    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/afbfd468-29d1-4f10-9703-52cfa77cf44d-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-csbw2\" (UID: \"afbfd468-29d1-4f10-9703-52cfa77cf44d\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-csbw2"
Sep 11 13:32:35.615316 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:32:35.615321    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/afbfd468-29d1-4f10-9703-52cfa77cf44d-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-csbw2\" (UID: \"afbfd468-29d1-4f10-9703-52cfa77cf44d\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-csbw2"
Sep 11 13:32:35.615590 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:32:35.615451    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/afbfd468-29d1-4f10-9703-52cfa77cf44d-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-csbw2\" (UID: \"afbfd468-29d1-4f10-9703-52cfa77cf44d\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-csbw2"
Sep 11 13:32:35.615590 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:32:35.615509    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/afbfd468-29d1-4f10-9703-52cfa77cf44d-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-csbw2\" (UID: \"afbfd468-29d1-4f10-9703-52cfa77cf44d\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-csbw2"
Sep 11 13:32:35.615590 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:32:35.615536    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmswz\" (UniqueName: \"kubernetes.io/projected/afbfd468-29d1-4f10-9703-52cfa77cf44d-kube-api-access-dmswz\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-csbw2\" (UID: \"afbfd468-29d1-4f10-9703-52cfa77cf44d\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-csbw2"
Sep 11 13:32:35.615733 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:32:35.615581    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/afbfd468-29d1-4f10-9703-52cfa77cf44d-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-csbw2\" (UID: \"afbfd468-29d1-4f10-9703-52cfa77cf44d\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-csbw2"
Sep 11 13:32:35.615899 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:32:35.615876    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/afbfd468-29d1-4f10-9703-52cfa77cf44d-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-csbw2\" (UID: \"afbfd468-29d1-4f10-9703-52cfa77cf44d\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-csbw2"
Sep 11 13:32:35.615963 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:32:35.615920    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/afbfd468-29d1-4f10-9703-52cfa77cf44d-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-csbw2\" (UID: \"afbfd468-29d1-4f10-9703-52cfa77cf44d\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-csbw2"
Sep 11 13:32:35.616133 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:32:35.616114    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/afbfd468-29d1-4f10-9703-52cfa77cf44d-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-csbw2\" (UID: \"afbfd468-29d1-4f10-9703-52cfa77cf44d\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-csbw2"
Sep 11 13:32:35.618461 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:32:35.618418    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/afbfd468-29d1-4f10-9703-52cfa77cf44d-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-csbw2\" (UID: \"afbfd468-29d1-4f10-9703-52cfa77cf44d\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-csbw2"
Sep 11 13:32:35.618693 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:32:35.618674    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/afbfd468-29d1-4f10-9703-52cfa77cf44d-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-csbw2\" (UID: \"afbfd468-29d1-4f10-9703-52cfa77cf44d\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-csbw2"
Sep 11 13:32:35.625677 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:32:35.625646    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmswz\" (UniqueName: \"kubernetes.io/projected/afbfd468-29d1-4f10-9703-52cfa77cf44d-kube-api-access-dmswz\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-csbw2\" (UID: \"afbfd468-29d1-4f10-9703-52cfa77cf44d\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-csbw2"
Sep 11 13:32:35.768549 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:32:35.768500    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-csbw2"
Sep 11 13:32:36.056621 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:32:36.056578    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-csbw2"]
Sep 11 13:32:36.059557 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:32:36.059514    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafbfd468_29d1_4f10_9703_52cfa77cf44d.slice/crio-7be421e0f7319336404c962627e1216c770e3c5682960877dca7963f186d3038 WatchSource:0}: Error finding container 7be421e0f7319336404c962627e1216c770e3c5682960877dca7963f186d3038: Status 404 returned error can't find the container with id 7be421e0f7319336404c962627e1216c770e3c5682960877dca7963f186d3038
Sep 11 13:32:36.061312 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:32:36.061289    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 13:32:36.924815 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:32:36.924760    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-csbw2" event={"ID":"afbfd468-29d1-4f10-9703-52cfa77cf44d","Type":"ContainerStarted","Data":"7be421e0f7319336404c962627e1216c770e3c5682960877dca7963f186d3038"}
Sep 11 13:32:37.932649 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:32:37.932608    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-csbw2" event={"ID":"afbfd468-29d1-4f10-9703-52cfa77cf44d","Type":"ContainerStarted","Data":"fc8c3e48cb7547d90ac8aee0bcb15f924486be85fbd569ecf93c0bebfc7b1c17"}
Sep 11 13:32:37.952200 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:32:37.952132    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-csbw2" podStartSLOduration=1.93069508 podStartE2EDuration="2.952111095s" totalImagesPullingTime="1.021416015s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:32:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:32:37.94968805 +0000 UTC m=+665.335791233" watchObservedRunningTime="2026-09-11 13:32:37.952111095 +0000 UTC m=+665.338214306"
Sep 11 13:32:45.769651 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:32:45.769595    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-csbw2"
Sep 11 13:32:45.770230 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:32:45.769689    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-csbw2"
Sep 11 13:32:45.783740 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:32:45.783697    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-csbw2"
Sep 11 13:32:45.994223 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:32:45.994184    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-csbw2"
Sep 11 13:33:05.844411 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:33:05.844355    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-h7xf4"]
Sep 11 13:33:05.849543 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:33:05.849509    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-h7xf4"
Sep 11 13:33:05.852181 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:33:05.852153    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Sep 11 13:33:05.858159 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:33:05.858128    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-h7xf4"]
Sep 11 13:33:05.969308 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:33:05.969257    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/375e4146-1062-42bd-83e7-c243c0ce730f-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-h7xf4\" (UID: \"375e4146-1062-42bd-83e7-c243c0ce730f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-h7xf4"
Sep 11 13:33:05.969551 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:33:05.969338    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/375e4146-1062-42bd-83e7-c243c0ce730f-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-h7xf4\" (UID: \"375e4146-1062-42bd-83e7-c243c0ce730f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-h7xf4"
Sep 11 13:33:05.969551 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:33:05.969410    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4tph\" (UniqueName: \"kubernetes.io/projected/375e4146-1062-42bd-83e7-c243c0ce730f-kube-api-access-q4tph\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-h7xf4\" (UID: \"375e4146-1062-42bd-83e7-c243c0ce730f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-h7xf4"
Sep 11 13:33:05.969551 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:33:05.969440    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/375e4146-1062-42bd-83e7-c243c0ce730f-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-h7xf4\" (UID: \"375e4146-1062-42bd-83e7-c243c0ce730f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-h7xf4"
Sep 11 13:33:05.969551 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:33:05.969492    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/375e4146-1062-42bd-83e7-c243c0ce730f-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-h7xf4\" (UID: \"375e4146-1062-42bd-83e7-c243c0ce730f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-h7xf4"
Sep 11 13:33:05.969711 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:33:05.969583    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/375e4146-1062-42bd-83e7-c243c0ce730f-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-h7xf4\" (UID: \"375e4146-1062-42bd-83e7-c243c0ce730f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-h7xf4"
Sep 11 13:33:06.071249 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:33:06.071168    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/375e4146-1062-42bd-83e7-c243c0ce730f-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-h7xf4\" (UID: \"375e4146-1062-42bd-83e7-c243c0ce730f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-h7xf4"
Sep 11 13:33:06.071554 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:33:06.071274    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/375e4146-1062-42bd-83e7-c243c0ce730f-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-h7xf4\" (UID: \"375e4146-1062-42bd-83e7-c243c0ce730f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-h7xf4"
Sep 11 13:33:06.071554 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:33:06.071342    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4tph\" (UniqueName: \"kubernetes.io/projected/375e4146-1062-42bd-83e7-c243c0ce730f-kube-api-access-q4tph\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-h7xf4\" (UID: \"375e4146-1062-42bd-83e7-c243c0ce730f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-h7xf4"
Sep 11 13:33:06.071554 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:33:06.071378    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/375e4146-1062-42bd-83e7-c243c0ce730f-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-h7xf4\" (UID: \"375e4146-1062-42bd-83e7-c243c0ce730f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-h7xf4"
Sep 11 13:33:06.071554 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:33:06.071435    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/375e4146-1062-42bd-83e7-c243c0ce730f-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-h7xf4\" (UID: \"375e4146-1062-42bd-83e7-c243c0ce730f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-h7xf4"
Sep 11 13:33:06.071554 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:33:06.071549    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/375e4146-1062-42bd-83e7-c243c0ce730f-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-h7xf4\" (UID: \"375e4146-1062-42bd-83e7-c243c0ce730f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-h7xf4"
Sep 11 13:33:06.071827 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:33:06.071747    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/375e4146-1062-42bd-83e7-c243c0ce730f-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-h7xf4\" (UID: \"375e4146-1062-42bd-83e7-c243c0ce730f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-h7xf4"
Sep 11 13:33:06.071827 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:33:06.071761    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/375e4146-1062-42bd-83e7-c243c0ce730f-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-h7xf4\" (UID: \"375e4146-1062-42bd-83e7-c243c0ce730f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-h7xf4"
Sep 11 13:33:06.071827 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:33:06.071883    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/375e4146-1062-42bd-83e7-c243c0ce730f-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-h7xf4\" (UID: \"375e4146-1062-42bd-83e7-c243c0ce730f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-h7xf4"
Sep 11 13:33:06.074886 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:33:06.074778    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/375e4146-1062-42bd-83e7-c243c0ce730f-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-h7xf4\" (UID: \"375e4146-1062-42bd-83e7-c243c0ce730f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-h7xf4"
Sep 11 13:33:06.075158 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:33:06.075142    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/375e4146-1062-42bd-83e7-c243c0ce730f-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-h7xf4\" (UID: \"375e4146-1062-42bd-83e7-c243c0ce730f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-h7xf4"
Sep 11 13:33:06.080379 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:33:06.080352    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4tph\" (UniqueName: \"kubernetes.io/projected/375e4146-1062-42bd-83e7-c243c0ce730f-kube-api-access-q4tph\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-h7xf4\" (UID: \"375e4146-1062-42bd-83e7-c243c0ce730f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-h7xf4"
Sep 11 13:33:06.164279 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:33:06.164161    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-h7xf4"
Sep 11 13:33:06.451244 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:33:06.451211    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-h7xf4"]
Sep 11 13:33:06.453592 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:33:06.453560    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod375e4146_1062_42bd_83e7_c243c0ce730f.slice/crio-9103745b9db0069ab85a6a50b445f56684d4c557a0af533e40867a9f0fae4300 WatchSource:0}: Error finding container 9103745b9db0069ab85a6a50b445f56684d4c557a0af533e40867a9f0fae4300: Status 404 returned error can't find the container with id 9103745b9db0069ab85a6a50b445f56684d4c557a0af533e40867a9f0fae4300
Sep 11 13:33:07.110293 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:33:07.110250    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-h7xf4" event={"ID":"375e4146-1062-42bd-83e7-c243c0ce730f","Type":"ContainerStarted","Data":"3d788b348e7df70af192df802fc117f5219c668bb34e2314a003e5bfb1810879"}
Sep 11 13:33:07.110798 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:33:07.110325    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-h7xf4" event={"ID":"375e4146-1062-42bd-83e7-c243c0ce730f","Type":"ContainerStarted","Data":"9103745b9db0069ab85a6a50b445f56684d4c557a0af533e40867a9f0fae4300"}
Sep 11 13:33:07.128962 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:33:07.128769    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-h7xf4" podStartSLOduration=1.922596368 podStartE2EDuration="2.128736175s" totalImagesPullingTime="206.139807ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:33:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:33:07.128218585 +0000 UTC m=+694.514321748" watchObservedRunningTime="2026-09-11 13:33:07.128736175 +0000 UTC m=+694.514839352"
Sep 11 13:33:16.164547 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:33:16.164499    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-h7xf4"
Sep 11 13:33:16.165098 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:33:16.164600    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-h7xf4"
Sep 11 13:33:16.178339 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:33:16.178300    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-h7xf4"
Sep 11 13:33:17.186137 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:33:17.186091    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-h7xf4"
Sep 11 13:35:11.966073 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:11.966026    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-371c38-openshift-default-668769478-66p8n"]
Sep 11 13:35:11.970194 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:11.970170    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-371c38-openshift-default-668769478-66p8n"
Sep 11 13:35:11.973873 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:11.973817    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-371c38-openshift-default-dockercfg-fz5bd\""
Sep 11 13:35:12.093979 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:12.093928    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/301e0569-db5c-45be-865c-491221d4883c-istio-data\") pod \"e2e-worker-w4-371c38-openshift-default-668769478-66p8n\" (UID: \"301e0569-db5c-45be-865c-491221d4883c\") " pod="openshift-ingress/e2e-worker-w4-371c38-openshift-default-668769478-66p8n"
Sep 11 13:35:12.094211 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:12.094000    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/301e0569-db5c-45be-865c-491221d4883c-workload-certs\") pod \"e2e-worker-w4-371c38-openshift-default-668769478-66p8n\" (UID: \"301e0569-db5c-45be-865c-491221d4883c\") " pod="openshift-ingress/e2e-worker-w4-371c38-openshift-default-668769478-66p8n"
Sep 11 13:35:12.094268 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:12.094226    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/301e0569-db5c-45be-865c-491221d4883c-istiod-ca-cert\") pod \"e2e-worker-w4-371c38-openshift-default-668769478-66p8n\" (UID: \"301e0569-db5c-45be-865c-491221d4883c\") " pod="openshift-ingress/e2e-worker-w4-371c38-openshift-default-668769478-66p8n"
Sep 11 13:35:12.094321 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:12.094303    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/301e0569-db5c-45be-865c-491221d4883c-workload-socket\") pod \"e2e-worker-w4-371c38-openshift-default-668769478-66p8n\" (UID: \"301e0569-db5c-45be-865c-491221d4883c\") " pod="openshift-ingress/e2e-worker-w4-371c38-openshift-default-668769478-66p8n"
Sep 11 13:35:12.094382 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:12.094331    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/301e0569-db5c-45be-865c-491221d4883c-credential-socket\") pod \"e2e-worker-w4-371c38-openshift-default-668769478-66p8n\" (UID: \"301e0569-db5c-45be-865c-491221d4883c\") " pod="openshift-ingress/e2e-worker-w4-371c38-openshift-default-668769478-66p8n"
Sep 11 13:35:12.094382 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:12.094350    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/301e0569-db5c-45be-865c-491221d4883c-istio-token\") pod \"e2e-worker-w4-371c38-openshift-default-668769478-66p8n\" (UID: \"301e0569-db5c-45be-865c-491221d4883c\") " pod="openshift-ingress/e2e-worker-w4-371c38-openshift-default-668769478-66p8n"
Sep 11 13:35:12.094486 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:12.094383    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/301e0569-db5c-45be-865c-491221d4883c-istio-podinfo\") pod \"e2e-worker-w4-371c38-openshift-default-668769478-66p8n\" (UID: \"301e0569-db5c-45be-865c-491221d4883c\") " pod="openshift-ingress/e2e-worker-w4-371c38-openshift-default-668769478-66p8n"
Sep 11 13:35:12.094486 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:12.094414    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/301e0569-db5c-45be-865c-491221d4883c-istio-envoy\") pod \"e2e-worker-w4-371c38-openshift-default-668769478-66p8n\" (UID: \"301e0569-db5c-45be-865c-491221d4883c\") " pod="openshift-ingress/e2e-worker-w4-371c38-openshift-default-668769478-66p8n"
Sep 11 13:35:12.094486 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:12.094439    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zd7vf\" (UniqueName: \"kubernetes.io/projected/301e0569-db5c-45be-865c-491221d4883c-kube-api-access-zd7vf\") pod \"e2e-worker-w4-371c38-openshift-default-668769478-66p8n\" (UID: \"301e0569-db5c-45be-865c-491221d4883c\") " pod="openshift-ingress/e2e-worker-w4-371c38-openshift-default-668769478-66p8n"
Sep 11 13:35:12.195264 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:12.195205    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/301e0569-db5c-45be-865c-491221d4883c-workload-socket\") pod \"e2e-worker-w4-371c38-openshift-default-668769478-66p8n\" (UID: \"301e0569-db5c-45be-865c-491221d4883c\") " pod="openshift-ingress/e2e-worker-w4-371c38-openshift-default-668769478-66p8n"
Sep 11 13:35:12.195264 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:12.195272    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/301e0569-db5c-45be-865c-491221d4883c-credential-socket\") pod \"e2e-worker-w4-371c38-openshift-default-668769478-66p8n\" (UID: \"301e0569-db5c-45be-865c-491221d4883c\") " pod="openshift-ingress/e2e-worker-w4-371c38-openshift-default-668769478-66p8n"
Sep 11 13:35:12.195624 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:12.195300    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/301e0569-db5c-45be-865c-491221d4883c-istio-token\") pod \"e2e-worker-w4-371c38-openshift-default-668769478-66p8n\" (UID: \"301e0569-db5c-45be-865c-491221d4883c\") " pod="openshift-ingress/e2e-worker-w4-371c38-openshift-default-668769478-66p8n"
Sep 11 13:35:12.195624 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:12.195340    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/301e0569-db5c-45be-865c-491221d4883c-istio-podinfo\") pod \"e2e-worker-w4-371c38-openshift-default-668769478-66p8n\" (UID: \"301e0569-db5c-45be-865c-491221d4883c\") " pod="openshift-ingress/e2e-worker-w4-371c38-openshift-default-668769478-66p8n"
Sep 11 13:35:12.195811 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:12.195776    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/301e0569-db5c-45be-865c-491221d4883c-workload-socket\") pod \"e2e-worker-w4-371c38-openshift-default-668769478-66p8n\" (UID: \"301e0569-db5c-45be-865c-491221d4883c\") " pod="openshift-ingress/e2e-worker-w4-371c38-openshift-default-668769478-66p8n"
Sep 11 13:35:12.195909 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:12.195812    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/301e0569-db5c-45be-865c-491221d4883c-credential-socket\") pod \"e2e-worker-w4-371c38-openshift-default-668769478-66p8n\" (UID: \"301e0569-db5c-45be-865c-491221d4883c\") " pod="openshift-ingress/e2e-worker-w4-371c38-openshift-default-668769478-66p8n"
Sep 11 13:35:12.195909 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:12.195813    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/301e0569-db5c-45be-865c-491221d4883c-istio-envoy\") pod \"e2e-worker-w4-371c38-openshift-default-668769478-66p8n\" (UID: \"301e0569-db5c-45be-865c-491221d4883c\") " pod="openshift-ingress/e2e-worker-w4-371c38-openshift-default-668769478-66p8n"
Sep 11 13:35:12.195909 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:12.195902    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zd7vf\" (UniqueName: \"kubernetes.io/projected/301e0569-db5c-45be-865c-491221d4883c-kube-api-access-zd7vf\") pod \"e2e-worker-w4-371c38-openshift-default-668769478-66p8n\" (UID: \"301e0569-db5c-45be-865c-491221d4883c\") " pod="openshift-ingress/e2e-worker-w4-371c38-openshift-default-668769478-66p8n"
Sep 11 13:35:12.196053 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:12.195980    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/301e0569-db5c-45be-865c-491221d4883c-istio-data\") pod \"e2e-worker-w4-371c38-openshift-default-668769478-66p8n\" (UID: \"301e0569-db5c-45be-865c-491221d4883c\") " pod="openshift-ingress/e2e-worker-w4-371c38-openshift-default-668769478-66p8n"
Sep 11 13:35:12.196296 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:12.196187    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/301e0569-db5c-45be-865c-491221d4883c-workload-certs\") pod \"e2e-worker-w4-371c38-openshift-default-668769478-66p8n\" (UID: \"301e0569-db5c-45be-865c-491221d4883c\") " pod="openshift-ingress/e2e-worker-w4-371c38-openshift-default-668769478-66p8n"
Sep 11 13:35:12.196502 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:12.196332    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/301e0569-db5c-45be-865c-491221d4883c-istio-data\") pod \"e2e-worker-w4-371c38-openshift-default-668769478-66p8n\" (UID: \"301e0569-db5c-45be-865c-491221d4883c\") " pod="openshift-ingress/e2e-worker-w4-371c38-openshift-default-668769478-66p8n"
Sep 11 13:35:12.196502 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:12.196447    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/301e0569-db5c-45be-865c-491221d4883c-istiod-ca-cert\") pod \"e2e-worker-w4-371c38-openshift-default-668769478-66p8n\" (UID: \"301e0569-db5c-45be-865c-491221d4883c\") " pod="openshift-ingress/e2e-worker-w4-371c38-openshift-default-668769478-66p8n"
Sep 11 13:35:12.196676 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:12.196640    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/301e0569-db5c-45be-865c-491221d4883c-workload-certs\") pod \"e2e-worker-w4-371c38-openshift-default-668769478-66p8n\" (UID: \"301e0569-db5c-45be-865c-491221d4883c\") " pod="openshift-ingress/e2e-worker-w4-371c38-openshift-default-668769478-66p8n"
Sep 11 13:35:12.197311 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:12.197273    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/301e0569-db5c-45be-865c-491221d4883c-istiod-ca-cert\") pod \"e2e-worker-w4-371c38-openshift-default-668769478-66p8n\" (UID: \"301e0569-db5c-45be-865c-491221d4883c\") " pod="openshift-ingress/e2e-worker-w4-371c38-openshift-default-668769478-66p8n"
Sep 11 13:35:12.199658 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:12.199627    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/301e0569-db5c-45be-865c-491221d4883c-istio-envoy\") pod \"e2e-worker-w4-371c38-openshift-default-668769478-66p8n\" (UID: \"301e0569-db5c-45be-865c-491221d4883c\") " pod="openshift-ingress/e2e-worker-w4-371c38-openshift-default-668769478-66p8n"
Sep 11 13:35:12.199896 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:12.199874    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/301e0569-db5c-45be-865c-491221d4883c-istio-podinfo\") pod \"e2e-worker-w4-371c38-openshift-default-668769478-66p8n\" (UID: \"301e0569-db5c-45be-865c-491221d4883c\") " pod="openshift-ingress/e2e-worker-w4-371c38-openshift-default-668769478-66p8n"
Sep 11 13:35:12.219945 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:12.219775    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/301e0569-db5c-45be-865c-491221d4883c-istio-token\") pod \"e2e-worker-w4-371c38-openshift-default-668769478-66p8n\" (UID: \"301e0569-db5c-45be-865c-491221d4883c\") " pod="openshift-ingress/e2e-worker-w4-371c38-openshift-default-668769478-66p8n"
Sep 11 13:35:12.239011 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:12.238965    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zd7vf\" (UniqueName: \"kubernetes.io/projected/301e0569-db5c-45be-865c-491221d4883c-kube-api-access-zd7vf\") pod \"e2e-worker-w4-371c38-openshift-default-668769478-66p8n\" (UID: \"301e0569-db5c-45be-865c-491221d4883c\") " pod="openshift-ingress/e2e-worker-w4-371c38-openshift-default-668769478-66p8n"
Sep 11 13:35:12.283485 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:12.283430    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-371c38-openshift-default-668769478-66p8n"
Sep 11 13:35:12.309433 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:12.309339    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-371c38-openshift-default-668769478-66p8n"]
Sep 11 13:35:12.572747 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:12.572656    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-082f62-openshift-default-7d446b6f4d-cvpwk"]
Sep 11 13:35:12.577986 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:12.577958    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-082f62-openshift-default-7d446b6f4d-cvpwk"
Sep 11 13:35:12.585656 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:12.585622    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-082f62-openshift-default-dockercfg-nvkws\""
Sep 11 13:35:12.659137 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:35:12.659080    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod301e0569_db5c_45be_865c_491221d4883c.slice/crio-46e8d25ea9afac6f6d67aadfb6cdf773480a44955679882094fb42460861ddc9 WatchSource:0}: Error finding container 46e8d25ea9afac6f6d67aadfb6cdf773480a44955679882094fb42460861ddc9: Status 404 returned error can't find the container with id 46e8d25ea9afac6f6d67aadfb6cdf773480a44955679882094fb42460861ddc9
Sep 11 13:35:12.661172 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:12.661126    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-371c38-openshift-default-668769478-66p8n"]
Sep 11 13:35:12.662592 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:12.662548    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 11 13:35:12.662791 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:12.662764    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 11 13:35:12.663012 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:12.662987    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 11 13:35:12.665646 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:12.665613    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-082f62-openshift-default-7d446b6f4d-cvpwk"]
Sep 11 13:35:12.702511 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:12.702466    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5eb98d8-6523-411d-bcfb-ad46743c0c72-workload-socket\") pod \"e2e-worker-w0-082f62-openshift-default-7d446b6f4d-cvpwk\" (UID: \"f5eb98d8-6523-411d-bcfb-ad46743c0c72\") " pod="openshift-ingress/e2e-worker-w0-082f62-openshift-default-7d446b6f4d-cvpwk"
Sep 11 13:35:12.702511 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:12.702512    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5eb98d8-6523-411d-bcfb-ad46743c0c72-workload-certs\") pod \"e2e-worker-w0-082f62-openshift-default-7d446b6f4d-cvpwk\" (UID: \"f5eb98d8-6523-411d-bcfb-ad46743c0c72\") " pod="openshift-ingress/e2e-worker-w0-082f62-openshift-default-7d446b6f4d-cvpwk"
Sep 11 13:35:12.702774 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:12.702585    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5eb98d8-6523-411d-bcfb-ad46743c0c72-istio-envoy\") pod \"e2e-worker-w0-082f62-openshift-default-7d446b6f4d-cvpwk\" (UID: \"f5eb98d8-6523-411d-bcfb-ad46743c0c72\") " pod="openshift-ingress/e2e-worker-w0-082f62-openshift-default-7d446b6f4d-cvpwk"
Sep 11 13:35:12.702774 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:12.702607    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5eb98d8-6523-411d-bcfb-ad46743c0c72-istio-token\") pod \"e2e-worker-w0-082f62-openshift-default-7d446b6f4d-cvpwk\" (UID: \"f5eb98d8-6523-411d-bcfb-ad46743c0c72\") " pod="openshift-ingress/e2e-worker-w0-082f62-openshift-default-7d446b6f4d-cvpwk"
Sep 11 13:35:12.702774 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:12.702722    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5eb98d8-6523-411d-bcfb-ad46743c0c72-credential-socket\") pod \"e2e-worker-w0-082f62-openshift-default-7d446b6f4d-cvpwk\" (UID: \"f5eb98d8-6523-411d-bcfb-ad46743c0c72\") " pod="openshift-ingress/e2e-worker-w0-082f62-openshift-default-7d446b6f4d-cvpwk"
Sep 11 13:35:12.702988 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:12.702771    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5eb98d8-6523-411d-bcfb-ad46743c0c72-istiod-ca-cert\") pod \"e2e-worker-w0-082f62-openshift-default-7d446b6f4d-cvpwk\" (UID: \"f5eb98d8-6523-411d-bcfb-ad46743c0c72\") " pod="openshift-ingress/e2e-worker-w0-082f62-openshift-default-7d446b6f4d-cvpwk"
Sep 11 13:35:12.702988 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:12.702941    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5eb98d8-6523-411d-bcfb-ad46743c0c72-istio-data\") pod \"e2e-worker-w0-082f62-openshift-default-7d446b6f4d-cvpwk\" (UID: \"f5eb98d8-6523-411d-bcfb-ad46743c0c72\") " pod="openshift-ingress/e2e-worker-w0-082f62-openshift-default-7d446b6f4d-cvpwk"
Sep 11 13:35:12.703093 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:12.703008    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5eb98d8-6523-411d-bcfb-ad46743c0c72-istio-podinfo\") pod \"e2e-worker-w0-082f62-openshift-default-7d446b6f4d-cvpwk\" (UID: \"f5eb98d8-6523-411d-bcfb-ad46743c0c72\") " pod="openshift-ingress/e2e-worker-w0-082f62-openshift-default-7d446b6f4d-cvpwk"
Sep 11 13:35:12.703093 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:12.703047    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltkwk\" (UniqueName: \"kubernetes.io/projected/f5eb98d8-6523-411d-bcfb-ad46743c0c72-kube-api-access-ltkwk\") pod \"e2e-worker-w0-082f62-openshift-default-7d446b6f4d-cvpwk\" (UID: \"f5eb98d8-6523-411d-bcfb-ad46743c0c72\") " pod="openshift-ingress/e2e-worker-w0-082f62-openshift-default-7d446b6f4d-cvpwk"
Sep 11 13:35:12.803817 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:12.803764    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5eb98d8-6523-411d-bcfb-ad46743c0c72-istio-podinfo\") pod \"e2e-worker-w0-082f62-openshift-default-7d446b6f4d-cvpwk\" (UID: \"f5eb98d8-6523-411d-bcfb-ad46743c0c72\") " pod="openshift-ingress/e2e-worker-w0-082f62-openshift-default-7d446b6f4d-cvpwk"
Sep 11 13:35:12.804067 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:12.803849    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltkwk\" (UniqueName: \"kubernetes.io/projected/f5eb98d8-6523-411d-bcfb-ad46743c0c72-kube-api-access-ltkwk\") pod \"e2e-worker-w0-082f62-openshift-default-7d446b6f4d-cvpwk\" (UID: \"f5eb98d8-6523-411d-bcfb-ad46743c0c72\") " pod="openshift-ingress/e2e-worker-w0-082f62-openshift-default-7d446b6f4d-cvpwk"
Sep 11 13:35:12.804067 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:12.803896    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5eb98d8-6523-411d-bcfb-ad46743c0c72-workload-socket\") pod \"e2e-worker-w0-082f62-openshift-default-7d446b6f4d-cvpwk\" (UID: \"f5eb98d8-6523-411d-bcfb-ad46743c0c72\") " pod="openshift-ingress/e2e-worker-w0-082f62-openshift-default-7d446b6f4d-cvpwk"
Sep 11 13:35:12.804067 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:12.803923    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5eb98d8-6523-411d-bcfb-ad46743c0c72-workload-certs\") pod \"e2e-worker-w0-082f62-openshift-default-7d446b6f4d-cvpwk\" (UID: \"f5eb98d8-6523-411d-bcfb-ad46743c0c72\") " pod="openshift-ingress/e2e-worker-w0-082f62-openshift-default-7d446b6f4d-cvpwk"
Sep 11 13:35:12.804067 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:12.803962    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5eb98d8-6523-411d-bcfb-ad46743c0c72-istio-envoy\") pod \"e2e-worker-w0-082f62-openshift-default-7d446b6f4d-cvpwk\" (UID: \"f5eb98d8-6523-411d-bcfb-ad46743c0c72\") " pod="openshift-ingress/e2e-worker-w0-082f62-openshift-default-7d446b6f4d-cvpwk"
Sep 11 13:35:12.804067 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:12.803988    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5eb98d8-6523-411d-bcfb-ad46743c0c72-istio-token\") pod \"e2e-worker-w0-082f62-openshift-default-7d446b6f4d-cvpwk\" (UID: \"f5eb98d8-6523-411d-bcfb-ad46743c0c72\") " pod="openshift-ingress/e2e-worker-w0-082f62-openshift-default-7d446b6f4d-cvpwk"
Sep 11 13:35:12.804410 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:12.804376    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5eb98d8-6523-411d-bcfb-ad46743c0c72-credential-socket\") pod \"e2e-worker-w0-082f62-openshift-default-7d446b6f4d-cvpwk\" (UID: \"f5eb98d8-6523-411d-bcfb-ad46743c0c72\") " pod="openshift-ingress/e2e-worker-w0-082f62-openshift-default-7d446b6f4d-cvpwk"
Sep 11 13:35:12.804504 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:12.804393    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5eb98d8-6523-411d-bcfb-ad46743c0c72-workload-socket\") pod \"e2e-worker-w0-082f62-openshift-default-7d446b6f4d-cvpwk\" (UID: \"f5eb98d8-6523-411d-bcfb-ad46743c0c72\") " pod="openshift-ingress/e2e-worker-w0-082f62-openshift-default-7d446b6f4d-cvpwk"
Sep 11 13:35:12.804504 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:12.804432    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5eb98d8-6523-411d-bcfb-ad46743c0c72-istiod-ca-cert\") pod \"e2e-worker-w0-082f62-openshift-default-7d446b6f4d-cvpwk\" (UID: \"f5eb98d8-6523-411d-bcfb-ad46743c0c72\") " pod="openshift-ingress/e2e-worker-w0-082f62-openshift-default-7d446b6f4d-cvpwk"
Sep 11 13:35:12.804504 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:12.804447    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5eb98d8-6523-411d-bcfb-ad46743c0c72-workload-certs\") pod \"e2e-worker-w0-082f62-openshift-default-7d446b6f4d-cvpwk\" (UID: \"f5eb98d8-6523-411d-bcfb-ad46743c0c72\") " pod="openshift-ingress/e2e-worker-w0-082f62-openshift-default-7d446b6f4d-cvpwk"
Sep 11 13:35:12.804660 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:12.804590    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5eb98d8-6523-411d-bcfb-ad46743c0c72-istio-data\") pod \"e2e-worker-w0-082f62-openshift-default-7d446b6f4d-cvpwk\" (UID: \"f5eb98d8-6523-411d-bcfb-ad46743c0c72\") " pod="openshift-ingress/e2e-worker-w0-082f62-openshift-default-7d446b6f4d-cvpwk"
Sep 11 13:35:12.804705 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:12.804660    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5eb98d8-6523-411d-bcfb-ad46743c0c72-credential-socket\") pod \"e2e-worker-w0-082f62-openshift-default-7d446b6f4d-cvpwk\" (UID: \"f5eb98d8-6523-411d-bcfb-ad46743c0c72\") " pod="openshift-ingress/e2e-worker-w0-082f62-openshift-default-7d446b6f4d-cvpwk"
Sep 11 13:35:12.805049 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:12.804986    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5eb98d8-6523-411d-bcfb-ad46743c0c72-istio-data\") pod \"e2e-worker-w0-082f62-openshift-default-7d446b6f4d-cvpwk\" (UID: \"f5eb98d8-6523-411d-bcfb-ad46743c0c72\") " pod="openshift-ingress/e2e-worker-w0-082f62-openshift-default-7d446b6f4d-cvpwk"
Sep 11 13:35:12.805159 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:12.805134    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5eb98d8-6523-411d-bcfb-ad46743c0c72-istiod-ca-cert\") pod \"e2e-worker-w0-082f62-openshift-default-7d446b6f4d-cvpwk\" (UID: \"f5eb98d8-6523-411d-bcfb-ad46743c0c72\") " pod="openshift-ingress/e2e-worker-w0-082f62-openshift-default-7d446b6f4d-cvpwk"
Sep 11 13:35:12.807525 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:12.807493    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5eb98d8-6523-411d-bcfb-ad46743c0c72-istio-envoy\") pod \"e2e-worker-w0-082f62-openshift-default-7d446b6f4d-cvpwk\" (UID: \"f5eb98d8-6523-411d-bcfb-ad46743c0c72\") " pod="openshift-ingress/e2e-worker-w0-082f62-openshift-default-7d446b6f4d-cvpwk"
Sep 11 13:35:12.807681 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:12.807663    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5eb98d8-6523-411d-bcfb-ad46743c0c72-istio-podinfo\") pod \"e2e-worker-w0-082f62-openshift-default-7d446b6f4d-cvpwk\" (UID: \"f5eb98d8-6523-411d-bcfb-ad46743c0c72\") " pod="openshift-ingress/e2e-worker-w0-082f62-openshift-default-7d446b6f4d-cvpwk"
Sep 11 13:35:12.821693 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:12.821646    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltkwk\" (UniqueName: \"kubernetes.io/projected/f5eb98d8-6523-411d-bcfb-ad46743c0c72-kube-api-access-ltkwk\") pod \"e2e-worker-w0-082f62-openshift-default-7d446b6f4d-cvpwk\" (UID: \"f5eb98d8-6523-411d-bcfb-ad46743c0c72\") " pod="openshift-ingress/e2e-worker-w0-082f62-openshift-default-7d446b6f4d-cvpwk"
Sep 11 13:35:12.846100 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:12.846005    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5eb98d8-6523-411d-bcfb-ad46743c0c72-istio-token\") pod \"e2e-worker-w0-082f62-openshift-default-7d446b6f4d-cvpwk\" (UID: \"f5eb98d8-6523-411d-bcfb-ad46743c0c72\") " pod="openshift-ingress/e2e-worker-w0-082f62-openshift-default-7d446b6f4d-cvpwk"
Sep 11 13:35:12.883980 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:12.883926    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-371c38-openshift-default-668769478-66p8n" event={"ID":"301e0569-db5c-45be-865c-491221d4883c","Type":"ContainerStarted","Data":"3769238e4b8be4d079ed73c23768099625c3d7aee61eca86e402877ba483b4f1"}
Sep 11 13:35:12.884282 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:12.884015    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-371c38-openshift-default-668769478-66p8n" event={"ID":"301e0569-db5c-45be-865c-491221d4883c","Type":"ContainerStarted","Data":"46e8d25ea9afac6f6d67aadfb6cdf773480a44955679882094fb42460861ddc9"}
Sep 11 13:35:12.893643 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:12.893589    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-082f62-openshift-default-7d446b6f4d-cvpwk"
Sep 11 13:35:12.992613 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:12.991961    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-371c38-openshift-default-668769478-66p8n" podStartSLOduration=1.991933865 podStartE2EDuration="1.991933865s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:35:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:35:12.991537872 +0000 UTC m=+820.377641060" watchObservedRunningTime="2026-09-11 13:35:12.991933865 +0000 UTC m=+820.378037030"
Sep 11 13:35:13.198689 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:13.198663    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-082f62-openshift-default-7d446b6f4d-cvpwk"]
Sep 11 13:35:13.201932 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:35:13.201891    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5eb98d8_6523_411d_bcfb_ad46743c0c72.slice/crio-be33f874ae891833b7fbf87dc2ce820740a53f9eef5ca47ff186e18f9e4506bb WatchSource:0}: Error finding container be33f874ae891833b7fbf87dc2ce820740a53f9eef5ca47ff186e18f9e4506bb: Status 404 returned error can't find the container with id be33f874ae891833b7fbf87dc2ce820740a53f9eef5ca47ff186e18f9e4506bb
Sep 11 13:35:13.205521 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:13.205474    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 11 13:35:13.205636 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:13.205570    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 11 13:35:13.205636 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:13.205610    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 11 13:35:13.283880 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:13.283818    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-371c38-openshift-default-668769478-66p8n"
Sep 11 13:35:13.290334 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:13.290299    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-371c38-openshift-default-668769478-66p8n"
Sep 11 13:35:13.894055 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:13.894000    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-082f62-openshift-default-7d446b6f4d-cvpwk" event={"ID":"f5eb98d8-6523-411d-bcfb-ad46743c0c72","Type":"ContainerStarted","Data":"1248566e39bbb57e4c0d11f4b0d9dbc25bb8ec98668d6c379f57521d66a53bea"}
Sep 11 13:35:13.894344 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:13.894120    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-082f62-openshift-default-7d446b6f4d-cvpwk" event={"ID":"f5eb98d8-6523-411d-bcfb-ad46743c0c72","Type":"ContainerStarted","Data":"be33f874ae891833b7fbf87dc2ce820740a53f9eef5ca47ff186e18f9e4506bb"}
Sep 11 13:35:13.894420 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:13.894343    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-371c38-openshift-default-668769478-66p8n"
Sep 11 13:35:13.895597 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:13.895570    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-371c38-openshift-default-668769478-66p8n"
Sep 11 13:35:13.929317 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:13.929228    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-082f62-openshift-default-7d446b6f4d-cvpwk" podStartSLOduration=1.929201792 podStartE2EDuration="1.929201792s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:35:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:35:13.92635708 +0000 UTC m=+821.312460265" watchObservedRunningTime="2026-09-11 13:35:13.929201792 +0000 UTC m=+821.315305042"
Sep 11 13:35:14.894871 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:14.894772    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-082f62-openshift-default-7d446b6f4d-cvpwk"
Sep 11 13:35:14.902303 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:14.902265    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-082f62-openshift-default-7d446b6f4d-cvpwk"
Sep 11 13:35:15.908779 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:15.908740    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-082f62-openshift-default-7d446b6f4d-cvpwk"
Sep 11 13:35:15.909875 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:15.909852    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-082f62-openshift-default-7d446b6f4d-cvpwk"
Sep 11 13:35:17.118990 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:17.118944    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b65337-58b646495c-w8ldq"]
Sep 11 13:35:17.123450 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:17.123420    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b65337-58b646495c-w8ldq"
Sep 11 13:35:17.138559 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:17.138528    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-b65337-serving-cert\""
Sep 11 13:35:17.165193 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:17.165145    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b65337-58b646495c-w8ldq"]
Sep 11 13:35:17.261553 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:17.261484    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4vpt\" (UniqueName: \"kubernetes.io/projected/c4cd43c6-1fe9-47ae-b24d-c9effce09fb8-kube-api-access-d4vpt\") pod \"maas-api-e2e-inf-a-b65337-58b646495c-w8ldq\" (UID: \"c4cd43c6-1fe9-47ae-b24d-c9effce09fb8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b65337-58b646495c-w8ldq"
Sep 11 13:35:17.261811 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:17.261586    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4cd43c6-1fe9-47ae-b24d-c9effce09fb8-openshift-service-ca\") pod \"maas-api-e2e-inf-a-b65337-58b646495c-w8ldq\" (UID: \"c4cd43c6-1fe9-47ae-b24d-c9effce09fb8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b65337-58b646495c-w8ldq"
Sep 11 13:35:17.261811 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:17.261714    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c4cd43c6-1fe9-47ae-b24d-c9effce09fb8-maas-api-tls\") pod \"maas-api-e2e-inf-a-b65337-58b646495c-w8ldq\" (UID: \"c4cd43c6-1fe9-47ae-b24d-c9effce09fb8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b65337-58b646495c-w8ldq"
Sep 11 13:35:17.261943 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:17.261875    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c4cd43c6-1fe9-47ae-b24d-c9effce09fb8-metrics-certs\") pod \"maas-api-e2e-inf-a-b65337-58b646495c-w8ldq\" (UID: \"c4cd43c6-1fe9-47ae-b24d-c9effce09fb8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b65337-58b646495c-w8ldq"
Sep 11 13:35:17.363818 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:17.363767    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4vpt\" (UniqueName: \"kubernetes.io/projected/c4cd43c6-1fe9-47ae-b24d-c9effce09fb8-kube-api-access-d4vpt\") pod \"maas-api-e2e-inf-a-b65337-58b646495c-w8ldq\" (UID: \"c4cd43c6-1fe9-47ae-b24d-c9effce09fb8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b65337-58b646495c-w8ldq"
Sep 11 13:35:17.363818 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:17.363852    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4cd43c6-1fe9-47ae-b24d-c9effce09fb8-openshift-service-ca\") pod \"maas-api-e2e-inf-a-b65337-58b646495c-w8ldq\" (UID: \"c4cd43c6-1fe9-47ae-b24d-c9effce09fb8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b65337-58b646495c-w8ldq"
Sep 11 13:35:17.364126 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:17.363925    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c4cd43c6-1fe9-47ae-b24d-c9effce09fb8-maas-api-tls\") pod \"maas-api-e2e-inf-a-b65337-58b646495c-w8ldq\" (UID: \"c4cd43c6-1fe9-47ae-b24d-c9effce09fb8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b65337-58b646495c-w8ldq"
Sep 11 13:35:17.364126 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:35:17.364070    2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-b65337-serving-cert: secret "maas-api-e2e-inf-a-b65337-serving-cert" not found
Sep 11 13:35:17.364126 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:17.364109    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c4cd43c6-1fe9-47ae-b24d-c9effce09fb8-metrics-certs\") pod \"maas-api-e2e-inf-a-b65337-58b646495c-w8ldq\" (UID: \"c4cd43c6-1fe9-47ae-b24d-c9effce09fb8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b65337-58b646495c-w8ldq"
Sep 11 13:35:17.364262 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:35:17.364157    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c4cd43c6-1fe9-47ae-b24d-c9effce09fb8-maas-api-tls podName:c4cd43c6-1fe9-47ae-b24d-c9effce09fb8 nodeName:}" failed. No retries permitted until 2026-09-11 13:35:17.864134215 +0000 UTC m=+825.250237364 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c4cd43c6-1fe9-47ae-b24d-c9effce09fb8-maas-api-tls") pod "maas-api-e2e-inf-a-b65337-58b646495c-w8ldq" (UID: "c4cd43c6-1fe9-47ae-b24d-c9effce09fb8") : secret "maas-api-e2e-inf-a-b65337-serving-cert" not found
Sep 11 13:35:17.364616 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:17.364593    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4cd43c6-1fe9-47ae-b24d-c9effce09fb8-openshift-service-ca\") pod \"maas-api-e2e-inf-a-b65337-58b646495c-w8ldq\" (UID: \"c4cd43c6-1fe9-47ae-b24d-c9effce09fb8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b65337-58b646495c-w8ldq"
Sep 11 13:35:17.367099 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:17.367073    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c4cd43c6-1fe9-47ae-b24d-c9effce09fb8-metrics-certs\") pod \"maas-api-e2e-inf-a-b65337-58b646495c-w8ldq\" (UID: \"c4cd43c6-1fe9-47ae-b24d-c9effce09fb8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b65337-58b646495c-w8ldq"
Sep 11 13:35:17.407639 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:17.407531    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4vpt\" (UniqueName: \"kubernetes.io/projected/c4cd43c6-1fe9-47ae-b24d-c9effce09fb8-kube-api-access-d4vpt\") pod \"maas-api-e2e-inf-a-b65337-58b646495c-w8ldq\" (UID: \"c4cd43c6-1fe9-47ae-b24d-c9effce09fb8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b65337-58b646495c-w8ldq"
Sep 11 13:35:17.870727 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:17.870675    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c4cd43c6-1fe9-47ae-b24d-c9effce09fb8-maas-api-tls\") pod \"maas-api-e2e-inf-a-b65337-58b646495c-w8ldq\" (UID: \"c4cd43c6-1fe9-47ae-b24d-c9effce09fb8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b65337-58b646495c-w8ldq"
Sep 11 13:35:17.873615 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:17.873575    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c4cd43c6-1fe9-47ae-b24d-c9effce09fb8-maas-api-tls\") pod \"maas-api-e2e-inf-a-b65337-58b646495c-w8ldq\" (UID: \"c4cd43c6-1fe9-47ae-b24d-c9effce09fb8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b65337-58b646495c-w8ldq"
Sep 11 13:35:18.035740 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:18.035697    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b65337-58b646495c-w8ldq"
Sep 11 13:35:18.339409 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:18.339365    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b65337-58b646495c-w8ldq"]
Sep 11 13:35:18.341368 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:35:18.341334    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4cd43c6_1fe9_47ae_b24d_c9effce09fb8.slice/crio-0e04e6f27ce94147d9812bc36093d697991e74f540b07f48a141a32f86a34118 WatchSource:0}: Error finding container 0e04e6f27ce94147d9812bc36093d697991e74f540b07f48a141a32f86a34118: Status 404 returned error can't find the container with id 0e04e6f27ce94147d9812bc36093d697991e74f540b07f48a141a32f86a34118
Sep 11 13:35:18.935126 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:18.935072    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b65337-58b646495c-w8ldq" event={"ID":"c4cd43c6-1fe9-47ae-b24d-c9effce09fb8","Type":"ContainerStarted","Data":"a4872d4814d5b81ef95c5c2f6b24765f80caa3e07e9107c9ed83269bbf0816c5"}
Sep 11 13:35:18.935361 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:18.935153    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b65337-58b646495c-w8ldq"
Sep 11 13:35:18.935361 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:18.935236    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b65337-58b646495c-w8ldq" event={"ID":"c4cd43c6-1fe9-47ae-b24d-c9effce09fb8","Type":"ContainerStarted","Data":"0e04e6f27ce94147d9812bc36093d697991e74f540b07f48a141a32f86a34118"}
Sep 11 13:35:18.963134 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:18.963042    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b65337-58b646495c-w8ldq" podStartSLOduration=1.474983984 podStartE2EDuration="1.963015384s" totalImagesPullingTime="488.0314ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:35:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:35:18.960048593 +0000 UTC m=+826.346151775" watchObservedRunningTime="2026-09-11 13:35:18.963015384 +0000 UTC m=+826.349118547"
Sep 11 13:35:19.328132 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:19.328095    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-082f62-5967dc89bd-clbhs"]
Sep 11 13:35:19.332262 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:19.332239    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-082f62-5967dc89bd-clbhs"
Sep 11 13:35:19.335173 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:19.335137    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-082f62-serving-cert\""
Sep 11 13:35:19.350335 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:19.350296    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-082f62-5967dc89bd-clbhs"]
Sep 11 13:35:19.492681 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:19.492632    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhncz\" (UniqueName: \"kubernetes.io/projected/bc25932e-1541-4894-848c-b1a86210d91f-kube-api-access-lhncz\") pod \"maas-api-e2e-worker-w0-082f62-5967dc89bd-clbhs\" (UID: \"bc25932e-1541-4894-848c-b1a86210d91f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-082f62-5967dc89bd-clbhs"
Sep 11 13:35:19.492681 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:19.492692    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc25932e-1541-4894-848c-b1a86210d91f-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-082f62-5967dc89bd-clbhs\" (UID: \"bc25932e-1541-4894-848c-b1a86210d91f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-082f62-5967dc89bd-clbhs"
Sep 11 13:35:19.493101 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:19.493040    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc25932e-1541-4894-848c-b1a86210d91f-maas-api-tls\") pod \"maas-api-e2e-worker-w0-082f62-5967dc89bd-clbhs\" (UID: \"bc25932e-1541-4894-848c-b1a86210d91f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-082f62-5967dc89bd-clbhs"
Sep 11 13:35:19.493166 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:19.493141    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc25932e-1541-4894-848c-b1a86210d91f-metrics-certs\") pod \"maas-api-e2e-worker-w0-082f62-5967dc89bd-clbhs\" (UID: \"bc25932e-1541-4894-848c-b1a86210d91f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-082f62-5967dc89bd-clbhs"
Sep 11 13:35:19.595369 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:19.594424    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhncz\" (UniqueName: \"kubernetes.io/projected/bc25932e-1541-4894-848c-b1a86210d91f-kube-api-access-lhncz\") pod \"maas-api-e2e-worker-w0-082f62-5967dc89bd-clbhs\" (UID: \"bc25932e-1541-4894-848c-b1a86210d91f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-082f62-5967dc89bd-clbhs"
Sep 11 13:35:19.595369 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:19.594514    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc25932e-1541-4894-848c-b1a86210d91f-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-082f62-5967dc89bd-clbhs\" (UID: \"bc25932e-1541-4894-848c-b1a86210d91f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-082f62-5967dc89bd-clbhs"
Sep 11 13:35:19.595369 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:19.594703    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc25932e-1541-4894-848c-b1a86210d91f-maas-api-tls\") pod \"maas-api-e2e-worker-w0-082f62-5967dc89bd-clbhs\" (UID: \"bc25932e-1541-4894-848c-b1a86210d91f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-082f62-5967dc89bd-clbhs"
Sep 11 13:35:19.595369 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:19.594745    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc25932e-1541-4894-848c-b1a86210d91f-metrics-certs\") pod \"maas-api-e2e-worker-w0-082f62-5967dc89bd-clbhs\" (UID: \"bc25932e-1541-4894-848c-b1a86210d91f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-082f62-5967dc89bd-clbhs"
Sep 11 13:35:19.595733 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:19.595459    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc25932e-1541-4894-848c-b1a86210d91f-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-082f62-5967dc89bd-clbhs\" (UID: \"bc25932e-1541-4894-848c-b1a86210d91f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-082f62-5967dc89bd-clbhs"
Sep 11 13:35:19.595733 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:35:19.595591    2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w0-082f62-serving-cert: secret "maas-api-e2e-worker-w0-082f62-serving-cert" not found
Sep 11 13:35:19.595733 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:35:19.595673    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc25932e-1541-4894-848c-b1a86210d91f-maas-api-tls podName:bc25932e-1541-4894-848c-b1a86210d91f nodeName:}" failed. No retries permitted until 2026-09-11 13:35:20.095644454 +0000 UTC m=+827.481747601 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/bc25932e-1541-4894-848c-b1a86210d91f-maas-api-tls") pod "maas-api-e2e-worker-w0-082f62-5967dc89bd-clbhs" (UID: "bc25932e-1541-4894-848c-b1a86210d91f") : secret "maas-api-e2e-worker-w0-082f62-serving-cert" not found
Sep 11 13:35:19.599303 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:19.599262    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc25932e-1541-4894-848c-b1a86210d91f-metrics-certs\") pod \"maas-api-e2e-worker-w0-082f62-5967dc89bd-clbhs\" (UID: \"bc25932e-1541-4894-848c-b1a86210d91f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-082f62-5967dc89bd-clbhs"
Sep 11 13:35:19.620709 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:19.620671    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhncz\" (UniqueName: \"kubernetes.io/projected/bc25932e-1541-4894-848c-b1a86210d91f-kube-api-access-lhncz\") pod \"maas-api-e2e-worker-w0-082f62-5967dc89bd-clbhs\" (UID: \"bc25932e-1541-4894-848c-b1a86210d91f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-082f62-5967dc89bd-clbhs"
Sep 11 13:35:20.099882 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:20.099795    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc25932e-1541-4894-848c-b1a86210d91f-maas-api-tls\") pod \"maas-api-e2e-worker-w0-082f62-5967dc89bd-clbhs\" (UID: \"bc25932e-1541-4894-848c-b1a86210d91f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-082f62-5967dc89bd-clbhs"
Sep 11 13:35:20.100161 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:35:20.100007    2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w0-082f62-serving-cert: secret "maas-api-e2e-worker-w0-082f62-serving-cert" not found
Sep 11 13:35:20.100161 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:35:20.100107    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc25932e-1541-4894-848c-b1a86210d91f-maas-api-tls podName:bc25932e-1541-4894-848c-b1a86210d91f nodeName:}" failed. No retries permitted until 2026-09-11 13:35:21.100083533 +0000 UTC m=+828.486186675 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/bc25932e-1541-4894-848c-b1a86210d91f-maas-api-tls") pod "maas-api-e2e-worker-w0-082f62-5967dc89bd-clbhs" (UID: "bc25932e-1541-4894-848c-b1a86210d91f") : secret "maas-api-e2e-worker-w0-082f62-serving-cert" not found
Sep 11 13:35:20.996351 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:20.996299    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-371c38-859578c76c-fgtss"]
Sep 11 13:35:21.000464 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:21.000428    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-371c38-859578c76c-fgtss"
Sep 11 13:35:21.002971 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:21.002938    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-371c38-serving-cert\""
Sep 11 13:35:21.036220 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:21.036173    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-371c38-859578c76c-fgtss"]
Sep 11 13:35:21.111895 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:21.111845    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5czw8\" (UniqueName: \"kubernetes.io/projected/dbb78093-57b6-4adf-9b31-7d2bf633100c-kube-api-access-5czw8\") pod \"maas-api-e2e-worker-w4-371c38-859578c76c-fgtss\" (UID: \"dbb78093-57b6-4adf-9b31-7d2bf633100c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-371c38-859578c76c-fgtss"
Sep 11 13:35:21.112111 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:21.111915    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbb78093-57b6-4adf-9b31-7d2bf633100c-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-371c38-859578c76c-fgtss\" (UID: \"dbb78093-57b6-4adf-9b31-7d2bf633100c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-371c38-859578c76c-fgtss"
Sep 11 13:35:21.112111 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:21.111953    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dbb78093-57b6-4adf-9b31-7d2bf633100c-maas-api-tls\") pod \"maas-api-e2e-worker-w4-371c38-859578c76c-fgtss\" (UID: \"dbb78093-57b6-4adf-9b31-7d2bf633100c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-371c38-859578c76c-fgtss"
Sep 11 13:35:21.112111 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:21.112010    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc25932e-1541-4894-848c-b1a86210d91f-maas-api-tls\") pod \"maas-api-e2e-worker-w0-082f62-5967dc89bd-clbhs\" (UID: \"bc25932e-1541-4894-848c-b1a86210d91f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-082f62-5967dc89bd-clbhs"
Sep 11 13:35:21.112111 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:21.112104    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dbb78093-57b6-4adf-9b31-7d2bf633100c-metrics-certs\") pod \"maas-api-e2e-worker-w4-371c38-859578c76c-fgtss\" (UID: \"dbb78093-57b6-4adf-9b31-7d2bf633100c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-371c38-859578c76c-fgtss"
Sep 11 13:35:21.115285 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:21.115248    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc25932e-1541-4894-848c-b1a86210d91f-maas-api-tls\") pod \"maas-api-e2e-worker-w0-082f62-5967dc89bd-clbhs\" (UID: \"bc25932e-1541-4894-848c-b1a86210d91f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-082f62-5967dc89bd-clbhs"
Sep 11 13:35:21.144665 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:21.144618    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-082f62-5967dc89bd-clbhs"
Sep 11 13:35:21.213654 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:21.213604    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbb78093-57b6-4adf-9b31-7d2bf633100c-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-371c38-859578c76c-fgtss\" (UID: \"dbb78093-57b6-4adf-9b31-7d2bf633100c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-371c38-859578c76c-fgtss"
Sep 11 13:35:21.213948 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:21.213721    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dbb78093-57b6-4adf-9b31-7d2bf633100c-maas-api-tls\") pod \"maas-api-e2e-worker-w4-371c38-859578c76c-fgtss\" (UID: \"dbb78093-57b6-4adf-9b31-7d2bf633100c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-371c38-859578c76c-fgtss"
Sep 11 13:35:21.213948 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:35:21.213925    2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w4-371c38-serving-cert: secret "maas-api-e2e-worker-w4-371c38-serving-cert" not found
Sep 11 13:35:21.214346 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:21.213969    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dbb78093-57b6-4adf-9b31-7d2bf633100c-metrics-certs\") pod \"maas-api-e2e-worker-w4-371c38-859578c76c-fgtss\" (UID: \"dbb78093-57b6-4adf-9b31-7d2bf633100c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-371c38-859578c76c-fgtss"
Sep 11 13:35:21.214346 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:35:21.214022    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dbb78093-57b6-4adf-9b31-7d2bf633100c-maas-api-tls podName:dbb78093-57b6-4adf-9b31-7d2bf633100c nodeName:}" failed. No retries permitted until 2026-09-11 13:35:21.713993742 +0000 UTC m=+829.100096901 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/dbb78093-57b6-4adf-9b31-7d2bf633100c-maas-api-tls") pod "maas-api-e2e-worker-w4-371c38-859578c76c-fgtss" (UID: "dbb78093-57b6-4adf-9b31-7d2bf633100c") : secret "maas-api-e2e-worker-w4-371c38-serving-cert" not found
Sep 11 13:35:21.214346 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:21.214213    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5czw8\" (UniqueName: \"kubernetes.io/projected/dbb78093-57b6-4adf-9b31-7d2bf633100c-kube-api-access-5czw8\") pod \"maas-api-e2e-worker-w4-371c38-859578c76c-fgtss\" (UID: \"dbb78093-57b6-4adf-9b31-7d2bf633100c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-371c38-859578c76c-fgtss"
Sep 11 13:35:21.215261 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:21.215234    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbb78093-57b6-4adf-9b31-7d2bf633100c-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-371c38-859578c76c-fgtss\" (UID: \"dbb78093-57b6-4adf-9b31-7d2bf633100c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-371c38-859578c76c-fgtss"
Sep 11 13:35:21.218045 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:21.218015    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dbb78093-57b6-4adf-9b31-7d2bf633100c-metrics-certs\") pod \"maas-api-e2e-worker-w4-371c38-859578c76c-fgtss\" (UID: \"dbb78093-57b6-4adf-9b31-7d2bf633100c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-371c38-859578c76c-fgtss"
Sep 11 13:35:21.252588 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:21.252440    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5czw8\" (UniqueName: \"kubernetes.io/projected/dbb78093-57b6-4adf-9b31-7d2bf633100c-kube-api-access-5czw8\") pod \"maas-api-e2e-worker-w4-371c38-859578c76c-fgtss\" (UID: \"dbb78093-57b6-4adf-9b31-7d2bf633100c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-371c38-859578c76c-fgtss"
Sep 11 13:35:21.481630 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:21.481584    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-082f62-5967dc89bd-clbhs"]
Sep 11 13:35:21.483960 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:35:21.483915    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc25932e_1541_4894_848c_b1a86210d91f.slice/crio-e5cf563d6b555a9357fac72e145f1d6ba5a096df51b655b791062008583ae68b WatchSource:0}: Error finding container e5cf563d6b555a9357fac72e145f1d6ba5a096df51b655b791062008583ae68b: Status 404 returned error can't find the container with id e5cf563d6b555a9357fac72e145f1d6ba5a096df51b655b791062008583ae68b
Sep 11 13:35:21.719562 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:21.719517    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dbb78093-57b6-4adf-9b31-7d2bf633100c-maas-api-tls\") pod \"maas-api-e2e-worker-w4-371c38-859578c76c-fgtss\" (UID: \"dbb78093-57b6-4adf-9b31-7d2bf633100c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-371c38-859578c76c-fgtss"
Sep 11 13:35:21.722514 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:21.722483    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dbb78093-57b6-4adf-9b31-7d2bf633100c-maas-api-tls\") pod \"maas-api-e2e-worker-w4-371c38-859578c76c-fgtss\" (UID: \"dbb78093-57b6-4adf-9b31-7d2bf633100c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-371c38-859578c76c-fgtss"
Sep 11 13:35:21.915000 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:21.914948    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-371c38-859578c76c-fgtss"
Sep 11 13:35:21.958665 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:21.958586    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-082f62-5967dc89bd-clbhs" event={"ID":"bc25932e-1541-4894-848c-b1a86210d91f","Type":"ContainerStarted","Data":"adbcf86ad6dffaac8b46a96e6e82cee6217e1ddf28ccfbf460f47285f5b15d69"}
Sep 11 13:35:21.958989 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:21.958766    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-082f62-5967dc89bd-clbhs"
Sep 11 13:35:21.958989 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:21.958798    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-082f62-5967dc89bd-clbhs" event={"ID":"bc25932e-1541-4894-848c-b1a86210d91f","Type":"ContainerStarted","Data":"e5cf563d6b555a9357fac72e145f1d6ba5a096df51b655b791062008583ae68b"}
Sep 11 13:35:21.997965 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:21.996643    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-082f62-5967dc89bd-clbhs" podStartSLOduration=2.61985893 podStartE2EDuration="2.99659232s" totalImagesPullingTime="376.73339ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:35:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:35:21.993076953 +0000 UTC m=+829.379180116" watchObservedRunningTime="2026-09-11 13:35:21.99659232 +0000 UTC m=+829.382695484"
Sep 11 13:35:22.226525 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:35:22.226477    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbb78093_57b6_4adf_9b31_7d2bf633100c.slice/crio-dfabaea28ab28d79d8ee6bc9a660a24387b15cc9e9f29c63ec9ba4386a3784ea WatchSource:0}: Error finding container dfabaea28ab28d79d8ee6bc9a660a24387b15cc9e9f29c63ec9ba4386a3784ea: Status 404 returned error can't find the container with id dfabaea28ab28d79d8ee6bc9a660a24387b15cc9e9f29c63ec9ba4386a3784ea
Sep 11 13:35:22.228824 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:22.228779    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-371c38-859578c76c-fgtss"]
Sep 11 13:35:22.967005 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:22.966945    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-371c38-859578c76c-fgtss" event={"ID":"dbb78093-57b6-4adf-9b31-7d2bf633100c","Type":"ContainerStarted","Data":"5f062c962eda648c1103b4f85eaafa6e0c63e84b901155db04e4fe18b5f9eec9"}
Sep 11 13:35:22.967270 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:22.967072    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-371c38-859578c76c-fgtss" event={"ID":"dbb78093-57b6-4adf-9b31-7d2bf633100c","Type":"ContainerStarted","Data":"dfabaea28ab28d79d8ee6bc9a660a24387b15cc9e9f29c63ec9ba4386a3784ea"}
Sep 11 13:35:22.967270 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:22.967222    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-371c38-859578c76c-fgtss"
Sep 11 13:35:24.947633 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:24.947586    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b65337-58b646495c-w8ldq"
Sep 11 13:35:24.966745 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:24.966675    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-371c38-859578c76c-fgtss" podStartSLOduration=4.596130335 podStartE2EDuration="4.966653351s" totalImagesPullingTime="370.523016ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:35:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:35:23.005765377 +0000 UTC m=+830.391868541" watchObservedRunningTime="2026-09-11 13:35:24.966653351 +0000 UTC m=+832.352756515"
Sep 11 13:35:26.992219 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:26.992165    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-f9f26e-fbc8976f7-l229k"]
Sep 11 13:35:26.996994 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:26.996958    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-f9f26e-fbc8976f7-l229k"
Sep 11 13:35:27.000192 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:27.000152    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-f9f26e-serving-cert\""
Sep 11 13:35:27.008195 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:27.008159    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-f9f26e-fbc8976f7-l229k"]
Sep 11 13:35:27.091889 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:27.091821    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/54749995-810d-417d-881d-c9f755795d3e-maas-api-tls\") pod \"maas-api-e2e-autores-f9f26e-fbc8976f7-l229k\" (UID: \"54749995-810d-417d-881d-c9f755795d3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-f9f26e-fbc8976f7-l229k"
Sep 11 13:35:27.092386 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:27.092347    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlxld\" (UniqueName: \"kubernetes.io/projected/54749995-810d-417d-881d-c9f755795d3e-kube-api-access-tlxld\") pod \"maas-api-e2e-autores-f9f26e-fbc8976f7-l229k\" (UID: \"54749995-810d-417d-881d-c9f755795d3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-f9f26e-fbc8976f7-l229k"
Sep 11 13:35:27.092560 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:27.092414    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/54749995-810d-417d-881d-c9f755795d3e-metrics-certs\") pod \"maas-api-e2e-autores-f9f26e-fbc8976f7-l229k\" (UID: \"54749995-810d-417d-881d-c9f755795d3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-f9f26e-fbc8976f7-l229k"
Sep 11 13:35:27.092560 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:27.092493    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/54749995-810d-417d-881d-c9f755795d3e-openshift-service-ca\") pod \"maas-api-e2e-autores-f9f26e-fbc8976f7-l229k\" (UID: \"54749995-810d-417d-881d-c9f755795d3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-f9f26e-fbc8976f7-l229k"
Sep 11 13:35:27.193801 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:27.193748    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/54749995-810d-417d-881d-c9f755795d3e-openshift-service-ca\") pod \"maas-api-e2e-autores-f9f26e-fbc8976f7-l229k\" (UID: \"54749995-810d-417d-881d-c9f755795d3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-f9f26e-fbc8976f7-l229k"
Sep 11 13:35:27.194110 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:27.193887    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/54749995-810d-417d-881d-c9f755795d3e-maas-api-tls\") pod \"maas-api-e2e-autores-f9f26e-fbc8976f7-l229k\" (UID: \"54749995-810d-417d-881d-c9f755795d3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-f9f26e-fbc8976f7-l229k"
Sep 11 13:35:27.194110 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:27.193954    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlxld\" (UniqueName: \"kubernetes.io/projected/54749995-810d-417d-881d-c9f755795d3e-kube-api-access-tlxld\") pod \"maas-api-e2e-autores-f9f26e-fbc8976f7-l229k\" (UID: \"54749995-810d-417d-881d-c9f755795d3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-f9f26e-fbc8976f7-l229k"
Sep 11 13:35:27.194110 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:27.193981    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/54749995-810d-417d-881d-c9f755795d3e-metrics-certs\") pod \"maas-api-e2e-autores-f9f26e-fbc8976f7-l229k\" (UID: \"54749995-810d-417d-881d-c9f755795d3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-f9f26e-fbc8976f7-l229k"
Sep 11 13:35:27.194110 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:35:27.194065    2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-autores-f9f26e-serving-cert: secret "maas-api-e2e-autores-f9f26e-serving-cert" not found
Sep 11 13:35:27.194332 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:35:27.194160    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54749995-810d-417d-881d-c9f755795d3e-maas-api-tls podName:54749995-810d-417d-881d-c9f755795d3e nodeName:}" failed. No retries permitted until 2026-09-11 13:35:27.694136729 +0000 UTC m=+835.080239871 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/54749995-810d-417d-881d-c9f755795d3e-maas-api-tls") pod "maas-api-e2e-autores-f9f26e-fbc8976f7-l229k" (UID: "54749995-810d-417d-881d-c9f755795d3e") : secret "maas-api-e2e-autores-f9f26e-serving-cert" not found
Sep 11 13:35:27.194719 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:27.194696    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/54749995-810d-417d-881d-c9f755795d3e-openshift-service-ca\") pod \"maas-api-e2e-autores-f9f26e-fbc8976f7-l229k\" (UID: \"54749995-810d-417d-881d-c9f755795d3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-f9f26e-fbc8976f7-l229k"
Sep 11 13:35:27.196981 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:27.196956    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/54749995-810d-417d-881d-c9f755795d3e-metrics-certs\") pod \"maas-api-e2e-autores-f9f26e-fbc8976f7-l229k\" (UID: \"54749995-810d-417d-881d-c9f755795d3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-f9f26e-fbc8976f7-l229k"
Sep 11 13:35:27.207363 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:27.207330    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlxld\" (UniqueName: \"kubernetes.io/projected/54749995-810d-417d-881d-c9f755795d3e-kube-api-access-tlxld\") pod \"maas-api-e2e-autores-f9f26e-fbc8976f7-l229k\" (UID: \"54749995-810d-417d-881d-c9f755795d3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-f9f26e-fbc8976f7-l229k"
Sep 11 13:35:27.699400 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:27.699346    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/54749995-810d-417d-881d-c9f755795d3e-maas-api-tls\") pod \"maas-api-e2e-autores-f9f26e-fbc8976f7-l229k\" (UID: \"54749995-810d-417d-881d-c9f755795d3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-f9f26e-fbc8976f7-l229k"
Sep 11 13:35:27.702401 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:27.702360    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/54749995-810d-417d-881d-c9f755795d3e-maas-api-tls\") pod \"maas-api-e2e-autores-f9f26e-fbc8976f7-l229k\" (UID: \"54749995-810d-417d-881d-c9f755795d3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-f9f26e-fbc8976f7-l229k"
Sep 11 13:35:27.913586 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:27.913533    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-f9f26e-fbc8976f7-l229k"
Sep 11 13:35:27.974187 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:27.974147    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-082f62-5967dc89bd-clbhs"
Sep 11 13:35:28.217400 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:28.217349    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-f9f26e-fbc8976f7-l229k"]
Sep 11 13:35:28.218996 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:35:28.218959    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54749995_810d_417d_881d_c9f755795d3e.slice/crio-a718ccc986193795b8e37c86d1bbcd07033974f7a241be24eb418b9d8379a3ee WatchSource:0}: Error finding container a718ccc986193795b8e37c86d1bbcd07033974f7a241be24eb418b9d8379a3ee: Status 404 returned error can't find the container with id a718ccc986193795b8e37c86d1bbcd07033974f7a241be24eb418b9d8379a3ee
Sep 11 13:35:28.982089 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:28.982046    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-371c38-859578c76c-fgtss"
Sep 11 13:35:29.038229 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:29.038181    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-f9f26e-fbc8976f7-l229k" event={"ID":"54749995-810d-417d-881d-c9f755795d3e","Type":"ContainerStarted","Data":"73132a03c561ce67304d4cb9396b6208c0459160fe531dca36cbb701afa765fd"}
Sep 11 13:35:29.038625 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:29.038603    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-f9f26e-fbc8976f7-l229k" event={"ID":"54749995-810d-417d-881d-c9f755795d3e","Type":"ContainerStarted","Data":"a718ccc986193795b8e37c86d1bbcd07033974f7a241be24eb418b9d8379a3ee"}
Sep 11 13:35:29.038779 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:29.038767    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-f9f26e-fbc8976f7-l229k"
Sep 11 13:35:29.116865 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:29.095862    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-f9f26e-fbc8976f7-l229k" podStartSLOduration=2.770698037 podStartE2EDuration="3.095802552s" totalImagesPullingTime="325.104515ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:35:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:35:29.092790411 +0000 UTC m=+836.478893601" watchObservedRunningTime="2026-09-11 13:35:29.095802552 +0000 UTC m=+836.481905717"
Sep 11 13:35:32.917515 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:32.917446    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ae1d6b61-9b6d65f54-77vfx"]
Sep 11 13:35:32.923562 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:32.923522    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ae1d6b61-9b6d65f54-77vfx"
Sep 11 13:35:32.926731 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:32.926687    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-ae1d6b61-serving-cert\""
Sep 11 13:35:32.949265 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:32.949226    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ae1d6b61-9b6d65f54-77vfx"]
Sep 11 13:35:32.972944 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:32.970094    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f4c59fa8-961c-489b-8168-7873bfd7b439-maas-api-tls\") pod \"maas-api-e2e-ait-ae1d6b61-9b6d65f54-77vfx\" (UID: \"f4c59fa8-961c-489b-8168-7873bfd7b439\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ae1d6b61-9b6d65f54-77vfx"
Sep 11 13:35:32.972944 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:32.970246    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f4c59fa8-961c-489b-8168-7873bfd7b439-metrics-certs\") pod \"maas-api-e2e-ait-ae1d6b61-9b6d65f54-77vfx\" (UID: \"f4c59fa8-961c-489b-8168-7873bfd7b439\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ae1d6b61-9b6d65f54-77vfx"
Sep 11 13:35:32.972944 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:32.970292    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4c59fa8-961c-489b-8168-7873bfd7b439-openshift-service-ca\") pod \"maas-api-e2e-ait-ae1d6b61-9b6d65f54-77vfx\" (UID: \"f4c59fa8-961c-489b-8168-7873bfd7b439\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ae1d6b61-9b6d65f54-77vfx"
Sep 11 13:35:32.972944 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:32.970351    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmfkd\" (UniqueName: \"kubernetes.io/projected/f4c59fa8-961c-489b-8168-7873bfd7b439-kube-api-access-rmfkd\") pod \"maas-api-e2e-ait-ae1d6b61-9b6d65f54-77vfx\" (UID: \"f4c59fa8-961c-489b-8168-7873bfd7b439\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ae1d6b61-9b6d65f54-77vfx"
Sep 11 13:35:33.071478 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:33.071426    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f4c59fa8-961c-489b-8168-7873bfd7b439-metrics-certs\") pod \"maas-api-e2e-ait-ae1d6b61-9b6d65f54-77vfx\" (UID: \"f4c59fa8-961c-489b-8168-7873bfd7b439\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ae1d6b61-9b6d65f54-77vfx"
Sep 11 13:35:33.071478 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:33.071486    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4c59fa8-961c-489b-8168-7873bfd7b439-openshift-service-ca\") pod \"maas-api-e2e-ait-ae1d6b61-9b6d65f54-77vfx\" (UID: \"f4c59fa8-961c-489b-8168-7873bfd7b439\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ae1d6b61-9b6d65f54-77vfx"
Sep 11 13:35:33.071771 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:33.071537    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmfkd\" (UniqueName: \"kubernetes.io/projected/f4c59fa8-961c-489b-8168-7873bfd7b439-kube-api-access-rmfkd\") pod \"maas-api-e2e-ait-ae1d6b61-9b6d65f54-77vfx\" (UID: \"f4c59fa8-961c-489b-8168-7873bfd7b439\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ae1d6b61-9b6d65f54-77vfx"
Sep 11 13:35:33.071771 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:33.071630    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f4c59fa8-961c-489b-8168-7873bfd7b439-maas-api-tls\") pod \"maas-api-e2e-ait-ae1d6b61-9b6d65f54-77vfx\" (UID: \"f4c59fa8-961c-489b-8168-7873bfd7b439\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ae1d6b61-9b6d65f54-77vfx"
Sep 11 13:35:33.072597 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:33.072510    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4c59fa8-961c-489b-8168-7873bfd7b439-openshift-service-ca\") pod \"maas-api-e2e-ait-ae1d6b61-9b6d65f54-77vfx\" (UID: \"f4c59fa8-961c-489b-8168-7873bfd7b439\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ae1d6b61-9b6d65f54-77vfx"
Sep 11 13:35:33.074906 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:33.074875    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-ae1d6b61-serving-cert\""
Sep 11 13:35:33.075089 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:33.075066    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f4c59fa8-961c-489b-8168-7873bfd7b439-metrics-certs\") pod \"maas-api-e2e-ait-ae1d6b61-9b6d65f54-77vfx\" (UID: \"f4c59fa8-961c-489b-8168-7873bfd7b439\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ae1d6b61-9b6d65f54-77vfx"
Sep 11 13:35:33.080703 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:33.080679    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmfkd\" (UniqueName: \"kubernetes.io/projected/f4c59fa8-961c-489b-8168-7873bfd7b439-kube-api-access-rmfkd\") pod \"maas-api-e2e-ait-ae1d6b61-9b6d65f54-77vfx\" (UID: \"f4c59fa8-961c-489b-8168-7873bfd7b439\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ae1d6b61-9b6d65f54-77vfx"
Sep 11 13:35:33.082430 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:35:33.082414    2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-ae1d6b61-serving-cert: secret "maas-api-e2e-ait-ae1d6b61-serving-cert" not found
Sep 11 13:35:33.082497 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:35:33.082491    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4c59fa8-961c-489b-8168-7873bfd7b439-maas-api-tls podName:f4c59fa8-961c-489b-8168-7873bfd7b439 nodeName:}" failed. No retries permitted until 2026-09-11 13:35:33.58247284 +0000 UTC m=+840.968575982 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f4c59fa8-961c-489b-8168-7873bfd7b439-maas-api-tls") pod "maas-api-e2e-ait-ae1d6b61-9b6d65f54-77vfx" (UID: "f4c59fa8-961c-489b-8168-7873bfd7b439") : secret "maas-api-e2e-ait-ae1d6b61-serving-cert" not found
Sep 11 13:35:33.680296 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:33.680234    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f4c59fa8-961c-489b-8168-7873bfd7b439-maas-api-tls\") pod \"maas-api-e2e-ait-ae1d6b61-9b6d65f54-77vfx\" (UID: \"f4c59fa8-961c-489b-8168-7873bfd7b439\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ae1d6b61-9b6d65f54-77vfx"
Sep 11 13:35:33.683219 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:33.683176    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f4c59fa8-961c-489b-8168-7873bfd7b439-maas-api-tls\") pod \"maas-api-e2e-ait-ae1d6b61-9b6d65f54-77vfx\" (UID: \"f4c59fa8-961c-489b-8168-7873bfd7b439\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ae1d6b61-9b6d65f54-77vfx"
Sep 11 13:35:33.837694 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:33.837635    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ae1d6b61-9b6d65f54-77vfx"
Sep 11 13:35:34.221480 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:35:34.221433    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4c59fa8_961c_489b_8168_7873bfd7b439.slice/crio-3879630e3785043a6db41ce9053c5a6b2b849156f8f4d09fd6c94ee98c6ff51f WatchSource:0}: Error finding container 3879630e3785043a6db41ce9053c5a6b2b849156f8f4d09fd6c94ee98c6ff51f: Status 404 returned error can't find the container with id 3879630e3785043a6db41ce9053c5a6b2b849156f8f4d09fd6c94ee98c6ff51f
Sep 11 13:35:34.222040 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:34.221640    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ae1d6b61-9b6d65f54-77vfx"]
Sep 11 13:35:35.081775 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:35.081723    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ae1d6b61-9b6d65f54-77vfx" event={"ID":"f4c59fa8-961c-489b-8168-7873bfd7b439","Type":"ContainerStarted","Data":"81660207b94c6fe34851a09e6ec13206b0c2ad3048452004e3781a7eddfd7cdf"}
Sep 11 13:35:35.082050 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:35.081807    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ae1d6b61-9b6d65f54-77vfx" event={"ID":"f4c59fa8-961c-489b-8168-7873bfd7b439","Type":"ContainerStarted","Data":"3879630e3785043a6db41ce9053c5a6b2b849156f8f4d09fd6c94ee98c6ff51f"}
Sep 11 13:35:35.082050 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:35.081861    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ae1d6b61-9b6d65f54-77vfx"
Sep 11 13:35:35.117331 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:35.117251    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ae1d6b61-9b6d65f54-77vfx" podStartSLOduration=2.7006203810000002 podStartE2EDuration="3.117222191s" totalImagesPullingTime="416.60181ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:35:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:35:35.115629372 +0000 UTC m=+842.501732555" watchObservedRunningTime="2026-09-11 13:35:35.117222191 +0000 UTC m=+842.503325356"
Sep 11 13:35:36.057153 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:36.057099    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-f9f26e-fbc8976f7-l229k"
Sep 11 13:35:40.141263 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:40.141222    2578 generic.go:356] "Generic (PLEG): container finished" podID="f5eb98d8-6523-411d-bcfb-ad46743c0c72" containerID="1248566e39bbb57e4c0d11f4b0d9dbc25bb8ec98668d6c379f57521d66a53bea" exitCode=0
Sep 11 13:35:40.141964 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:40.141303    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-082f62-openshift-default-7d446b6f4d-cvpwk" event={"ID":"f5eb98d8-6523-411d-bcfb-ad46743c0c72","Type":"ContainerDied","Data":"1248566e39bbb57e4c0d11f4b0d9dbc25bb8ec98668d6c379f57521d66a53bea"}
Sep 11 13:35:40.141964 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:40.141931    2578 scope.go:117] "RemoveContainer" containerID="1248566e39bbb57e4c0d11f4b0d9dbc25bb8ec98668d6c379f57521d66a53bea"
Sep 11 13:35:40.143595 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:40.143564    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 11 13:35:40.143692 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:40.143675    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 11 13:35:40.143754 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:40.143733    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 11 13:35:41.120625 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:41.120581    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ae1d6b61-9b6d65f54-77vfx"
Sep 11 13:35:41.152580 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:41.152521    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-082f62-openshift-default-7d446b6f4d-cvpwk" event={"ID":"f5eb98d8-6523-411d-bcfb-ad46743c0c72","Type":"ContainerStarted","Data":"1fb9677dad6b50981cb02f1a14e4532e4bc2f632f20a4e13875b29f1dba28066"}
Sep 11 13:35:41.896331 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:41.896277    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-082f62-openshift-default-7d446b6f4d-cvpwk"
Sep 11 13:35:41.896635 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:41.896566    2578 patch_prober.go:28] interesting pod/e2e-worker-w0-082f62-openshift-default-7d446b6f4d-cvpwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body=
Sep 11 13:35:41.896720 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:41.896634    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-082f62-openshift-default-7d446b6f4d-cvpwk" podUID="f5eb98d8-6523-411d-bcfb-ad46743c0c72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused"
Sep 11 13:35:42.894979 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:42.894929    2578 patch_prober.go:28] interesting pod/e2e-worker-w0-082f62-openshift-default-7d446b6f4d-cvpwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body=
Sep 11 13:35:42.895909 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:42.895865    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-082f62-openshift-default-7d446b6f4d-cvpwk" podUID="f5eb98d8-6523-411d-bcfb-ad46743c0c72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused"
Sep 11 13:35:43.894868 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:43.894216    2578 patch_prober.go:28] interesting pod/e2e-worker-w0-082f62-openshift-default-7d446b6f4d-cvpwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body=
Sep 11 13:35:43.894868 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:43.894352    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-082f62-openshift-default-7d446b6f4d-cvpwk" podUID="f5eb98d8-6523-411d-bcfb-ad46743c0c72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused"
Sep 11 13:35:44.894870 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:44.894795    2578 patch_prober.go:28] interesting pod/e2e-worker-w0-082f62-openshift-default-7d446b6f4d-cvpwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body=
Sep 11 13:35:44.895301 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:44.895252    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-082f62-openshift-default-7d446b6f4d-cvpwk" podUID="f5eb98d8-6523-411d-bcfb-ad46743c0c72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused"
Sep 11 13:35:44.937231 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:44.937169    2578 patch_prober.go:28] interesting pod/e2e-worker-w4-371c38-openshift-default-668769478-66p8n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 11 13:35:44.937482 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:44.937267    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-371c38-openshift-default-668769478-66p8n" podUID="301e0569-db5c-45be-865c-491221d4883c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 11 13:35:45.895215 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:45.895151    2578 patch_prober.go:28] interesting pod/e2e-worker-w0-082f62-openshift-default-7d446b6f4d-cvpwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body=
Sep 11 13:35:45.895707 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:45.895656    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-082f62-openshift-default-7d446b6f4d-cvpwk" podUID="f5eb98d8-6523-411d-bcfb-ad46743c0c72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused"
Sep 11 13:35:45.908634 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:45.908583    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-082f62-openshift-default-7d446b6f4d-cvpwk"
Sep 11 13:35:46.903463 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:46.903411    2578 patch_prober.go:28] interesting pod/e2e-worker-w0-082f62-openshift-default-7d446b6f4d-cvpwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body=
Sep 11 13:35:46.904702 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:46.904166    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-082f62-openshift-default-7d446b6f4d-cvpwk" podUID="f5eb98d8-6523-411d-bcfb-ad46743c0c72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused"
Sep 11 13:35:46.908072 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:46.908027    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-a649d3-6f5f65cc57-kjbpx"]
Sep 11 13:35:46.937655 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:46.937603    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-a649d3-6f5f65cc57-kjbpx"]
Sep 11 13:35:46.938068 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:46.937737    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a649d3-6f5f65cc57-kjbpx"
Sep 11 13:35:46.941170 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:46.941145    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-a649d3-serving-cert\""
Sep 11 13:35:47.009346 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:47.004748    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b902823-ca13-41f9-90f5-42ceacbc5576-metrics-certs\") pod \"maas-api-e2e-inf-b-a649d3-6f5f65cc57-kjbpx\" (UID: \"5b902823-ca13-41f9-90f5-42ceacbc5576\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a649d3-6f5f65cc57-kjbpx"
Sep 11 13:35:47.009346 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:47.004987    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8v8v\" (UniqueName: \"kubernetes.io/projected/5b902823-ca13-41f9-90f5-42ceacbc5576-kube-api-access-z8v8v\") pod \"maas-api-e2e-inf-b-a649d3-6f5f65cc57-kjbpx\" (UID: \"5b902823-ca13-41f9-90f5-42ceacbc5576\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a649d3-6f5f65cc57-kjbpx"
Sep 11 13:35:47.009346 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:47.005165    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b902823-ca13-41f9-90f5-42ceacbc5576-maas-api-tls\") pod \"maas-api-e2e-inf-b-a649d3-6f5f65cc57-kjbpx\" (UID: \"5b902823-ca13-41f9-90f5-42ceacbc5576\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a649d3-6f5f65cc57-kjbpx"
Sep 11 13:35:47.009346 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:47.005212    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b902823-ca13-41f9-90f5-42ceacbc5576-openshift-service-ca\") pod \"maas-api-e2e-inf-b-a649d3-6f5f65cc57-kjbpx\" (UID: \"5b902823-ca13-41f9-90f5-42ceacbc5576\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a649d3-6f5f65cc57-kjbpx"
Sep 11 13:35:47.106860 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:47.106775    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8v8v\" (UniqueName: \"kubernetes.io/projected/5b902823-ca13-41f9-90f5-42ceacbc5576-kube-api-access-z8v8v\") pod \"maas-api-e2e-inf-b-a649d3-6f5f65cc57-kjbpx\" (UID: \"5b902823-ca13-41f9-90f5-42ceacbc5576\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a649d3-6f5f65cc57-kjbpx"
Sep 11 13:35:47.107154 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:47.106995    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b902823-ca13-41f9-90f5-42ceacbc5576-maas-api-tls\") pod \"maas-api-e2e-inf-b-a649d3-6f5f65cc57-kjbpx\" (UID: \"5b902823-ca13-41f9-90f5-42ceacbc5576\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a649d3-6f5f65cc57-kjbpx"
Sep 11 13:35:47.107154 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:47.107029    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b902823-ca13-41f9-90f5-42ceacbc5576-openshift-service-ca\") pod \"maas-api-e2e-inf-b-a649d3-6f5f65cc57-kjbpx\" (UID: \"5b902823-ca13-41f9-90f5-42ceacbc5576\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a649d3-6f5f65cc57-kjbpx"
Sep 11 13:35:47.107154 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:47.107107    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b902823-ca13-41f9-90f5-42ceacbc5576-metrics-certs\") pod \"maas-api-e2e-inf-b-a649d3-6f5f65cc57-kjbpx\" (UID: \"5b902823-ca13-41f9-90f5-42ceacbc5576\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a649d3-6f5f65cc57-kjbpx"
Sep 11 13:35:47.109980 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:35:47.109401    2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-a649d3-serving-cert: secret "maas-api-e2e-inf-b-a649d3-serving-cert" not found
Sep 11 13:35:47.109980 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:35:47.109511    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b902823-ca13-41f9-90f5-42ceacbc5576-maas-api-tls podName:5b902823-ca13-41f9-90f5-42ceacbc5576 nodeName:}" failed. No retries permitted until 2026-09-11 13:35:47.60948062 +0000 UTC m=+854.995583767 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5b902823-ca13-41f9-90f5-42ceacbc5576-maas-api-tls") pod "maas-api-e2e-inf-b-a649d3-6f5f65cc57-kjbpx" (UID: "5b902823-ca13-41f9-90f5-42ceacbc5576") : secret "maas-api-e2e-inf-b-a649d3-serving-cert" not found
Sep 11 13:35:47.110728 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:47.110615    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b902823-ca13-41f9-90f5-42ceacbc5576-openshift-service-ca\") pod \"maas-api-e2e-inf-b-a649d3-6f5f65cc57-kjbpx\" (UID: \"5b902823-ca13-41f9-90f5-42ceacbc5576\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a649d3-6f5f65cc57-kjbpx"
Sep 11 13:35:47.111794 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:47.111751    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b902823-ca13-41f9-90f5-42ceacbc5576-metrics-certs\") pod \"maas-api-e2e-inf-b-a649d3-6f5f65cc57-kjbpx\" (UID: \"5b902823-ca13-41f9-90f5-42ceacbc5576\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a649d3-6f5f65cc57-kjbpx"
Sep 11 13:35:47.122701 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:47.122658    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8v8v\" (UniqueName: \"kubernetes.io/projected/5b902823-ca13-41f9-90f5-42ceacbc5576-kube-api-access-z8v8v\") pod \"maas-api-e2e-inf-b-a649d3-6f5f65cc57-kjbpx\" (UID: \"5b902823-ca13-41f9-90f5-42ceacbc5576\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a649d3-6f5f65cc57-kjbpx"
Sep 11 13:35:47.205121 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:47.205060    2578 generic.go:356] "Generic (PLEG): container finished" podID="301e0569-db5c-45be-865c-491221d4883c" containerID="3769238e4b8be4d079ed73c23768099625c3d7aee61eca86e402877ba483b4f1" exitCode=0
Sep 11 13:35:47.205373 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:47.205161    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-371c38-openshift-default-668769478-66p8n" event={"ID":"301e0569-db5c-45be-865c-491221d4883c","Type":"ContainerDied","Data":"3769238e4b8be4d079ed73c23768099625c3d7aee61eca86e402877ba483b4f1"}
Sep 11 13:35:47.205802 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:47.205782    2578 scope.go:117] "RemoveContainer" containerID="3769238e4b8be4d079ed73c23768099625c3d7aee61eca86e402877ba483b4f1"
Sep 11 13:35:47.207417 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:47.207384    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 11 13:35:47.207555 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:47.207537    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 11 13:35:47.207626 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:47.207610    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 11 13:35:47.614862 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:47.614775    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b902823-ca13-41f9-90f5-42ceacbc5576-maas-api-tls\") pod \"maas-api-e2e-inf-b-a649d3-6f5f65cc57-kjbpx\" (UID: \"5b902823-ca13-41f9-90f5-42ceacbc5576\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a649d3-6f5f65cc57-kjbpx"
Sep 11 13:35:47.617971 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:47.617936    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b902823-ca13-41f9-90f5-42ceacbc5576-maas-api-tls\") pod \"maas-api-e2e-inf-b-a649d3-6f5f65cc57-kjbpx\" (UID: \"5b902823-ca13-41f9-90f5-42ceacbc5576\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a649d3-6f5f65cc57-kjbpx"
Sep 11 13:35:47.862867 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:47.862219    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a649d3-6f5f65cc57-kjbpx"
Sep 11 13:35:47.936638 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:47.934381    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-082f62-openshift-default-7d446b6f4d-cvpwk"
Sep 11 13:35:47.941525 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:47.941435    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-082f62-openshift-default-7d446b6f4d-cvpwk"
Sep 11 13:35:48.233303 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:48.232459    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-a649d3-6f5f65cc57-kjbpx"]
Sep 11 13:35:48.239031 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:48.238981    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-371c38-openshift-default-668769478-66p8n" event={"ID":"301e0569-db5c-45be-865c-491221d4883c","Type":"ContainerStarted","Data":"0aa2c4f8407c4a21a81095171f51f7843d615426f87544c5f99ca0e5d7f02483"}
Sep 11 13:35:48.284526 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:48.284472    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-371c38-openshift-default-668769478-66p8n"
Sep 11 13:35:48.284819 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:48.284787    2578 patch_prober.go:28] interesting pod/e2e-worker-w4-371c38-openshift-default-668769478-66p8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body=
Sep 11 13:35:48.284942 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:48.284889    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-371c38-openshift-default-668769478-66p8n" podUID="301e0569-db5c-45be-865c-491221d4883c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused"
Sep 11 13:35:49.248973 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:49.248918    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a649d3-6f5f65cc57-kjbpx" event={"ID":"5b902823-ca13-41f9-90f5-42ceacbc5576","Type":"ContainerStarted","Data":"1d0586a212cb06bd49c1b568b03dfdd228119377db65e9f94b9eb6148b430692"}
Sep 11 13:35:49.249558 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:49.249002    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a649d3-6f5f65cc57-kjbpx" event={"ID":"5b902823-ca13-41f9-90f5-42ceacbc5576","Type":"ContainerStarted","Data":"391d758e776169cd8fd315f9b1407318e2ce11539a75a368a0b13e2c491fb15e"}
Sep 11 13:35:49.249558 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:49.249237    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a649d3-6f5f65cc57-kjbpx"
Sep 11 13:35:49.277422 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:49.277351    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a649d3-6f5f65cc57-kjbpx" podStartSLOduration=2.960682287 podStartE2EDuration="3.27732801s" totalImagesPullingTime="316.645723ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:35:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:35:49.274382799 +0000 UTC m=+856.660485982" watchObservedRunningTime="2026-09-11 13:35:49.27732801 +0000 UTC m=+856.663431173"
Sep 11 13:35:49.284740 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:49.284687    2578 patch_prober.go:28] interesting pod/e2e-worker-w4-371c38-openshift-default-668769478-66p8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body=
Sep 11 13:35:49.285040 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:49.284774    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-371c38-openshift-default-668769478-66p8n" podUID="301e0569-db5c-45be-865c-491221d4883c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused"
Sep 11 13:35:50.284823 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:50.284774    2578 patch_prober.go:28] interesting pod/e2e-worker-w4-371c38-openshift-default-668769478-66p8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body=
Sep 11 13:35:50.285338 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:50.284880    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-371c38-openshift-default-668769478-66p8n" podUID="301e0569-db5c-45be-865c-491221d4883c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused"
Sep 11 13:35:51.284369 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:51.284309    2578 patch_prober.go:28] interesting pod/e2e-worker-w4-371c38-openshift-default-668769478-66p8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body=
Sep 11 13:35:51.284687 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:51.284414    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-371c38-openshift-default-668769478-66p8n" podUID="301e0569-db5c-45be-865c-491221d4883c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused"
Sep 11 13:35:52.284094 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:52.284046    2578 patch_prober.go:28] interesting pod/e2e-worker-w4-371c38-openshift-default-668769478-66p8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body=
Sep 11 13:35:52.284541 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:52.284124    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-371c38-openshift-default-668769478-66p8n" podUID="301e0569-db5c-45be-865c-491221d4883c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused"
Sep 11 13:35:53.284365 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:53.284327    2578 patch_prober.go:28] interesting pod/e2e-worker-w4-371c38-openshift-default-668769478-66p8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body=
Sep 11 13:35:53.284787 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:53.284391    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-371c38-openshift-default-668769478-66p8n" podUID="301e0569-db5c-45be-865c-491221d4883c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused"
Sep 11 13:35:54.289507 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:54.289462    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-371c38-openshift-default-668769478-66p8n"
Sep 11 13:35:54.290367 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:54.290334    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-371c38-openshift-default-668769478-66p8n"
Sep 11 13:35:54.291588 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:54.291560    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-371c38-openshift-default-668769478-66p8n"
Sep 11 13:35:55.262872 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:35:55.262824    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a649d3-6f5f65cc57-kjbpx"
Sep 11 13:36:03.894976 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:36:03.894916    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ae1d6b61-9b6d65f54-77vfx"]
Sep 11 13:36:03.895516 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:36:03.895300    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ae1d6b61-9b6d65f54-77vfx" podUID="f4c59fa8-961c-489b-8168-7873bfd7b439" containerName="maas-api" containerID="cri-o://81660207b94c6fe34851a09e6ec13206b0c2ad3048452004e3781a7eddfd7cdf" gracePeriod=30
Sep 11 13:36:04.197609 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:36:04.195667    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ae1d6b61-9b6d65f54-77vfx"
Sep 11 13:36:04.344101 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:36:04.344041    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f4c59fa8-961c-489b-8168-7873bfd7b439-maas-api-tls\") pod \"f4c59fa8-961c-489b-8168-7873bfd7b439\" (UID: \"f4c59fa8-961c-489b-8168-7873bfd7b439\") "
Sep 11 13:36:04.344422 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:36:04.344114    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4c59fa8-961c-489b-8168-7873bfd7b439-openshift-service-ca\") pod \"f4c59fa8-961c-489b-8168-7873bfd7b439\" (UID: \"f4c59fa8-961c-489b-8168-7873bfd7b439\") "
Sep 11 13:36:04.344422 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:36:04.344305    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rmfkd\" (UniqueName: \"kubernetes.io/projected/f4c59fa8-961c-489b-8168-7873bfd7b439-kube-api-access-rmfkd\") pod \"f4c59fa8-961c-489b-8168-7873bfd7b439\" (UID: \"f4c59fa8-961c-489b-8168-7873bfd7b439\") "
Sep 11 13:36:04.344422 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:36:04.344358    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f4c59fa8-961c-489b-8168-7873bfd7b439-metrics-certs\") pod \"f4c59fa8-961c-489b-8168-7873bfd7b439\" (UID: \"f4c59fa8-961c-489b-8168-7873bfd7b439\") "
Sep 11 13:36:04.344631 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:36:04.344575    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4c59fa8-961c-489b-8168-7873bfd7b439-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f4c59fa8-961c-489b-8168-7873bfd7b439" (UID: "f4c59fa8-961c-489b-8168-7873bfd7b439"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 13:36:04.344934 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:36:04.344906    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4c59fa8-961c-489b-8168-7873bfd7b439-openshift-service-ca\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:36:04.347645 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:36:04.347554    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4c59fa8-961c-489b-8168-7873bfd7b439-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f4c59fa8-961c-489b-8168-7873bfd7b439" (UID: "f4c59fa8-961c-489b-8168-7873bfd7b439"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 13:36:04.347783 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:36:04.347659    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4c59fa8-961c-489b-8168-7873bfd7b439-kube-api-access-rmfkd" (OuterVolumeSpecName: "kube-api-access-rmfkd") pod "f4c59fa8-961c-489b-8168-7873bfd7b439" (UID: "f4c59fa8-961c-489b-8168-7873bfd7b439"). InnerVolumeSpecName "kube-api-access-rmfkd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 13:36:04.347881 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:36:04.347857    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4c59fa8-961c-489b-8168-7873bfd7b439-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "f4c59fa8-961c-489b-8168-7873bfd7b439" (UID: "f4c59fa8-961c-489b-8168-7873bfd7b439"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 13:36:04.369053 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:36:04.369009    2578 generic.go:356] "Generic (PLEG): container finished" podID="f4c59fa8-961c-489b-8168-7873bfd7b439" containerID="81660207b94c6fe34851a09e6ec13206b0c2ad3048452004e3781a7eddfd7cdf" exitCode=0
Sep 11 13:36:04.369297 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:36:04.369072    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ae1d6b61-9b6d65f54-77vfx"
Sep 11 13:36:04.369297 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:36:04.369093    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ae1d6b61-9b6d65f54-77vfx" event={"ID":"f4c59fa8-961c-489b-8168-7873bfd7b439","Type":"ContainerDied","Data":"81660207b94c6fe34851a09e6ec13206b0c2ad3048452004e3781a7eddfd7cdf"}
Sep 11 13:36:04.369297 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:36:04.369160    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ae1d6b61-9b6d65f54-77vfx" event={"ID":"f4c59fa8-961c-489b-8168-7873bfd7b439","Type":"ContainerDied","Data":"3879630e3785043a6db41ce9053c5a6b2b849156f8f4d09fd6c94ee98c6ff51f"}
Sep 11 13:36:04.369297 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:36:04.369187    2578 scope.go:117] "RemoveContainer" containerID="81660207b94c6fe34851a09e6ec13206b0c2ad3048452004e3781a7eddfd7cdf"
Sep 11 13:36:04.380952 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:36:04.380930    2578 scope.go:117] "RemoveContainer" containerID="81660207b94c6fe34851a09e6ec13206b0c2ad3048452004e3781a7eddfd7cdf"
Sep 11 13:36:04.381390 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:36:04.381360    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81660207b94c6fe34851a09e6ec13206b0c2ad3048452004e3781a7eddfd7cdf\": container with ID starting with 81660207b94c6fe34851a09e6ec13206b0c2ad3048452004e3781a7eddfd7cdf not found: ID does not exist" containerID="81660207b94c6fe34851a09e6ec13206b0c2ad3048452004e3781a7eddfd7cdf"
Sep 11 13:36:04.381461 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:36:04.381407    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81660207b94c6fe34851a09e6ec13206b0c2ad3048452004e3781a7eddfd7cdf"} err="failed to get container status \"81660207b94c6fe34851a09e6ec13206b0c2ad3048452004e3781a7eddfd7cdf\": rpc error: code = NotFound desc = could not find container \"81660207b94c6fe34851a09e6ec13206b0c2ad3048452004e3781a7eddfd7cdf\": container with ID starting with 81660207b94c6fe34851a09e6ec13206b0c2ad3048452004e3781a7eddfd7cdf not found: ID does not exist"
Sep 11 13:36:04.394323 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:36:04.394232    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ae1d6b61-9b6d65f54-77vfx"]
Sep 11 13:36:04.399252 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:36:04.399199    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ae1d6b61-9b6d65f54-77vfx"]
Sep 11 13:36:04.446735 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:36:04.446613    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rmfkd\" (UniqueName: \"kubernetes.io/projected/f4c59fa8-961c-489b-8168-7873bfd7b439-kube-api-access-rmfkd\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:36:04.446735 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:36:04.446660    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f4c59fa8-961c-489b-8168-7873bfd7b439-metrics-certs\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:36:04.446735 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:36:04.446672    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f4c59fa8-961c-489b-8168-7873bfd7b439-maas-api-tls\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:36:05.108224 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:36:05.108184    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f4c59fa8-961c-489b-8168-7873bfd7b439" path="/var/lib/kubelet/pods/f4c59fa8-961c-489b-8168-7873bfd7b439/volumes"
Sep 11 13:36:33.117932 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:36:33.117810    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6gltj_e384caae-3cb1-4264-8a25-55b40ac8ac45/ovn-acl-logging/0.log"
Sep 11 13:36:33.118857 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:36:33.118812    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6gltj_e384caae-3cb1-4264-8a25-55b40ac8ac45/ovn-acl-logging/0.log"
Sep 11 13:36:38.000355 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:36:38.000308    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-5d979b8c-64dcd95c6d-fkw76"]
Sep 11 13:36:38.001282 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:36:38.001256    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4c59fa8-961c-489b-8168-7873bfd7b439" containerName="maas-api"
Sep 11 13:36:38.001376 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:36:38.001288    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4c59fa8-961c-489b-8168-7873bfd7b439" containerName="maas-api"
Sep 11 13:36:38.001447 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:36:38.001426    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="f4c59fa8-961c-489b-8168-7873bfd7b439" containerName="maas-api"
Sep 11 13:36:38.017044 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:36:38.016997    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5d979b8c-64dcd95c6d-fkw76"
Sep 11 13:36:38.019543 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:36:38.019484    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-5d979b8c-serving-cert\""
Sep 11 13:36:38.020642 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:36:38.020612    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-5d979b8c-64dcd95c6d-fkw76"]
Sep 11 13:36:38.059597 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:36:38.059550    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e56fe761-7235-4170-9d33-7c36add06dcb-metrics-certs\") pod \"maas-api-e2e-migrate-5d979b8c-64dcd95c6d-fkw76\" (UID: \"e56fe761-7235-4170-9d33-7c36add06dcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5d979b8c-64dcd95c6d-fkw76"
Sep 11 13:36:38.059820 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:36:38.059689    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhtcv\" (UniqueName: \"kubernetes.io/projected/e56fe761-7235-4170-9d33-7c36add06dcb-kube-api-access-jhtcv\") pod \"maas-api-e2e-migrate-5d979b8c-64dcd95c6d-fkw76\" (UID: \"e56fe761-7235-4170-9d33-7c36add06dcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5d979b8c-64dcd95c6d-fkw76"
Sep 11 13:36:38.059820 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:36:38.059723    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e56fe761-7235-4170-9d33-7c36add06dcb-openshift-service-ca\") pod \"maas-api-e2e-migrate-5d979b8c-64dcd95c6d-fkw76\" (UID: \"e56fe761-7235-4170-9d33-7c36add06dcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5d979b8c-64dcd95c6d-fkw76"
Sep 11 13:36:38.059977 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:36:38.059865    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e56fe761-7235-4170-9d33-7c36add06dcb-maas-api-tls\") pod \"maas-api-e2e-migrate-5d979b8c-64dcd95c6d-fkw76\" (UID: \"e56fe761-7235-4170-9d33-7c36add06dcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5d979b8c-64dcd95c6d-fkw76"
Sep 11 13:36:38.160913 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:36:38.160818    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e56fe761-7235-4170-9d33-7c36add06dcb-metrics-certs\") pod \"maas-api-e2e-migrate-5d979b8c-64dcd95c6d-fkw76\" (UID: \"e56fe761-7235-4170-9d33-7c36add06dcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5d979b8c-64dcd95c6d-fkw76"
Sep 11 13:36:38.161143 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:36:38.160954    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhtcv\" (UniqueName: \"kubernetes.io/projected/e56fe761-7235-4170-9d33-7c36add06dcb-kube-api-access-jhtcv\") pod \"maas-api-e2e-migrate-5d979b8c-64dcd95c6d-fkw76\" (UID: \"e56fe761-7235-4170-9d33-7c36add06dcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5d979b8c-64dcd95c6d-fkw76"
Sep 11 13:36:38.161143 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:36:38.160975    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e56fe761-7235-4170-9d33-7c36add06dcb-openshift-service-ca\") pod \"maas-api-e2e-migrate-5d979b8c-64dcd95c6d-fkw76\" (UID: \"e56fe761-7235-4170-9d33-7c36add06dcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5d979b8c-64dcd95c6d-fkw76"
Sep 11 13:36:38.161143 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:36:38.161061    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e56fe761-7235-4170-9d33-7c36add06dcb-maas-api-tls\") pod \"maas-api-e2e-migrate-5d979b8c-64dcd95c6d-fkw76\" (UID: \"e56fe761-7235-4170-9d33-7c36add06dcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5d979b8c-64dcd95c6d-fkw76"
Sep 11 13:36:38.161329 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:36:38.161167    2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-5d979b8c-serving-cert: secret "maas-api-e2e-migrate-5d979b8c-serving-cert" not found
Sep 11 13:36:38.161329 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:36:38.161271    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e56fe761-7235-4170-9d33-7c36add06dcb-maas-api-tls podName:e56fe761-7235-4170-9d33-7c36add06dcb nodeName:}" failed. No retries permitted until 2026-09-11 13:36:38.661246669 +0000 UTC m=+906.047349833 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e56fe761-7235-4170-9d33-7c36add06dcb-maas-api-tls") pod "maas-api-e2e-migrate-5d979b8c-64dcd95c6d-fkw76" (UID: "e56fe761-7235-4170-9d33-7c36add06dcb") : secret "maas-api-e2e-migrate-5d979b8c-serving-cert" not found
Sep 11 13:36:38.161774 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:36:38.161753    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e56fe761-7235-4170-9d33-7c36add06dcb-openshift-service-ca\") pod \"maas-api-e2e-migrate-5d979b8c-64dcd95c6d-fkw76\" (UID: \"e56fe761-7235-4170-9d33-7c36add06dcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5d979b8c-64dcd95c6d-fkw76"
Sep 11 13:36:38.163937 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:36:38.163912    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e56fe761-7235-4170-9d33-7c36add06dcb-metrics-certs\") pod \"maas-api-e2e-migrate-5d979b8c-64dcd95c6d-fkw76\" (UID: \"e56fe761-7235-4170-9d33-7c36add06dcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5d979b8c-64dcd95c6d-fkw76"
Sep 11 13:36:38.172354 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:36:38.172318    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhtcv\" (UniqueName: \"kubernetes.io/projected/e56fe761-7235-4170-9d33-7c36add06dcb-kube-api-access-jhtcv\") pod \"maas-api-e2e-migrate-5d979b8c-64dcd95c6d-fkw76\" (UID: \"e56fe761-7235-4170-9d33-7c36add06dcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5d979b8c-64dcd95c6d-fkw76"
Sep 11 13:36:38.668856 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:36:38.668781    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e56fe761-7235-4170-9d33-7c36add06dcb-maas-api-tls\") pod \"maas-api-e2e-migrate-5d979b8c-64dcd95c6d-fkw76\" (UID: \"e56fe761-7235-4170-9d33-7c36add06dcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5d979b8c-64dcd95c6d-fkw76"
Sep 11 13:36:38.669203 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:36:38.668979    2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-5d979b8c-serving-cert: secret "maas-api-e2e-migrate-5d979b8c-serving-cert" not found
Sep 11 13:36:38.669203 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:36:38.669079    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e56fe761-7235-4170-9d33-7c36add06dcb-maas-api-tls podName:e56fe761-7235-4170-9d33-7c36add06dcb nodeName:}" failed. No retries permitted until 2026-09-11 13:36:39.669060461 +0000 UTC m=+907.055163603 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e56fe761-7235-4170-9d33-7c36add06dcb-maas-api-tls") pod "maas-api-e2e-migrate-5d979b8c-64dcd95c6d-fkw76" (UID: "e56fe761-7235-4170-9d33-7c36add06dcb") : secret "maas-api-e2e-migrate-5d979b8c-serving-cert" not found
Sep 11 13:36:39.682411 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:36:39.682354    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e56fe761-7235-4170-9d33-7c36add06dcb-maas-api-tls\") pod \"maas-api-e2e-migrate-5d979b8c-64dcd95c6d-fkw76\" (UID: \"e56fe761-7235-4170-9d33-7c36add06dcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5d979b8c-64dcd95c6d-fkw76"
Sep 11 13:36:39.685574 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:36:39.685533    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e56fe761-7235-4170-9d33-7c36add06dcb-maas-api-tls\") pod \"maas-api-e2e-migrate-5d979b8c-64dcd95c6d-fkw76\" (UID: \"e56fe761-7235-4170-9d33-7c36add06dcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5d979b8c-64dcd95c6d-fkw76"
Sep 11 13:36:39.832311 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:36:39.832261    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5d979b8c-64dcd95c6d-fkw76"
Sep 11 13:36:40.124754 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:36:40.124715    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-5d979b8c-64dcd95c6d-fkw76"]
Sep 11 13:36:40.128063 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:36:40.128017    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode56fe761_7235_4170_9d33_7c36add06dcb.slice/crio-e7c010c155400d90b0eda6170721b80e4890b45dac50eb77acc1d802172d2a01 WatchSource:0}: Error finding container e7c010c155400d90b0eda6170721b80e4890b45dac50eb77acc1d802172d2a01: Status 404 returned error can't find the container with id e7c010c155400d90b0eda6170721b80e4890b45dac50eb77acc1d802172d2a01
Sep 11 13:36:40.617117 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:36:40.617069    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5d979b8c-64dcd95c6d-fkw76" event={"ID":"e56fe761-7235-4170-9d33-7c36add06dcb","Type":"ContainerStarted","Data":"85ebd60f1ac37f3d813f31ed95488e0a54e9270d5d3bf15b90a9f53d3522cc89"}
Sep 11 13:36:40.617353 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:36:40.617137    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5d979b8c-64dcd95c6d-fkw76"
Sep 11 13:36:40.617353 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:36:40.617164    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5d979b8c-64dcd95c6d-fkw76" event={"ID":"e56fe761-7235-4170-9d33-7c36add06dcb","Type":"ContainerStarted","Data":"e7c010c155400d90b0eda6170721b80e4890b45dac50eb77acc1d802172d2a01"}
Sep 11 13:36:40.642204 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:36:40.642116    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5d979b8c-64dcd95c6d-fkw76" podStartSLOduration=3.280229484 podStartE2EDuration="3.6420885s" totalImagesPullingTime="361.859016ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:36:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:36:40.641492108 +0000 UTC m=+908.027595294" watchObservedRunningTime="2026-09-11 13:36:40.6420885 +0000 UTC m=+908.028191675"
Sep 11 13:36:46.631471 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:36:46.631433    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5d979b8c-64dcd95c6d-fkw76"
Sep 11 13:37:16.274641 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:37:16.274604    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-5d979b8c-64dcd95c6d-fkw76"]
Sep 11 13:37:16.275156 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:37:16.274810    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5d979b8c-64dcd95c6d-fkw76" podUID="e56fe761-7235-4170-9d33-7c36add06dcb" containerName="maas-api" containerID="cri-o://85ebd60f1ac37f3d813f31ed95488e0a54e9270d5d3bf15b90a9f53d3522cc89" gracePeriod=30
Sep 11 13:37:16.572963 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:37:16.572929    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5d979b8c-64dcd95c6d-fkw76"
Sep 11 13:37:16.739959 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:37:16.739925    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jhtcv\" (UniqueName: \"kubernetes.io/projected/e56fe761-7235-4170-9d33-7c36add06dcb-kube-api-access-jhtcv\") pod \"e56fe761-7235-4170-9d33-7c36add06dcb\" (UID: \"e56fe761-7235-4170-9d33-7c36add06dcb\") "
Sep 11 13:37:16.740212 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:37:16.740015    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e56fe761-7235-4170-9d33-7c36add06dcb-maas-api-tls\") pod \"e56fe761-7235-4170-9d33-7c36add06dcb\" (UID: \"e56fe761-7235-4170-9d33-7c36add06dcb\") "
Sep 11 13:37:16.740212 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:37:16.740075    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e56fe761-7235-4170-9d33-7c36add06dcb-metrics-certs\") pod \"e56fe761-7235-4170-9d33-7c36add06dcb\" (UID: \"e56fe761-7235-4170-9d33-7c36add06dcb\") "
Sep 11 13:37:16.740929 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:37:16.740396    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e56fe761-7235-4170-9d33-7c36add06dcb-openshift-service-ca\") pod \"e56fe761-7235-4170-9d33-7c36add06dcb\" (UID: \"e56fe761-7235-4170-9d33-7c36add06dcb\") "
Sep 11 13:37:16.740929 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:37:16.740821    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e56fe761-7235-4170-9d33-7c36add06dcb-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e56fe761-7235-4170-9d33-7c36add06dcb" (UID: "e56fe761-7235-4170-9d33-7c36add06dcb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 13:37:16.741146 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:37:16.741082    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e56fe761-7235-4170-9d33-7c36add06dcb-openshift-service-ca\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:37:16.742996 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:37:16.742964    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e56fe761-7235-4170-9d33-7c36add06dcb-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e56fe761-7235-4170-9d33-7c36add06dcb" (UID: "e56fe761-7235-4170-9d33-7c36add06dcb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 13:37:16.743216 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:37:16.743194    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e56fe761-7235-4170-9d33-7c36add06dcb-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "e56fe761-7235-4170-9d33-7c36add06dcb" (UID: "e56fe761-7235-4170-9d33-7c36add06dcb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 13:37:16.743317 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:37:16.743247    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e56fe761-7235-4170-9d33-7c36add06dcb-kube-api-access-jhtcv" (OuterVolumeSpecName: "kube-api-access-jhtcv") pod "e56fe761-7235-4170-9d33-7c36add06dcb" (UID: "e56fe761-7235-4170-9d33-7c36add06dcb"). InnerVolumeSpecName "kube-api-access-jhtcv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 13:37:16.842404 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:37:16.842295    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e56fe761-7235-4170-9d33-7c36add06dcb-metrics-certs\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:37:16.842404 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:37:16.842340    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhtcv\" (UniqueName: \"kubernetes.io/projected/e56fe761-7235-4170-9d33-7c36add06dcb-kube-api-access-jhtcv\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:37:16.842404 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:37:16.842355    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e56fe761-7235-4170-9d33-7c36add06dcb-maas-api-tls\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:37:16.865240 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:37:16.865196    2578 generic.go:356] "Generic (PLEG): container finished" podID="e56fe761-7235-4170-9d33-7c36add06dcb" containerID="85ebd60f1ac37f3d813f31ed95488e0a54e9270d5d3bf15b90a9f53d3522cc89" exitCode=0
Sep 11 13:37:16.865492 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:37:16.865267    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5d979b8c-64dcd95c6d-fkw76"
Sep 11 13:37:16.865492 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:37:16.865272    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5d979b8c-64dcd95c6d-fkw76" event={"ID":"e56fe761-7235-4170-9d33-7c36add06dcb","Type":"ContainerDied","Data":"85ebd60f1ac37f3d813f31ed95488e0a54e9270d5d3bf15b90a9f53d3522cc89"}
Sep 11 13:37:16.865492 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:37:16.865337    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5d979b8c-64dcd95c6d-fkw76" event={"ID":"e56fe761-7235-4170-9d33-7c36add06dcb","Type":"ContainerDied","Data":"e7c010c155400d90b0eda6170721b80e4890b45dac50eb77acc1d802172d2a01"}
Sep 11 13:37:16.865492 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:37:16.865364    2578 scope.go:117] "RemoveContainer" containerID="85ebd60f1ac37f3d813f31ed95488e0a54e9270d5d3bf15b90a9f53d3522cc89"
Sep 11 13:37:16.878866 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:37:16.878823    2578 scope.go:117] "RemoveContainer" containerID="85ebd60f1ac37f3d813f31ed95488e0a54e9270d5d3bf15b90a9f53d3522cc89"
Sep 11 13:37:16.879291 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:37:16.879232    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85ebd60f1ac37f3d813f31ed95488e0a54e9270d5d3bf15b90a9f53d3522cc89\": container with ID starting with 85ebd60f1ac37f3d813f31ed95488e0a54e9270d5d3bf15b90a9f53d3522cc89 not found: ID does not exist" containerID="85ebd60f1ac37f3d813f31ed95488e0a54e9270d5d3bf15b90a9f53d3522cc89"
Sep 11 13:37:16.879406 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:37:16.879288    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85ebd60f1ac37f3d813f31ed95488e0a54e9270d5d3bf15b90a9f53d3522cc89"} err="failed to get container status \"85ebd60f1ac37f3d813f31ed95488e0a54e9270d5d3bf15b90a9f53d3522cc89\": rpc error: code = NotFound desc = could not find container \"85ebd60f1ac37f3d813f31ed95488e0a54e9270d5d3bf15b90a9f53d3522cc89\": container with ID starting with 85ebd60f1ac37f3d813f31ed95488e0a54e9270d5d3bf15b90a9f53d3522cc89 not found: ID does not exist"
Sep 11 13:37:16.896700 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:37:16.896657    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-5d979b8c-64dcd95c6d-fkw76"]
Sep 11 13:37:16.904455 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:37:16.904409    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-5d979b8c-64dcd95c6d-fkw76"]
Sep 11 13:37:17.109404 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:37:17.109300    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e56fe761-7235-4170-9d33-7c36add06dcb" path="/var/lib/kubelet/pods/e56fe761-7235-4170-9d33-7c36add06dcb/volumes"
Sep 11 13:37:26.032676 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:37:26.032594    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-eeb77b9d-79db4df455-nxxq5"]
Sep 11 13:37:26.033533 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:37:26.033509    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e56fe761-7235-4170-9d33-7c36add06dcb" containerName="maas-api"
Sep 11 13:37:26.033601 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:37:26.033539    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="e56fe761-7235-4170-9d33-7c36add06dcb" containerName="maas-api"
Sep 11 13:37:26.033659 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:37:26.033648    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="e56fe761-7235-4170-9d33-7c36add06dcb" containerName="maas-api"
Sep 11 13:37:26.039060 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:37:26.039031    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-eeb77b9d-79db4df455-nxxq5"
Sep 11 13:37:26.043871 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:37:26.043841    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-eeb77b9d-serving-cert\""
Sep 11 13:37:26.060123 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:37:26.060084    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-eeb77b9d-79db4df455-nxxq5"]
Sep 11 13:37:26.162075 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:37:26.162028    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8458f343-f74e-49a6-a130-02848f58dcbc-metrics-certs\") pod \"maas-api-e2e-ait-eeb77b9d-79db4df455-nxxq5\" (UID: \"8458f343-f74e-49a6-a130-02848f58dcbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-eeb77b9d-79db4df455-nxxq5"
Sep 11 13:37:26.162270 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:37:26.162129    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8458f343-f74e-49a6-a130-02848f58dcbc-openshift-service-ca\") pod \"maas-api-e2e-ait-eeb77b9d-79db4df455-nxxq5\" (UID: \"8458f343-f74e-49a6-a130-02848f58dcbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-eeb77b9d-79db4df455-nxxq5"
Sep 11 13:37:26.162270 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:37:26.162234    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nz6dp\" (UniqueName: \"kubernetes.io/projected/8458f343-f74e-49a6-a130-02848f58dcbc-kube-api-access-nz6dp\") pod \"maas-api-e2e-ait-eeb77b9d-79db4df455-nxxq5\" (UID: \"8458f343-f74e-49a6-a130-02848f58dcbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-eeb77b9d-79db4df455-nxxq5"
Sep 11 13:37:26.162351 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:37:26.162335    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8458f343-f74e-49a6-a130-02848f58dcbc-maas-api-tls\") pod \"maas-api-e2e-ait-eeb77b9d-79db4df455-nxxq5\" (UID: \"8458f343-f74e-49a6-a130-02848f58dcbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-eeb77b9d-79db4df455-nxxq5"
Sep 11 13:37:26.263565 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:37:26.263522    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8458f343-f74e-49a6-a130-02848f58dcbc-openshift-service-ca\") pod \"maas-api-e2e-ait-eeb77b9d-79db4df455-nxxq5\" (UID: \"8458f343-f74e-49a6-a130-02848f58dcbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-eeb77b9d-79db4df455-nxxq5"
Sep 11 13:37:26.263797 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:37:26.263605    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nz6dp\" (UniqueName: \"kubernetes.io/projected/8458f343-f74e-49a6-a130-02848f58dcbc-kube-api-access-nz6dp\") pod \"maas-api-e2e-ait-eeb77b9d-79db4df455-nxxq5\" (UID: \"8458f343-f74e-49a6-a130-02848f58dcbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-eeb77b9d-79db4df455-nxxq5"
Sep 11 13:37:26.263797 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:37:26.263667    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8458f343-f74e-49a6-a130-02848f58dcbc-maas-api-tls\") pod \"maas-api-e2e-ait-eeb77b9d-79db4df455-nxxq5\" (UID: \"8458f343-f74e-49a6-a130-02848f58dcbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-eeb77b9d-79db4df455-nxxq5"
Sep 11 13:37:26.263797 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:37:26.263727    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8458f343-f74e-49a6-a130-02848f58dcbc-metrics-certs\") pod \"maas-api-e2e-ait-eeb77b9d-79db4df455-nxxq5\" (UID: \"8458f343-f74e-49a6-a130-02848f58dcbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-eeb77b9d-79db4df455-nxxq5"
Sep 11 13:37:26.264003 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:37:26.263867    2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-eeb77b9d-serving-cert: secret "maas-api-e2e-ait-eeb77b9d-serving-cert" not found
Sep 11 13:37:26.264003 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:37:26.263947    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8458f343-f74e-49a6-a130-02848f58dcbc-maas-api-tls podName:8458f343-f74e-49a6-a130-02848f58dcbc nodeName:}" failed. No retries permitted until 2026-09-11 13:37:26.763928518 +0000 UTC m=+954.150031659 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8458f343-f74e-49a6-a130-02848f58dcbc-maas-api-tls") pod "maas-api-e2e-ait-eeb77b9d-79db4df455-nxxq5" (UID: "8458f343-f74e-49a6-a130-02848f58dcbc") : secret "maas-api-e2e-ait-eeb77b9d-serving-cert" not found
Sep 11 13:37:26.264357 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:37:26.264331    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8458f343-f74e-49a6-a130-02848f58dcbc-openshift-service-ca\") pod \"maas-api-e2e-ait-eeb77b9d-79db4df455-nxxq5\" (UID: \"8458f343-f74e-49a6-a130-02848f58dcbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-eeb77b9d-79db4df455-nxxq5"
Sep 11 13:37:26.266377 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:37:26.266353    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8458f343-f74e-49a6-a130-02848f58dcbc-metrics-certs\") pod \"maas-api-e2e-ait-eeb77b9d-79db4df455-nxxq5\" (UID: \"8458f343-f74e-49a6-a130-02848f58dcbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-eeb77b9d-79db4df455-nxxq5"
Sep 11 13:37:26.282055 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:37:26.282018    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nz6dp\" (UniqueName: \"kubernetes.io/projected/8458f343-f74e-49a6-a130-02848f58dcbc-kube-api-access-nz6dp\") pod \"maas-api-e2e-ait-eeb77b9d-79db4df455-nxxq5\" (UID: \"8458f343-f74e-49a6-a130-02848f58dcbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-eeb77b9d-79db4df455-nxxq5"
Sep 11 13:37:26.769392 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:37:26.769350    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8458f343-f74e-49a6-a130-02848f58dcbc-maas-api-tls\") pod \"maas-api-e2e-ait-eeb77b9d-79db4df455-nxxq5\" (UID: \"8458f343-f74e-49a6-a130-02848f58dcbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-eeb77b9d-79db4df455-nxxq5"
Sep 11 13:37:26.772220 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:37:26.772191    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8458f343-f74e-49a6-a130-02848f58dcbc-maas-api-tls\") pod \"maas-api-e2e-ait-eeb77b9d-79db4df455-nxxq5\" (UID: \"8458f343-f74e-49a6-a130-02848f58dcbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-eeb77b9d-79db4df455-nxxq5"
Sep 11 13:37:26.955445 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:37:26.955401    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-eeb77b9d-79db4df455-nxxq5"
Sep 11 13:37:27.256900 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:37:27.256804    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-eeb77b9d-79db4df455-nxxq5"]
Sep 11 13:37:27.259796 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:37:27.259758    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8458f343_f74e_49a6_a130_02848f58dcbc.slice/crio-35eb1b928d6074ea5de3c6c8ed173a294e4522bbd8b0f2bfc3e08c7a1bd5b543 WatchSource:0}: Error finding container 35eb1b928d6074ea5de3c6c8ed173a294e4522bbd8b0f2bfc3e08c7a1bd5b543: Status 404 returned error can't find the container with id 35eb1b928d6074ea5de3c6c8ed173a294e4522bbd8b0f2bfc3e08c7a1bd5b543
Sep 11 13:37:27.946815 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:37:27.946701    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-eeb77b9d-79db4df455-nxxq5" event={"ID":"8458f343-f74e-49a6-a130-02848f58dcbc","Type":"ContainerStarted","Data":"2a0ea972795d027c71e9da0b0201fab14a339d12bfd8c714d9b01ffec48b08e2"}
Sep 11 13:37:27.946815 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:37:27.946770    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-eeb77b9d-79db4df455-nxxq5" event={"ID":"8458f343-f74e-49a6-a130-02848f58dcbc","Type":"ContainerStarted","Data":"35eb1b928d6074ea5de3c6c8ed173a294e4522bbd8b0f2bfc3e08c7a1bd5b543"}
Sep 11 13:37:27.947043 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:37:27.946909    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-eeb77b9d-79db4df455-nxxq5"
Sep 11 13:37:27.977944 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:37:27.977887    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-eeb77b9d-79db4df455-nxxq5" podStartSLOduration=2.6294221110000002 podStartE2EDuration="2.977869559s" totalImagesPullingTime="348.447448ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:37:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:37:27.975873645 +0000 UTC m=+955.361976835" watchObservedRunningTime="2026-09-11 13:37:27.977869559 +0000 UTC m=+955.363972744"
Sep 11 13:37:29.290166 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:37:29.290127    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-f9f26e-fbc8976f7-l229k"]
Sep 11 13:37:29.291144 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:37:29.291101    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-f9f26e-fbc8976f7-l229k" podUID="54749995-810d-417d-881d-c9f755795d3e" containerName="maas-api" containerID="cri-o://73132a03c561ce67304d4cb9396b6208c0459160fe531dca36cbb701afa765fd" gracePeriod=30
Sep 11 13:37:29.576242 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:37:29.576209    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-f9f26e-fbc8976f7-l229k"
Sep 11 13:37:29.704792 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:37:29.704749    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tlxld\" (UniqueName: \"kubernetes.io/projected/54749995-810d-417d-881d-c9f755795d3e-kube-api-access-tlxld\") pod \"54749995-810d-417d-881d-c9f755795d3e\" (UID: \"54749995-810d-417d-881d-c9f755795d3e\") "
Sep 11 13:37:29.705327 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:37:29.705246    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/54749995-810d-417d-881d-c9f755795d3e-maas-api-tls\") pod \"54749995-810d-417d-881d-c9f755795d3e\" (UID: \"54749995-810d-417d-881d-c9f755795d3e\") "
Sep 11 13:37:29.705421 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:37:29.705401    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/54749995-810d-417d-881d-c9f755795d3e-openshift-service-ca\") pod \"54749995-810d-417d-881d-c9f755795d3e\" (UID: \"54749995-810d-417d-881d-c9f755795d3e\") "
Sep 11 13:37:29.705546 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:37:29.705449    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/54749995-810d-417d-881d-c9f755795d3e-metrics-certs\") pod \"54749995-810d-417d-881d-c9f755795d3e\" (UID: \"54749995-810d-417d-881d-c9f755795d3e\") "
Sep 11 13:37:29.705988 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:37:29.705952    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54749995-810d-417d-881d-c9f755795d3e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "54749995-810d-417d-881d-c9f755795d3e" (UID: "54749995-810d-417d-881d-c9f755795d3e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 13:37:29.706293 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:37:29.706270    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/54749995-810d-417d-881d-c9f755795d3e-openshift-service-ca\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:37:29.708166 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:37:29.707977    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54749995-810d-417d-881d-c9f755795d3e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "54749995-810d-417d-881d-c9f755795d3e" (UID: "54749995-810d-417d-881d-c9f755795d3e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 13:37:29.708555 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:37:29.708526    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54749995-810d-417d-881d-c9f755795d3e-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "54749995-810d-417d-881d-c9f755795d3e" (UID: "54749995-810d-417d-881d-c9f755795d3e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 13:37:29.708804 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:37:29.708772    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54749995-810d-417d-881d-c9f755795d3e-kube-api-access-tlxld" (OuterVolumeSpecName: "kube-api-access-tlxld") pod "54749995-810d-417d-881d-c9f755795d3e" (UID: "54749995-810d-417d-881d-c9f755795d3e"). InnerVolumeSpecName "kube-api-access-tlxld". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 13:37:29.808089 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:37:29.807989    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tlxld\" (UniqueName: \"kubernetes.io/projected/54749995-810d-417d-881d-c9f755795d3e-kube-api-access-tlxld\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:37:29.808089 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:37:29.808036    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/54749995-810d-417d-881d-c9f755795d3e-maas-api-tls\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:37:29.808089 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:37:29.808050    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/54749995-810d-417d-881d-c9f755795d3e-metrics-certs\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:37:29.960526 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:37:29.960484    2578 generic.go:356] "Generic (PLEG): container finished" podID="54749995-810d-417d-881d-c9f755795d3e" containerID="73132a03c561ce67304d4cb9396b6208c0459160fe531dca36cbb701afa765fd" exitCode=0
Sep 11 13:37:29.960767 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:37:29.960571    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-f9f26e-fbc8976f7-l229k"
Sep 11 13:37:29.960767 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:37:29.960581    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-f9f26e-fbc8976f7-l229k" event={"ID":"54749995-810d-417d-881d-c9f755795d3e","Type":"ContainerDied","Data":"73132a03c561ce67304d4cb9396b6208c0459160fe531dca36cbb701afa765fd"}
Sep 11 13:37:29.960767 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:37:29.960671    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-f9f26e-fbc8976f7-l229k" event={"ID":"54749995-810d-417d-881d-c9f755795d3e","Type":"ContainerDied","Data":"a718ccc986193795b8e37c86d1bbcd07033974f7a241be24eb418b9d8379a3ee"}
Sep 11 13:37:29.960767 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:37:29.960699    2578 scope.go:117] "RemoveContainer" containerID="73132a03c561ce67304d4cb9396b6208c0459160fe531dca36cbb701afa765fd"
Sep 11 13:37:29.971752 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:37:29.971732    2578 scope.go:117] "RemoveContainer" containerID="73132a03c561ce67304d4cb9396b6208c0459160fe531dca36cbb701afa765fd"
Sep 11 13:37:29.972162 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:37:29.972138    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73132a03c561ce67304d4cb9396b6208c0459160fe531dca36cbb701afa765fd\": container with ID starting with 73132a03c561ce67304d4cb9396b6208c0459160fe531dca36cbb701afa765fd not found: ID does not exist" containerID="73132a03c561ce67304d4cb9396b6208c0459160fe531dca36cbb701afa765fd"
Sep 11 13:37:29.972280 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:37:29.972169    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73132a03c561ce67304d4cb9396b6208c0459160fe531dca36cbb701afa765fd"} err="failed to get container status \"73132a03c561ce67304d4cb9396b6208c0459160fe531dca36cbb701afa765fd\": rpc error: code = NotFound desc = could not find container \"73132a03c561ce67304d4cb9396b6208c0459160fe531dca36cbb701afa765fd\": container with ID starting with 73132a03c561ce67304d4cb9396b6208c0459160fe531dca36cbb701afa765fd not found: ID does not exist"
Sep 11 13:37:29.985594 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:37:29.985550    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-f9f26e-fbc8976f7-l229k"]
Sep 11 13:37:29.990309 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:37:29.990270    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-f9f26e-fbc8976f7-l229k"]
Sep 11 13:37:31.110235 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:37:31.110199    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="54749995-810d-417d-881d-c9f755795d3e" path="/var/lib/kubelet/pods/54749995-810d-417d-881d-c9f755795d3e/volumes"
Sep 11 13:37:33.959488 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:37:33.959454    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-eeb77b9d-79db4df455-nxxq5"
Sep 11 13:38:02.875453 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:02.875409    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-eeb77b9d-79db4df455-nxxq5"]
Sep 11 13:38:02.876163 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:02.875689    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-eeb77b9d-79db4df455-nxxq5" podUID="8458f343-f74e-49a6-a130-02848f58dcbc" containerName="maas-api" containerID="cri-o://2a0ea972795d027c71e9da0b0201fab14a339d12bfd8c714d9b01ffec48b08e2" gracePeriod=30
Sep 11 13:38:03.164319 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:03.164279    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-eeb77b9d-79db4df455-nxxq5"
Sep 11 13:38:03.183234 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:03.183192    2578 generic.go:356] "Generic (PLEG): container finished" podID="8458f343-f74e-49a6-a130-02848f58dcbc" containerID="2a0ea972795d027c71e9da0b0201fab14a339d12bfd8c714d9b01ffec48b08e2" exitCode=0
Sep 11 13:38:03.183459 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:03.183249    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-eeb77b9d-79db4df455-nxxq5"
Sep 11 13:38:03.183459 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:03.183281    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-eeb77b9d-79db4df455-nxxq5" event={"ID":"8458f343-f74e-49a6-a130-02848f58dcbc","Type":"ContainerDied","Data":"2a0ea972795d027c71e9da0b0201fab14a339d12bfd8c714d9b01ffec48b08e2"}
Sep 11 13:38:03.183459 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:03.183348    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-eeb77b9d-79db4df455-nxxq5" event={"ID":"8458f343-f74e-49a6-a130-02848f58dcbc","Type":"ContainerDied","Data":"35eb1b928d6074ea5de3c6c8ed173a294e4522bbd8b0f2bfc3e08c7a1bd5b543"}
Sep 11 13:38:03.183459 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:03.183381    2578 scope.go:117] "RemoveContainer" containerID="2a0ea972795d027c71e9da0b0201fab14a339d12bfd8c714d9b01ffec48b08e2"
Sep 11 13:38:03.201775 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:03.201743    2578 scope.go:117] "RemoveContainer" containerID="2a0ea972795d027c71e9da0b0201fab14a339d12bfd8c714d9b01ffec48b08e2"
Sep 11 13:38:03.202563 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:38:03.202528    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a0ea972795d027c71e9da0b0201fab14a339d12bfd8c714d9b01ffec48b08e2\": container with ID starting with 2a0ea972795d027c71e9da0b0201fab14a339d12bfd8c714d9b01ffec48b08e2 not found: ID does not exist" containerID="2a0ea972795d027c71e9da0b0201fab14a339d12bfd8c714d9b01ffec48b08e2"
Sep 11 13:38:03.202679 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:03.202579    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a0ea972795d027c71e9da0b0201fab14a339d12bfd8c714d9b01ffec48b08e2"} err="failed to get container status \"2a0ea972795d027c71e9da0b0201fab14a339d12bfd8c714d9b01ffec48b08e2\": rpc error: code = NotFound desc = could not find container \"2a0ea972795d027c71e9da0b0201fab14a339d12bfd8c714d9b01ffec48b08e2\": container with ID starting with 2a0ea972795d027c71e9da0b0201fab14a339d12bfd8c714d9b01ffec48b08e2 not found: ID does not exist"
Sep 11 13:38:03.315252 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:03.315182    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nz6dp\" (UniqueName: \"kubernetes.io/projected/8458f343-f74e-49a6-a130-02848f58dcbc-kube-api-access-nz6dp\") pod \"8458f343-f74e-49a6-a130-02848f58dcbc\" (UID: \"8458f343-f74e-49a6-a130-02848f58dcbc\") "
Sep 11 13:38:03.315537 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:03.315276    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8458f343-f74e-49a6-a130-02848f58dcbc-openshift-service-ca\") pod \"8458f343-f74e-49a6-a130-02848f58dcbc\" (UID: \"8458f343-f74e-49a6-a130-02848f58dcbc\") "
Sep 11 13:38:03.315537 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:03.315392    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8458f343-f74e-49a6-a130-02848f58dcbc-maas-api-tls\") pod \"8458f343-f74e-49a6-a130-02848f58dcbc\" (UID: \"8458f343-f74e-49a6-a130-02848f58dcbc\") "
Sep 11 13:38:03.315537 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:03.315500    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8458f343-f74e-49a6-a130-02848f58dcbc-metrics-certs\") pod \"8458f343-f74e-49a6-a130-02848f58dcbc\" (UID: \"8458f343-f74e-49a6-a130-02848f58dcbc\") "
Sep 11 13:38:03.315741 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:03.315672    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8458f343-f74e-49a6-a130-02848f58dcbc-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8458f343-f74e-49a6-a130-02848f58dcbc" (UID: "8458f343-f74e-49a6-a130-02848f58dcbc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 13:38:03.316185 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:03.316142    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8458f343-f74e-49a6-a130-02848f58dcbc-openshift-service-ca\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:38:03.318567 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:03.318518    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8458f343-f74e-49a6-a130-02848f58dcbc-kube-api-access-nz6dp" (OuterVolumeSpecName: "kube-api-access-nz6dp") pod "8458f343-f74e-49a6-a130-02848f58dcbc" (UID: "8458f343-f74e-49a6-a130-02848f58dcbc"). InnerVolumeSpecName "kube-api-access-nz6dp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 13:38:03.318887 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:03.318862    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8458f343-f74e-49a6-a130-02848f58dcbc-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8458f343-f74e-49a6-a130-02848f58dcbc" (UID: "8458f343-f74e-49a6-a130-02848f58dcbc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 13:38:03.319011 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:03.318973    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8458f343-f74e-49a6-a130-02848f58dcbc-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "8458f343-f74e-49a6-a130-02848f58dcbc" (UID: "8458f343-f74e-49a6-a130-02848f58dcbc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 13:38:03.417253 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:03.417126    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8458f343-f74e-49a6-a130-02848f58dcbc-metrics-certs\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:38:03.417253 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:03.417166    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nz6dp\" (UniqueName: \"kubernetes.io/projected/8458f343-f74e-49a6-a130-02848f58dcbc-kube-api-access-nz6dp\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:38:03.417253 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:03.417183    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8458f343-f74e-49a6-a130-02848f58dcbc-maas-api-tls\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:38:03.507839 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:03.507773    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-eeb77b9d-79db4df455-nxxq5"]
Sep 11 13:38:03.512075 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:03.512040    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-eeb77b9d-79db4df455-nxxq5"]
Sep 11 13:38:04.996857 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:04.996792    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-2a5fb372-openshift-default-5bfb49554d-7klgx"]
Sep 11 13:38:04.997514 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:04.997493    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8458f343-f74e-49a6-a130-02848f58dcbc" containerName="maas-api"
Sep 11 13:38:04.997604 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:04.997518    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="8458f343-f74e-49a6-a130-02848f58dcbc" containerName="maas-api"
Sep 11 13:38:04.997604 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:04.997576    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54749995-810d-417d-881d-c9f755795d3e" containerName="maas-api"
Sep 11 13:38:04.997604 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:04.997586    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="54749995-810d-417d-881d-c9f755795d3e" containerName="maas-api"
Sep 11 13:38:04.997727 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:04.997715    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="8458f343-f74e-49a6-a130-02848f58dcbc" containerName="maas-api"
Sep 11 13:38:04.997764 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:04.997729    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="54749995-810d-417d-881d-c9f755795d3e" containerName="maas-api"
Sep 11 13:38:05.004361 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:05.004317    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-2a5fb372-openshift-default-5bfb49554d-7klgx"
Sep 11 13:38:05.007311 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:05.007285    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-2a5fb372-openshift-default-dockercfg-5fndv\""
Sep 11 13:38:05.017609 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:05.017569    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-2a5fb372-openshift-default-5bfb49554d-7klgx"]
Sep 11 13:38:05.111638 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:05.111584    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8458f343-f74e-49a6-a130-02848f58dcbc" path="/var/lib/kubelet/pods/8458f343-f74e-49a6-a130-02848f58dcbc/volumes"
Sep 11 13:38:05.137327 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:05.137283    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c5140cf-195d-4604-9325-40d5b485e127-workload-socket\") pod \"e2e-derive-2a5fb372-openshift-default-5bfb49554d-7klgx\" (UID: \"7c5140cf-195d-4604-9325-40d5b485e127\") " pod="openshift-ingress/e2e-derive-2a5fb372-openshift-default-5bfb49554d-7klgx"
Sep 11 13:38:05.137533 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:05.137340    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c5140cf-195d-4604-9325-40d5b485e127-istio-token\") pod \"e2e-derive-2a5fb372-openshift-default-5bfb49554d-7klgx\" (UID: \"7c5140cf-195d-4604-9325-40d5b485e127\") " pod="openshift-ingress/e2e-derive-2a5fb372-openshift-default-5bfb49554d-7klgx"
Sep 11 13:38:05.137533 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:05.137397    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c5140cf-195d-4604-9325-40d5b485e127-istiod-ca-cert\") pod \"e2e-derive-2a5fb372-openshift-default-5bfb49554d-7klgx\" (UID: \"7c5140cf-195d-4604-9325-40d5b485e127\") " pod="openshift-ingress/e2e-derive-2a5fb372-openshift-default-5bfb49554d-7klgx"
Sep 11 13:38:05.137533 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:05.137435    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c5140cf-195d-4604-9325-40d5b485e127-istio-podinfo\") pod \"e2e-derive-2a5fb372-openshift-default-5bfb49554d-7klgx\" (UID: \"7c5140cf-195d-4604-9325-40d5b485e127\") " pod="openshift-ingress/e2e-derive-2a5fb372-openshift-default-5bfb49554d-7klgx"
Sep 11 13:38:05.137533 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:05.137464    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c5140cf-195d-4604-9325-40d5b485e127-istio-data\") pod \"e2e-derive-2a5fb372-openshift-default-5bfb49554d-7klgx\" (UID: \"7c5140cf-195d-4604-9325-40d5b485e127\") " pod="openshift-ingress/e2e-derive-2a5fb372-openshift-default-5bfb49554d-7klgx"
Sep 11 13:38:05.137533 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:05.137499    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7l8bp\" (UniqueName: \"kubernetes.io/projected/7c5140cf-195d-4604-9325-40d5b485e127-kube-api-access-7l8bp\") pod \"e2e-derive-2a5fb372-openshift-default-5bfb49554d-7klgx\" (UID: \"7c5140cf-195d-4604-9325-40d5b485e127\") " pod="openshift-ingress/e2e-derive-2a5fb372-openshift-default-5bfb49554d-7klgx"
Sep 11 13:38:05.137712 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:05.137557    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c5140cf-195d-4604-9325-40d5b485e127-credential-socket\") pod \"e2e-derive-2a5fb372-openshift-default-5bfb49554d-7klgx\" (UID: \"7c5140cf-195d-4604-9325-40d5b485e127\") " pod="openshift-ingress/e2e-derive-2a5fb372-openshift-default-5bfb49554d-7klgx"
Sep 11 13:38:05.137712 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:05.137589    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c5140cf-195d-4604-9325-40d5b485e127-workload-certs\") pod \"e2e-derive-2a5fb372-openshift-default-5bfb49554d-7klgx\" (UID: \"7c5140cf-195d-4604-9325-40d5b485e127\") " pod="openshift-ingress/e2e-derive-2a5fb372-openshift-default-5bfb49554d-7klgx"
Sep 11 13:38:05.137712 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:05.137631    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c5140cf-195d-4604-9325-40d5b485e127-istio-envoy\") pod \"e2e-derive-2a5fb372-openshift-default-5bfb49554d-7klgx\" (UID: \"7c5140cf-195d-4604-9325-40d5b485e127\") " pod="openshift-ingress/e2e-derive-2a5fb372-openshift-default-5bfb49554d-7klgx"
Sep 11 13:38:05.239409 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:05.239357    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c5140cf-195d-4604-9325-40d5b485e127-workload-socket\") pod \"e2e-derive-2a5fb372-openshift-default-5bfb49554d-7klgx\" (UID: \"7c5140cf-195d-4604-9325-40d5b485e127\") " pod="openshift-ingress/e2e-derive-2a5fb372-openshift-default-5bfb49554d-7klgx"
Sep 11 13:38:05.239695 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:05.239461    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c5140cf-195d-4604-9325-40d5b485e127-istio-token\") pod \"e2e-derive-2a5fb372-openshift-default-5bfb49554d-7klgx\" (UID: \"7c5140cf-195d-4604-9325-40d5b485e127\") " pod="openshift-ingress/e2e-derive-2a5fb372-openshift-default-5bfb49554d-7klgx"
Sep 11 13:38:05.239695 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:05.239484    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c5140cf-195d-4604-9325-40d5b485e127-istiod-ca-cert\") pod \"e2e-derive-2a5fb372-openshift-default-5bfb49554d-7klgx\" (UID: \"7c5140cf-195d-4604-9325-40d5b485e127\") " pod="openshift-ingress/e2e-derive-2a5fb372-openshift-default-5bfb49554d-7klgx"
Sep 11 13:38:05.239695 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:05.239514    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c5140cf-195d-4604-9325-40d5b485e127-istio-podinfo\") pod \"e2e-derive-2a5fb372-openshift-default-5bfb49554d-7klgx\" (UID: \"7c5140cf-195d-4604-9325-40d5b485e127\") " pod="openshift-ingress/e2e-derive-2a5fb372-openshift-default-5bfb49554d-7klgx"
Sep 11 13:38:05.239695 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:05.239543    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c5140cf-195d-4604-9325-40d5b485e127-istio-data\") pod \"e2e-derive-2a5fb372-openshift-default-5bfb49554d-7klgx\" (UID: \"7c5140cf-195d-4604-9325-40d5b485e127\") " pod="openshift-ingress/e2e-derive-2a5fb372-openshift-default-5bfb49554d-7klgx"
Sep 11 13:38:05.239990 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:05.239917    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7l8bp\" (UniqueName: \"kubernetes.io/projected/7c5140cf-195d-4604-9325-40d5b485e127-kube-api-access-7l8bp\") pod \"e2e-derive-2a5fb372-openshift-default-5bfb49554d-7klgx\" (UID: \"7c5140cf-195d-4604-9325-40d5b485e127\") " pod="openshift-ingress/e2e-derive-2a5fb372-openshift-default-5bfb49554d-7klgx"
Sep 11 13:38:05.240047 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:05.239982    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c5140cf-195d-4604-9325-40d5b485e127-workload-socket\") pod \"e2e-derive-2a5fb372-openshift-default-5bfb49554d-7klgx\" (UID: \"7c5140cf-195d-4604-9325-40d5b485e127\") " pod="openshift-ingress/e2e-derive-2a5fb372-openshift-default-5bfb49554d-7klgx"
Sep 11 13:38:05.240047 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:05.240016    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c5140cf-195d-4604-9325-40d5b485e127-istio-data\") pod \"e2e-derive-2a5fb372-openshift-default-5bfb49554d-7klgx\" (UID: \"7c5140cf-195d-4604-9325-40d5b485e127\") " pod="openshift-ingress/e2e-derive-2a5fb372-openshift-default-5bfb49554d-7klgx"
Sep 11 13:38:05.240176 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:05.240122    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c5140cf-195d-4604-9325-40d5b485e127-credential-socket\") pod \"e2e-derive-2a5fb372-openshift-default-5bfb49554d-7klgx\" (UID: \"7c5140cf-195d-4604-9325-40d5b485e127\") " pod="openshift-ingress/e2e-derive-2a5fb372-openshift-default-5bfb49554d-7klgx"
Sep 11 13:38:05.240241 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:05.240223    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c5140cf-195d-4604-9325-40d5b485e127-workload-certs\") pod \"e2e-derive-2a5fb372-openshift-default-5bfb49554d-7klgx\" (UID: \"7c5140cf-195d-4604-9325-40d5b485e127\") " pod="openshift-ingress/e2e-derive-2a5fb372-openshift-default-5bfb49554d-7klgx"
Sep 11 13:38:05.240321 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:05.240307    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c5140cf-195d-4604-9325-40d5b485e127-istio-envoy\") pod \"e2e-derive-2a5fb372-openshift-default-5bfb49554d-7klgx\" (UID: \"7c5140cf-195d-4604-9325-40d5b485e127\") " pod="openshift-ingress/e2e-derive-2a5fb372-openshift-default-5bfb49554d-7klgx"
Sep 11 13:38:05.240447 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:05.240417    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c5140cf-195d-4604-9325-40d5b485e127-credential-socket\") pod \"e2e-derive-2a5fb372-openshift-default-5bfb49554d-7klgx\" (UID: \"7c5140cf-195d-4604-9325-40d5b485e127\") " pod="openshift-ingress/e2e-derive-2a5fb372-openshift-default-5bfb49554d-7klgx"
Sep 11 13:38:05.240593 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:05.240447    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c5140cf-195d-4604-9325-40d5b485e127-istiod-ca-cert\") pod \"e2e-derive-2a5fb372-openshift-default-5bfb49554d-7klgx\" (UID: \"7c5140cf-195d-4604-9325-40d5b485e127\") " pod="openshift-ingress/e2e-derive-2a5fb372-openshift-default-5bfb49554d-7klgx"
Sep 11 13:38:05.240593 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:05.240529    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c5140cf-195d-4604-9325-40d5b485e127-workload-certs\") pod \"e2e-derive-2a5fb372-openshift-default-5bfb49554d-7klgx\" (UID: \"7c5140cf-195d-4604-9325-40d5b485e127\") " pod="openshift-ingress/e2e-derive-2a5fb372-openshift-default-5bfb49554d-7klgx"
Sep 11 13:38:05.243524 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:05.243494    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c5140cf-195d-4604-9325-40d5b485e127-istio-envoy\") pod \"e2e-derive-2a5fb372-openshift-default-5bfb49554d-7klgx\" (UID: \"7c5140cf-195d-4604-9325-40d5b485e127\") " pod="openshift-ingress/e2e-derive-2a5fb372-openshift-default-5bfb49554d-7klgx"
Sep 11 13:38:05.243747 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:05.243720    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c5140cf-195d-4604-9325-40d5b485e127-istio-podinfo\") pod \"e2e-derive-2a5fb372-openshift-default-5bfb49554d-7klgx\" (UID: \"7c5140cf-195d-4604-9325-40d5b485e127\") " pod="openshift-ingress/e2e-derive-2a5fb372-openshift-default-5bfb49554d-7klgx"
Sep 11 13:38:05.250601 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:05.250508    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c5140cf-195d-4604-9325-40d5b485e127-istio-token\") pod \"e2e-derive-2a5fb372-openshift-default-5bfb49554d-7klgx\" (UID: \"7c5140cf-195d-4604-9325-40d5b485e127\") " pod="openshift-ingress/e2e-derive-2a5fb372-openshift-default-5bfb49554d-7klgx"
Sep 11 13:38:05.250862 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:05.250822    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7l8bp\" (UniqueName: \"kubernetes.io/projected/7c5140cf-195d-4604-9325-40d5b485e127-kube-api-access-7l8bp\") pod \"e2e-derive-2a5fb372-openshift-default-5bfb49554d-7klgx\" (UID: \"7c5140cf-195d-4604-9325-40d5b485e127\") " pod="openshift-ingress/e2e-derive-2a5fb372-openshift-default-5bfb49554d-7klgx"
Sep 11 13:38:05.319439 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:05.319394    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-2a5fb372-openshift-default-5bfb49554d-7klgx"
Sep 11 13:38:05.623171 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:05.623127    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-2a5fb372-openshift-default-5bfb49554d-7klgx"]
Sep 11 13:38:05.624231 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:38:05.624194    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c5140cf_195d_4604_9325_40d5b485e127.slice/crio-3acdf3bcfb4749b27b300f329db035a04ef11c94aaecc922311295fd79d0e417 WatchSource:0}: Error finding container 3acdf3bcfb4749b27b300f329db035a04ef11c94aaecc922311295fd79d0e417: Status 404 returned error can't find the container with id 3acdf3bcfb4749b27b300f329db035a04ef11c94aaecc922311295fd79d0e417
Sep 11 13:38:05.627208 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:05.627161    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 11 13:38:05.627312 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:05.627267    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 11 13:38:05.627312 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:05.627303    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 11 13:38:06.210908 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:06.210853    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-2a5fb372-openshift-default-5bfb49554d-7klgx" event={"ID":"7c5140cf-195d-4604-9325-40d5b485e127","Type":"ContainerStarted","Data":"d7220556846a6ce5cc08dbef2d891c1765dbe63bc17ac394a813e1e3b324ddc7"}
Sep 11 13:38:06.211431 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:06.210923    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-2a5fb372-openshift-default-5bfb49554d-7klgx" event={"ID":"7c5140cf-195d-4604-9325-40d5b485e127","Type":"ContainerStarted","Data":"3acdf3bcfb4749b27b300f329db035a04ef11c94aaecc922311295fd79d0e417"}
Sep 11 13:38:06.238943 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:06.238863    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-2a5fb372-openshift-default-5bfb49554d-7klgx" podStartSLOduration=2.238820096 podStartE2EDuration="2.238820096s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:38:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:38:06.237142629 +0000 UTC m=+993.623245819" watchObservedRunningTime="2026-09-11 13:38:06.238820096 +0000 UTC m=+993.624923261"
Sep 11 13:38:06.320061 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:06.320016    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-2a5fb372-openshift-default-5bfb49554d-7klgx"
Sep 11 13:38:06.326728 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:06.326685    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-2a5fb372-openshift-default-5bfb49554d-7klgx"
Sep 11 13:38:07.217815 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:07.217772    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-2a5fb372-openshift-default-5bfb49554d-7klgx"
Sep 11 13:38:07.218990 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:07.218964    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-2a5fb372-openshift-default-5bfb49554d-7klgx"
Sep 11 13:38:12.385095 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:12.385052    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-2a5fb372-74b5dcf5b7-cdt49"]
Sep 11 13:38:12.390339 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:12.390301    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2a5fb372-74b5dcf5b7-cdt49"
Sep 11 13:38:12.394764 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:12.394740    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-2a5fb372-serving-cert\""
Sep 11 13:38:12.407484 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:12.407451    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-2a5fb372-74b5dcf5b7-cdt49"]
Sep 11 13:38:12.427263 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:12.427222    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d5cc1e7-c0b9-4438-837d-beeeeb496273-metrics-certs\") pod \"maas-api-e2e-derive-2a5fb372-74b5dcf5b7-cdt49\" (UID: \"8d5cc1e7-c0b9-4438-837d-beeeeb496273\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2a5fb372-74b5dcf5b7-cdt49"
Sep 11 13:38:12.427490 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:12.427288    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jq7l\" (UniqueName: \"kubernetes.io/projected/8d5cc1e7-c0b9-4438-837d-beeeeb496273-kube-api-access-4jq7l\") pod \"maas-api-e2e-derive-2a5fb372-74b5dcf5b7-cdt49\" (UID: \"8d5cc1e7-c0b9-4438-837d-beeeeb496273\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2a5fb372-74b5dcf5b7-cdt49"
Sep 11 13:38:12.427490 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:12.427352    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d5cc1e7-c0b9-4438-837d-beeeeb496273-openshift-service-ca\") pod \"maas-api-e2e-derive-2a5fb372-74b5dcf5b7-cdt49\" (UID: \"8d5cc1e7-c0b9-4438-837d-beeeeb496273\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2a5fb372-74b5dcf5b7-cdt49"
Sep 11 13:38:12.427627 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:12.427514    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d5cc1e7-c0b9-4438-837d-beeeeb496273-maas-api-tls\") pod \"maas-api-e2e-derive-2a5fb372-74b5dcf5b7-cdt49\" (UID: \"8d5cc1e7-c0b9-4438-837d-beeeeb496273\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2a5fb372-74b5dcf5b7-cdt49"
Sep 11 13:38:12.528550 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:12.528492    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d5cc1e7-c0b9-4438-837d-beeeeb496273-maas-api-tls\") pod \"maas-api-e2e-derive-2a5fb372-74b5dcf5b7-cdt49\" (UID: \"8d5cc1e7-c0b9-4438-837d-beeeeb496273\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2a5fb372-74b5dcf5b7-cdt49"
Sep 11 13:38:12.528875 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:12.528636    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d5cc1e7-c0b9-4438-837d-beeeeb496273-metrics-certs\") pod \"maas-api-e2e-derive-2a5fb372-74b5dcf5b7-cdt49\" (UID: \"8d5cc1e7-c0b9-4438-837d-beeeeb496273\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2a5fb372-74b5dcf5b7-cdt49"
Sep 11 13:38:12.528875 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:12.528673    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jq7l\" (UniqueName: \"kubernetes.io/projected/8d5cc1e7-c0b9-4438-837d-beeeeb496273-kube-api-access-4jq7l\") pod \"maas-api-e2e-derive-2a5fb372-74b5dcf5b7-cdt49\" (UID: \"8d5cc1e7-c0b9-4438-837d-beeeeb496273\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2a5fb372-74b5dcf5b7-cdt49"
Sep 11 13:38:12.528875 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:12.528709    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d5cc1e7-c0b9-4438-837d-beeeeb496273-openshift-service-ca\") pod \"maas-api-e2e-derive-2a5fb372-74b5dcf5b7-cdt49\" (UID: \"8d5cc1e7-c0b9-4438-837d-beeeeb496273\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2a5fb372-74b5dcf5b7-cdt49"
Sep 11 13:38:12.529639 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:12.529556    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d5cc1e7-c0b9-4438-837d-beeeeb496273-openshift-service-ca\") pod \"maas-api-e2e-derive-2a5fb372-74b5dcf5b7-cdt49\" (UID: \"8d5cc1e7-c0b9-4438-837d-beeeeb496273\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2a5fb372-74b5dcf5b7-cdt49"
Sep 11 13:38:12.532134 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:12.532092    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d5cc1e7-c0b9-4438-837d-beeeeb496273-metrics-certs\") pod \"maas-api-e2e-derive-2a5fb372-74b5dcf5b7-cdt49\" (UID: \"8d5cc1e7-c0b9-4438-837d-beeeeb496273\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2a5fb372-74b5dcf5b7-cdt49"
Sep 11 13:38:12.532134 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:12.532123    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d5cc1e7-c0b9-4438-837d-beeeeb496273-maas-api-tls\") pod \"maas-api-e2e-derive-2a5fb372-74b5dcf5b7-cdt49\" (UID: \"8d5cc1e7-c0b9-4438-837d-beeeeb496273\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2a5fb372-74b5dcf5b7-cdt49"
Sep 11 13:38:12.537801 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:12.537761    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jq7l\" (UniqueName: \"kubernetes.io/projected/8d5cc1e7-c0b9-4438-837d-beeeeb496273-kube-api-access-4jq7l\") pod \"maas-api-e2e-derive-2a5fb372-74b5dcf5b7-cdt49\" (UID: \"8d5cc1e7-c0b9-4438-837d-beeeeb496273\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2a5fb372-74b5dcf5b7-cdt49"
Sep 11 13:38:12.704287 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:12.704146    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2a5fb372-74b5dcf5b7-cdt49"
Sep 11 13:38:13.006954 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:13.006885    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-2a5fb372-74b5dcf5b7-cdt49"]
Sep 11 13:38:13.008848 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:38:13.008785    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d5cc1e7_c0b9_4438_837d_beeeeb496273.slice/crio-d2b730cd9243e612ccbee91bacaadc77071e5a4e561a3354ee62202e17c18f50 WatchSource:0}: Error finding container d2b730cd9243e612ccbee91bacaadc77071e5a4e561a3354ee62202e17c18f50: Status 404 returned error can't find the container with id d2b730cd9243e612ccbee91bacaadc77071e5a4e561a3354ee62202e17c18f50
Sep 11 13:38:13.011291 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:13.011257    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 13:38:13.264210 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:13.264082    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2a5fb372-74b5dcf5b7-cdt49" event={"ID":"8d5cc1e7-c0b9-4438-837d-beeeeb496273","Type":"ContainerStarted","Data":"d2b730cd9243e612ccbee91bacaadc77071e5a4e561a3354ee62202e17c18f50"}
Sep 11 13:38:14.273034 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:14.272988    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2a5fb372-74b5dcf5b7-cdt49" event={"ID":"8d5cc1e7-c0b9-4438-837d-beeeeb496273","Type":"ContainerStarted","Data":"6c6ad11e0a413f80ab450c3a6050b4aff4ecf607cec45b6482e28925c6f561fa"}
Sep 11 13:38:14.273727 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:14.273085    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2a5fb372-74b5dcf5b7-cdt49"
Sep 11 13:38:14.296619 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:14.296539    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2a5fb372-74b5dcf5b7-cdt49" podStartSLOduration=1.828086129 podStartE2EDuration="2.296513934s" totalImagesPullingTime="468.427805ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:38:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:38:14.294290774 +0000 UTC m=+1001.680393957" watchObservedRunningTime="2026-09-11 13:38:14.296513934 +0000 UTC m=+1001.682617099"
Sep 11 13:38:20.286512 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:20.286453    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2a5fb372-74b5dcf5b7-cdt49"
Sep 11 13:38:23.767211 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:23.767156    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-5a0fa1-57454c47c4-vns7s"]
Sep 11 13:38:23.772561 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:23.772519    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5a0fa1-57454c47c4-vns7s"
Sep 11 13:38:23.775103 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:23.775068    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-5a0fa1-serving-cert\""
Sep 11 13:38:23.782277 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:23.782232    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-5a0fa1-57454c47c4-vns7s"]
Sep 11 13:38:23.881024 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:23.880980    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xsqd8\" (UniqueName: \"kubernetes.io/projected/2ae059be-ab6a-457a-82a2-7d59faa24517-kube-api-access-xsqd8\") pod \"maas-api-e2e-explicit-5a0fa1-57454c47c4-vns7s\" (UID: \"2ae059be-ab6a-457a-82a2-7d59faa24517\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5a0fa1-57454c47c4-vns7s"
Sep 11 13:38:23.881280 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:23.881041    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ae059be-ab6a-457a-82a2-7d59faa24517-openshift-service-ca\") pod \"maas-api-e2e-explicit-5a0fa1-57454c47c4-vns7s\" (UID: \"2ae059be-ab6a-457a-82a2-7d59faa24517\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5a0fa1-57454c47c4-vns7s"
Sep 11 13:38:23.881280 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:23.881185    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ae059be-ab6a-457a-82a2-7d59faa24517-metrics-certs\") pod \"maas-api-e2e-explicit-5a0fa1-57454c47c4-vns7s\" (UID: \"2ae059be-ab6a-457a-82a2-7d59faa24517\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5a0fa1-57454c47c4-vns7s"
Sep 11 13:38:23.881655 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:23.881448    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2ae059be-ab6a-457a-82a2-7d59faa24517-maas-api-tls\") pod \"maas-api-e2e-explicit-5a0fa1-57454c47c4-vns7s\" (UID: \"2ae059be-ab6a-457a-82a2-7d59faa24517\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5a0fa1-57454c47c4-vns7s"
Sep 11 13:38:23.982924 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:23.982825    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xsqd8\" (UniqueName: \"kubernetes.io/projected/2ae059be-ab6a-457a-82a2-7d59faa24517-kube-api-access-xsqd8\") pod \"maas-api-e2e-explicit-5a0fa1-57454c47c4-vns7s\" (UID: \"2ae059be-ab6a-457a-82a2-7d59faa24517\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5a0fa1-57454c47c4-vns7s"
Sep 11 13:38:23.982924 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:23.982922    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ae059be-ab6a-457a-82a2-7d59faa24517-openshift-service-ca\") pod \"maas-api-e2e-explicit-5a0fa1-57454c47c4-vns7s\" (UID: \"2ae059be-ab6a-457a-82a2-7d59faa24517\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5a0fa1-57454c47c4-vns7s"
Sep 11 13:38:23.983305 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:23.982978    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ae059be-ab6a-457a-82a2-7d59faa24517-metrics-certs\") pod \"maas-api-e2e-explicit-5a0fa1-57454c47c4-vns7s\" (UID: \"2ae059be-ab6a-457a-82a2-7d59faa24517\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5a0fa1-57454c47c4-vns7s"
Sep 11 13:38:23.983305 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:23.983119    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2ae059be-ab6a-457a-82a2-7d59faa24517-maas-api-tls\") pod \"maas-api-e2e-explicit-5a0fa1-57454c47c4-vns7s\" (UID: \"2ae059be-ab6a-457a-82a2-7d59faa24517\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5a0fa1-57454c47c4-vns7s"
Sep 11 13:38:23.983305 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:38:23.983217    2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-explicit-5a0fa1-serving-cert: secret "maas-api-e2e-explicit-5a0fa1-serving-cert" not found
Sep 11 13:38:23.983457 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:38:23.983322    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2ae059be-ab6a-457a-82a2-7d59faa24517-maas-api-tls podName:2ae059be-ab6a-457a-82a2-7d59faa24517 nodeName:}" failed. No retries permitted until 2026-09-11 13:38:24.483294245 +0000 UTC m=+1011.869397414 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2ae059be-ab6a-457a-82a2-7d59faa24517-maas-api-tls") pod "maas-api-e2e-explicit-5a0fa1-57454c47c4-vns7s" (UID: "2ae059be-ab6a-457a-82a2-7d59faa24517") : secret "maas-api-e2e-explicit-5a0fa1-serving-cert" not found
Sep 11 13:38:23.983848 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:23.983803    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ae059be-ab6a-457a-82a2-7d59faa24517-openshift-service-ca\") pod \"maas-api-e2e-explicit-5a0fa1-57454c47c4-vns7s\" (UID: \"2ae059be-ab6a-457a-82a2-7d59faa24517\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5a0fa1-57454c47c4-vns7s"
Sep 11 13:38:23.986397 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:23.986366    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ae059be-ab6a-457a-82a2-7d59faa24517-metrics-certs\") pod \"maas-api-e2e-explicit-5a0fa1-57454c47c4-vns7s\" (UID: \"2ae059be-ab6a-457a-82a2-7d59faa24517\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5a0fa1-57454c47c4-vns7s"
Sep 11 13:38:23.992502 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:23.992449    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xsqd8\" (UniqueName: \"kubernetes.io/projected/2ae059be-ab6a-457a-82a2-7d59faa24517-kube-api-access-xsqd8\") pod \"maas-api-e2e-explicit-5a0fa1-57454c47c4-vns7s\" (UID: \"2ae059be-ab6a-457a-82a2-7d59faa24517\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5a0fa1-57454c47c4-vns7s"
Sep 11 13:38:24.502759 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:24.502702    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2ae059be-ab6a-457a-82a2-7d59faa24517-maas-api-tls\") pod \"maas-api-e2e-explicit-5a0fa1-57454c47c4-vns7s\" (UID: \"2ae059be-ab6a-457a-82a2-7d59faa24517\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5a0fa1-57454c47c4-vns7s"
Sep 11 13:38:24.519389 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:24.519325    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2ae059be-ab6a-457a-82a2-7d59faa24517-maas-api-tls\") pod \"maas-api-e2e-explicit-5a0fa1-57454c47c4-vns7s\" (UID: \"2ae059be-ab6a-457a-82a2-7d59faa24517\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5a0fa1-57454c47c4-vns7s"
Sep 11 13:38:24.690329 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:24.690201    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5a0fa1-57454c47c4-vns7s"
Sep 11 13:38:25.057158 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:25.057125    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-5a0fa1-57454c47c4-vns7s"]
Sep 11 13:38:25.060937 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:38:25.060893    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ae059be_ab6a_457a_82a2_7d59faa24517.slice/crio-0c546e686187813d1b3f5057aa90367e82387514031ea2d4103381edbbf8f57c WatchSource:0}: Error finding container 0c546e686187813d1b3f5057aa90367e82387514031ea2d4103381edbbf8f57c: Status 404 returned error can't find the container with id 0c546e686187813d1b3f5057aa90367e82387514031ea2d4103381edbbf8f57c
Sep 11 13:38:25.362026 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:25.361917    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5a0fa1-57454c47c4-vns7s" event={"ID":"2ae059be-ab6a-457a-82a2-7d59faa24517","Type":"ContainerStarted","Data":"0c546e686187813d1b3f5057aa90367e82387514031ea2d4103381edbbf8f57c"}
Sep 11 13:38:26.370929 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:26.370873    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5a0fa1-57454c47c4-vns7s" event={"ID":"2ae059be-ab6a-457a-82a2-7d59faa24517","Type":"ContainerStarted","Data":"699a9c7a3ebdc19f5acb8c739b03d6fd39b379b592abf269986ca5cf0431d987"}
Sep 11 13:38:26.371433 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:26.370986    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5a0fa1-57454c47c4-vns7s"
Sep 11 13:38:26.393742 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:26.393641    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5a0fa1-57454c47c4-vns7s" podStartSLOduration=2.908244068 podStartE2EDuration="3.393608031s" totalImagesPullingTime="485.363963ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:38:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:38:26.388788772 +0000 UTC m=+1013.774891960" watchObservedRunningTime="2026-09-11 13:38:26.393608031 +0000 UTC m=+1013.779711196"
Sep 11 13:38:32.385003 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:32.384964    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5a0fa1-57454c47c4-vns7s"
Sep 11 13:38:34.456455 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:34.456408    2578 generic.go:356] "Generic (PLEG): container finished" podID="7c5140cf-195d-4604-9325-40d5b485e127" containerID="d7220556846a6ce5cc08dbef2d891c1765dbe63bc17ac394a813e1e3b324ddc7" exitCode=0
Sep 11 13:38:34.457003 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:34.456489    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-2a5fb372-openshift-default-5bfb49554d-7klgx" event={"ID":"7c5140cf-195d-4604-9325-40d5b485e127","Type":"ContainerDied","Data":"d7220556846a6ce5cc08dbef2d891c1765dbe63bc17ac394a813e1e3b324ddc7"}
Sep 11 13:38:34.457206 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:34.457186    2578 scope.go:117] "RemoveContainer" containerID="d7220556846a6ce5cc08dbef2d891c1765dbe63bc17ac394a813e1e3b324ddc7"
Sep 11 13:38:34.458614 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:34.458581    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 11 13:38:34.458753 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:34.458666    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 11 13:38:34.458753 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:34.458704    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 11 13:38:35.466726 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:35.466677    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-2a5fb372-openshift-default-5bfb49554d-7klgx" event={"ID":"7c5140cf-195d-4604-9325-40d5b485e127","Type":"ContainerStarted","Data":"826221b6e0cdcf15ed8f50d9354caac2dacb7d01fb6e0f561c271e6b0b608ef4"}
Sep 11 13:38:36.320165 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:36.320103    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-2a5fb372-openshift-default-5bfb49554d-7klgx"
Sep 11 13:38:36.321607 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:36.321569    2578 patch_prober.go:28] interesting pod/e2e-derive-2a5fb372-openshift-default-5bfb49554d-7klgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Sep 11 13:38:36.321802 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:36.321641    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-2a5fb372-openshift-default-5bfb49554d-7klgx" podUID="7c5140cf-195d-4604-9325-40d5b485e127" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Sep 11 13:38:37.218136 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:37.218084    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-2a5fb372-openshift-default-5bfb49554d-7klgx"
Sep 11 13:38:37.321053 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:37.320915    2578 patch_prober.go:28] interesting pod/e2e-derive-2a5fb372-openshift-default-5bfb49554d-7klgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Sep 11 13:38:37.321288 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:37.321170    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-2a5fb372-openshift-default-5bfb49554d-7klgx" podUID="7c5140cf-195d-4604-9325-40d5b485e127" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Sep 11 13:38:38.320892 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:38.320844    2578 patch_prober.go:28] interesting pod/e2e-derive-2a5fb372-openshift-default-5bfb49554d-7klgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Sep 11 13:38:38.321370 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:38.320922    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-2a5fb372-openshift-default-5bfb49554d-7klgx" podUID="7c5140cf-195d-4604-9325-40d5b485e127" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Sep 11 13:38:39.320518 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:39.320468    2578 patch_prober.go:28] interesting pod/e2e-derive-2a5fb372-openshift-default-5bfb49554d-7klgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Sep 11 13:38:39.320793 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:39.320562    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-2a5fb372-openshift-default-5bfb49554d-7klgx" podUID="7c5140cf-195d-4604-9325-40d5b485e127" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Sep 11 13:38:40.320997 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:40.320940    2578 patch_prober.go:28] interesting pod/e2e-derive-2a5fb372-openshift-default-5bfb49554d-7klgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Sep 11 13:38:40.321615 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:40.321029    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-2a5fb372-openshift-default-5bfb49554d-7klgx" podUID="7c5140cf-195d-4604-9325-40d5b485e127" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Sep 11 13:38:41.320685 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:41.320618    2578 patch_prober.go:28] interesting pod/e2e-derive-2a5fb372-openshift-default-5bfb49554d-7klgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Sep 11 13:38:41.321074 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:41.320736    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-2a5fb372-openshift-default-5bfb49554d-7klgx" podUID="7c5140cf-195d-4604-9325-40d5b485e127" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Sep 11 13:38:42.324606 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:42.324571    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-2a5fb372-openshift-default-5bfb49554d-7klgx"
Sep 11 13:38:42.325759 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:42.325736    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-2a5fb372-openshift-default-5bfb49554d-7klgx"
Sep 11 13:38:49.046951 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:49.046815    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-2a5fb372-74b5dcf5b7-cdt49"]
Sep 11 13:38:49.048072 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:49.048031    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2a5fb372-74b5dcf5b7-cdt49" podUID="8d5cc1e7-c0b9-4438-837d-beeeeb496273" containerName="maas-api" containerID="cri-o://6c6ad11e0a413f80ab450c3a6050b4aff4ecf607cec45b6482e28925c6f561fa" gracePeriod=30
Sep 11 13:38:49.355566 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:49.355535    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2a5fb372-74b5dcf5b7-cdt49"
Sep 11 13:38:49.445476 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:49.445423    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d5cc1e7-c0b9-4438-837d-beeeeb496273-openshift-service-ca\") pod \"8d5cc1e7-c0b9-4438-837d-beeeeb496273\" (UID: \"8d5cc1e7-c0b9-4438-837d-beeeeb496273\") "
Sep 11 13:38:49.445476 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:49.445485    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d5cc1e7-c0b9-4438-837d-beeeeb496273-metrics-certs\") pod \"8d5cc1e7-c0b9-4438-837d-beeeeb496273\" (UID: \"8d5cc1e7-c0b9-4438-837d-beeeeb496273\") "
Sep 11 13:38:49.445963 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:49.445549    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4jq7l\" (UniqueName: \"kubernetes.io/projected/8d5cc1e7-c0b9-4438-837d-beeeeb496273-kube-api-access-4jq7l\") pod \"8d5cc1e7-c0b9-4438-837d-beeeeb496273\" (UID: \"8d5cc1e7-c0b9-4438-837d-beeeeb496273\") "
Sep 11 13:38:49.445963 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:49.445597    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d5cc1e7-c0b9-4438-837d-beeeeb496273-maas-api-tls\") pod \"8d5cc1e7-c0b9-4438-837d-beeeeb496273\" (UID: \"8d5cc1e7-c0b9-4438-837d-beeeeb496273\") "
Sep 11 13:38:49.446078 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:49.446031    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d5cc1e7-c0b9-4438-837d-beeeeb496273-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8d5cc1e7-c0b9-4438-837d-beeeeb496273" (UID: "8d5cc1e7-c0b9-4438-837d-beeeeb496273"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 13:38:49.448560 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:49.448493    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d5cc1e7-c0b9-4438-837d-beeeeb496273-kube-api-access-4jq7l" (OuterVolumeSpecName: "kube-api-access-4jq7l") pod "8d5cc1e7-c0b9-4438-837d-beeeeb496273" (UID: "8d5cc1e7-c0b9-4438-837d-beeeeb496273"). InnerVolumeSpecName "kube-api-access-4jq7l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 13:38:49.448765 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:49.448588    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d5cc1e7-c0b9-4438-837d-beeeeb496273-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "8d5cc1e7-c0b9-4438-837d-beeeeb496273" (UID: "8d5cc1e7-c0b9-4438-837d-beeeeb496273"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 13:38:49.448765 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:49.448614    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d5cc1e7-c0b9-4438-837d-beeeeb496273-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8d5cc1e7-c0b9-4438-837d-beeeeb496273" (UID: "8d5cc1e7-c0b9-4438-837d-beeeeb496273"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 13:38:49.487139 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:49.486152    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-2a5fb372-openshift-default-5bfb49554d-7klgx"]
Sep 11 13:38:49.487139 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:49.486589    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-2a5fb372-openshift-default-5bfb49554d-7klgx" podUID="7c5140cf-195d-4604-9325-40d5b485e127" containerName="istio-proxy" containerID="cri-o://826221b6e0cdcf15ed8f50d9354caac2dacb7d01fb6e0f561c271e6b0b608ef4" gracePeriod=30
Sep 11 13:38:49.547858 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:49.547798    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d5cc1e7-c0b9-4438-837d-beeeeb496273-openshift-service-ca\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:38:49.548108 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:49.547896    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d5cc1e7-c0b9-4438-837d-beeeeb496273-metrics-certs\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:38:49.548108 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:49.547915    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jq7l\" (UniqueName: \"kubernetes.io/projected/8d5cc1e7-c0b9-4438-837d-beeeeb496273-kube-api-access-4jq7l\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:38:49.548108 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:49.547931    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d5cc1e7-c0b9-4438-837d-beeeeb496273-maas-api-tls\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:38:49.582095 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:49.581949    2578 generic.go:356] "Generic (PLEG): container finished" podID="8d5cc1e7-c0b9-4438-837d-beeeeb496273" containerID="6c6ad11e0a413f80ab450c3a6050b4aff4ecf607cec45b6482e28925c6f561fa" exitCode=0
Sep 11 13:38:49.582095 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:49.582068    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2a5fb372-74b5dcf5b7-cdt49"
Sep 11 13:38:49.582095 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:49.582079    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2a5fb372-74b5dcf5b7-cdt49" event={"ID":"8d5cc1e7-c0b9-4438-837d-beeeeb496273","Type":"ContainerDied","Data":"6c6ad11e0a413f80ab450c3a6050b4aff4ecf607cec45b6482e28925c6f561fa"}
Sep 11 13:38:49.582505 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:49.582144    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2a5fb372-74b5dcf5b7-cdt49" event={"ID":"8d5cc1e7-c0b9-4438-837d-beeeeb496273","Type":"ContainerDied","Data":"d2b730cd9243e612ccbee91bacaadc77071e5a4e561a3354ee62202e17c18f50"}
Sep 11 13:38:49.582505 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:49.582180    2578 scope.go:117] "RemoveContainer" containerID="6c6ad11e0a413f80ab450c3a6050b4aff4ecf607cec45b6482e28925c6f561fa"
Sep 11 13:38:49.615424 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:49.615376    2578 scope.go:117] "RemoveContainer" containerID="6c6ad11e0a413f80ab450c3a6050b4aff4ecf607cec45b6482e28925c6f561fa"
Sep 11 13:38:49.616178 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:38:49.616148    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c6ad11e0a413f80ab450c3a6050b4aff4ecf607cec45b6482e28925c6f561fa\": container with ID starting with 6c6ad11e0a413f80ab450c3a6050b4aff4ecf607cec45b6482e28925c6f561fa not found: ID does not exist" containerID="6c6ad11e0a413f80ab450c3a6050b4aff4ecf607cec45b6482e28925c6f561fa"
Sep 11 13:38:49.616382 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:49.616354    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c6ad11e0a413f80ab450c3a6050b4aff4ecf607cec45b6482e28925c6f561fa"} err="failed to get container status \"6c6ad11e0a413f80ab450c3a6050b4aff4ecf607cec45b6482e28925c6f561fa\": rpc error: code = NotFound desc = could not find container \"6c6ad11e0a413f80ab450c3a6050b4aff4ecf607cec45b6482e28925c6f561fa\": container with ID starting with 6c6ad11e0a413f80ab450c3a6050b4aff4ecf607cec45b6482e28925c6f561fa not found: ID does not exist"
Sep 11 13:38:49.623485 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:49.622958    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-2a5fb372-74b5dcf5b7-cdt49"]
Sep 11 13:38:49.627494 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:49.627435    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-2a5fb372-74b5dcf5b7-cdt49"]
Sep 11 13:38:51.108678 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:51.108636    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8d5cc1e7-c0b9-4438-837d-beeeeb496273" path="/var/lib/kubelet/pods/8d5cc1e7-c0b9-4438-837d-beeeeb496273/volumes"
Sep 11 13:38:54.625441 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:54.625388    2578 generic.go:356] "Generic (PLEG): container finished" podID="7c5140cf-195d-4604-9325-40d5b485e127" containerID="826221b6e0cdcf15ed8f50d9354caac2dacb7d01fb6e0f561c271e6b0b608ef4" exitCode=0
Sep 11 13:38:54.626039 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:54.625456    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-2a5fb372-openshift-default-5bfb49554d-7klgx" event={"ID":"7c5140cf-195d-4604-9325-40d5b485e127","Type":"ContainerDied","Data":"826221b6e0cdcf15ed8f50d9354caac2dacb7d01fb6e0f561c271e6b0b608ef4"}
Sep 11 13:38:54.626039 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:54.625550    2578 scope.go:117] "RemoveContainer" containerID="d7220556846a6ce5cc08dbef2d891c1765dbe63bc17ac394a813e1e3b324ddc7"
Sep 11 13:38:54.778952 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:54.778523    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-2a5fb372-openshift-default-5bfb49554d-7klgx"
Sep 11 13:38:54.812322 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:54.812233    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c5140cf-195d-4604-9325-40d5b485e127-istio-envoy\") pod \"7c5140cf-195d-4604-9325-40d5b485e127\" (UID: \"7c5140cf-195d-4604-9325-40d5b485e127\") "
Sep 11 13:38:54.812508 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:54.812335    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c5140cf-195d-4604-9325-40d5b485e127-istio-podinfo\") pod \"7c5140cf-195d-4604-9325-40d5b485e127\" (UID: \"7c5140cf-195d-4604-9325-40d5b485e127\") "
Sep 11 13:38:54.812508 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:54.812379    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7l8bp\" (UniqueName: \"kubernetes.io/projected/7c5140cf-195d-4604-9325-40d5b485e127-kube-api-access-7l8bp\") pod \"7c5140cf-195d-4604-9325-40d5b485e127\" (UID: \"7c5140cf-195d-4604-9325-40d5b485e127\") "
Sep 11 13:38:54.812508 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:54.812437    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c5140cf-195d-4604-9325-40d5b485e127-istiod-ca-cert\") pod \"7c5140cf-195d-4604-9325-40d5b485e127\" (UID: \"7c5140cf-195d-4604-9325-40d5b485e127\") "
Sep 11 13:38:54.812508 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:54.812457    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c5140cf-195d-4604-9325-40d5b485e127-credential-socket\") pod \"7c5140cf-195d-4604-9325-40d5b485e127\" (UID: \"7c5140cf-195d-4604-9325-40d5b485e127\") "
Sep 11 13:38:54.812701 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:54.812512    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c5140cf-195d-4604-9325-40d5b485e127-workload-certs\") pod \"7c5140cf-195d-4604-9325-40d5b485e127\" (UID: \"7c5140cf-195d-4604-9325-40d5b485e127\") "
Sep 11 13:38:54.812701 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:54.812534    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c5140cf-195d-4604-9325-40d5b485e127-istio-token\") pod \"7c5140cf-195d-4604-9325-40d5b485e127\" (UID: \"7c5140cf-195d-4604-9325-40d5b485e127\") "
Sep 11 13:38:54.812701 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:54.812598    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c5140cf-195d-4604-9325-40d5b485e127-workload-socket\") pod \"7c5140cf-195d-4604-9325-40d5b485e127\" (UID: \"7c5140cf-195d-4604-9325-40d5b485e127\") "
Sep 11 13:38:54.812701 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:54.812670    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c5140cf-195d-4604-9325-40d5b485e127-istio-data\") pod \"7c5140cf-195d-4604-9325-40d5b485e127\" (UID: \"7c5140cf-195d-4604-9325-40d5b485e127\") "
Sep 11 13:38:54.812920 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:54.812792    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c5140cf-195d-4604-9325-40d5b485e127-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7c5140cf-195d-4604-9325-40d5b485e127" (UID: "7c5140cf-195d-4604-9325-40d5b485e127"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 13:38:54.813024 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:54.813001    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c5140cf-195d-4604-9325-40d5b485e127-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7c5140cf-195d-4604-9325-40d5b485e127" (UID: "7c5140cf-195d-4604-9325-40d5b485e127"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 13:38:54.813081 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:54.813014    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c5140cf-195d-4604-9325-40d5b485e127-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7c5140cf-195d-4604-9325-40d5b485e127" (UID: "7c5140cf-195d-4604-9325-40d5b485e127"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 13:38:54.813081 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:54.813060    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c5140cf-195d-4604-9325-40d5b485e127-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7c5140cf-195d-4604-9325-40d5b485e127" (UID: "7c5140cf-195d-4604-9325-40d5b485e127"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 13:38:54.813187 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:54.813167    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c5140cf-195d-4604-9325-40d5b485e127-istio-data" (OuterVolumeSpecName: "istio-data") pod "7c5140cf-195d-4604-9325-40d5b485e127" (UID: "7c5140cf-195d-4604-9325-40d5b485e127"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 13:38:54.813648 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:54.813587    2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c5140cf-195d-4604-9325-40d5b485e127-istiod-ca-cert\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:38:54.813779 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:54.813741    2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c5140cf-195d-4604-9325-40d5b485e127-credential-socket\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:38:54.813779 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:54.813758    2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c5140cf-195d-4604-9325-40d5b485e127-workload-certs\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:38:54.813779 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:54.813773    2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c5140cf-195d-4604-9325-40d5b485e127-workload-socket\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:38:54.814015 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:54.813785    2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c5140cf-195d-4604-9325-40d5b485e127-istio-data\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:38:54.815773 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:54.815723    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c5140cf-195d-4604-9325-40d5b485e127-kube-api-access-7l8bp" (OuterVolumeSpecName: "kube-api-access-7l8bp") pod "7c5140cf-195d-4604-9325-40d5b485e127" (UID: "7c5140cf-195d-4604-9325-40d5b485e127"). InnerVolumeSpecName "kube-api-access-7l8bp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 13:38:54.816073 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:54.816044    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c5140cf-195d-4604-9325-40d5b485e127-istio-token" (OuterVolumeSpecName: "istio-token") pod "7c5140cf-195d-4604-9325-40d5b485e127" (UID: "7c5140cf-195d-4604-9325-40d5b485e127"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 13:38:54.816386 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:54.816360    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7c5140cf-195d-4604-9325-40d5b485e127-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7c5140cf-195d-4604-9325-40d5b485e127" (UID: "7c5140cf-195d-4604-9325-40d5b485e127"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 13:38:54.816900 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:54.816874    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c5140cf-195d-4604-9325-40d5b485e127-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7c5140cf-195d-4604-9325-40d5b485e127" (UID: "7c5140cf-195d-4604-9325-40d5b485e127"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 13:38:54.926009 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:54.925967    2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c5140cf-195d-4604-9325-40d5b485e127-istio-envoy\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:38:54.926009 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:54.926003    2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c5140cf-195d-4604-9325-40d5b485e127-istio-podinfo\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:38:54.926009 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:54.926013    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7l8bp\" (UniqueName: \"kubernetes.io/projected/7c5140cf-195d-4604-9325-40d5b485e127-kube-api-access-7l8bp\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:38:54.926009 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:54.926023    2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c5140cf-195d-4604-9325-40d5b485e127-istio-token\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:38:55.636459 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:55.636420    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-2a5fb372-openshift-default-5bfb49554d-7klgx"
Sep 11 13:38:55.636459 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:55.636437    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-2a5fb372-openshift-default-5bfb49554d-7klgx" event={"ID":"7c5140cf-195d-4604-9325-40d5b485e127","Type":"ContainerDied","Data":"3acdf3bcfb4749b27b300f329db035a04ef11c94aaecc922311295fd79d0e417"}
Sep 11 13:38:55.637218 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:55.636538    2578 scope.go:117] "RemoveContainer" containerID="826221b6e0cdcf15ed8f50d9354caac2dacb7d01fb6e0f561c271e6b0b608ef4"
Sep 11 13:38:55.657305 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:55.657266    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-2a5fb372-openshift-default-5bfb49554d-7klgx"]
Sep 11 13:38:55.675229 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:55.675180    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-2a5fb372-openshift-default-5bfb49554d-7klgx"]
Sep 11 13:38:57.107849 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:38:57.107786    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7c5140cf-195d-4604-9325-40d5b485e127" path="/var/lib/kubelet/pods/7c5140cf-195d-4604-9325-40d5b485e127/volumes"
Sep 11 13:39:25.661006 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:39:25.660898    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-371c38/premium-w4-371c38-kserve-5d8c655487-qw6kh"]
Sep 11 13:39:25.661579 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:39:25.661561    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c5140cf-195d-4604-9325-40d5b485e127" containerName="istio-proxy"
Sep 11 13:39:25.661628 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:39:25.661581    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c5140cf-195d-4604-9325-40d5b485e127" containerName="istio-proxy"
Sep 11 13:39:25.661628 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:39:25.661615    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d5cc1e7-c0b9-4438-837d-beeeeb496273" containerName="maas-api"
Sep 11 13:39:25.661628 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:39:25.661621    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d5cc1e7-c0b9-4438-837d-beeeeb496273" containerName="maas-api"
Sep 11 13:39:25.661732 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:39:25.661715    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d5cc1e7-c0b9-4438-837d-beeeeb496273" containerName="maas-api"
Sep 11 13:39:25.661732 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:39:25.661725    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c5140cf-195d-4604-9325-40d5b485e127" containerName="istio-proxy"
Sep 11 13:39:25.664440 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:39:25.664417    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-371c38/premium-w4-371c38-kserve-5d8c655487-qw6kh"
Sep 11 13:39:25.667151 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:39:25.667114    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-371c38\"/\"kube-root-ca.crt\""
Sep 11 13:39:25.667656 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:39:25.667633    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-371c38\"/\"openshift-service-ca.crt\""
Sep 11 13:39:25.667801 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:39:25.667686    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-371c38\"/\"premium-w4-371c38-kserve-self-signed-certs\""
Sep 11 13:39:25.667801 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:39:25.667716    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-371c38\"/\"default-dockercfg-xvr7g\""
Sep 11 13:39:25.677679 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:39:25.677643    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-371c38/premium-w4-371c38-kserve-5d8c655487-qw6kh"]
Sep 11 13:39:25.800455 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:39:25.800394    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/848ec50c-75af-4c07-be1a-cb8bf34c0c08-tmp-dir\") pod \"premium-w4-371c38-kserve-5d8c655487-qw6kh\" (UID: \"848ec50c-75af-4c07-be1a-cb8bf34c0c08\") " pod="e2e-models-e2e-worker-w4-371c38/premium-w4-371c38-kserve-5d8c655487-qw6kh"
Sep 11 13:39:25.800711 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:39:25.800476    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/848ec50c-75af-4c07-be1a-cb8bf34c0c08-dshm\") pod \"premium-w4-371c38-kserve-5d8c655487-qw6kh\" (UID: \"848ec50c-75af-4c07-be1a-cb8bf34c0c08\") " pod="e2e-models-e2e-worker-w4-371c38/premium-w4-371c38-kserve-5d8c655487-qw6kh"
Sep 11 13:39:25.800711 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:39:25.800539    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/848ec50c-75af-4c07-be1a-cb8bf34c0c08-home\") pod \"premium-w4-371c38-kserve-5d8c655487-qw6kh\" (UID: \"848ec50c-75af-4c07-be1a-cb8bf34c0c08\") " pod="e2e-models-e2e-worker-w4-371c38/premium-w4-371c38-kserve-5d8c655487-qw6kh"
Sep 11 13:39:25.800711 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:39:25.800613    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/848ec50c-75af-4c07-be1a-cb8bf34c0c08-model-cache\") pod \"premium-w4-371c38-kserve-5d8c655487-qw6kh\" (UID: \"848ec50c-75af-4c07-be1a-cb8bf34c0c08\") " pod="e2e-models-e2e-worker-w4-371c38/premium-w4-371c38-kserve-5d8c655487-qw6kh"
Sep 11 13:39:25.800711 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:39:25.800647    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66ccf\" (UniqueName: \"kubernetes.io/projected/848ec50c-75af-4c07-be1a-cb8bf34c0c08-kube-api-access-66ccf\") pod \"premium-w4-371c38-kserve-5d8c655487-qw6kh\" (UID: \"848ec50c-75af-4c07-be1a-cb8bf34c0c08\") " pod="e2e-models-e2e-worker-w4-371c38/premium-w4-371c38-kserve-5d8c655487-qw6kh"
Sep 11 13:39:25.800711 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:39:25.800707    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/848ec50c-75af-4c07-be1a-cb8bf34c0c08-tls-certs\") pod \"premium-w4-371c38-kserve-5d8c655487-qw6kh\" (UID: \"848ec50c-75af-4c07-be1a-cb8bf34c0c08\") " pod="e2e-models-e2e-worker-w4-371c38/premium-w4-371c38-kserve-5d8c655487-qw6kh"
Sep 11 13:39:25.902410 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:39:25.902356    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/848ec50c-75af-4c07-be1a-cb8bf34c0c08-dshm\") pod \"premium-w4-371c38-kserve-5d8c655487-qw6kh\" (UID: \"848ec50c-75af-4c07-be1a-cb8bf34c0c08\") " pod="e2e-models-e2e-worker-w4-371c38/premium-w4-371c38-kserve-5d8c655487-qw6kh"
Sep 11 13:39:25.902668 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:39:25.902480    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/848ec50c-75af-4c07-be1a-cb8bf34c0c08-home\") pod \"premium-w4-371c38-kserve-5d8c655487-qw6kh\" (UID: \"848ec50c-75af-4c07-be1a-cb8bf34c0c08\") " pod="e2e-models-e2e-worker-w4-371c38/premium-w4-371c38-kserve-5d8c655487-qw6kh"
Sep 11 13:39:25.902668 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:39:25.902521    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/848ec50c-75af-4c07-be1a-cb8bf34c0c08-model-cache\") pod \"premium-w4-371c38-kserve-5d8c655487-qw6kh\" (UID: \"848ec50c-75af-4c07-be1a-cb8bf34c0c08\") " pod="e2e-models-e2e-worker-w4-371c38/premium-w4-371c38-kserve-5d8c655487-qw6kh"
Sep 11 13:39:25.902668 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:39:25.902540    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66ccf\" (UniqueName: \"kubernetes.io/projected/848ec50c-75af-4c07-be1a-cb8bf34c0c08-kube-api-access-66ccf\") pod \"premium-w4-371c38-kserve-5d8c655487-qw6kh\" (UID: \"848ec50c-75af-4c07-be1a-cb8bf34c0c08\") " pod="e2e-models-e2e-worker-w4-371c38/premium-w4-371c38-kserve-5d8c655487-qw6kh"
Sep 11 13:39:25.902668 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:39:25.902594    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/848ec50c-75af-4c07-be1a-cb8bf34c0c08-tls-certs\") pod \"premium-w4-371c38-kserve-5d8c655487-qw6kh\" (UID: \"848ec50c-75af-4c07-be1a-cb8bf34c0c08\") " pod="e2e-models-e2e-worker-w4-371c38/premium-w4-371c38-kserve-5d8c655487-qw6kh"
Sep 11 13:39:25.902945 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:39:25.902677    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/848ec50c-75af-4c07-be1a-cb8bf34c0c08-tmp-dir\") pod \"premium-w4-371c38-kserve-5d8c655487-qw6kh\" (UID: \"848ec50c-75af-4c07-be1a-cb8bf34c0c08\") " pod="e2e-models-e2e-worker-w4-371c38/premium-w4-371c38-kserve-5d8c655487-qw6kh"
Sep 11 13:39:25.903084 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:39:25.903055    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/848ec50c-75af-4c07-be1a-cb8bf34c0c08-home\") pod \"premium-w4-371c38-kserve-5d8c655487-qw6kh\" (UID: \"848ec50c-75af-4c07-be1a-cb8bf34c0c08\") " pod="e2e-models-e2e-worker-w4-371c38/premium-w4-371c38-kserve-5d8c655487-qw6kh"
Sep 11 13:39:25.903161 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:39:25.903095    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/848ec50c-75af-4c07-be1a-cb8bf34c0c08-model-cache\") pod \"premium-w4-371c38-kserve-5d8c655487-qw6kh\" (UID: \"848ec50c-75af-4c07-be1a-cb8bf34c0c08\") " pod="e2e-models-e2e-worker-w4-371c38/premium-w4-371c38-kserve-5d8c655487-qw6kh"
Sep 11 13:39:25.903255 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:39:25.903236    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/848ec50c-75af-4c07-be1a-cb8bf34c0c08-tmp-dir\") pod \"premium-w4-371c38-kserve-5d8c655487-qw6kh\" (UID: \"848ec50c-75af-4c07-be1a-cb8bf34c0c08\") " pod="e2e-models-e2e-worker-w4-371c38/premium-w4-371c38-kserve-5d8c655487-qw6kh"
Sep 11 13:39:25.905468 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:39:25.905434    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/848ec50c-75af-4c07-be1a-cb8bf34c0c08-dshm\") pod \"premium-w4-371c38-kserve-5d8c655487-qw6kh\" (UID: \"848ec50c-75af-4c07-be1a-cb8bf34c0c08\") " pod="e2e-models-e2e-worker-w4-371c38/premium-w4-371c38-kserve-5d8c655487-qw6kh"
Sep 11 13:39:25.905591 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:39:25.905543    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/848ec50c-75af-4c07-be1a-cb8bf34c0c08-tls-certs\") pod \"premium-w4-371c38-kserve-5d8c655487-qw6kh\" (UID: \"848ec50c-75af-4c07-be1a-cb8bf34c0c08\") " pod="e2e-models-e2e-worker-w4-371c38/premium-w4-371c38-kserve-5d8c655487-qw6kh"
Sep 11 13:39:25.912167 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:39:25.912086    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-66ccf\" (UniqueName: \"kubernetes.io/projected/848ec50c-75af-4c07-be1a-cb8bf34c0c08-kube-api-access-66ccf\") pod \"premium-w4-371c38-kserve-5d8c655487-qw6kh\" (UID: \"848ec50c-75af-4c07-be1a-cb8bf34c0c08\") " pod="e2e-models-e2e-worker-w4-371c38/premium-w4-371c38-kserve-5d8c655487-qw6kh"
Sep 11 13:39:25.977749 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:39:25.977699    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-371c38/premium-w4-371c38-kserve-5d8c655487-qw6kh"
Sep 11 13:39:26.270257 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:39:26.270223    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-371c38/premium-w4-371c38-kserve-5d8c655487-qw6kh"]
Sep 11 13:39:26.273215 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:39:26.273179    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod848ec50c_75af_4c07_be1a_cb8bf34c0c08.slice/crio-81622ba7e088298ca00efcf36e0330a69ea9408a299bd08b2116767b2d318d6c WatchSource:0}: Error finding container 81622ba7e088298ca00efcf36e0330a69ea9408a299bd08b2116767b2d318d6c: Status 404 returned error can't find the container with id 81622ba7e088298ca00efcf36e0330a69ea9408a299bd08b2116767b2d318d6c
Sep 11 13:39:26.858688 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:39:26.858625    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-371c38/premium-w4-371c38-kserve-5d8c655487-qw6kh" event={"ID":"848ec50c-75af-4c07-be1a-cb8bf34c0c08","Type":"ContainerStarted","Data":"9f44f8e81aebc5e94280961db301fab0ec54644de1ac42c545ebe58eac075f13"}
Sep 11 13:39:26.859201 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:39:26.858702    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-371c38/premium-w4-371c38-kserve-5d8c655487-qw6kh" event={"ID":"848ec50c-75af-4c07-be1a-cb8bf34c0c08","Type":"ContainerStarted","Data":"81622ba7e088298ca00efcf36e0330a69ea9408a299bd08b2116767b2d318d6c"}
Sep 11 13:39:26.882652 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:39:26.882590    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-371c38/premium-w4-371c38-kserve-5d8c655487-qw6kh" podStartSLOduration=1.88256992 podStartE2EDuration="1.88256992s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:39:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:39:26.878503177 +0000 UTC m=+1074.264606343" watchObservedRunningTime="2026-09-11 13:39:26.88256992 +0000 UTC m=+1074.268673143"
Sep 11 13:39:35.978646 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:39:35.978590    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-371c38/premium-w4-371c38-kserve-5d8c655487-qw6kh"
Sep 11 13:39:35.978646 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:39:35.978658    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-371c38/premium-w4-371c38-kserve-5d8c655487-qw6kh"
Sep 11 13:39:36.002373 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:39:36.002338    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-371c38/premium-w4-371c38-kserve-5d8c655487-qw6kh"
Sep 11 13:39:36.941469 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:39:36.941423    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-371c38/premium-w4-371c38-kserve-5d8c655487-qw6kh"
Sep 11 13:40:43.330946 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:40:43.330893    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-1a269e4e-675c8d4ffb-nvj7h"]
Sep 11 13:40:43.331480 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:40:43.331467    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c5140cf-195d-4604-9325-40d5b485e127" containerName="istio-proxy"
Sep 11 13:40:43.331480 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:40:43.331480    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c5140cf-195d-4604-9325-40d5b485e127" containerName="istio-proxy"
Sep 11 13:40:43.331599 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:40:43.331586    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c5140cf-195d-4604-9325-40d5b485e127" containerName="istio-proxy"
Sep 11 13:40:43.334175 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:40:43.334153    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1a269e4e-675c8d4ffb-nvj7h"
Sep 11 13:40:43.336627 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:40:43.336599    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-1a269e4e-serving-cert\""
Sep 11 13:40:43.347274 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:40:43.347226    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-1a269e4e-675c8d4ffb-nvj7h"]
Sep 11 13:40:43.480570 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:40:43.480520    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac20b42b-e82d-4276-b63f-9a8b902baa88-openshift-service-ca\") pod \"maas-api-e2e-mt-1a269e4e-675c8d4ffb-nvj7h\" (UID: \"ac20b42b-e82d-4276-b63f-9a8b902baa88\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1a269e4e-675c8d4ffb-nvj7h"
Sep 11 13:40:43.480570 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:40:43.480570    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac20b42b-e82d-4276-b63f-9a8b902baa88-metrics-certs\") pod \"maas-api-e2e-mt-1a269e4e-675c8d4ffb-nvj7h\" (UID: \"ac20b42b-e82d-4276-b63f-9a8b902baa88\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1a269e4e-675c8d4ffb-nvj7h"
Sep 11 13:40:43.480967 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:40:43.480689    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kv9w\" (UniqueName: \"kubernetes.io/projected/ac20b42b-e82d-4276-b63f-9a8b902baa88-kube-api-access-7kv9w\") pod \"maas-api-e2e-mt-1a269e4e-675c8d4ffb-nvj7h\" (UID: \"ac20b42b-e82d-4276-b63f-9a8b902baa88\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1a269e4e-675c8d4ffb-nvj7h"
Sep 11 13:40:43.480967 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:40:43.480745    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac20b42b-e82d-4276-b63f-9a8b902baa88-maas-api-tls\") pod \"maas-api-e2e-mt-1a269e4e-675c8d4ffb-nvj7h\" (UID: \"ac20b42b-e82d-4276-b63f-9a8b902baa88\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1a269e4e-675c8d4ffb-nvj7h"
Sep 11 13:40:43.583476 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:40:43.583319    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kv9w\" (UniqueName: \"kubernetes.io/projected/ac20b42b-e82d-4276-b63f-9a8b902baa88-kube-api-access-7kv9w\") pod \"maas-api-e2e-mt-1a269e4e-675c8d4ffb-nvj7h\" (UID: \"ac20b42b-e82d-4276-b63f-9a8b902baa88\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1a269e4e-675c8d4ffb-nvj7h"
Sep 11 13:40:43.584065 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:40:43.584011    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac20b42b-e82d-4276-b63f-9a8b902baa88-maas-api-tls\") pod \"maas-api-e2e-mt-1a269e4e-675c8d4ffb-nvj7h\" (UID: \"ac20b42b-e82d-4276-b63f-9a8b902baa88\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1a269e4e-675c8d4ffb-nvj7h"
Sep 11 13:40:43.584384 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:40:43.584353    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac20b42b-e82d-4276-b63f-9a8b902baa88-openshift-service-ca\") pod \"maas-api-e2e-mt-1a269e4e-675c8d4ffb-nvj7h\" (UID: \"ac20b42b-e82d-4276-b63f-9a8b902baa88\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1a269e4e-675c8d4ffb-nvj7h"
Sep 11 13:40:43.584555 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:40:43.584397    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac20b42b-e82d-4276-b63f-9a8b902baa88-metrics-certs\") pod \"maas-api-e2e-mt-1a269e4e-675c8d4ffb-nvj7h\" (UID: \"ac20b42b-e82d-4276-b63f-9a8b902baa88\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1a269e4e-675c8d4ffb-nvj7h"
Sep 11 13:40:43.585456 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:40:43.585414    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac20b42b-e82d-4276-b63f-9a8b902baa88-openshift-service-ca\") pod \"maas-api-e2e-mt-1a269e4e-675c8d4ffb-nvj7h\" (UID: \"ac20b42b-e82d-4276-b63f-9a8b902baa88\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1a269e4e-675c8d4ffb-nvj7h"
Sep 11 13:40:43.592881 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:40:43.588871    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac20b42b-e82d-4276-b63f-9a8b902baa88-metrics-certs\") pod \"maas-api-e2e-mt-1a269e4e-675c8d4ffb-nvj7h\" (UID: \"ac20b42b-e82d-4276-b63f-9a8b902baa88\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1a269e4e-675c8d4ffb-nvj7h"
Sep 11 13:40:43.592881 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:40:43.589401    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac20b42b-e82d-4276-b63f-9a8b902baa88-maas-api-tls\") pod \"maas-api-e2e-mt-1a269e4e-675c8d4ffb-nvj7h\" (UID: \"ac20b42b-e82d-4276-b63f-9a8b902baa88\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1a269e4e-675c8d4ffb-nvj7h"
Sep 11 13:40:43.594306 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:40:43.594272    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kv9w\" (UniqueName: \"kubernetes.io/projected/ac20b42b-e82d-4276-b63f-9a8b902baa88-kube-api-access-7kv9w\") pod \"maas-api-e2e-mt-1a269e4e-675c8d4ffb-nvj7h\" (UID: \"ac20b42b-e82d-4276-b63f-9a8b902baa88\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1a269e4e-675c8d4ffb-nvj7h"
Sep 11 13:40:43.649301 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:40:43.649228    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1a269e4e-675c8d4ffb-nvj7h"
Sep 11 13:40:43.952646 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:40:43.952601    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-1a269e4e-675c8d4ffb-nvj7h"]
Sep 11 13:40:43.954732 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:40:43.954674    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac20b42b_e82d_4276_b63f_9a8b902baa88.slice/crio-34b87bf79d6493b736f218b67f97648c204b7ddbaf49977c05a5c6949be2ab05 WatchSource:0}: Error finding container 34b87bf79d6493b736f218b67f97648c204b7ddbaf49977c05a5c6949be2ab05: Status 404 returned error can't find the container with id 34b87bf79d6493b736f218b67f97648c204b7ddbaf49977c05a5c6949be2ab05
Sep 11 13:40:44.397229 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:40:44.397163    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1a269e4e-675c8d4ffb-nvj7h" event={"ID":"ac20b42b-e82d-4276-b63f-9a8b902baa88","Type":"ContainerStarted","Data":"34b87bf79d6493b736f218b67f97648c204b7ddbaf49977c05a5c6949be2ab05"}
Sep 11 13:40:45.406445 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:40:45.406406    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1a269e4e-675c8d4ffb-nvj7h" event={"ID":"ac20b42b-e82d-4276-b63f-9a8b902baa88","Type":"ContainerStarted","Data":"d2b7bcc4a58db1d2c0062c873407d17a2f31fe82fea6e82a0e72a0de2af27150"}
Sep 11 13:40:45.407000 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:40:45.406604    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1a269e4e-675c8d4ffb-nvj7h"
Sep 11 13:40:45.428258 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:40:45.428189    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1a269e4e-675c8d4ffb-nvj7h" podStartSLOduration=1.9403562939999999 podStartE2EDuration="2.428168321s" totalImagesPullingTime="487.812027ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:40:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:40:45.427608801 +0000 UTC m=+1152.813711975" watchObservedRunningTime="2026-09-11 13:40:45.428168321 +0000 UTC m=+1152.814271572"
Sep 11 13:40:49.727468 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:40:49.727418    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-5a0fa1-57454c47c4-vns7s"]
Sep 11 13:40:49.728126 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:40:49.728088    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5a0fa1-57454c47c4-vns7s" podUID="2ae059be-ab6a-457a-82a2-7d59faa24517" containerName="maas-api" containerID="cri-o://699a9c7a3ebdc19f5acb8c739b03d6fd39b379b592abf269986ca5cf0431d987" gracePeriod=30
Sep 11 13:40:50.050268 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:40:50.050227    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5a0fa1-57454c47c4-vns7s"
Sep 11 13:40:50.171627 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:40:50.171572    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ae059be-ab6a-457a-82a2-7d59faa24517-metrics-certs\") pod \"2ae059be-ab6a-457a-82a2-7d59faa24517\" (UID: \"2ae059be-ab6a-457a-82a2-7d59faa24517\") "
Sep 11 13:40:50.171924 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:40:50.171748    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2ae059be-ab6a-457a-82a2-7d59faa24517-maas-api-tls\") pod \"2ae059be-ab6a-457a-82a2-7d59faa24517\" (UID: \"2ae059be-ab6a-457a-82a2-7d59faa24517\") "
Sep 11 13:40:50.171924 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:40:50.171772    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ae059be-ab6a-457a-82a2-7d59faa24517-openshift-service-ca\") pod \"2ae059be-ab6a-457a-82a2-7d59faa24517\" (UID: \"2ae059be-ab6a-457a-82a2-7d59faa24517\") "
Sep 11 13:40:50.171924 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:40:50.171791    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xsqd8\" (UniqueName: \"kubernetes.io/projected/2ae059be-ab6a-457a-82a2-7d59faa24517-kube-api-access-xsqd8\") pod \"2ae059be-ab6a-457a-82a2-7d59faa24517\" (UID: \"2ae059be-ab6a-457a-82a2-7d59faa24517\") "
Sep 11 13:40:50.172394 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:40:50.172220    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ae059be-ab6a-457a-82a2-7d59faa24517-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2ae059be-ab6a-457a-82a2-7d59faa24517" (UID: "2ae059be-ab6a-457a-82a2-7d59faa24517"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 13:40:50.172547 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:40:50.172514    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ae059be-ab6a-457a-82a2-7d59faa24517-openshift-service-ca\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:40:50.174564 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:40:50.174528    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ae059be-ab6a-457a-82a2-7d59faa24517-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2ae059be-ab6a-457a-82a2-7d59faa24517" (UID: "2ae059be-ab6a-457a-82a2-7d59faa24517"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 13:40:50.174957 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:40:50.174929    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ae059be-ab6a-457a-82a2-7d59faa24517-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "2ae059be-ab6a-457a-82a2-7d59faa24517" (UID: "2ae059be-ab6a-457a-82a2-7d59faa24517"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 13:40:50.175040 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:40:50.174927    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ae059be-ab6a-457a-82a2-7d59faa24517-kube-api-access-xsqd8" (OuterVolumeSpecName: "kube-api-access-xsqd8") pod "2ae059be-ab6a-457a-82a2-7d59faa24517" (UID: "2ae059be-ab6a-457a-82a2-7d59faa24517"). InnerVolumeSpecName "kube-api-access-xsqd8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 13:40:50.274311 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:40:50.274181    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ae059be-ab6a-457a-82a2-7d59faa24517-metrics-certs\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:40:50.274311 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:40:50.274231    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2ae059be-ab6a-457a-82a2-7d59faa24517-maas-api-tls\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:40:50.274311 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:40:50.274243    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xsqd8\" (UniqueName: \"kubernetes.io/projected/2ae059be-ab6a-457a-82a2-7d59faa24517-kube-api-access-xsqd8\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:40:50.443746 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:40:50.443700    2578 generic.go:356] "Generic (PLEG): container finished" podID="2ae059be-ab6a-457a-82a2-7d59faa24517" containerID="699a9c7a3ebdc19f5acb8c739b03d6fd39b379b592abf269986ca5cf0431d987" exitCode=0
Sep 11 13:40:50.444044 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:40:50.443768    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5a0fa1-57454c47c4-vns7s"
Sep 11 13:40:50.444044 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:40:50.443795    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5a0fa1-57454c47c4-vns7s" event={"ID":"2ae059be-ab6a-457a-82a2-7d59faa24517","Type":"ContainerDied","Data":"699a9c7a3ebdc19f5acb8c739b03d6fd39b379b592abf269986ca5cf0431d987"}
Sep 11 13:40:50.444044 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:40:50.443889    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5a0fa1-57454c47c4-vns7s" event={"ID":"2ae059be-ab6a-457a-82a2-7d59faa24517","Type":"ContainerDied","Data":"0c546e686187813d1b3f5057aa90367e82387514031ea2d4103381edbbf8f57c"}
Sep 11 13:40:50.444044 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:40:50.443922    2578 scope.go:117] "RemoveContainer" containerID="699a9c7a3ebdc19f5acb8c739b03d6fd39b379b592abf269986ca5cf0431d987"
Sep 11 13:40:50.456508 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:40:50.456461    2578 scope.go:117] "RemoveContainer" containerID="699a9c7a3ebdc19f5acb8c739b03d6fd39b379b592abf269986ca5cf0431d987"
Sep 11 13:40:50.457135 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:40:50.457098    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"699a9c7a3ebdc19f5acb8c739b03d6fd39b379b592abf269986ca5cf0431d987\": container with ID starting with 699a9c7a3ebdc19f5acb8c739b03d6fd39b379b592abf269986ca5cf0431d987 not found: ID does not exist" containerID="699a9c7a3ebdc19f5acb8c739b03d6fd39b379b592abf269986ca5cf0431d987"
Sep 11 13:40:50.457281 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:40:50.457145    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"699a9c7a3ebdc19f5acb8c739b03d6fd39b379b592abf269986ca5cf0431d987"} err="failed to get container status \"699a9c7a3ebdc19f5acb8c739b03d6fd39b379b592abf269986ca5cf0431d987\": rpc error: code = NotFound desc = could not find container \"699a9c7a3ebdc19f5acb8c739b03d6fd39b379b592abf269986ca5cf0431d987\": container with ID starting with 699a9c7a3ebdc19f5acb8c739b03d6fd39b379b592abf269986ca5cf0431d987 not found: ID does not exist"
Sep 11 13:40:50.468267 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:40:50.468211    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-5a0fa1-57454c47c4-vns7s"]
Sep 11 13:40:50.472712 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:40:50.472672    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-5a0fa1-57454c47c4-vns7s"]
Sep 11 13:40:51.107963 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:40:51.107927    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2ae059be-ab6a-457a-82a2-7d59faa24517" path="/var/lib/kubelet/pods/2ae059be-ab6a-457a-82a2-7d59faa24517/volumes"
Sep 11 13:40:51.419631 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:40:51.419525    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1a269e4e-675c8d4ffb-nvj7h"
Sep 11 13:41:20.291229 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:41:20.291157    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-1a269e4e-675c8d4ffb-nvj7h"]
Sep 11 13:41:20.292102 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:41:20.291587    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1a269e4e-675c8d4ffb-nvj7h" podUID="ac20b42b-e82d-4276-b63f-9a8b902baa88" containerName="maas-api" containerID="cri-o://d2b7bcc4a58db1d2c0062c873407d17a2f31fe82fea6e82a0e72a0de2af27150" gracePeriod=30
Sep 11 13:41:20.595002 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:41:20.594952    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1a269e4e-675c8d4ffb-nvj7h"
Sep 11 13:41:20.661555 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:41:20.661513    2578 generic.go:356] "Generic (PLEG): container finished" podID="ac20b42b-e82d-4276-b63f-9a8b902baa88" containerID="d2b7bcc4a58db1d2c0062c873407d17a2f31fe82fea6e82a0e72a0de2af27150" exitCode=0
Sep 11 13:41:20.661795 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:41:20.661574    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1a269e4e-675c8d4ffb-nvj7h"
Sep 11 13:41:20.661795 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:41:20.661594    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1a269e4e-675c8d4ffb-nvj7h" event={"ID":"ac20b42b-e82d-4276-b63f-9a8b902baa88","Type":"ContainerDied","Data":"d2b7bcc4a58db1d2c0062c873407d17a2f31fe82fea6e82a0e72a0de2af27150"}
Sep 11 13:41:20.661795 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:41:20.661663    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1a269e4e-675c8d4ffb-nvj7h" event={"ID":"ac20b42b-e82d-4276-b63f-9a8b902baa88","Type":"ContainerDied","Data":"34b87bf79d6493b736f218b67f97648c204b7ddbaf49977c05a5c6949be2ab05"}
Sep 11 13:41:20.661795 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:41:20.661698    2578 scope.go:117] "RemoveContainer" containerID="d2b7bcc4a58db1d2c0062c873407d17a2f31fe82fea6e82a0e72a0de2af27150"
Sep 11 13:41:20.674043 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:41:20.673998    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac20b42b-e82d-4276-b63f-9a8b902baa88-openshift-service-ca\") pod \"ac20b42b-e82d-4276-b63f-9a8b902baa88\" (UID: \"ac20b42b-e82d-4276-b63f-9a8b902baa88\") "
Sep 11 13:41:20.674790 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:41:20.674420    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac20b42b-e82d-4276-b63f-9a8b902baa88-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ac20b42b-e82d-4276-b63f-9a8b902baa88" (UID: "ac20b42b-e82d-4276-b63f-9a8b902baa88"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 13:41:20.674790 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:41:20.674639    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac20b42b-e82d-4276-b63f-9a8b902baa88-maas-api-tls\") pod \"ac20b42b-e82d-4276-b63f-9a8b902baa88\" (UID: \"ac20b42b-e82d-4276-b63f-9a8b902baa88\") "
Sep 11 13:41:20.674790 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:41:20.674695    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7kv9w\" (UniqueName: \"kubernetes.io/projected/ac20b42b-e82d-4276-b63f-9a8b902baa88-kube-api-access-7kv9w\") pod \"ac20b42b-e82d-4276-b63f-9a8b902baa88\" (UID: \"ac20b42b-e82d-4276-b63f-9a8b902baa88\") "
Sep 11 13:41:20.675622 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:41:20.675478    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac20b42b-e82d-4276-b63f-9a8b902baa88-metrics-certs\") pod \"ac20b42b-e82d-4276-b63f-9a8b902baa88\" (UID: \"ac20b42b-e82d-4276-b63f-9a8b902baa88\") "
Sep 11 13:41:20.676021 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:41:20.675908    2578 scope.go:117] "RemoveContainer" containerID="d2b7bcc4a58db1d2c0062c873407d17a2f31fe82fea6e82a0e72a0de2af27150"
Sep 11 13:41:20.676599 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:41:20.676572    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac20b42b-e82d-4276-b63f-9a8b902baa88-openshift-service-ca\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:41:20.677546 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:41:20.677500    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2b7bcc4a58db1d2c0062c873407d17a2f31fe82fea6e82a0e72a0de2af27150\": container with ID starting with d2b7bcc4a58db1d2c0062c873407d17a2f31fe82fea6e82a0e72a0de2af27150 not found: ID does not exist" containerID="d2b7bcc4a58db1d2c0062c873407d17a2f31fe82fea6e82a0e72a0de2af27150"
Sep 11 13:41:20.677679 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:41:20.677552    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2b7bcc4a58db1d2c0062c873407d17a2f31fe82fea6e82a0e72a0de2af27150"} err="failed to get container status \"d2b7bcc4a58db1d2c0062c873407d17a2f31fe82fea6e82a0e72a0de2af27150\": rpc error: code = NotFound desc = could not find container \"d2b7bcc4a58db1d2c0062c873407d17a2f31fe82fea6e82a0e72a0de2af27150\": container with ID starting with d2b7bcc4a58db1d2c0062c873407d17a2f31fe82fea6e82a0e72a0de2af27150 not found: ID does not exist"
Sep 11 13:41:20.679223 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:41:20.679188    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac20b42b-e82d-4276-b63f-9a8b902baa88-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "ac20b42b-e82d-4276-b63f-9a8b902baa88" (UID: "ac20b42b-e82d-4276-b63f-9a8b902baa88"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 13:41:20.679340 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:41:20.679283    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac20b42b-e82d-4276-b63f-9a8b902baa88-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ac20b42b-e82d-4276-b63f-9a8b902baa88" (UID: "ac20b42b-e82d-4276-b63f-9a8b902baa88"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 13:41:20.680028 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:41:20.679990    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac20b42b-e82d-4276-b63f-9a8b902baa88-kube-api-access-7kv9w" (OuterVolumeSpecName: "kube-api-access-7kv9w") pod "ac20b42b-e82d-4276-b63f-9a8b902baa88" (UID: "ac20b42b-e82d-4276-b63f-9a8b902baa88"). InnerVolumeSpecName "kube-api-access-7kv9w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 13:41:20.777714 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:41:20.777671    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac20b42b-e82d-4276-b63f-9a8b902baa88-maas-api-tls\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:41:20.777714 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:41:20.777710    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7kv9w\" (UniqueName: \"kubernetes.io/projected/ac20b42b-e82d-4276-b63f-9a8b902baa88-kube-api-access-7kv9w\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:41:20.777714 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:41:20.777724    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac20b42b-e82d-4276-b63f-9a8b902baa88-metrics-certs\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:41:20.986095 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:41:20.986046    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-1a269e4e-675c8d4ffb-nvj7h"]
Sep 11 13:41:20.991810 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:41:20.991745    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-1a269e4e-675c8d4ffb-nvj7h"]
Sep 11 13:41:21.108001 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:41:21.107964    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ac20b42b-e82d-4276-b63f-9a8b902baa88" path="/var/lib/kubelet/pods/ac20b42b-e82d-4276-b63f-9a8b902baa88/volumes"
Sep 11 13:41:33.163880 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:41:33.163813    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6gltj_e384caae-3cb1-4264-8a25-55b40ac8ac45/ovn-acl-logging/0.log"
Sep 11 13:41:33.166789 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:41:33.166764    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6gltj_e384caae-3cb1-4264-8a25-55b40ac8ac45/ovn-acl-logging/0.log"
Sep 11 13:41:49.966800 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:41:49.966757    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-470eeb-6d66876586-4mrvl"]
Sep 11 13:41:49.967361 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:41:49.967344    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac20b42b-e82d-4276-b63f-9a8b902baa88" containerName="maas-api"
Sep 11 13:41:49.967412 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:41:49.967363    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac20b42b-e82d-4276-b63f-9a8b902baa88" containerName="maas-api"
Sep 11 13:41:49.967412 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:41:49.967376    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ae059be-ab6a-457a-82a2-7d59faa24517" containerName="maas-api"
Sep 11 13:41:49.967412 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:41:49.967382    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ae059be-ab6a-457a-82a2-7d59faa24517" containerName="maas-api"
Sep 11 13:41:49.967516 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:41:49.967474    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac20b42b-e82d-4276-b63f-9a8b902baa88" containerName="maas-api"
Sep 11 13:41:49.967516 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:41:49.967487    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ae059be-ab6a-457a-82a2-7d59faa24517" containerName="maas-api"
Sep 11 13:41:49.972990 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:41:49.972955    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-470eeb-6d66876586-4mrvl"
Sep 11 13:41:49.975773 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:41:49.975746    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-470eeb-serving-cert\""
Sep 11 13:41:49.983385 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:41:49.983352    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-470eeb-6d66876586-4mrvl"]
Sep 11 13:41:50.044267 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:41:50.044213    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwvpw\" (UniqueName: \"kubernetes.io/projected/e62b849e-6379-4858-aeb3-14eea3fc2f52-kube-api-access-vwvpw\") pod \"maas-api-e2e-shared-a-w2-470eeb-6d66876586-4mrvl\" (UID: \"e62b849e-6379-4858-aeb3-14eea3fc2f52\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-470eeb-6d66876586-4mrvl"
Sep 11 13:41:50.044511 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:41:50.044298    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e62b849e-6379-4858-aeb3-14eea3fc2f52-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-470eeb-6d66876586-4mrvl\" (UID: \"e62b849e-6379-4858-aeb3-14eea3fc2f52\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-470eeb-6d66876586-4mrvl"
Sep 11 13:41:50.044511 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:41:50.044344    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e62b849e-6379-4858-aeb3-14eea3fc2f52-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-470eeb-6d66876586-4mrvl\" (UID: \"e62b849e-6379-4858-aeb3-14eea3fc2f52\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-470eeb-6d66876586-4mrvl"
Sep 11 13:41:50.044511 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:41:50.044374    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e62b849e-6379-4858-aeb3-14eea3fc2f52-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-470eeb-6d66876586-4mrvl\" (UID: \"e62b849e-6379-4858-aeb3-14eea3fc2f52\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-470eeb-6d66876586-4mrvl"
Sep 11 13:41:50.145559 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:41:50.145487    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwvpw\" (UniqueName: \"kubernetes.io/projected/e62b849e-6379-4858-aeb3-14eea3fc2f52-kube-api-access-vwvpw\") pod \"maas-api-e2e-shared-a-w2-470eeb-6d66876586-4mrvl\" (UID: \"e62b849e-6379-4858-aeb3-14eea3fc2f52\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-470eeb-6d66876586-4mrvl"
Sep 11 13:41:50.145818 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:41:50.145604    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e62b849e-6379-4858-aeb3-14eea3fc2f52-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-470eeb-6d66876586-4mrvl\" (UID: \"e62b849e-6379-4858-aeb3-14eea3fc2f52\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-470eeb-6d66876586-4mrvl"
Sep 11 13:41:50.145818 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:41:50.145630    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e62b849e-6379-4858-aeb3-14eea3fc2f52-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-470eeb-6d66876586-4mrvl\" (UID: \"e62b849e-6379-4858-aeb3-14eea3fc2f52\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-470eeb-6d66876586-4mrvl"
Sep 11 13:41:50.145818 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:41:50.145653    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e62b849e-6379-4858-aeb3-14eea3fc2f52-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-470eeb-6d66876586-4mrvl\" (UID: \"e62b849e-6379-4858-aeb3-14eea3fc2f52\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-470eeb-6d66876586-4mrvl"
Sep 11 13:41:50.146435 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:41:50.146399    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e62b849e-6379-4858-aeb3-14eea3fc2f52-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-470eeb-6d66876586-4mrvl\" (UID: \"e62b849e-6379-4858-aeb3-14eea3fc2f52\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-470eeb-6d66876586-4mrvl"
Sep 11 13:41:50.148666 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:41:50.148643    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e62b849e-6379-4858-aeb3-14eea3fc2f52-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-470eeb-6d66876586-4mrvl\" (UID: \"e62b849e-6379-4858-aeb3-14eea3fc2f52\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-470eeb-6d66876586-4mrvl"
Sep 11 13:41:50.148774 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:41:50.148686    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e62b849e-6379-4858-aeb3-14eea3fc2f52-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-470eeb-6d66876586-4mrvl\" (UID: \"e62b849e-6379-4858-aeb3-14eea3fc2f52\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-470eeb-6d66876586-4mrvl"
Sep 11 13:41:50.154624 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:41:50.154596    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwvpw\" (UniqueName: \"kubernetes.io/projected/e62b849e-6379-4858-aeb3-14eea3fc2f52-kube-api-access-vwvpw\") pod \"maas-api-e2e-shared-a-w2-470eeb-6d66876586-4mrvl\" (UID: \"e62b849e-6379-4858-aeb3-14eea3fc2f52\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-470eeb-6d66876586-4mrvl"
Sep 11 13:41:50.287253 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:41:50.287125    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-470eeb-6d66876586-4mrvl"
Sep 11 13:41:50.595423 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:41:50.595219    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-470eeb-6d66876586-4mrvl"]
Sep 11 13:41:50.599771 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:41:50.599717    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode62b849e_6379_4858_aeb3_14eea3fc2f52.slice/crio-c8be9aaf69acd4e819b36892a197024391ba74e7ac309984eeb7ae24e8cce78e WatchSource:0}: Error finding container c8be9aaf69acd4e819b36892a197024391ba74e7ac309984eeb7ae24e8cce78e: Status 404 returned error can't find the container with id c8be9aaf69acd4e819b36892a197024391ba74e7ac309984eeb7ae24e8cce78e
Sep 11 13:41:50.875326 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:41:50.875202    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-470eeb-6d66876586-4mrvl" event={"ID":"e62b849e-6379-4858-aeb3-14eea3fc2f52","Type":"ContainerStarted","Data":"c8be9aaf69acd4e819b36892a197024391ba74e7ac309984eeb7ae24e8cce78e"}
Sep 11 13:41:51.883761 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:41:51.883712    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-470eeb-6d66876586-4mrvl" event={"ID":"e62b849e-6379-4858-aeb3-14eea3fc2f52","Type":"ContainerStarted","Data":"7bfcd81265fc739f3a58a9a773804872f393521c5e3bd9543d59a970371e5578"}
Sep 11 13:41:51.884338 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:41:51.883961    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-470eeb-6d66876586-4mrvl"
Sep 11 13:41:51.916161 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:41:51.916079    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-470eeb-6d66876586-4mrvl" podStartSLOduration=2.489159233 podStartE2EDuration="2.916048469s" totalImagesPullingTime="426.889236ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:41:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:41:51.91298751 +0000 UTC m=+1219.299090706" watchObservedRunningTime="2026-09-11 13:41:51.916048469 +0000 UTC m=+1219.302151634"
Sep 11 13:41:57.897933 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:41:57.897888    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-470eeb-6d66876586-4mrvl"
Sep 11 13:42:22.358346 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:42:22.358294    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-371c38/e2e-conflict-a-f8ce9a34-kserve-67b74b49b4-p52gz"]
Sep 11 13:42:22.364012 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:42:22.363971    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-371c38/e2e-conflict-a-f8ce9a34-kserve-67b74b49b4-p52gz"
Sep 11 13:42:22.366570 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:42:22.366537    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-371c38\"/\"e2e-conflict-a-f8ce9a34-kserve-self-signed-certs\""
Sep 11 13:42:22.390534 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:42:22.390466    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-371c38/e2e-conflict-a-f8ce9a34-kserve-67b74b49b4-p52gz"]
Sep 11 13:42:22.464804 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:42:22.464737    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/02db85b9-8b34-40e1-9518-d60e3971228f-dshm\") pod \"e2e-conflict-a-f8ce9a34-kserve-67b74b49b4-p52gz\" (UID: \"02db85b9-8b34-40e1-9518-d60e3971228f\") " pod="e2e-models-e2e-worker-w4-371c38/e2e-conflict-a-f8ce9a34-kserve-67b74b49b4-p52gz"
Sep 11 13:42:22.465210 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:42:22.464878    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02db85b9-8b34-40e1-9518-d60e3971228f-model-cache\") pod \"e2e-conflict-a-f8ce9a34-kserve-67b74b49b4-p52gz\" (UID: \"02db85b9-8b34-40e1-9518-d60e3971228f\") " pod="e2e-models-e2e-worker-w4-371c38/e2e-conflict-a-f8ce9a34-kserve-67b74b49b4-p52gz"
Sep 11 13:42:22.465210 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:42:22.464948    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02db85b9-8b34-40e1-9518-d60e3971228f-tmp-dir\") pod \"e2e-conflict-a-f8ce9a34-kserve-67b74b49b4-p52gz\" (UID: \"02db85b9-8b34-40e1-9518-d60e3971228f\") " pod="e2e-models-e2e-worker-w4-371c38/e2e-conflict-a-f8ce9a34-kserve-67b74b49b4-p52gz"
Sep 11 13:42:22.465359 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:42:22.465234    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lstns\" (UniqueName: \"kubernetes.io/projected/02db85b9-8b34-40e1-9518-d60e3971228f-kube-api-access-lstns\") pod \"e2e-conflict-a-f8ce9a34-kserve-67b74b49b4-p52gz\" (UID: \"02db85b9-8b34-40e1-9518-d60e3971228f\") " pod="e2e-models-e2e-worker-w4-371c38/e2e-conflict-a-f8ce9a34-kserve-67b74b49b4-p52gz"
Sep 11 13:42:22.465359 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:42:22.465273    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/02db85b9-8b34-40e1-9518-d60e3971228f-home\") pod \"e2e-conflict-a-f8ce9a34-kserve-67b74b49b4-p52gz\" (UID: \"02db85b9-8b34-40e1-9518-d60e3971228f\") " pod="e2e-models-e2e-worker-w4-371c38/e2e-conflict-a-f8ce9a34-kserve-67b74b49b4-p52gz"
Sep 11 13:42:22.465359 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:42:22.465300    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02db85b9-8b34-40e1-9518-d60e3971228f-tls-certs\") pod \"e2e-conflict-a-f8ce9a34-kserve-67b74b49b4-p52gz\" (UID: \"02db85b9-8b34-40e1-9518-d60e3971228f\") " pod="e2e-models-e2e-worker-w4-371c38/e2e-conflict-a-f8ce9a34-kserve-67b74b49b4-p52gz"
Sep 11 13:42:22.567096 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:42:22.567044    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02db85b9-8b34-40e1-9518-d60e3971228f-model-cache\") pod \"e2e-conflict-a-f8ce9a34-kserve-67b74b49b4-p52gz\" (UID: \"02db85b9-8b34-40e1-9518-d60e3971228f\") " pod="e2e-models-e2e-worker-w4-371c38/e2e-conflict-a-f8ce9a34-kserve-67b74b49b4-p52gz"
Sep 11 13:42:22.567328 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:42:22.567130    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02db85b9-8b34-40e1-9518-d60e3971228f-tmp-dir\") pod \"e2e-conflict-a-f8ce9a34-kserve-67b74b49b4-p52gz\" (UID: \"02db85b9-8b34-40e1-9518-d60e3971228f\") " pod="e2e-models-e2e-worker-w4-371c38/e2e-conflict-a-f8ce9a34-kserve-67b74b49b4-p52gz"
Sep 11 13:42:22.567328 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:42:22.567284    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lstns\" (UniqueName: \"kubernetes.io/projected/02db85b9-8b34-40e1-9518-d60e3971228f-kube-api-access-lstns\") pod \"e2e-conflict-a-f8ce9a34-kserve-67b74b49b4-p52gz\" (UID: \"02db85b9-8b34-40e1-9518-d60e3971228f\") " pod="e2e-models-e2e-worker-w4-371c38/e2e-conflict-a-f8ce9a34-kserve-67b74b49b4-p52gz"
Sep 11 13:42:22.567328 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:42:22.567319    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/02db85b9-8b34-40e1-9518-d60e3971228f-home\") pod \"e2e-conflict-a-f8ce9a34-kserve-67b74b49b4-p52gz\" (UID: \"02db85b9-8b34-40e1-9518-d60e3971228f\") " pod="e2e-models-e2e-worker-w4-371c38/e2e-conflict-a-f8ce9a34-kserve-67b74b49b4-p52gz"
Sep 11 13:42:22.567514 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:42:22.567345    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02db85b9-8b34-40e1-9518-d60e3971228f-tls-certs\") pod \"e2e-conflict-a-f8ce9a34-kserve-67b74b49b4-p52gz\" (UID: \"02db85b9-8b34-40e1-9518-d60e3971228f\") " pod="e2e-models-e2e-worker-w4-371c38/e2e-conflict-a-f8ce9a34-kserve-67b74b49b4-p52gz"
Sep 11 13:42:22.567514 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:42:22.567398    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/02db85b9-8b34-40e1-9518-d60e3971228f-dshm\") pod \"e2e-conflict-a-f8ce9a34-kserve-67b74b49b4-p52gz\" (UID: \"02db85b9-8b34-40e1-9518-d60e3971228f\") " pod="e2e-models-e2e-worker-w4-371c38/e2e-conflict-a-f8ce9a34-kserve-67b74b49b4-p52gz"
Sep 11 13:42:22.567623 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:42:22.567547    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02db85b9-8b34-40e1-9518-d60e3971228f-model-cache\") pod \"e2e-conflict-a-f8ce9a34-kserve-67b74b49b4-p52gz\" (UID: \"02db85b9-8b34-40e1-9518-d60e3971228f\") " pod="e2e-models-e2e-worker-w4-371c38/e2e-conflict-a-f8ce9a34-kserve-67b74b49b4-p52gz"
Sep 11 13:42:22.567684 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:42:22.567633    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02db85b9-8b34-40e1-9518-d60e3971228f-tmp-dir\") pod \"e2e-conflict-a-f8ce9a34-kserve-67b74b49b4-p52gz\" (UID: \"02db85b9-8b34-40e1-9518-d60e3971228f\") " pod="e2e-models-e2e-worker-w4-371c38/e2e-conflict-a-f8ce9a34-kserve-67b74b49b4-p52gz"
Sep 11 13:42:22.567780 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:42:22.567736    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/02db85b9-8b34-40e1-9518-d60e3971228f-home\") pod \"e2e-conflict-a-f8ce9a34-kserve-67b74b49b4-p52gz\" (UID: \"02db85b9-8b34-40e1-9518-d60e3971228f\") " pod="e2e-models-e2e-worker-w4-371c38/e2e-conflict-a-f8ce9a34-kserve-67b74b49b4-p52gz"
Sep 11 13:42:22.570222 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:42:22.570193    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/02db85b9-8b34-40e1-9518-d60e3971228f-dshm\") pod \"e2e-conflict-a-f8ce9a34-kserve-67b74b49b4-p52gz\" (UID: \"02db85b9-8b34-40e1-9518-d60e3971228f\") " pod="e2e-models-e2e-worker-w4-371c38/e2e-conflict-a-f8ce9a34-kserve-67b74b49b4-p52gz"
Sep 11 13:42:22.570479 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:42:22.570455    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02db85b9-8b34-40e1-9518-d60e3971228f-tls-certs\") pod \"e2e-conflict-a-f8ce9a34-kserve-67b74b49b4-p52gz\" (UID: \"02db85b9-8b34-40e1-9518-d60e3971228f\") " pod="e2e-models-e2e-worker-w4-371c38/e2e-conflict-a-f8ce9a34-kserve-67b74b49b4-p52gz"
Sep 11 13:42:22.586287 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:42:22.586241    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lstns\" (UniqueName: \"kubernetes.io/projected/02db85b9-8b34-40e1-9518-d60e3971228f-kube-api-access-lstns\") pod \"e2e-conflict-a-f8ce9a34-kserve-67b74b49b4-p52gz\" (UID: \"02db85b9-8b34-40e1-9518-d60e3971228f\") " pod="e2e-models-e2e-worker-w4-371c38/e2e-conflict-a-f8ce9a34-kserve-67b74b49b4-p52gz"
Sep 11 13:42:22.678464 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:42:22.678325    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-371c38/e2e-conflict-a-f8ce9a34-kserve-67b74b49b4-p52gz"
Sep 11 13:42:22.979069 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:42:22.979034    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-371c38/e2e-conflict-a-f8ce9a34-kserve-67b74b49b4-p52gz"]
Sep 11 13:42:22.981869 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:42:22.981792    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod02db85b9_8b34_40e1_9518_d60e3971228f.slice/crio-b4db58ea58d632f9f22681ef229b89bb92fb6aa525d2e59869a633906201c0d9 WatchSource:0}: Error finding container b4db58ea58d632f9f22681ef229b89bb92fb6aa525d2e59869a633906201c0d9: Status 404 returned error can't find the container with id b4db58ea58d632f9f22681ef229b89bb92fb6aa525d2e59869a633906201c0d9
Sep 11 13:42:23.114051 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:42:23.113971    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-371c38/e2e-conflict-a-f8ce9a34-kserve-67b74b49b4-p52gz" event={"ID":"02db85b9-8b34-40e1-9518-d60e3971228f","Type":"ContainerStarted","Data":"748a3296b9ded3fcd1a65d30785560e02cbe35e69ee275f06a5043e837e4271d"}
Sep 11 13:42:23.114051 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:42:23.114037    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-371c38/e2e-conflict-a-f8ce9a34-kserve-67b74b49b4-p52gz" event={"ID":"02db85b9-8b34-40e1-9518-d60e3971228f","Type":"ContainerStarted","Data":"b4db58ea58d632f9f22681ef229b89bb92fb6aa525d2e59869a633906201c0d9"}
Sep 11 13:42:23.137707 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:42:23.137635    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-371c38/e2e-conflict-a-f8ce9a34-kserve-67b74b49b4-p52gz" podStartSLOduration=1.137612065 podStartE2EDuration="1.137612065s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:42:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:42:23.135957384 +0000 UTC m=+1250.522060567" watchObservedRunningTime="2026-09-11 13:42:23.137612065 +0000 UTC m=+1250.523715255"
Sep 11 13:42:26.109014 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:42:26.108957    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e52aaa-68484dc9c-68kbs"]
Sep 11 13:42:26.114725 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:42:26.114678    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e52aaa-68484dc9c-68kbs"
Sep 11 13:42:26.120344 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:42:26.120311    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-e52aaa-serving-cert\""
Sep 11 13:42:26.159598 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:42:26.159562    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e52aaa-68484dc9c-68kbs"]
Sep 11 13:42:26.221057 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:42:26.221009    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7dc1eb52-2060-473a-a0d5-4fe00eca2972-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-e52aaa-68484dc9c-68kbs\" (UID: \"7dc1eb52-2060-473a-a0d5-4fe00eca2972\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e52aaa-68484dc9c-68kbs"
Sep 11 13:42:26.221342 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:42:26.221310    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7dc1eb52-2060-473a-a0d5-4fe00eca2972-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-e52aaa-68484dc9c-68kbs\" (UID: \"7dc1eb52-2060-473a-a0d5-4fe00eca2972\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e52aaa-68484dc9c-68kbs"
Sep 11 13:42:26.221406 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:42:26.221383    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7dc1eb52-2060-473a-a0d5-4fe00eca2972-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-e52aaa-68484dc9c-68kbs\" (UID: \"7dc1eb52-2060-473a-a0d5-4fe00eca2972\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e52aaa-68484dc9c-68kbs"
Sep 11 13:42:26.221494 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:42:26.221475    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2vkk\" (UniqueName: \"kubernetes.io/projected/7dc1eb52-2060-473a-a0d5-4fe00eca2972-kube-api-access-m2vkk\") pod \"maas-api-e2e-shared-b-w2-e52aaa-68484dc9c-68kbs\" (UID: \"7dc1eb52-2060-473a-a0d5-4fe00eca2972\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e52aaa-68484dc9c-68kbs"
Sep 11 13:42:26.323204 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:42:26.323146    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7dc1eb52-2060-473a-a0d5-4fe00eca2972-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-e52aaa-68484dc9c-68kbs\" (UID: \"7dc1eb52-2060-473a-a0d5-4fe00eca2972\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e52aaa-68484dc9c-68kbs"
Sep 11 13:42:26.323204 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:42:26.323214    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7dc1eb52-2060-473a-a0d5-4fe00eca2972-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-e52aaa-68484dc9c-68kbs\" (UID: \"7dc1eb52-2060-473a-a0d5-4fe00eca2972\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e52aaa-68484dc9c-68kbs"
Sep 11 13:42:26.323592 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:42:26.323258    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2vkk\" (UniqueName: \"kubernetes.io/projected/7dc1eb52-2060-473a-a0d5-4fe00eca2972-kube-api-access-m2vkk\") pod \"maas-api-e2e-shared-b-w2-e52aaa-68484dc9c-68kbs\" (UID: \"7dc1eb52-2060-473a-a0d5-4fe00eca2972\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e52aaa-68484dc9c-68kbs"
Sep 11 13:42:26.323592 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:42:26.323286    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7dc1eb52-2060-473a-a0d5-4fe00eca2972-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-e52aaa-68484dc9c-68kbs\" (UID: \"7dc1eb52-2060-473a-a0d5-4fe00eca2972\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e52aaa-68484dc9c-68kbs"
Sep 11 13:42:26.323592 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:42:26.323425    2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e52aaa-serving-cert: secret "maas-api-e2e-shared-b-w2-e52aaa-serving-cert" not found
Sep 11 13:42:26.323592 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:42:26.323508    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7dc1eb52-2060-473a-a0d5-4fe00eca2972-maas-api-tls podName:7dc1eb52-2060-473a-a0d5-4fe00eca2972 nodeName:}" failed. No retries permitted until 2026-09-11 13:42:26.823483438 +0000 UTC m=+1254.209586593 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7dc1eb52-2060-473a-a0d5-4fe00eca2972-maas-api-tls") pod "maas-api-e2e-shared-b-w2-e52aaa-68484dc9c-68kbs" (UID: "7dc1eb52-2060-473a-a0d5-4fe00eca2972") : secret "maas-api-e2e-shared-b-w2-e52aaa-serving-cert" not found
Sep 11 13:42:26.324259 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:42:26.324226    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7dc1eb52-2060-473a-a0d5-4fe00eca2972-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-e52aaa-68484dc9c-68kbs\" (UID: \"7dc1eb52-2060-473a-a0d5-4fe00eca2972\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e52aaa-68484dc9c-68kbs"
Sep 11 13:42:26.326540 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:42:26.326506    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7dc1eb52-2060-473a-a0d5-4fe00eca2972-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-e52aaa-68484dc9c-68kbs\" (UID: \"7dc1eb52-2060-473a-a0d5-4fe00eca2972\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e52aaa-68484dc9c-68kbs"
Sep 11 13:42:26.335505 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:42:26.335467    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2vkk\" (UniqueName: \"kubernetes.io/projected/7dc1eb52-2060-473a-a0d5-4fe00eca2972-kube-api-access-m2vkk\") pod \"maas-api-e2e-shared-b-w2-e52aaa-68484dc9c-68kbs\" (UID: \"7dc1eb52-2060-473a-a0d5-4fe00eca2972\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e52aaa-68484dc9c-68kbs"
Sep 11 13:42:26.830810 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:42:26.830751    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7dc1eb52-2060-473a-a0d5-4fe00eca2972-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-e52aaa-68484dc9c-68kbs\" (UID: \"7dc1eb52-2060-473a-a0d5-4fe00eca2972\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e52aaa-68484dc9c-68kbs"
Sep 11 13:42:26.834791 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:42:26.834745    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7dc1eb52-2060-473a-a0d5-4fe00eca2972-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-e52aaa-68484dc9c-68kbs\" (UID: \"7dc1eb52-2060-473a-a0d5-4fe00eca2972\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e52aaa-68484dc9c-68kbs"
Sep 11 13:42:27.029144 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:42:27.029088    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e52aaa-68484dc9c-68kbs"
Sep 11 13:42:27.332413 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:42:27.332378    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e52aaa-68484dc9c-68kbs"]
Sep 11 13:42:27.335414 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:42:27.335366    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7dc1eb52_2060_473a_a0d5_4fe00eca2972.slice/crio-be2470d58cb9d2c295afdb0c19bd5c02322ef779581695993d15ea921722be0d WatchSource:0}: Error finding container be2470d58cb9d2c295afdb0c19bd5c02322ef779581695993d15ea921722be0d: Status 404 returned error can't find the container with id be2470d58cb9d2c295afdb0c19bd5c02322ef779581695993d15ea921722be0d
Sep 11 13:42:28.152160 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:42:28.152115    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e52aaa-68484dc9c-68kbs" event={"ID":"7dc1eb52-2060-473a-a0d5-4fe00eca2972","Type":"ContainerStarted","Data":"cacfb4df4b2719431ca7423d1cd4fd62bf02c7f11c254639033a3048abbd9883"}
Sep 11 13:42:28.152446 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:42:28.152330    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e52aaa-68484dc9c-68kbs"
Sep 11 13:42:28.152446 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:42:28.152381    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e52aaa-68484dc9c-68kbs" event={"ID":"7dc1eb52-2060-473a-a0d5-4fe00eca2972","Type":"ContainerStarted","Data":"be2470d58cb9d2c295afdb0c19bd5c02322ef779581695993d15ea921722be0d"}
Sep 11 13:42:28.185729 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:42:28.185585    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e52aaa-68484dc9c-68kbs" podStartSLOduration=1.636327792 podStartE2EDuration="2.185561907s" totalImagesPullingTime="549.234115ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:42:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:42:28.183175456 +0000 UTC m=+1255.569278633" watchObservedRunningTime="2026-09-11 13:42:28.185561907 +0000 UTC m=+1255.571665071"
Sep 11 13:42:32.679627 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:42:32.679567    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-371c38/e2e-conflict-a-f8ce9a34-kserve-67b74b49b4-p52gz"
Sep 11 13:42:32.679627 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:42:32.679647    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-371c38/e2e-conflict-a-f8ce9a34-kserve-67b74b49b4-p52gz"
Sep 11 13:42:32.695672 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:42:32.695623    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-371c38/e2e-conflict-a-f8ce9a34-kserve-67b74b49b4-p52gz"
Sep 11 13:42:33.199713 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:42:33.199659    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-371c38/e2e-conflict-a-f8ce9a34-kserve-67b74b49b4-p52gz"
Sep 11 13:42:34.166042 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:42:34.166009    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e52aaa-68484dc9c-68kbs"
Sep 11 13:43:07.879927 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:43:07.879877    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b65337-58b646495c-w8ldq"]
Sep 11 13:43:07.881339 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:43:07.880190    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b65337-58b646495c-w8ldq" podUID="c4cd43c6-1fe9-47ae-b24d-c9effce09fb8" containerName="maas-api" containerID="cri-o://a4872d4814d5b81ef95c5c2f6b24765f80caa3e07e9107c9ed83269bbf0816c5" gracePeriod=30
Sep 11 13:43:08.176365 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:43:08.176328    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b65337-58b646495c-w8ldq"
Sep 11 13:43:08.272680 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:43:08.272637    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4cd43c6-1fe9-47ae-b24d-c9effce09fb8-openshift-service-ca\") pod \"c4cd43c6-1fe9-47ae-b24d-c9effce09fb8\" (UID: \"c4cd43c6-1fe9-47ae-b24d-c9effce09fb8\") "
Sep 11 13:43:08.272974 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:43:08.272763    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c4cd43c6-1fe9-47ae-b24d-c9effce09fb8-metrics-certs\") pod \"c4cd43c6-1fe9-47ae-b24d-c9effce09fb8\" (UID: \"c4cd43c6-1fe9-47ae-b24d-c9effce09fb8\") "
Sep 11 13:43:08.272974 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:43:08.272799    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d4vpt\" (UniqueName: \"kubernetes.io/projected/c4cd43c6-1fe9-47ae-b24d-c9effce09fb8-kube-api-access-d4vpt\") pod \"c4cd43c6-1fe9-47ae-b24d-c9effce09fb8\" (UID: \"c4cd43c6-1fe9-47ae-b24d-c9effce09fb8\") "
Sep 11 13:43:08.272974 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:43:08.272931    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c4cd43c6-1fe9-47ae-b24d-c9effce09fb8-maas-api-tls\") pod \"c4cd43c6-1fe9-47ae-b24d-c9effce09fb8\" (UID: \"c4cd43c6-1fe9-47ae-b24d-c9effce09fb8\") "
Sep 11 13:43:08.273140 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:43:08.273108    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4cd43c6-1fe9-47ae-b24d-c9effce09fb8-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c4cd43c6-1fe9-47ae-b24d-c9effce09fb8" (UID: "c4cd43c6-1fe9-47ae-b24d-c9effce09fb8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 13:43:08.273825 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:43:08.273525    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4cd43c6-1fe9-47ae-b24d-c9effce09fb8-openshift-service-ca\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:43:08.275625 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:43:08.275590    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4cd43c6-1fe9-47ae-b24d-c9effce09fb8-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c4cd43c6-1fe9-47ae-b24d-c9effce09fb8" (UID: "c4cd43c6-1fe9-47ae-b24d-c9effce09fb8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 13:43:08.275625 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:43:08.275604    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4cd43c6-1fe9-47ae-b24d-c9effce09fb8-kube-api-access-d4vpt" (OuterVolumeSpecName: "kube-api-access-d4vpt") pod "c4cd43c6-1fe9-47ae-b24d-c9effce09fb8" (UID: "c4cd43c6-1fe9-47ae-b24d-c9effce09fb8"). InnerVolumeSpecName "kube-api-access-d4vpt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 13:43:08.275821 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:43:08.275582    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4cd43c6-1fe9-47ae-b24d-c9effce09fb8-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "c4cd43c6-1fe9-47ae-b24d-c9effce09fb8" (UID: "c4cd43c6-1fe9-47ae-b24d-c9effce09fb8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 13:43:08.374969 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:43:08.374921    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c4cd43c6-1fe9-47ae-b24d-c9effce09fb8-metrics-certs\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:43:08.374969 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:43:08.374958    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4vpt\" (UniqueName: \"kubernetes.io/projected/c4cd43c6-1fe9-47ae-b24d-c9effce09fb8-kube-api-access-d4vpt\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:43:08.374969 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:43:08.374967    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c4cd43c6-1fe9-47ae-b24d-c9effce09fb8-maas-api-tls\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:43:08.436283 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:43:08.436181    2578 generic.go:356] "Generic (PLEG): container finished" podID="c4cd43c6-1fe9-47ae-b24d-c9effce09fb8" containerID="a4872d4814d5b81ef95c5c2f6b24765f80caa3e07e9107c9ed83269bbf0816c5" exitCode=0
Sep 11 13:43:08.436283 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:43:08.436236    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b65337-58b646495c-w8ldq"
Sep 11 13:43:08.436283 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:43:08.436265    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b65337-58b646495c-w8ldq" event={"ID":"c4cd43c6-1fe9-47ae-b24d-c9effce09fb8","Type":"ContainerDied","Data":"a4872d4814d5b81ef95c5c2f6b24765f80caa3e07e9107c9ed83269bbf0816c5"}
Sep 11 13:43:08.436573 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:43:08.436337    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b65337-58b646495c-w8ldq" event={"ID":"c4cd43c6-1fe9-47ae-b24d-c9effce09fb8","Type":"ContainerDied","Data":"0e04e6f27ce94147d9812bc36093d697991e74f540b07f48a141a32f86a34118"}
Sep 11 13:43:08.436573 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:43:08.436353    2578 scope.go:117] "RemoveContainer" containerID="a4872d4814d5b81ef95c5c2f6b24765f80caa3e07e9107c9ed83269bbf0816c5"
Sep 11 13:43:08.449248 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:43:08.449218    2578 scope.go:117] "RemoveContainer" containerID="a4872d4814d5b81ef95c5c2f6b24765f80caa3e07e9107c9ed83269bbf0816c5"
Sep 11 13:43:08.449745 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:43:08.449713    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4872d4814d5b81ef95c5c2f6b24765f80caa3e07e9107c9ed83269bbf0816c5\": container with ID starting with a4872d4814d5b81ef95c5c2f6b24765f80caa3e07e9107c9ed83269bbf0816c5 not found: ID does not exist" containerID="a4872d4814d5b81ef95c5c2f6b24765f80caa3e07e9107c9ed83269bbf0816c5"
Sep 11 13:43:08.449940 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:43:08.449753    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4872d4814d5b81ef95c5c2f6b24765f80caa3e07e9107c9ed83269bbf0816c5"} err="failed to get container status \"a4872d4814d5b81ef95c5c2f6b24765f80caa3e07e9107c9ed83269bbf0816c5\": rpc error: code = NotFound desc = could not find container \"a4872d4814d5b81ef95c5c2f6b24765f80caa3e07e9107c9ed83269bbf0816c5\": container with ID starting with a4872d4814d5b81ef95c5c2f6b24765f80caa3e07e9107c9ed83269bbf0816c5 not found: ID does not exist"
Sep 11 13:43:08.467299 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:43:08.467250    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b65337-58b646495c-w8ldq"]
Sep 11 13:43:08.472010 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:43:08.471967    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b65337-58b646495c-w8ldq"]
Sep 11 13:43:09.108123 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:43:09.108083    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c4cd43c6-1fe9-47ae-b24d-c9effce09fb8" path="/var/lib/kubelet/pods/c4cd43c6-1fe9-47ae-b24d-c9effce09fb8/volumes"
Sep 11 13:43:09.572173 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:43:09.572130    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c0f589fb-755b68d8bb-p4vnf"]
Sep 11 13:43:09.572784 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:43:09.572768    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4cd43c6-1fe9-47ae-b24d-c9effce09fb8" containerName="maas-api"
Sep 11 13:43:09.572875 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:43:09.572786    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4cd43c6-1fe9-47ae-b24d-c9effce09fb8" containerName="maas-api"
Sep 11 13:43:09.572921 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:43:09.572912    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="c4cd43c6-1fe9-47ae-b24d-c9effce09fb8" containerName="maas-api"
Sep 11 13:43:09.578999 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:43:09.578961    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c0f589fb-755b68d8bb-p4vnf"
Sep 11 13:43:09.582117 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:43:09.582081    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-c0f589fb-serving-cert\""
Sep 11 13:43:09.593019 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:43:09.592975    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c0f589fb-755b68d8bb-p4vnf"]
Sep 11 13:43:09.692505 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:43:09.692442    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/190cdc1b-8da3-497d-ab8b-9648eda088db-maas-api-tls\") pod \"maas-api-e2e-mt-c0f589fb-755b68d8bb-p4vnf\" (UID: \"190cdc1b-8da3-497d-ab8b-9648eda088db\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c0f589fb-755b68d8bb-p4vnf"
Sep 11 13:43:09.693006 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:43:09.692974    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/190cdc1b-8da3-497d-ab8b-9648eda088db-metrics-certs\") pod \"maas-api-e2e-mt-c0f589fb-755b68d8bb-p4vnf\" (UID: \"190cdc1b-8da3-497d-ab8b-9648eda088db\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c0f589fb-755b68d8bb-p4vnf"
Sep 11 13:43:09.693214 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:43:09.693016    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-674gt\" (UniqueName: \"kubernetes.io/projected/190cdc1b-8da3-497d-ab8b-9648eda088db-kube-api-access-674gt\") pod \"maas-api-e2e-mt-c0f589fb-755b68d8bb-p4vnf\" (UID: \"190cdc1b-8da3-497d-ab8b-9648eda088db\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c0f589fb-755b68d8bb-p4vnf"
Sep 11 13:43:09.693214 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:43:09.693100    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/190cdc1b-8da3-497d-ab8b-9648eda088db-openshift-service-ca\") pod \"maas-api-e2e-mt-c0f589fb-755b68d8bb-p4vnf\" (UID: \"190cdc1b-8da3-497d-ab8b-9648eda088db\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c0f589fb-755b68d8bb-p4vnf"
Sep 11 13:43:09.794247 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:43:09.794185    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/190cdc1b-8da3-497d-ab8b-9648eda088db-metrics-certs\") pod \"maas-api-e2e-mt-c0f589fb-755b68d8bb-p4vnf\" (UID: \"190cdc1b-8da3-497d-ab8b-9648eda088db\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c0f589fb-755b68d8bb-p4vnf"
Sep 11 13:43:09.794247 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:43:09.794241    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-674gt\" (UniqueName: \"kubernetes.io/projected/190cdc1b-8da3-497d-ab8b-9648eda088db-kube-api-access-674gt\") pod \"maas-api-e2e-mt-c0f589fb-755b68d8bb-p4vnf\" (UID: \"190cdc1b-8da3-497d-ab8b-9648eda088db\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c0f589fb-755b68d8bb-p4vnf"
Sep 11 13:43:09.794586 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:43:09.794291    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/190cdc1b-8da3-497d-ab8b-9648eda088db-openshift-service-ca\") pod \"maas-api-e2e-mt-c0f589fb-755b68d8bb-p4vnf\" (UID: \"190cdc1b-8da3-497d-ab8b-9648eda088db\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c0f589fb-755b68d8bb-p4vnf"
Sep 11 13:43:09.794586 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:43:09.794441    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/190cdc1b-8da3-497d-ab8b-9648eda088db-maas-api-tls\") pod \"maas-api-e2e-mt-c0f589fb-755b68d8bb-p4vnf\" (UID: \"190cdc1b-8da3-497d-ab8b-9648eda088db\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c0f589fb-755b68d8bb-p4vnf"
Sep 11 13:43:09.794586 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:43:09.794557    2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-c0f589fb-serving-cert: secret "maas-api-e2e-mt-c0f589fb-serving-cert" not found
Sep 11 13:43:09.794757 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:43:09.794623    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/190cdc1b-8da3-497d-ab8b-9648eda088db-maas-api-tls podName:190cdc1b-8da3-497d-ab8b-9648eda088db nodeName:}" failed. No retries permitted until 2026-09-11 13:43:10.294602676 +0000 UTC m=+1297.680705818 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/190cdc1b-8da3-497d-ab8b-9648eda088db-maas-api-tls") pod "maas-api-e2e-mt-c0f589fb-755b68d8bb-p4vnf" (UID: "190cdc1b-8da3-497d-ab8b-9648eda088db") : secret "maas-api-e2e-mt-c0f589fb-serving-cert" not found
Sep 11 13:43:09.795239 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:43:09.795210    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/190cdc1b-8da3-497d-ab8b-9648eda088db-openshift-service-ca\") pod \"maas-api-e2e-mt-c0f589fb-755b68d8bb-p4vnf\" (UID: \"190cdc1b-8da3-497d-ab8b-9648eda088db\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c0f589fb-755b68d8bb-p4vnf"
Sep 11 13:43:09.797578 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:43:09.797548    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/190cdc1b-8da3-497d-ab8b-9648eda088db-metrics-certs\") pod \"maas-api-e2e-mt-c0f589fb-755b68d8bb-p4vnf\" (UID: \"190cdc1b-8da3-497d-ab8b-9648eda088db\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c0f589fb-755b68d8bb-p4vnf"
Sep 11 13:43:09.806007 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:43:09.805968    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-674gt\" (UniqueName: \"kubernetes.io/projected/190cdc1b-8da3-497d-ab8b-9648eda088db-kube-api-access-674gt\") pod \"maas-api-e2e-mt-c0f589fb-755b68d8bb-p4vnf\" (UID: \"190cdc1b-8da3-497d-ab8b-9648eda088db\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c0f589fb-755b68d8bb-p4vnf"
Sep 11 13:43:10.302416 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:43:10.302350    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/190cdc1b-8da3-497d-ab8b-9648eda088db-maas-api-tls\") pod \"maas-api-e2e-mt-c0f589fb-755b68d8bb-p4vnf\" (UID: \"190cdc1b-8da3-497d-ab8b-9648eda088db\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c0f589fb-755b68d8bb-p4vnf"
Sep 11 13:43:10.305947 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:43:10.305911    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/190cdc1b-8da3-497d-ab8b-9648eda088db-maas-api-tls\") pod \"maas-api-e2e-mt-c0f589fb-755b68d8bb-p4vnf\" (UID: \"190cdc1b-8da3-497d-ab8b-9648eda088db\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c0f589fb-755b68d8bb-p4vnf"
Sep 11 13:43:10.497303 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:43:10.497246    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c0f589fb-755b68d8bb-p4vnf"
Sep 11 13:43:10.801673 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:43:10.801630    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c0f589fb-755b68d8bb-p4vnf"]
Sep 11 13:43:10.803864 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:43:10.803816    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod190cdc1b_8da3_497d_ab8b_9648eda088db.slice/crio-e8107fd91658583173ac1ffc0705bf67f70df24cd192417d2908591ceddd2140 WatchSource:0}: Error finding container e8107fd91658583173ac1ffc0705bf67f70df24cd192417d2908591ceddd2140: Status 404 returned error can't find the container with id e8107fd91658583173ac1ffc0705bf67f70df24cd192417d2908591ceddd2140
Sep 11 13:43:11.463155 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:43:11.463097    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c0f589fb-755b68d8bb-p4vnf" event={"ID":"190cdc1b-8da3-497d-ab8b-9648eda088db","Type":"ContainerStarted","Data":"2719a60189a5a0fb37a6d971e896bc1bfae00e55d11b8ab34192ecc5f75743d1"}
Sep 11 13:43:11.463874 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:43:11.463196    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c0f589fb-755b68d8bb-p4vnf" event={"ID":"190cdc1b-8da3-497d-ab8b-9648eda088db","Type":"ContainerStarted","Data":"e8107fd91658583173ac1ffc0705bf67f70df24cd192417d2908591ceddd2140"}
Sep 11 13:43:11.463874 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:43:11.463445    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c0f589fb-755b68d8bb-p4vnf"
Sep 11 13:43:11.496375 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:43:11.496314    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c0f589fb-755b68d8bb-p4vnf" podStartSLOduration=2.022543184 podStartE2EDuration="2.496293909s" totalImagesPullingTime="473.750725ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:43:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:43:11.496069538 +0000 UTC m=+1298.882172720" watchObservedRunningTime="2026-09-11 13:43:11.496293909 +0000 UTC m=+1298.882397100"
Sep 11 13:43:17.476638 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:43:17.476602    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c0f589fb-755b68d8bb-p4vnf"
Sep 11 13:44:00.625337 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:44:00.625206    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-a649d3-6f5f65cc57-kjbpx"]
Sep 11 13:44:00.627946 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:44:00.625604    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a649d3-6f5f65cc57-kjbpx" podUID="5b902823-ca13-41f9-90f5-42ceacbc5576" containerName="maas-api" containerID="cri-o://1d0586a212cb06bd49c1b568b03dfdd228119377db65e9f94b9eb6148b430692" gracePeriod=30
Sep 11 13:44:00.829322 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:44:00.829201    2578 generic.go:356] "Generic (PLEG): container finished" podID="5b902823-ca13-41f9-90f5-42ceacbc5576" containerID="1d0586a212cb06bd49c1b568b03dfdd228119377db65e9f94b9eb6148b430692" exitCode=0
Sep 11 13:44:00.829322 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:44:00.829267    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a649d3-6f5f65cc57-kjbpx" event={"ID":"5b902823-ca13-41f9-90f5-42ceacbc5576","Type":"ContainerDied","Data":"1d0586a212cb06bd49c1b568b03dfdd228119377db65e9f94b9eb6148b430692"}
Sep 11 13:44:00.932507 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:44:00.932461    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a649d3-6f5f65cc57-kjbpx"
Sep 11 13:44:01.056461 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:44:01.056408    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z8v8v\" (UniqueName: \"kubernetes.io/projected/5b902823-ca13-41f9-90f5-42ceacbc5576-kube-api-access-z8v8v\") pod \"5b902823-ca13-41f9-90f5-42ceacbc5576\" (UID: \"5b902823-ca13-41f9-90f5-42ceacbc5576\") "
Sep 11 13:44:01.056729 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:44:01.056474    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b902823-ca13-41f9-90f5-42ceacbc5576-maas-api-tls\") pod \"5b902823-ca13-41f9-90f5-42ceacbc5576\" (UID: \"5b902823-ca13-41f9-90f5-42ceacbc5576\") "
Sep 11 13:44:01.056729 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:44:01.056507    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b902823-ca13-41f9-90f5-42ceacbc5576-metrics-certs\") pod \"5b902823-ca13-41f9-90f5-42ceacbc5576\" (UID: \"5b902823-ca13-41f9-90f5-42ceacbc5576\") "
Sep 11 13:44:01.056729 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:44:01.056670    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b902823-ca13-41f9-90f5-42ceacbc5576-openshift-service-ca\") pod \"5b902823-ca13-41f9-90f5-42ceacbc5576\" (UID: \"5b902823-ca13-41f9-90f5-42ceacbc5576\") "
Sep 11 13:44:01.057252 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:44:01.057212    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b902823-ca13-41f9-90f5-42ceacbc5576-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5b902823-ca13-41f9-90f5-42ceacbc5576" (UID: "5b902823-ca13-41f9-90f5-42ceacbc5576"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 13:44:01.059286 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:44:01.059241    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b902823-ca13-41f9-90f5-42ceacbc5576-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "5b902823-ca13-41f9-90f5-42ceacbc5576" (UID: "5b902823-ca13-41f9-90f5-42ceacbc5576"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 13:44:01.059621 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:44:01.059595    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b902823-ca13-41f9-90f5-42ceacbc5576-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5b902823-ca13-41f9-90f5-42ceacbc5576" (UID: "5b902823-ca13-41f9-90f5-42ceacbc5576"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 13:44:01.059685 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:44:01.059634    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b902823-ca13-41f9-90f5-42ceacbc5576-kube-api-access-z8v8v" (OuterVolumeSpecName: "kube-api-access-z8v8v") pod "5b902823-ca13-41f9-90f5-42ceacbc5576" (UID: "5b902823-ca13-41f9-90f5-42ceacbc5576"). InnerVolumeSpecName "kube-api-access-z8v8v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 13:44:01.158352 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:44:01.158238    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8v8v\" (UniqueName: \"kubernetes.io/projected/5b902823-ca13-41f9-90f5-42ceacbc5576-kube-api-access-z8v8v\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:44:01.158352 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:44:01.158277    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b902823-ca13-41f9-90f5-42ceacbc5576-maas-api-tls\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:44:01.158352 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:44:01.158291    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b902823-ca13-41f9-90f5-42ceacbc5576-metrics-certs\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:44:01.158352 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:44:01.158305    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b902823-ca13-41f9-90f5-42ceacbc5576-openshift-service-ca\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:44:01.837716 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:44:01.837673    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a649d3-6f5f65cc57-kjbpx" event={"ID":"5b902823-ca13-41f9-90f5-42ceacbc5576","Type":"ContainerDied","Data":"391d758e776169cd8fd315f9b1407318e2ce11539a75a368a0b13e2c491fb15e"}
Sep 11 13:44:01.837716 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:44:01.837705    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a649d3-6f5f65cc57-kjbpx"
Sep 11 13:44:01.838385 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:44:01.837748    2578 scope.go:117] "RemoveContainer" containerID="1d0586a212cb06bd49c1b568b03dfdd228119377db65e9f94b9eb6148b430692"
Sep 11 13:44:01.880596 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:44:01.880555    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-a649d3-6f5f65cc57-kjbpx"]
Sep 11 13:44:01.897304 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:44:01.897255    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-a649d3-6f5f65cc57-kjbpx"]
Sep 11 13:44:03.111146 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:44:03.111102    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5b902823-ca13-41f9-90f5-42ceacbc5576" path="/var/lib/kubelet/pods/5b902823-ca13-41f9-90f5-42ceacbc5576/volumes"
Sep 11 13:44:20.280543 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:44:20.280485    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0317c3-5544cf86cf-ps7g5"]
Sep 11 13:44:20.282403 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:44:20.282365    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b902823-ca13-41f9-90f5-42ceacbc5576" containerName="maas-api"
Sep 11 13:44:20.282403 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:44:20.282400    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b902823-ca13-41f9-90f5-42ceacbc5576" containerName="maas-api"
Sep 11 13:44:20.282678 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:44:20.282509    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b902823-ca13-41f9-90f5-42ceacbc5576" containerName="maas-api"
Sep 11 13:44:20.287736 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:44:20.287699    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0317c3-5544cf86cf-ps7g5"
Sep 11 13:44:20.290912 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:44:20.290876    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-0317c3-serving-cert\""
Sep 11 13:44:20.301160 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:44:20.301121    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0317c3-5544cf86cf-ps7g5"]
Sep 11 13:44:20.397724 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:44:20.397672    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99c8676f-2468-4393-ac09-bd0ebe7e185b-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-0317c3-5544cf86cf-ps7g5\" (UID: \"99c8676f-2468-4393-ac09-bd0ebe7e185b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0317c3-5544cf86cf-ps7g5"
Sep 11 13:44:20.398018 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:44:20.397924    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p29rl\" (UniqueName: \"kubernetes.io/projected/99c8676f-2468-4393-ac09-bd0ebe7e185b-kube-api-access-p29rl\") pod \"maas-api-e2e-shared-a-w3-0317c3-5544cf86cf-ps7g5\" (UID: \"99c8676f-2468-4393-ac09-bd0ebe7e185b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0317c3-5544cf86cf-ps7g5"
Sep 11 13:44:20.398018 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:44:20.398009    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/99c8676f-2468-4393-ac09-bd0ebe7e185b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-0317c3-5544cf86cf-ps7g5\" (UID: \"99c8676f-2468-4393-ac09-bd0ebe7e185b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0317c3-5544cf86cf-ps7g5"
Sep 11 13:44:20.398158 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:44:20.398072    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/99c8676f-2468-4393-ac09-bd0ebe7e185b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-0317c3-5544cf86cf-ps7g5\" (UID: \"99c8676f-2468-4393-ac09-bd0ebe7e185b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0317c3-5544cf86cf-ps7g5"
Sep 11 13:44:20.499282 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:44:20.499236    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99c8676f-2468-4393-ac09-bd0ebe7e185b-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-0317c3-5544cf86cf-ps7g5\" (UID: \"99c8676f-2468-4393-ac09-bd0ebe7e185b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0317c3-5544cf86cf-ps7g5"
Sep 11 13:44:20.499537 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:44:20.499332    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p29rl\" (UniqueName: \"kubernetes.io/projected/99c8676f-2468-4393-ac09-bd0ebe7e185b-kube-api-access-p29rl\") pod \"maas-api-e2e-shared-a-w3-0317c3-5544cf86cf-ps7g5\" (UID: \"99c8676f-2468-4393-ac09-bd0ebe7e185b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0317c3-5544cf86cf-ps7g5"
Sep 11 13:44:20.499537 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:44:20.499369    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/99c8676f-2468-4393-ac09-bd0ebe7e185b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-0317c3-5544cf86cf-ps7g5\" (UID: \"99c8676f-2468-4393-ac09-bd0ebe7e185b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0317c3-5544cf86cf-ps7g5"
Sep 11 13:44:20.499537 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:44:20.499413    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/99c8676f-2468-4393-ac09-bd0ebe7e185b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-0317c3-5544cf86cf-ps7g5\" (UID: \"99c8676f-2468-4393-ac09-bd0ebe7e185b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0317c3-5544cf86cf-ps7g5"
Sep 11 13:44:20.499728 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:44:20.499559    2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0317c3-serving-cert: secret "maas-api-e2e-shared-a-w3-0317c3-serving-cert" not found
Sep 11 13:44:20.499728 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:44:20.499640    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99c8676f-2468-4393-ac09-bd0ebe7e185b-maas-api-tls podName:99c8676f-2468-4393-ac09-bd0ebe7e185b nodeName:}" failed. No retries permitted until 2026-09-11 13:44:20.999618037 +0000 UTC m=+1368.385721179 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/99c8676f-2468-4393-ac09-bd0ebe7e185b-maas-api-tls") pod "maas-api-e2e-shared-a-w3-0317c3-5544cf86cf-ps7g5" (UID: "99c8676f-2468-4393-ac09-bd0ebe7e185b") : secret "maas-api-e2e-shared-a-w3-0317c3-serving-cert" not found
Sep 11 13:44:20.500227 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:44:20.500199    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/99c8676f-2468-4393-ac09-bd0ebe7e185b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-0317c3-5544cf86cf-ps7g5\" (UID: \"99c8676f-2468-4393-ac09-bd0ebe7e185b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0317c3-5544cf86cf-ps7g5"
Sep 11 13:44:20.502472 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:44:20.502446    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99c8676f-2468-4393-ac09-bd0ebe7e185b-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-0317c3-5544cf86cf-ps7g5\" (UID: \"99c8676f-2468-4393-ac09-bd0ebe7e185b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0317c3-5544cf86cf-ps7g5"
Sep 11 13:44:20.508611 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:44:20.508577    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p29rl\" (UniqueName: \"kubernetes.io/projected/99c8676f-2468-4393-ac09-bd0ebe7e185b-kube-api-access-p29rl\") pod \"maas-api-e2e-shared-a-w3-0317c3-5544cf86cf-ps7g5\" (UID: \"99c8676f-2468-4393-ac09-bd0ebe7e185b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0317c3-5544cf86cf-ps7g5"
Sep 11 13:44:21.005693 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:44:21.005639    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/99c8676f-2468-4393-ac09-bd0ebe7e185b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-0317c3-5544cf86cf-ps7g5\" (UID: \"99c8676f-2468-4393-ac09-bd0ebe7e185b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0317c3-5544cf86cf-ps7g5"
Sep 11 13:44:21.008847 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:44:21.008783    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/99c8676f-2468-4393-ac09-bd0ebe7e185b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-0317c3-5544cf86cf-ps7g5\" (UID: \"99c8676f-2468-4393-ac09-bd0ebe7e185b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0317c3-5544cf86cf-ps7g5"
Sep 11 13:44:21.205672 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:44:21.205619    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0317c3-5544cf86cf-ps7g5"
Sep 11 13:44:21.513161 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:44:21.513112    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0317c3-5544cf86cf-ps7g5"]
Sep 11 13:44:21.515443 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:44:21.515408    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99c8676f_2468_4393_ac09_bd0ebe7e185b.slice/crio-d80b7ccf6f4f1ddb7731f71f51f9167b8906824d331f5e6013861bc6c2b2e410 WatchSource:0}: Error finding container d80b7ccf6f4f1ddb7731f71f51f9167b8906824d331f5e6013861bc6c2b2e410: Status 404 returned error can't find the container with id d80b7ccf6f4f1ddb7731f71f51f9167b8906824d331f5e6013861bc6c2b2e410
Sep 11 13:44:21.517104 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:44:21.517074    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 13:44:21.985476 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:44:21.985435    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0317c3-5544cf86cf-ps7g5" event={"ID":"99c8676f-2468-4393-ac09-bd0ebe7e185b","Type":"ContainerStarted","Data":"9b996073196cb74bd5f8f0b6825ce43d8b868f242edfcb9ad96af5f4bd59cd8b"}
Sep 11 13:44:21.985698 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:44:21.985667    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0317c3-5544cf86cf-ps7g5"
Sep 11 13:44:21.985775 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:44:21.985713    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0317c3-5544cf86cf-ps7g5" event={"ID":"99c8676f-2468-4393-ac09-bd0ebe7e185b","Type":"ContainerStarted","Data":"d80b7ccf6f4f1ddb7731f71f51f9167b8906824d331f5e6013861bc6c2b2e410"}
Sep 11 13:44:22.005509 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:44:22.005442    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0317c3-5544cf86cf-ps7g5" podStartSLOduration=1.648629604 podStartE2EDuration="2.00541918s" totalImagesPullingTime="356.789576ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:44:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:44:22.003733858 +0000 UTC m=+1369.389837041" watchObservedRunningTime="2026-09-11 13:44:22.00541918 +0000 UTC m=+1369.391522390"
Sep 11 13:44:28.000725 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:44:28.000682    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0317c3-5544cf86cf-ps7g5"
Sep 11 13:44:51.800947 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:44:51.800889    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-9edb53-69dbb74897-9w4g8"]
Sep 11 13:44:51.806941 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:44:51.806908    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-9edb53-69dbb74897-9w4g8"
Sep 11 13:44:51.810264 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:44:51.810229    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-9edb53-serving-cert\""
Sep 11 13:44:51.821909 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:44:51.821868    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-9edb53-69dbb74897-9w4g8"]
Sep 11 13:44:51.829247 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:44:51.829195    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06530615-4817-4218-9591-7e2eb7dea6a0-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-9edb53-69dbb74897-9w4g8\" (UID: \"06530615-4817-4218-9591-7e2eb7dea6a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-9edb53-69dbb74897-9w4g8"
Sep 11 13:44:51.829456 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:44:51.829371    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87fbl\" (UniqueName: \"kubernetes.io/projected/06530615-4817-4218-9591-7e2eb7dea6a0-kube-api-access-87fbl\") pod \"maas-api-e2e-shared-b-w3-9edb53-69dbb74897-9w4g8\" (UID: \"06530615-4817-4218-9591-7e2eb7dea6a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-9edb53-69dbb74897-9w4g8"
Sep 11 13:44:51.829522 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:44:51.829480    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06530615-4817-4218-9591-7e2eb7dea6a0-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-9edb53-69dbb74897-9w4g8\" (UID: \"06530615-4817-4218-9591-7e2eb7dea6a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-9edb53-69dbb74897-9w4g8"
Sep 11 13:44:51.829575 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:44:51.829561    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06530615-4817-4218-9591-7e2eb7dea6a0-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-9edb53-69dbb74897-9w4g8\" (UID: \"06530615-4817-4218-9591-7e2eb7dea6a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-9edb53-69dbb74897-9w4g8"
Sep 11 13:44:51.930548 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:44:51.930494    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06530615-4817-4218-9591-7e2eb7dea6a0-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-9edb53-69dbb74897-9w4g8\" (UID: \"06530615-4817-4218-9591-7e2eb7dea6a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-9edb53-69dbb74897-9w4g8"
Sep 11 13:44:51.930780 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:44:51.930574    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87fbl\" (UniqueName: \"kubernetes.io/projected/06530615-4817-4218-9591-7e2eb7dea6a0-kube-api-access-87fbl\") pod \"maas-api-e2e-shared-b-w3-9edb53-69dbb74897-9w4g8\" (UID: \"06530615-4817-4218-9591-7e2eb7dea6a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-9edb53-69dbb74897-9w4g8"
Sep 11 13:44:51.930780 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:44:51.930621    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06530615-4817-4218-9591-7e2eb7dea6a0-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-9edb53-69dbb74897-9w4g8\" (UID: \"06530615-4817-4218-9591-7e2eb7dea6a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-9edb53-69dbb74897-9w4g8"
Sep 11 13:44:51.930780 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:44:51.930658    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06530615-4817-4218-9591-7e2eb7dea6a0-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-9edb53-69dbb74897-9w4g8\" (UID: \"06530615-4817-4218-9591-7e2eb7dea6a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-9edb53-69dbb74897-9w4g8"
Sep 11 13:44:51.930780 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:44:51.930754    2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-9edb53-serving-cert: secret "maas-api-e2e-shared-b-w3-9edb53-serving-cert" not found
Sep 11 13:44:51.930974 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:44:51.930819    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06530615-4817-4218-9591-7e2eb7dea6a0-maas-api-tls podName:06530615-4817-4218-9591-7e2eb7dea6a0 nodeName:}" failed. No retries permitted until 2026-09-11 13:44:52.430801711 +0000 UTC m=+1399.816904853 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/06530615-4817-4218-9591-7e2eb7dea6a0-maas-api-tls") pod "maas-api-e2e-shared-b-w3-9edb53-69dbb74897-9w4g8" (UID: "06530615-4817-4218-9591-7e2eb7dea6a0") : secret "maas-api-e2e-shared-b-w3-9edb53-serving-cert" not found
Sep 11 13:44:51.931399 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:44:51.931367    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06530615-4817-4218-9591-7e2eb7dea6a0-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-9edb53-69dbb74897-9w4g8\" (UID: \"06530615-4817-4218-9591-7e2eb7dea6a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-9edb53-69dbb74897-9w4g8"
Sep 11 13:44:51.933445 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:44:51.933405    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06530615-4817-4218-9591-7e2eb7dea6a0-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-9edb53-69dbb74897-9w4g8\" (UID: \"06530615-4817-4218-9591-7e2eb7dea6a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-9edb53-69dbb74897-9w4g8"
Sep 11 13:44:51.940625 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:44:51.940578    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-87fbl\" (UniqueName: \"kubernetes.io/projected/06530615-4817-4218-9591-7e2eb7dea6a0-kube-api-access-87fbl\") pod \"maas-api-e2e-shared-b-w3-9edb53-69dbb74897-9w4g8\" (UID: \"06530615-4817-4218-9591-7e2eb7dea6a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-9edb53-69dbb74897-9w4g8"
Sep 11 13:44:52.437612 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:44:52.437562    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06530615-4817-4218-9591-7e2eb7dea6a0-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-9edb53-69dbb74897-9w4g8\" (UID: \"06530615-4817-4218-9591-7e2eb7dea6a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-9edb53-69dbb74897-9w4g8"
Sep 11 13:44:52.440426 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:44:52.440401    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06530615-4817-4218-9591-7e2eb7dea6a0-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-9edb53-69dbb74897-9w4g8\" (UID: \"06530615-4817-4218-9591-7e2eb7dea6a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-9edb53-69dbb74897-9w4g8"
Sep 11 13:44:52.721773 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:44:52.721733    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-9edb53-69dbb74897-9w4g8"
Sep 11 13:44:53.029404 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:44:53.029358    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-9edb53-69dbb74897-9w4g8"]
Sep 11 13:44:53.032436 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:44:53.032389    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06530615_4817_4218_9591_7e2eb7dea6a0.slice/crio-d6c565feb7cd2cf285905df1f08e3f12f4625021d49e50a513cfc85354ab392f WatchSource:0}: Error finding container d6c565feb7cd2cf285905df1f08e3f12f4625021d49e50a513cfc85354ab392f: Status 404 returned error can't find the container with id d6c565feb7cd2cf285905df1f08e3f12f4625021d49e50a513cfc85354ab392f
Sep 11 13:44:53.210518 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:44:53.210478    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-9edb53-69dbb74897-9w4g8" event={"ID":"06530615-4817-4218-9591-7e2eb7dea6a0","Type":"ContainerStarted","Data":"d6c565feb7cd2cf285905df1f08e3f12f4625021d49e50a513cfc85354ab392f"}
Sep 11 13:44:54.220197 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:44:54.220136    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-9edb53-69dbb74897-9w4g8" event={"ID":"06530615-4817-4218-9591-7e2eb7dea6a0","Type":"ContainerStarted","Data":"c3caa65a0e473877bcd3247f6c4bf7eb3aa6fa4074c6a98878aeb3356c78badb"}
Sep 11 13:44:54.220848 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:44:54.220323    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-9edb53-69dbb74897-9w4g8"
Sep 11 13:44:54.241501 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:44:54.241436    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-9edb53-69dbb74897-9w4g8" podStartSLOduration=2.905149516 podStartE2EDuration="3.24141018s" totalImagesPullingTime="336.260664ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:44:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:44:54.239952143 +0000 UTC m=+1401.626055329" watchObservedRunningTime="2026-09-11 13:44:54.24141018 +0000 UTC m=+1401.627513344"
Sep 11 13:45:00.234647 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:45:00.234597    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-9edb53-69dbb74897-9w4g8"
Sep 11 13:46:16.177624 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:46:16.177573    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c0f589fb-755b68d8bb-p4vnf"]
Sep 11 13:46:16.178363 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:46:16.177901    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c0f589fb-755b68d8bb-p4vnf" podUID="190cdc1b-8da3-497d-ab8b-9648eda088db" containerName="maas-api" containerID="cri-o://2719a60189a5a0fb37a6d971e896bc1bfae00e55d11b8ab34192ecc5f75743d1" gracePeriod=30
Sep 11 13:46:16.498740 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:46:16.498701    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c0f589fb-755b68d8bb-p4vnf"
Sep 11 13:46:16.646356 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:46:16.646292    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/190cdc1b-8da3-497d-ab8b-9648eda088db-maas-api-tls\") pod \"190cdc1b-8da3-497d-ab8b-9648eda088db\" (UID: \"190cdc1b-8da3-497d-ab8b-9648eda088db\") "
Sep 11 13:46:16.646670 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:46:16.646560    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/190cdc1b-8da3-497d-ab8b-9648eda088db-openshift-service-ca\") pod \"190cdc1b-8da3-497d-ab8b-9648eda088db\" (UID: \"190cdc1b-8da3-497d-ab8b-9648eda088db\") "
Sep 11 13:46:16.646670 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:46:16.646617    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-674gt\" (UniqueName: \"kubernetes.io/projected/190cdc1b-8da3-497d-ab8b-9648eda088db-kube-api-access-674gt\") pod \"190cdc1b-8da3-497d-ab8b-9648eda088db\" (UID: \"190cdc1b-8da3-497d-ab8b-9648eda088db\") "
Sep 11 13:46:16.646670 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:46:16.646663    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/190cdc1b-8da3-497d-ab8b-9648eda088db-metrics-certs\") pod \"190cdc1b-8da3-497d-ab8b-9648eda088db\" (UID: \"190cdc1b-8da3-497d-ab8b-9648eda088db\") "
Sep 11 13:46:16.647017 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:46:16.646967    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/190cdc1b-8da3-497d-ab8b-9648eda088db-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "190cdc1b-8da3-497d-ab8b-9648eda088db" (UID: "190cdc1b-8da3-497d-ab8b-9648eda088db"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 13:46:16.647392 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:46:16.647360    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/190cdc1b-8da3-497d-ab8b-9648eda088db-openshift-service-ca\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:46:16.649327 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:46:16.649300    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/190cdc1b-8da3-497d-ab8b-9648eda088db-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "190cdc1b-8da3-497d-ab8b-9648eda088db" (UID: "190cdc1b-8da3-497d-ab8b-9648eda088db"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 13:46:16.649404 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:46:16.649317    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/190cdc1b-8da3-497d-ab8b-9648eda088db-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "190cdc1b-8da3-497d-ab8b-9648eda088db" (UID: "190cdc1b-8da3-497d-ab8b-9648eda088db"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 13:46:16.649404 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:46:16.649331    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/190cdc1b-8da3-497d-ab8b-9648eda088db-kube-api-access-674gt" (OuterVolumeSpecName: "kube-api-access-674gt") pod "190cdc1b-8da3-497d-ab8b-9648eda088db" (UID: "190cdc1b-8da3-497d-ab8b-9648eda088db"). InnerVolumeSpecName "kube-api-access-674gt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 13:46:16.748595 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:46:16.748464    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-674gt\" (UniqueName: \"kubernetes.io/projected/190cdc1b-8da3-497d-ab8b-9648eda088db-kube-api-access-674gt\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:46:16.748595 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:46:16.748511    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/190cdc1b-8da3-497d-ab8b-9648eda088db-metrics-certs\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:46:16.748595 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:46:16.748521    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/190cdc1b-8da3-497d-ab8b-9648eda088db-maas-api-tls\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:46:16.822033 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:46:16.821985    2578 generic.go:356] "Generic (PLEG): container finished" podID="190cdc1b-8da3-497d-ab8b-9648eda088db" containerID="2719a60189a5a0fb37a6d971e896bc1bfae00e55d11b8ab34192ecc5f75743d1" exitCode=0
Sep 11 13:46:16.822286 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:46:16.822054    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c0f589fb-755b68d8bb-p4vnf"
Sep 11 13:46:16.822286 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:46:16.822073    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c0f589fb-755b68d8bb-p4vnf" event={"ID":"190cdc1b-8da3-497d-ab8b-9648eda088db","Type":"ContainerDied","Data":"2719a60189a5a0fb37a6d971e896bc1bfae00e55d11b8ab34192ecc5f75743d1"}
Sep 11 13:46:16.822286 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:46:16.822156    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c0f589fb-755b68d8bb-p4vnf" event={"ID":"190cdc1b-8da3-497d-ab8b-9648eda088db","Type":"ContainerDied","Data":"e8107fd91658583173ac1ffc0705bf67f70df24cd192417d2908591ceddd2140"}
Sep 11 13:46:16.822286 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:46:16.822187    2578 scope.go:117] "RemoveContainer" containerID="2719a60189a5a0fb37a6d971e896bc1bfae00e55d11b8ab34192ecc5f75743d1"
Sep 11 13:46:16.835199 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:46:16.835171    2578 scope.go:117] "RemoveContainer" containerID="2719a60189a5a0fb37a6d971e896bc1bfae00e55d11b8ab34192ecc5f75743d1"
Sep 11 13:46:16.835601 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:46:16.835574    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2719a60189a5a0fb37a6d971e896bc1bfae00e55d11b8ab34192ecc5f75743d1\": container with ID starting with 2719a60189a5a0fb37a6d971e896bc1bfae00e55d11b8ab34192ecc5f75743d1 not found: ID does not exist" containerID="2719a60189a5a0fb37a6d971e896bc1bfae00e55d11b8ab34192ecc5f75743d1"
Sep 11 13:46:16.835685 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:46:16.835621    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2719a60189a5a0fb37a6d971e896bc1bfae00e55d11b8ab34192ecc5f75743d1"} err="failed to get container status \"2719a60189a5a0fb37a6d971e896bc1bfae00e55d11b8ab34192ecc5f75743d1\": rpc error: code = NotFound desc = could not find container \"2719a60189a5a0fb37a6d971e896bc1bfae00e55d11b8ab34192ecc5f75743d1\": container with ID starting with 2719a60189a5a0fb37a6d971e896bc1bfae00e55d11b8ab34192ecc5f75743d1 not found: ID does not exist"
Sep 11 13:46:16.850910 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:46:16.850871    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c0f589fb-755b68d8bb-p4vnf"]
Sep 11 13:46:16.853773 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:46:16.853740    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c0f589fb-755b68d8bb-p4vnf"]
Sep 11 13:46:17.109298 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:46:17.109194    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="190cdc1b-8da3-497d-ab8b-9648eda088db" path="/var/lib/kubelet/pods/190cdc1b-8da3-497d-ab8b-9648eda088db/volumes"
Sep 11 13:46:33.214277 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:46:33.214230    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6gltj_e384caae-3cb1-4264-8a25-55b40ac8ac45/ovn-acl-logging/0.log"
Sep 11 13:46:33.217743 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:46:33.217710    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6gltj_e384caae-3cb1-4264-8a25-55b40ac8ac45/ovn-acl-logging/0.log"
Sep 11 13:46:46.549622 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:46:46.549569    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-371c38/e2e-conflict-a-f8ce9a34-kserve-67b74b49b4-p52gz"]
Sep 11 13:47:01.572633 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:47:01.572427    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-371c38/e2e-conflict-a-f8ce9a34-kserve-67b74b49b4-p52gz" podUID="02db85b9-8b34-40e1-9518-d60e3971228f" containerName="main" containerID="cri-o://748a3296b9ded3fcd1a65d30785560e02cbe35e69ee275f06a5043e837e4271d" gracePeriod=45
Sep 11 13:47:01.856270 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:47:01.856233    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-371c38/e2e-conflict-a-f8ce9a34-kserve-67b74b49b4-p52gz"
Sep 11 13:47:01.938412 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:47:01.938358    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lstns\" (UniqueName: \"kubernetes.io/projected/02db85b9-8b34-40e1-9518-d60e3971228f-kube-api-access-lstns\") pod \"02db85b9-8b34-40e1-9518-d60e3971228f\" (UID: \"02db85b9-8b34-40e1-9518-d60e3971228f\") "
Sep 11 13:47:01.938700 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:47:01.938425    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/02db85b9-8b34-40e1-9518-d60e3971228f-dshm\") pod \"02db85b9-8b34-40e1-9518-d60e3971228f\" (UID: \"02db85b9-8b34-40e1-9518-d60e3971228f\") "
Sep 11 13:47:01.938700 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:47:01.938466    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02db85b9-8b34-40e1-9518-d60e3971228f-model-cache\") pod \"02db85b9-8b34-40e1-9518-d60e3971228f\" (UID: \"02db85b9-8b34-40e1-9518-d60e3971228f\") "
Sep 11 13:47:01.938700 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:47:01.938500    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02db85b9-8b34-40e1-9518-d60e3971228f-tls-certs\") pod \"02db85b9-8b34-40e1-9518-d60e3971228f\" (UID: \"02db85b9-8b34-40e1-9518-d60e3971228f\") "
Sep 11 13:47:01.938700 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:47:01.938566    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/02db85b9-8b34-40e1-9518-d60e3971228f-home\") pod \"02db85b9-8b34-40e1-9518-d60e3971228f\" (UID: \"02db85b9-8b34-40e1-9518-d60e3971228f\") "
Sep 11 13:47:01.938961 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:47:01.938742    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02db85b9-8b34-40e1-9518-d60e3971228f-tmp-dir\") pod \"02db85b9-8b34-40e1-9518-d60e3971228f\" (UID: \"02db85b9-8b34-40e1-9518-d60e3971228f\") "
Sep 11 13:47:01.938961 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:47:01.938768    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02db85b9-8b34-40e1-9518-d60e3971228f-model-cache" (OuterVolumeSpecName: "model-cache") pod "02db85b9-8b34-40e1-9518-d60e3971228f" (UID: "02db85b9-8b34-40e1-9518-d60e3971228f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 13:47:01.939060 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:47:01.938979    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02db85b9-8b34-40e1-9518-d60e3971228f-home" (OuterVolumeSpecName: "home") pod "02db85b9-8b34-40e1-9518-d60e3971228f" (UID: "02db85b9-8b34-40e1-9518-d60e3971228f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 13:47:01.939122 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:47:01.939088    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02db85b9-8b34-40e1-9518-d60e3971228f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "02db85b9-8b34-40e1-9518-d60e3971228f" (UID: "02db85b9-8b34-40e1-9518-d60e3971228f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 13:47:01.939684 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:47:01.939659    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02db85b9-8b34-40e1-9518-d60e3971228f-model-cache\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:47:01.939777 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:47:01.939691    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/02db85b9-8b34-40e1-9518-d60e3971228f-home\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:47:01.939777 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:47:01.939706    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02db85b9-8b34-40e1-9518-d60e3971228f-tmp-dir\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:47:01.941486 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:47:01.941448    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02db85b9-8b34-40e1-9518-d60e3971228f-dshm" (OuterVolumeSpecName: "dshm") pod "02db85b9-8b34-40e1-9518-d60e3971228f" (UID: "02db85b9-8b34-40e1-9518-d60e3971228f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 13:47:01.941621 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:47:01.941486    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02db85b9-8b34-40e1-9518-d60e3971228f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "02db85b9-8b34-40e1-9518-d60e3971228f" (UID: "02db85b9-8b34-40e1-9518-d60e3971228f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 13:47:01.941621 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:47:01.941497    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02db85b9-8b34-40e1-9518-d60e3971228f-kube-api-access-lstns" (OuterVolumeSpecName: "kube-api-access-lstns") pod "02db85b9-8b34-40e1-9518-d60e3971228f" (UID: "02db85b9-8b34-40e1-9518-d60e3971228f"). InnerVolumeSpecName "kube-api-access-lstns". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 13:47:02.041412 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:47:02.041339    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lstns\" (UniqueName: \"kubernetes.io/projected/02db85b9-8b34-40e1-9518-d60e3971228f-kube-api-access-lstns\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:47:02.041412 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:47:02.041400    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/02db85b9-8b34-40e1-9518-d60e3971228f-dshm\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:47:02.041412 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:47:02.041413    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02db85b9-8b34-40e1-9518-d60e3971228f-tls-certs\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:47:02.144529 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:47:02.144401    2578 generic.go:356] "Generic (PLEG): container finished" podID="02db85b9-8b34-40e1-9518-d60e3971228f" containerID="748a3296b9ded3fcd1a65d30785560e02cbe35e69ee275f06a5043e837e4271d" exitCode=0
Sep 11 13:47:02.144529 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:47:02.144459    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-371c38/e2e-conflict-a-f8ce9a34-kserve-67b74b49b4-p52gz"
Sep 11 13:47:02.144862 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:47:02.144458    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-371c38/e2e-conflict-a-f8ce9a34-kserve-67b74b49b4-p52gz" event={"ID":"02db85b9-8b34-40e1-9518-d60e3971228f","Type":"ContainerDied","Data":"748a3296b9ded3fcd1a65d30785560e02cbe35e69ee275f06a5043e837e4271d"}
Sep 11 13:47:02.144862 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:47:02.144583    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-371c38/e2e-conflict-a-f8ce9a34-kserve-67b74b49b4-p52gz" event={"ID":"02db85b9-8b34-40e1-9518-d60e3971228f","Type":"ContainerDied","Data":"b4db58ea58d632f9f22681ef229b89bb92fb6aa525d2e59869a633906201c0d9"}
Sep 11 13:47:02.144862 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:47:02.144613    2578 scope.go:117] "RemoveContainer" containerID="748a3296b9ded3fcd1a65d30785560e02cbe35e69ee275f06a5043e837e4271d"
Sep 11 13:47:02.157177 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:47:02.157153    2578 scope.go:117] "RemoveContainer" containerID="748a3296b9ded3fcd1a65d30785560e02cbe35e69ee275f06a5043e837e4271d"
Sep 11 13:47:02.157630 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:47:02.157605    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"748a3296b9ded3fcd1a65d30785560e02cbe35e69ee275f06a5043e837e4271d\": container with ID starting with 748a3296b9ded3fcd1a65d30785560e02cbe35e69ee275f06a5043e837e4271d not found: ID does not exist" containerID="748a3296b9ded3fcd1a65d30785560e02cbe35e69ee275f06a5043e837e4271d"
Sep 11 13:47:02.157683 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:47:02.157645    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"748a3296b9ded3fcd1a65d30785560e02cbe35e69ee275f06a5043e837e4271d"} err="failed to get container status \"748a3296b9ded3fcd1a65d30785560e02cbe35e69ee275f06a5043e837e4271d\": rpc error: code = NotFound desc = could not find container \"748a3296b9ded3fcd1a65d30785560e02cbe35e69ee275f06a5043e837e4271d\": container with ID starting with 748a3296b9ded3fcd1a65d30785560e02cbe35e69ee275f06a5043e837e4271d not found: ID does not exist"
Sep 11 13:47:02.175341 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:47:02.175304    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-371c38/e2e-conflict-a-f8ce9a34-kserve-67b74b49b4-p52gz"]
Sep 11 13:47:02.181138 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:47:02.181099    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-371c38/e2e-conflict-a-f8ce9a34-kserve-67b74b49b4-p52gz"]
Sep 11 13:47:03.110439 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:47:03.110391    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="02db85b9-8b34-40e1-9518-d60e3971228f" path="/var/lib/kubelet/pods/02db85b9-8b34-40e1-9518-d60e3971228f/volumes"
Sep 11 13:47:21.073799 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:47:21.073757    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-9edb53/auth-test-model-9edb53-kserve-5d97c5f9c9-nmzxg"]
Sep 11 13:47:21.074500 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:47:21.074483    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02db85b9-8b34-40e1-9518-d60e3971228f" containerName="main"
Sep 11 13:47:21.074559 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:47:21.074504    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="02db85b9-8b34-40e1-9518-d60e3971228f" containerName="main"
Sep 11 13:47:21.074559 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:47:21.074521    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="190cdc1b-8da3-497d-ab8b-9648eda088db" containerName="maas-api"
Sep 11 13:47:21.074559 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:47:21.074526    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="190cdc1b-8da3-497d-ab8b-9648eda088db" containerName="maas-api"
Sep 11 13:47:21.074656 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:47:21.074619    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="02db85b9-8b34-40e1-9518-d60e3971228f" containerName="main"
Sep 11 13:47:21.074656 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:47:21.074632    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="190cdc1b-8da3-497d-ab8b-9648eda088db" containerName="maas-api"
Sep 11 13:47:21.079986 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:47:21.079957    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-9edb53/auth-test-model-9edb53-kserve-5d97c5f9c9-nmzxg"
Sep 11 13:47:21.083916 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:47:21.083873    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-9edb53\"/\"default-dockercfg-4shkp\""
Sep 11 13:47:21.084687 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:47:21.084651    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-9edb53\"/\"openshift-service-ca.crt\""
Sep 11 13:47:21.085014 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:47:21.084991    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-9edb53\"/\"kube-root-ca.crt\""
Sep 11 13:47:21.085123 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:47:21.085070    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-9edb53\"/\"auth-test-model-9edb53-kserve-self-signed-certs\""
Sep 11 13:47:21.109549 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:47:21.109510    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-9edb53/auth-test-model-9edb53-kserve-5d97c5f9c9-nmzxg"]
Sep 11 13:47:21.182525 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:47:21.182454    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0d24dd1-a65c-4ce4-92e4-42e8781ee8e6-model-cache\") pod \"auth-test-model-9edb53-kserve-5d97c5f9c9-nmzxg\" (UID: \"c0d24dd1-a65c-4ce4-92e4-42e8781ee8e6\") " pod="ai-tenant-e2e-shared-b-w3-9edb53/auth-test-model-9edb53-kserve-5d97c5f9c9-nmzxg"
Sep 11 13:47:21.182525 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:47:21.182524    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0d24dd1-a65c-4ce4-92e4-42e8781ee8e6-tls-certs\") pod \"auth-test-model-9edb53-kserve-5d97c5f9c9-nmzxg\" (UID: \"c0d24dd1-a65c-4ce4-92e4-42e8781ee8e6\") " pod="ai-tenant-e2e-shared-b-w3-9edb53/auth-test-model-9edb53-kserve-5d97c5f9c9-nmzxg"
Sep 11 13:47:21.182916 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:47:21.182560    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0d24dd1-a65c-4ce4-92e4-42e8781ee8e6-tmp-dir\") pod \"auth-test-model-9edb53-kserve-5d97c5f9c9-nmzxg\" (UID: \"c0d24dd1-a65c-4ce4-92e4-42e8781ee8e6\") " pod="ai-tenant-e2e-shared-b-w3-9edb53/auth-test-model-9edb53-kserve-5d97c5f9c9-nmzxg"
Sep 11 13:47:21.182916 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:47:21.182716    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0d24dd1-a65c-4ce4-92e4-42e8781ee8e6-dshm\") pod \"auth-test-model-9edb53-kserve-5d97c5f9c9-nmzxg\" (UID: \"c0d24dd1-a65c-4ce4-92e4-42e8781ee8e6\") " pod="ai-tenant-e2e-shared-b-w3-9edb53/auth-test-model-9edb53-kserve-5d97c5f9c9-nmzxg"
Sep 11 13:47:21.182916 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:47:21.182758    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zm5w9\" (UniqueName: \"kubernetes.io/projected/c0d24dd1-a65c-4ce4-92e4-42e8781ee8e6-kube-api-access-zm5w9\") pod \"auth-test-model-9edb53-kserve-5d97c5f9c9-nmzxg\" (UID: \"c0d24dd1-a65c-4ce4-92e4-42e8781ee8e6\") " pod="ai-tenant-e2e-shared-b-w3-9edb53/auth-test-model-9edb53-kserve-5d97c5f9c9-nmzxg"
Sep 11 13:47:21.182916 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:47:21.182824    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c0d24dd1-a65c-4ce4-92e4-42e8781ee8e6-home\") pod \"auth-test-model-9edb53-kserve-5d97c5f9c9-nmzxg\" (UID: \"c0d24dd1-a65c-4ce4-92e4-42e8781ee8e6\") " pod="ai-tenant-e2e-shared-b-w3-9edb53/auth-test-model-9edb53-kserve-5d97c5f9c9-nmzxg"
Sep 11 13:47:21.283937 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:47:21.283866    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0d24dd1-a65c-4ce4-92e4-42e8781ee8e6-model-cache\") pod \"auth-test-model-9edb53-kserve-5d97c5f9c9-nmzxg\" (UID: \"c0d24dd1-a65c-4ce4-92e4-42e8781ee8e6\") " pod="ai-tenant-e2e-shared-b-w3-9edb53/auth-test-model-9edb53-kserve-5d97c5f9c9-nmzxg"
Sep 11 13:47:21.283937 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:47:21.283942    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0d24dd1-a65c-4ce4-92e4-42e8781ee8e6-tls-certs\") pod \"auth-test-model-9edb53-kserve-5d97c5f9c9-nmzxg\" (UID: \"c0d24dd1-a65c-4ce4-92e4-42e8781ee8e6\") " pod="ai-tenant-e2e-shared-b-w3-9edb53/auth-test-model-9edb53-kserve-5d97c5f9c9-nmzxg"
Sep 11 13:47:21.284362 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:47:21.283977    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0d24dd1-a65c-4ce4-92e4-42e8781ee8e6-tmp-dir\") pod \"auth-test-model-9edb53-kserve-5d97c5f9c9-nmzxg\" (UID: \"c0d24dd1-a65c-4ce4-92e4-42e8781ee8e6\") " pod="ai-tenant-e2e-shared-b-w3-9edb53/auth-test-model-9edb53-kserve-5d97c5f9c9-nmzxg"
Sep 11 13:47:21.284362 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:47:21.284159    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0d24dd1-a65c-4ce4-92e4-42e8781ee8e6-dshm\") pod \"auth-test-model-9edb53-kserve-5d97c5f9c9-nmzxg\" (UID: \"c0d24dd1-a65c-4ce4-92e4-42e8781ee8e6\") " pod="ai-tenant-e2e-shared-b-w3-9edb53/auth-test-model-9edb53-kserve-5d97c5f9c9-nmzxg"
Sep 11 13:47:21.284362 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:47:21.284201    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zm5w9\" (UniqueName: \"kubernetes.io/projected/c0d24dd1-a65c-4ce4-92e4-42e8781ee8e6-kube-api-access-zm5w9\") pod \"auth-test-model-9edb53-kserve-5d97c5f9c9-nmzxg\" (UID: \"c0d24dd1-a65c-4ce4-92e4-42e8781ee8e6\") " pod="ai-tenant-e2e-shared-b-w3-9edb53/auth-test-model-9edb53-kserve-5d97c5f9c9-nmzxg"
Sep 11 13:47:21.284362 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:47:21.284248    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c0d24dd1-a65c-4ce4-92e4-42e8781ee8e6-home\") pod \"auth-test-model-9edb53-kserve-5d97c5f9c9-nmzxg\" (UID: \"c0d24dd1-a65c-4ce4-92e4-42e8781ee8e6\") " pod="ai-tenant-e2e-shared-b-w3-9edb53/auth-test-model-9edb53-kserve-5d97c5f9c9-nmzxg"
Sep 11 13:47:21.284573 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:47:21.284408    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0d24dd1-a65c-4ce4-92e4-42e8781ee8e6-model-cache\") pod \"auth-test-model-9edb53-kserve-5d97c5f9c9-nmzxg\" (UID: \"c0d24dd1-a65c-4ce4-92e4-42e8781ee8e6\") " pod="ai-tenant-e2e-shared-b-w3-9edb53/auth-test-model-9edb53-kserve-5d97c5f9c9-nmzxg"
Sep 11 13:47:21.284573 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:47:21.284438    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0d24dd1-a65c-4ce4-92e4-42e8781ee8e6-tmp-dir\") pod \"auth-test-model-9edb53-kserve-5d97c5f9c9-nmzxg\" (UID: \"c0d24dd1-a65c-4ce4-92e4-42e8781ee8e6\") " pod="ai-tenant-e2e-shared-b-w3-9edb53/auth-test-model-9edb53-kserve-5d97c5f9c9-nmzxg"
Sep 11 13:47:21.284671 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:47:21.284603    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c0d24dd1-a65c-4ce4-92e4-42e8781ee8e6-home\") pod \"auth-test-model-9edb53-kserve-5d97c5f9c9-nmzxg\" (UID: \"c0d24dd1-a65c-4ce4-92e4-42e8781ee8e6\") " pod="ai-tenant-e2e-shared-b-w3-9edb53/auth-test-model-9edb53-kserve-5d97c5f9c9-nmzxg"
Sep 11 13:47:21.287127 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:47:21.287087    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0d24dd1-a65c-4ce4-92e4-42e8781ee8e6-dshm\") pod \"auth-test-model-9edb53-kserve-5d97c5f9c9-nmzxg\" (UID: \"c0d24dd1-a65c-4ce4-92e4-42e8781ee8e6\") " pod="ai-tenant-e2e-shared-b-w3-9edb53/auth-test-model-9edb53-kserve-5d97c5f9c9-nmzxg"
Sep 11 13:47:21.287385 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:47:21.287360    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0d24dd1-a65c-4ce4-92e4-42e8781ee8e6-tls-certs\") pod \"auth-test-model-9edb53-kserve-5d97c5f9c9-nmzxg\" (UID: \"c0d24dd1-a65c-4ce4-92e4-42e8781ee8e6\") " pod="ai-tenant-e2e-shared-b-w3-9edb53/auth-test-model-9edb53-kserve-5d97c5f9c9-nmzxg"
Sep 11 13:47:21.295383 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:47:21.295345    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zm5w9\" (UniqueName: \"kubernetes.io/projected/c0d24dd1-a65c-4ce4-92e4-42e8781ee8e6-kube-api-access-zm5w9\") pod \"auth-test-model-9edb53-kserve-5d97c5f9c9-nmzxg\" (UID: \"c0d24dd1-a65c-4ce4-92e4-42e8781ee8e6\") " pod="ai-tenant-e2e-shared-b-w3-9edb53/auth-test-model-9edb53-kserve-5d97c5f9c9-nmzxg"
Sep 11 13:47:21.394161 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:47:21.394035    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-9edb53/auth-test-model-9edb53-kserve-5d97c5f9c9-nmzxg"
Sep 11 13:47:21.727538 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:47:21.727513    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-9edb53/auth-test-model-9edb53-kserve-5d97c5f9c9-nmzxg"]
Sep 11 13:47:21.730475 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:47:21.730434    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc0d24dd1_a65c_4ce4_92e4_42e8781ee8e6.slice/crio-ccefda09fe8a2f26996b31e6244086347c99e17ada1224d99f549cc62e0590d4 WatchSource:0}: Error finding container ccefda09fe8a2f26996b31e6244086347c99e17ada1224d99f549cc62e0590d4: Status 404 returned error can't find the container with id ccefda09fe8a2f26996b31e6244086347c99e17ada1224d99f549cc62e0590d4
Sep 11 13:47:22.290914 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:47:22.290866    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-9edb53/auth-test-model-9edb53-kserve-5d97c5f9c9-nmzxg" event={"ID":"c0d24dd1-a65c-4ce4-92e4-42e8781ee8e6","Type":"ContainerStarted","Data":"a81ae7d957c456608512d87ebd9443c5e7a3e8b386687468c914903d8531f2da"}
Sep 11 13:47:22.291552 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:47:22.291519    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-9edb53/auth-test-model-9edb53-kserve-5d97c5f9c9-nmzxg" event={"ID":"c0d24dd1-a65c-4ce4-92e4-42e8781ee8e6","Type":"ContainerStarted","Data":"ccefda09fe8a2f26996b31e6244086347c99e17ada1224d99f549cc62e0590d4"}
Sep 11 13:47:31.394510 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:47:31.394463    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-9edb53/auth-test-model-9edb53-kserve-5d97c5f9c9-nmzxg"
Sep 11 13:47:31.395072 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:47:31.394544    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-9edb53/auth-test-model-9edb53-kserve-5d97c5f9c9-nmzxg"
Sep 11 13:47:31.409179 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:47:31.409142    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-9edb53/auth-test-model-9edb53-kserve-5d97c5f9c9-nmzxg"
Sep 11 13:47:31.440490 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:47:31.440410    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-9edb53/auth-test-model-9edb53-kserve-5d97c5f9c9-nmzxg" podStartSLOduration=10.440385478 podStartE2EDuration="10.440385478s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:47:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:47:22.328467899 +0000 UTC m=+1549.714571063" watchObservedRunningTime="2026-09-11 13:47:31.440385478 +0000 UTC m=+1558.826488641"
Sep 11 13:47:32.382846 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:47:32.382800    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-9edb53/auth-test-model-9edb53-kserve-5d97c5f9c9-nmzxg"
Sep 11 13:48:35.770200 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:48:35.770092    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-0a2395-6546fc7994-k8vfz"]
Sep 11 13:48:35.775508 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:48:35.775477    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0a2395-6546fc7994-k8vfz"
Sep 11 13:48:35.784978 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:48:35.784947    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-0a2395-serving-cert\""
Sep 11 13:48:35.798503 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:48:35.798462    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-0a2395-6546fc7994-k8vfz"]
Sep 11 13:48:35.850587 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:48:35.850541    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kmfq\" (UniqueName: \"kubernetes.io/projected/83e64ec1-0915-4ff5-8be0-2bfb03873398-kube-api-access-8kmfq\") pod \"maas-api-e2e-api-a-0a2395-6546fc7994-k8vfz\" (UID: \"83e64ec1-0915-4ff5-8be0-2bfb03873398\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0a2395-6546fc7994-k8vfz"
Sep 11 13:48:35.850587 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:48:35.850587    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/83e64ec1-0915-4ff5-8be0-2bfb03873398-openshift-service-ca\") pod \"maas-api-e2e-api-a-0a2395-6546fc7994-k8vfz\" (UID: \"83e64ec1-0915-4ff5-8be0-2bfb03873398\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0a2395-6546fc7994-k8vfz"
Sep 11 13:48:35.850886 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:48:35.850667    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/83e64ec1-0915-4ff5-8be0-2bfb03873398-maas-api-tls\") pod \"maas-api-e2e-api-a-0a2395-6546fc7994-k8vfz\" (UID: \"83e64ec1-0915-4ff5-8be0-2bfb03873398\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0a2395-6546fc7994-k8vfz"
Sep 11 13:48:35.850886 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:48:35.850805    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/83e64ec1-0915-4ff5-8be0-2bfb03873398-metrics-certs\") pod \"maas-api-e2e-api-a-0a2395-6546fc7994-k8vfz\" (UID: \"83e64ec1-0915-4ff5-8be0-2bfb03873398\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0a2395-6546fc7994-k8vfz"
Sep 11 13:48:35.952292 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:48:35.952232    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/83e64ec1-0915-4ff5-8be0-2bfb03873398-metrics-certs\") pod \"maas-api-e2e-api-a-0a2395-6546fc7994-k8vfz\" (UID: \"83e64ec1-0915-4ff5-8be0-2bfb03873398\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0a2395-6546fc7994-k8vfz"
Sep 11 13:48:35.952580 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:48:35.952420    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kmfq\" (UniqueName: \"kubernetes.io/projected/83e64ec1-0915-4ff5-8be0-2bfb03873398-kube-api-access-8kmfq\") pod \"maas-api-e2e-api-a-0a2395-6546fc7994-k8vfz\" (UID: \"83e64ec1-0915-4ff5-8be0-2bfb03873398\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0a2395-6546fc7994-k8vfz"
Sep 11 13:48:35.952580 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:48:35.952459    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/83e64ec1-0915-4ff5-8be0-2bfb03873398-openshift-service-ca\") pod \"maas-api-e2e-api-a-0a2395-6546fc7994-k8vfz\" (UID: \"83e64ec1-0915-4ff5-8be0-2bfb03873398\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0a2395-6546fc7994-k8vfz"
Sep 11 13:48:35.952580 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:48:35.952534    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/83e64ec1-0915-4ff5-8be0-2bfb03873398-maas-api-tls\") pod \"maas-api-e2e-api-a-0a2395-6546fc7994-k8vfz\" (UID: \"83e64ec1-0915-4ff5-8be0-2bfb03873398\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0a2395-6546fc7994-k8vfz"
Sep 11 13:48:35.952809 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:48:35.952778    2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-0a2395-serving-cert: secret "maas-api-e2e-api-a-0a2395-serving-cert" not found
Sep 11 13:48:35.952972 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:48:35.952945    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/83e64ec1-0915-4ff5-8be0-2bfb03873398-maas-api-tls podName:83e64ec1-0915-4ff5-8be0-2bfb03873398 nodeName:}" failed. No retries permitted until 2026-09-11 13:48:36.45291286 +0000 UTC m=+1623.839016016 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/83e64ec1-0915-4ff5-8be0-2bfb03873398-maas-api-tls") pod "maas-api-e2e-api-a-0a2395-6546fc7994-k8vfz" (UID: "83e64ec1-0915-4ff5-8be0-2bfb03873398") : secret "maas-api-e2e-api-a-0a2395-serving-cert" not found
Sep 11 13:48:35.953328 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:48:35.953265    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/83e64ec1-0915-4ff5-8be0-2bfb03873398-openshift-service-ca\") pod \"maas-api-e2e-api-a-0a2395-6546fc7994-k8vfz\" (UID: \"83e64ec1-0915-4ff5-8be0-2bfb03873398\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0a2395-6546fc7994-k8vfz"
Sep 11 13:48:35.955741 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:48:35.955711    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/83e64ec1-0915-4ff5-8be0-2bfb03873398-metrics-certs\") pod \"maas-api-e2e-api-a-0a2395-6546fc7994-k8vfz\" (UID: \"83e64ec1-0915-4ff5-8be0-2bfb03873398\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0a2395-6546fc7994-k8vfz"
Sep 11 13:48:35.978101 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:48:35.978067    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kmfq\" (UniqueName: \"kubernetes.io/projected/83e64ec1-0915-4ff5-8be0-2bfb03873398-kube-api-access-8kmfq\") pod \"maas-api-e2e-api-a-0a2395-6546fc7994-k8vfz\" (UID: \"83e64ec1-0915-4ff5-8be0-2bfb03873398\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0a2395-6546fc7994-k8vfz"
Sep 11 13:48:36.459920 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:48:36.459866    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/83e64ec1-0915-4ff5-8be0-2bfb03873398-maas-api-tls\") pod \"maas-api-e2e-api-a-0a2395-6546fc7994-k8vfz\" (UID: \"83e64ec1-0915-4ff5-8be0-2bfb03873398\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0a2395-6546fc7994-k8vfz"
Sep 11 13:48:36.463747 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:48:36.463707    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/83e64ec1-0915-4ff5-8be0-2bfb03873398-maas-api-tls\") pod \"maas-api-e2e-api-a-0a2395-6546fc7994-k8vfz\" (UID: \"83e64ec1-0915-4ff5-8be0-2bfb03873398\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0a2395-6546fc7994-k8vfz"
Sep 11 13:48:36.688502 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:48:36.688436    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0a2395-6546fc7994-k8vfz"
Sep 11 13:48:37.035637 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:48:37.035605    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-0a2395-6546fc7994-k8vfz"]
Sep 11 13:48:37.039322 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:48:37.039279    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83e64ec1_0915_4ff5_8be0_2bfb03873398.slice/crio-793f5932c558ec36352bbcb16436892d6007c13269424840db8ff35de118e112 WatchSource:0}: Error finding container 793f5932c558ec36352bbcb16436892d6007c13269424840db8ff35de118e112: Status 404 returned error can't find the container with id 793f5932c558ec36352bbcb16436892d6007c13269424840db8ff35de118e112
Sep 11 13:48:37.863656 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:48:37.863547    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0a2395-6546fc7994-k8vfz" event={"ID":"83e64ec1-0915-4ff5-8be0-2bfb03873398","Type":"ContainerStarted","Data":"0383c99adf415352989ff008fb2dc6a54818815192f1b94aeb4c07cdaeaafa55"}
Sep 11 13:48:37.863656 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:48:37.863614    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0a2395-6546fc7994-k8vfz" event={"ID":"83e64ec1-0915-4ff5-8be0-2bfb03873398","Type":"ContainerStarted","Data":"793f5932c558ec36352bbcb16436892d6007c13269424840db8ff35de118e112"}
Sep 11 13:48:37.863656 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:48:37.863646    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0a2395-6546fc7994-k8vfz"
Sep 11 13:48:37.906257 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:48:37.906187    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0a2395-6546fc7994-k8vfz" podStartSLOduration=2.4182273260000002 podStartE2EDuration="2.906167391s" totalImagesPullingTime="487.940065ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:48:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:48:37.905782842 +0000 UTC m=+1625.291886018" watchObservedRunningTime="2026-09-11 13:48:37.906167391 +0000 UTC m=+1625.292270598"
Sep 11 13:48:43.880553 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:48:43.880511    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0a2395-6546fc7994-k8vfz"
Sep 11 13:49:03.227843 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:49:03.227786    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-c8d2f6-openshift-default-77cccb6ccb-2cdcx"]
Sep 11 13:49:03.232552 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:49:03.232517    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-c8d2f6-openshift-default-77cccb6ccb-2cdcx"
Sep 11 13:49:03.235139 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:49:03.235113    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-c8d2f6-openshift-default-dockercfg-d9wwv\""
Sep 11 13:49:03.248093 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:49:03.248058    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-c8d2f6-openshift-default-77cccb6ccb-2cdcx"]
Sep 11 13:49:03.298798 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:49:03.298745    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14cd118e-eb64-4b60-a329-0964cc60faf8-workload-certs\") pod \"e2e-api-b-c8d2f6-openshift-default-77cccb6ccb-2cdcx\" (UID: \"14cd118e-eb64-4b60-a329-0964cc60faf8\") " pod="openshift-ingress/e2e-api-b-c8d2f6-openshift-default-77cccb6ccb-2cdcx"
Sep 11 13:49:03.298798 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:49:03.298803    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14cd118e-eb64-4b60-a329-0964cc60faf8-workload-socket\") pod \"e2e-api-b-c8d2f6-openshift-default-77cccb6ccb-2cdcx\" (UID: \"14cd118e-eb64-4b60-a329-0964cc60faf8\") " pod="openshift-ingress/e2e-api-b-c8d2f6-openshift-default-77cccb6ccb-2cdcx"
Sep 11 13:49:03.299177 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:49:03.298914    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14cd118e-eb64-4b60-a329-0964cc60faf8-istiod-ca-cert\") pod \"e2e-api-b-c8d2f6-openshift-default-77cccb6ccb-2cdcx\" (UID: \"14cd118e-eb64-4b60-a329-0964cc60faf8\") " pod="openshift-ingress/e2e-api-b-c8d2f6-openshift-default-77cccb6ccb-2cdcx"
Sep 11 13:49:03.299177 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:49:03.298948    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14cd118e-eb64-4b60-a329-0964cc60faf8-credential-socket\") pod \"e2e-api-b-c8d2f6-openshift-default-77cccb6ccb-2cdcx\" (UID: \"14cd118e-eb64-4b60-a329-0964cc60faf8\") " pod="openshift-ingress/e2e-api-b-c8d2f6-openshift-default-77cccb6ccb-2cdcx"
Sep 11 13:49:03.299177 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:49:03.299003    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14cd118e-eb64-4b60-a329-0964cc60faf8-istio-podinfo\") pod \"e2e-api-b-c8d2f6-openshift-default-77cccb6ccb-2cdcx\" (UID: \"14cd118e-eb64-4b60-a329-0964cc60faf8\") " pod="openshift-ingress/e2e-api-b-c8d2f6-openshift-default-77cccb6ccb-2cdcx"
Sep 11 13:49:03.299177 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:49:03.299113    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zb8jm\" (UniqueName: \"kubernetes.io/projected/14cd118e-eb64-4b60-a329-0964cc60faf8-kube-api-access-zb8jm\") pod \"e2e-api-b-c8d2f6-openshift-default-77cccb6ccb-2cdcx\" (UID: \"14cd118e-eb64-4b60-a329-0964cc60faf8\") " pod="openshift-ingress/e2e-api-b-c8d2f6-openshift-default-77cccb6ccb-2cdcx"
Sep 11 13:49:03.299333 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:49:03.299204    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14cd118e-eb64-4b60-a329-0964cc60faf8-istio-envoy\") pod \"e2e-api-b-c8d2f6-openshift-default-77cccb6ccb-2cdcx\" (UID: \"14cd118e-eb64-4b60-a329-0964cc60faf8\") " pod="openshift-ingress/e2e-api-b-c8d2f6-openshift-default-77cccb6ccb-2cdcx"
Sep 11 13:49:03.299333 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:49:03.299239    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14cd118e-eb64-4b60-a329-0964cc60faf8-istio-token\") pod \"e2e-api-b-c8d2f6-openshift-default-77cccb6ccb-2cdcx\" (UID: \"14cd118e-eb64-4b60-a329-0964cc60faf8\") " pod="openshift-ingress/e2e-api-b-c8d2f6-openshift-default-77cccb6ccb-2cdcx"
Sep 11 13:49:03.299333 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:49:03.299260    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14cd118e-eb64-4b60-a329-0964cc60faf8-istio-data\") pod \"e2e-api-b-c8d2f6-openshift-default-77cccb6ccb-2cdcx\" (UID: \"14cd118e-eb64-4b60-a329-0964cc60faf8\") " pod="openshift-ingress/e2e-api-b-c8d2f6-openshift-default-77cccb6ccb-2cdcx"
Sep 11 13:49:03.400513 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:49:03.400452    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14cd118e-eb64-4b60-a329-0964cc60faf8-workload-socket\") pod \"e2e-api-b-c8d2f6-openshift-default-77cccb6ccb-2cdcx\" (UID: \"14cd118e-eb64-4b60-a329-0964cc60faf8\") " pod="openshift-ingress/e2e-api-b-c8d2f6-openshift-default-77cccb6ccb-2cdcx"
Sep 11 13:49:03.400801 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:49:03.400546    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14cd118e-eb64-4b60-a329-0964cc60faf8-istiod-ca-cert\") pod \"e2e-api-b-c8d2f6-openshift-default-77cccb6ccb-2cdcx\" (UID: \"14cd118e-eb64-4b60-a329-0964cc60faf8\") " pod="openshift-ingress/e2e-api-b-c8d2f6-openshift-default-77cccb6ccb-2cdcx"
Sep 11 13:49:03.400801 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:49:03.400583    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14cd118e-eb64-4b60-a329-0964cc60faf8-credential-socket\") pod \"e2e-api-b-c8d2f6-openshift-default-77cccb6ccb-2cdcx\" (UID: \"14cd118e-eb64-4b60-a329-0964cc60faf8\") " pod="openshift-ingress/e2e-api-b-c8d2f6-openshift-default-77cccb6ccb-2cdcx"
Sep 11 13:49:03.400801 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:49:03.400627    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14cd118e-eb64-4b60-a329-0964cc60faf8-istio-podinfo\") pod \"e2e-api-b-c8d2f6-openshift-default-77cccb6ccb-2cdcx\" (UID: \"14cd118e-eb64-4b60-a329-0964cc60faf8\") " pod="openshift-ingress/e2e-api-b-c8d2f6-openshift-default-77cccb6ccb-2cdcx"
Sep 11 13:49:03.400801 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:49:03.400697    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zb8jm\" (UniqueName: \"kubernetes.io/projected/14cd118e-eb64-4b60-a329-0964cc60faf8-kube-api-access-zb8jm\") pod \"e2e-api-b-c8d2f6-openshift-default-77cccb6ccb-2cdcx\" (UID: \"14cd118e-eb64-4b60-a329-0964cc60faf8\") " pod="openshift-ingress/e2e-api-b-c8d2f6-openshift-default-77cccb6ccb-2cdcx"
Sep 11 13:49:03.400801 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:49:03.400789    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14cd118e-eb64-4b60-a329-0964cc60faf8-istio-envoy\") pod \"e2e-api-b-c8d2f6-openshift-default-77cccb6ccb-2cdcx\" (UID: \"14cd118e-eb64-4b60-a329-0964cc60faf8\") " pod="openshift-ingress/e2e-api-b-c8d2f6-openshift-default-77cccb6ccb-2cdcx"
Sep 11 13:49:03.401176 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:49:03.400861    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14cd118e-eb64-4b60-a329-0964cc60faf8-istio-token\") pod \"e2e-api-b-c8d2f6-openshift-default-77cccb6ccb-2cdcx\" (UID: \"14cd118e-eb64-4b60-a329-0964cc60faf8\") " pod="openshift-ingress/e2e-api-b-c8d2f6-openshift-default-77cccb6ccb-2cdcx"
Sep 11 13:49:03.401176 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:49:03.400900    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14cd118e-eb64-4b60-a329-0964cc60faf8-istio-data\") pod \"e2e-api-b-c8d2f6-openshift-default-77cccb6ccb-2cdcx\" (UID: \"14cd118e-eb64-4b60-a329-0964cc60faf8\") " pod="openshift-ingress/e2e-api-b-c8d2f6-openshift-default-77cccb6ccb-2cdcx"
Sep 11 13:49:03.401176 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:49:03.400969    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14cd118e-eb64-4b60-a329-0964cc60faf8-workload-certs\") pod \"e2e-api-b-c8d2f6-openshift-default-77cccb6ccb-2cdcx\" (UID: \"14cd118e-eb64-4b60-a329-0964cc60faf8\") " pod="openshift-ingress/e2e-api-b-c8d2f6-openshift-default-77cccb6ccb-2cdcx"
Sep 11 13:49:03.401176 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:49:03.401065    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14cd118e-eb64-4b60-a329-0964cc60faf8-workload-socket\") pod \"e2e-api-b-c8d2f6-openshift-default-77cccb6ccb-2cdcx\" (UID: \"14cd118e-eb64-4b60-a329-0964cc60faf8\") " pod="openshift-ingress/e2e-api-b-c8d2f6-openshift-default-77cccb6ccb-2cdcx"
Sep 11 13:49:03.401176 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:49:03.401122    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14cd118e-eb64-4b60-a329-0964cc60faf8-credential-socket\") pod \"e2e-api-b-c8d2f6-openshift-default-77cccb6ccb-2cdcx\" (UID: \"14cd118e-eb64-4b60-a329-0964cc60faf8\") " pod="openshift-ingress/e2e-api-b-c8d2f6-openshift-default-77cccb6ccb-2cdcx"
Sep 11 13:49:03.401423 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:49:03.401312    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14cd118e-eb64-4b60-a329-0964cc60faf8-workload-certs\") pod \"e2e-api-b-c8d2f6-openshift-default-77cccb6ccb-2cdcx\" (UID: \"14cd118e-eb64-4b60-a329-0964cc60faf8\") " pod="openshift-ingress/e2e-api-b-c8d2f6-openshift-default-77cccb6ccb-2cdcx"
Sep 11 13:49:03.401423 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:49:03.401382    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14cd118e-eb64-4b60-a329-0964cc60faf8-istio-data\") pod \"e2e-api-b-c8d2f6-openshift-default-77cccb6ccb-2cdcx\" (UID: \"14cd118e-eb64-4b60-a329-0964cc60faf8\") " pod="openshift-ingress/e2e-api-b-c8d2f6-openshift-default-77cccb6ccb-2cdcx"
Sep 11 13:49:03.401791 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:49:03.401748    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14cd118e-eb64-4b60-a329-0964cc60faf8-istiod-ca-cert\") pod \"e2e-api-b-c8d2f6-openshift-default-77cccb6ccb-2cdcx\" (UID: \"14cd118e-eb64-4b60-a329-0964cc60faf8\") " pod="openshift-ingress/e2e-api-b-c8d2f6-openshift-default-77cccb6ccb-2cdcx"
Sep 11 13:49:03.403676 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:49:03.403646    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14cd118e-eb64-4b60-a329-0964cc60faf8-istio-envoy\") pod \"e2e-api-b-c8d2f6-openshift-default-77cccb6ccb-2cdcx\" (UID: \"14cd118e-eb64-4b60-a329-0964cc60faf8\") " pod="openshift-ingress/e2e-api-b-c8d2f6-openshift-default-77cccb6ccb-2cdcx"
Sep 11 13:49:03.404041 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:49:03.404017    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14cd118e-eb64-4b60-a329-0964cc60faf8-istio-podinfo\") pod \"e2e-api-b-c8d2f6-openshift-default-77cccb6ccb-2cdcx\" (UID: \"14cd118e-eb64-4b60-a329-0964cc60faf8\") " pod="openshift-ingress/e2e-api-b-c8d2f6-openshift-default-77cccb6ccb-2cdcx"
Sep 11 13:49:03.419626 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:49:03.419576    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zb8jm\" (UniqueName: \"kubernetes.io/projected/14cd118e-eb64-4b60-a329-0964cc60faf8-kube-api-access-zb8jm\") pod \"e2e-api-b-c8d2f6-openshift-default-77cccb6ccb-2cdcx\" (UID: \"14cd118e-eb64-4b60-a329-0964cc60faf8\") " pod="openshift-ingress/e2e-api-b-c8d2f6-openshift-default-77cccb6ccb-2cdcx"
Sep 11 13:49:03.419626 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:49:03.419603    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14cd118e-eb64-4b60-a329-0964cc60faf8-istio-token\") pod \"e2e-api-b-c8d2f6-openshift-default-77cccb6ccb-2cdcx\" (UID: \"14cd118e-eb64-4b60-a329-0964cc60faf8\") " pod="openshift-ingress/e2e-api-b-c8d2f6-openshift-default-77cccb6ccb-2cdcx"
Sep 11 13:49:03.546263 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:49:03.546129    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-c8d2f6-openshift-default-77cccb6ccb-2cdcx"
Sep 11 13:49:03.899926 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:49:03.899890    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-c8d2f6-openshift-default-77cccb6ccb-2cdcx"]
Sep 11 13:49:03.902746 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:49:03.902709    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14cd118e_eb64_4b60_a329_0964cc60faf8.slice/crio-3352ff10c1014c15f9dbd197076a04b8bd87b3a49d0d2bc3376f41a5f9b032a6 WatchSource:0}: Error finding container 3352ff10c1014c15f9dbd197076a04b8bd87b3a49d0d2bc3376f41a5f9b032a6: Status 404 returned error can't find the container with id 3352ff10c1014c15f9dbd197076a04b8bd87b3a49d0d2bc3376f41a5f9b032a6
Sep 11 13:49:03.906517 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:49:03.906465    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 11 13:49:03.906708 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:49:03.906582    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 11 13:49:03.906708 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:49:03.906632    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 11 13:49:04.059277 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:49:04.059226    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c8d2f6-openshift-default-77cccb6ccb-2cdcx" event={"ID":"14cd118e-eb64-4b60-a329-0964cc60faf8","Type":"ContainerStarted","Data":"5b5ed2a6fa340d03eb7e84016817c7cb7b5ab96874bfcdcfea3afb97035a3e8a"}
Sep 11 13:49:04.059570 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:49:04.059312    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c8d2f6-openshift-default-77cccb6ccb-2cdcx" event={"ID":"14cd118e-eb64-4b60-a329-0964cc60faf8","Type":"ContainerStarted","Data":"3352ff10c1014c15f9dbd197076a04b8bd87b3a49d0d2bc3376f41a5f9b032a6"}
Sep 11 13:49:04.098980 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:49:04.098922    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-c8d2f6-openshift-default-77cccb6ccb-2cdcx" podStartSLOduration=1.098900687 podStartE2EDuration="1.098900687s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:49:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:49:04.094357705 +0000 UTC m=+1651.480460895" watchObservedRunningTime="2026-09-11 13:49:04.098900687 +0000 UTC m=+1651.485003937"
Sep 11 13:49:04.547018 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:49:04.546959    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-c8d2f6-openshift-default-77cccb6ccb-2cdcx"
Sep 11 13:49:04.553721 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:49:04.553678    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-c8d2f6-openshift-default-77cccb6ccb-2cdcx"
Sep 11 13:49:05.066817 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:49:05.066776    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-c8d2f6-openshift-default-77cccb6ccb-2cdcx"
Sep 11 13:49:05.068086 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:49:05.068060    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-c8d2f6-openshift-default-77cccb6ccb-2cdcx"
Sep 11 13:49:10.507501 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:49:10.507459    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c8d2f6-996c49f78-xnw4d"]
Sep 11 13:49:10.513315 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:49:10.513290    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c8d2f6-996c49f78-xnw4d"
Sep 11 13:49:10.517218 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:49:10.517170    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-c8d2f6-serving-cert\""
Sep 11 13:49:10.543339 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:49:10.543285    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c8d2f6-996c49f78-xnw4d"]
Sep 11 13:49:10.600320 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:49:10.600260    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/26968216-f646-4245-91a5-eb8f0710c82e-openshift-service-ca\") pod \"maas-api-e2e-api-b-c8d2f6-996c49f78-xnw4d\" (UID: \"26968216-f646-4245-91a5-eb8f0710c82e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c8d2f6-996c49f78-xnw4d"
Sep 11 13:49:10.600586 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:49:10.600348    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/26968216-f646-4245-91a5-eb8f0710c82e-maas-api-tls\") pod \"maas-api-e2e-api-b-c8d2f6-996c49f78-xnw4d\" (UID: \"26968216-f646-4245-91a5-eb8f0710c82e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c8d2f6-996c49f78-xnw4d"
Sep 11 13:49:10.600586 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:49:10.600489    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4r66\" (UniqueName: \"kubernetes.io/projected/26968216-f646-4245-91a5-eb8f0710c82e-kube-api-access-z4r66\") pod \"maas-api-e2e-api-b-c8d2f6-996c49f78-xnw4d\" (UID: \"26968216-f646-4245-91a5-eb8f0710c82e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c8d2f6-996c49f78-xnw4d"
Sep 11 13:49:10.600586 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:49:10.600550    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26968216-f646-4245-91a5-eb8f0710c82e-metrics-certs\") pod \"maas-api-e2e-api-b-c8d2f6-996c49f78-xnw4d\" (UID: \"26968216-f646-4245-91a5-eb8f0710c82e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c8d2f6-996c49f78-xnw4d"
Sep 11 13:49:10.701398 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:49:10.701346    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/26968216-f646-4245-91a5-eb8f0710c82e-maas-api-tls\") pod \"maas-api-e2e-api-b-c8d2f6-996c49f78-xnw4d\" (UID: \"26968216-f646-4245-91a5-eb8f0710c82e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c8d2f6-996c49f78-xnw4d"
Sep 11 13:49:10.701630 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:49:10.701511    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4r66\" (UniqueName: \"kubernetes.io/projected/26968216-f646-4245-91a5-eb8f0710c82e-kube-api-access-z4r66\") pod \"maas-api-e2e-api-b-c8d2f6-996c49f78-xnw4d\" (UID: \"26968216-f646-4245-91a5-eb8f0710c82e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c8d2f6-996c49f78-xnw4d"
Sep 11 13:49:10.701630 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:49:10.701560    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26968216-f646-4245-91a5-eb8f0710c82e-metrics-certs\") pod \"maas-api-e2e-api-b-c8d2f6-996c49f78-xnw4d\" (UID: \"26968216-f646-4245-91a5-eb8f0710c82e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c8d2f6-996c49f78-xnw4d"
Sep 11 13:49:10.701782 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:49:10.701680    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/26968216-f646-4245-91a5-eb8f0710c82e-openshift-service-ca\") pod \"maas-api-e2e-api-b-c8d2f6-996c49f78-xnw4d\" (UID: \"26968216-f646-4245-91a5-eb8f0710c82e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c8d2f6-996c49f78-xnw4d"
Sep 11 13:49:10.702552 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:49:10.702483    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/26968216-f646-4245-91a5-eb8f0710c82e-openshift-service-ca\") pod \"maas-api-e2e-api-b-c8d2f6-996c49f78-xnw4d\" (UID: \"26968216-f646-4245-91a5-eb8f0710c82e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c8d2f6-996c49f78-xnw4d"
Sep 11 13:49:10.704493 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:49:10.704464    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/26968216-f646-4245-91a5-eb8f0710c82e-maas-api-tls\") pod \"maas-api-e2e-api-b-c8d2f6-996c49f78-xnw4d\" (UID: \"26968216-f646-4245-91a5-eb8f0710c82e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c8d2f6-996c49f78-xnw4d"
Sep 11 13:49:10.704645 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:49:10.704625    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26968216-f646-4245-91a5-eb8f0710c82e-metrics-certs\") pod \"maas-api-e2e-api-b-c8d2f6-996c49f78-xnw4d\" (UID: \"26968216-f646-4245-91a5-eb8f0710c82e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c8d2f6-996c49f78-xnw4d"
Sep 11 13:49:10.712039 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:49:10.712006    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4r66\" (UniqueName: \"kubernetes.io/projected/26968216-f646-4245-91a5-eb8f0710c82e-kube-api-access-z4r66\") pod \"maas-api-e2e-api-b-c8d2f6-996c49f78-xnw4d\" (UID: \"26968216-f646-4245-91a5-eb8f0710c82e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c8d2f6-996c49f78-xnw4d"
Sep 11 13:49:10.828588 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:49:10.828439    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c8d2f6-996c49f78-xnw4d"
Sep 11 13:49:11.145664 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:49:11.145616    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c8d2f6-996c49f78-xnw4d"]
Sep 11 13:49:11.148599 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:49:11.148562    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26968216_f646_4245_91a5_eb8f0710c82e.slice/crio-550fcf34bca23001e92f3b5047affd8d24cd4bbdc2984a9ff53f979d95fc4636 WatchSource:0}: Error finding container 550fcf34bca23001e92f3b5047affd8d24cd4bbdc2984a9ff53f979d95fc4636: Status 404 returned error can't find the container with id 550fcf34bca23001e92f3b5047affd8d24cd4bbdc2984a9ff53f979d95fc4636
Sep 11 13:49:12.124970 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:49:12.124821    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c8d2f6-996c49f78-xnw4d" event={"ID":"26968216-f646-4245-91a5-eb8f0710c82e","Type":"ContainerStarted","Data":"d46891ffb9a23d46a9a2de0825fbed504ac3c51c91870f17734cbe9e20f5e412"}
Sep 11 13:49:12.125524 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:49:12.125066    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c8d2f6-996c49f78-xnw4d"
Sep 11 13:49:12.125524 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:49:12.125099    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c8d2f6-996c49f78-xnw4d" event={"ID":"26968216-f646-4245-91a5-eb8f0710c82e","Type":"ContainerStarted","Data":"550fcf34bca23001e92f3b5047affd8d24cd4bbdc2984a9ff53f979d95fc4636"}
Sep 11 13:49:12.157359 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:49:12.157282    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c8d2f6-996c49f78-xnw4d" podStartSLOduration=1.851063141 podStartE2EDuration="2.157260239s" totalImagesPullingTime="306.197098ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:49:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:49:12.154378038 +0000 UTC m=+1659.540481202" watchObservedRunningTime="2026-09-11 13:49:12.157260239 +0000 UTC m=+1659.543363402"
Sep 11 13:49:18.138466 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:49:18.138428    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c8d2f6-996c49f78-xnw4d"
Sep 11 13:49:33.297851 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:49:33.297796    2578 generic.go:356] "Generic (PLEG): container finished" podID="944e63d2-014c-40e1-9b2c-caa5f1259b21" containerID="716f452cecdef11943e418b4748eec2e32cd57c43fc5161d7eff1d3f3683dd88" exitCode=137
Sep 11 13:49:33.298417 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:49:33.297877    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-d8lgv" event={"ID":"944e63d2-014c-40e1-9b2c-caa5f1259b21","Type":"ContainerDied","Data":"716f452cecdef11943e418b4748eec2e32cd57c43fc5161d7eff1d3f3683dd88"}
Sep 11 13:49:33.298463 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:49:33.298442    2578 scope.go:117] "RemoveContainer" containerID="716f452cecdef11943e418b4748eec2e32cd57c43fc5161d7eff1d3f3683dd88"
Sep 11 13:49:34.308394 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:49:34.308354    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-d8lgv" event={"ID":"944e63d2-014c-40e1-9b2c-caa5f1259b21","Type":"ContainerStarted","Data":"147696462aa3448ab971010e2e89a5b4f17388693b83170659305cbede8d7f0e"}
Sep 11 13:49:34.308860 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:49:34.308599    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-d8lgv"
Sep 11 13:49:41.877015 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:49:41.876965    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-0a2395-6546fc7994-k8vfz"]
Sep 11 13:49:41.877634 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:49:41.877256    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0a2395-6546fc7994-k8vfz" podUID="83e64ec1-0915-4ff5-8be0-2bfb03873398" containerName="maas-api" containerID="cri-o://0383c99adf415352989ff008fb2dc6a54818815192f1b94aeb4c07cdaeaafa55" gracePeriod=30
Sep 11 13:49:42.167287 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:49:42.167252    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0a2395-6546fc7994-k8vfz"
Sep 11 13:49:42.238723 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:49:42.238667    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/83e64ec1-0915-4ff5-8be0-2bfb03873398-openshift-service-ca\") pod \"83e64ec1-0915-4ff5-8be0-2bfb03873398\" (UID: \"83e64ec1-0915-4ff5-8be0-2bfb03873398\") "
Sep 11 13:49:42.238983 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:49:42.238795    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/83e64ec1-0915-4ff5-8be0-2bfb03873398-metrics-certs\") pod \"83e64ec1-0915-4ff5-8be0-2bfb03873398\" (UID: \"83e64ec1-0915-4ff5-8be0-2bfb03873398\") "
Sep 11 13:49:42.238983 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:49:42.238875    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/83e64ec1-0915-4ff5-8be0-2bfb03873398-maas-api-tls\") pod \"83e64ec1-0915-4ff5-8be0-2bfb03873398\" (UID: \"83e64ec1-0915-4ff5-8be0-2bfb03873398\") "
Sep 11 13:49:42.239224 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:49:42.239190    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83e64ec1-0915-4ff5-8be0-2bfb03873398-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "83e64ec1-0915-4ff5-8be0-2bfb03873398" (UID: "83e64ec1-0915-4ff5-8be0-2bfb03873398"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 13:49:42.239563 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:49:42.239534    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8kmfq\" (UniqueName: \"kubernetes.io/projected/83e64ec1-0915-4ff5-8be0-2bfb03873398-kube-api-access-8kmfq\") pod \"83e64ec1-0915-4ff5-8be0-2bfb03873398\" (UID: \"83e64ec1-0915-4ff5-8be0-2bfb03873398\") "
Sep 11 13:49:42.240598 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:49:42.240570    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/83e64ec1-0915-4ff5-8be0-2bfb03873398-openshift-service-ca\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:49:42.242043 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:49:42.241964    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83e64ec1-0915-4ff5-8be0-2bfb03873398-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "83e64ec1-0915-4ff5-8be0-2bfb03873398" (UID: "83e64ec1-0915-4ff5-8be0-2bfb03873398"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 13:49:42.242161 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:49:42.242079    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83e64ec1-0915-4ff5-8be0-2bfb03873398-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "83e64ec1-0915-4ff5-8be0-2bfb03873398" (UID: "83e64ec1-0915-4ff5-8be0-2bfb03873398"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 13:49:42.242161 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:49:42.242123    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83e64ec1-0915-4ff5-8be0-2bfb03873398-kube-api-access-8kmfq" (OuterVolumeSpecName: "kube-api-access-8kmfq") pod "83e64ec1-0915-4ff5-8be0-2bfb03873398" (UID: "83e64ec1-0915-4ff5-8be0-2bfb03873398"). InnerVolumeSpecName "kube-api-access-8kmfq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 13:49:42.342617 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:49:42.342568    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/83e64ec1-0915-4ff5-8be0-2bfb03873398-metrics-certs\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:49:42.342617 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:49:42.342608    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/83e64ec1-0915-4ff5-8be0-2bfb03873398-maas-api-tls\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:49:42.342617 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:49:42.342621    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8kmfq\" (UniqueName: \"kubernetes.io/projected/83e64ec1-0915-4ff5-8be0-2bfb03873398-kube-api-access-8kmfq\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:49:42.365100 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:49:42.365058    2578 generic.go:356] "Generic (PLEG): container finished" podID="83e64ec1-0915-4ff5-8be0-2bfb03873398" containerID="0383c99adf415352989ff008fb2dc6a54818815192f1b94aeb4c07cdaeaafa55" exitCode=0
Sep 11 13:49:42.365340 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:49:42.365116    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0a2395-6546fc7994-k8vfz"
Sep 11 13:49:42.365340 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:49:42.365141    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0a2395-6546fc7994-k8vfz" event={"ID":"83e64ec1-0915-4ff5-8be0-2bfb03873398","Type":"ContainerDied","Data":"0383c99adf415352989ff008fb2dc6a54818815192f1b94aeb4c07cdaeaafa55"}
Sep 11 13:49:42.365340 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:49:42.365220    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0a2395-6546fc7994-k8vfz" event={"ID":"83e64ec1-0915-4ff5-8be0-2bfb03873398","Type":"ContainerDied","Data":"793f5932c558ec36352bbcb16436892d6007c13269424840db8ff35de118e112"}
Sep 11 13:49:42.365340 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:49:42.365255    2578 scope.go:117] "RemoveContainer" containerID="0383c99adf415352989ff008fb2dc6a54818815192f1b94aeb4c07cdaeaafa55"
Sep 11 13:49:42.378921 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:49:42.378892    2578 scope.go:117] "RemoveContainer" containerID="0383c99adf415352989ff008fb2dc6a54818815192f1b94aeb4c07cdaeaafa55"
Sep 11 13:49:42.379417 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:49:42.379358    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0383c99adf415352989ff008fb2dc6a54818815192f1b94aeb4c07cdaeaafa55\": container with ID starting with 0383c99adf415352989ff008fb2dc6a54818815192f1b94aeb4c07cdaeaafa55 not found: ID does not exist" containerID="0383c99adf415352989ff008fb2dc6a54818815192f1b94aeb4c07cdaeaafa55"
Sep 11 13:49:42.379482 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:49:42.379432    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0383c99adf415352989ff008fb2dc6a54818815192f1b94aeb4c07cdaeaafa55"} err="failed to get container status \"0383c99adf415352989ff008fb2dc6a54818815192f1b94aeb4c07cdaeaafa55\": rpc error: code = NotFound desc = could not find container \"0383c99adf415352989ff008fb2dc6a54818815192f1b94aeb4c07cdaeaafa55\": container with ID starting with 0383c99adf415352989ff008fb2dc6a54818815192f1b94aeb4c07cdaeaafa55 not found: ID does not exist"
Sep 11 13:49:42.403674 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:49:42.403564    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-0a2395-6546fc7994-k8vfz"]
Sep 11 13:49:42.412683 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:49:42.412638    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-0a2395-6546fc7994-k8vfz"]
Sep 11 13:49:43.109359 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:49:43.109264    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="83e64ec1-0915-4ff5-8be0-2bfb03873398" path="/var/lib/kubelet/pods/83e64ec1-0915-4ff5-8be0-2bfb03873398/volumes"
Sep 11 13:49:45.318851 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:49:45.318793    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-d8lgv"
Sep 11 13:50:06.792507 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:50:06.792445    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c8d2f6-996c49f78-xnw4d"]
Sep 11 13:50:06.793187 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:50:06.793043    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c8d2f6-996c49f78-xnw4d" podUID="26968216-f646-4245-91a5-eb8f0710c82e" containerName="maas-api" containerID="cri-o://d46891ffb9a23d46a9a2de0825fbed504ac3c51c91870f17734cbe9e20f5e412" gracePeriod=30
Sep 11 13:50:07.099253 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:50:07.099214    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c8d2f6-996c49f78-xnw4d"
Sep 11 13:50:07.167289 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:50:07.167235    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z4r66\" (UniqueName: \"kubernetes.io/projected/26968216-f646-4245-91a5-eb8f0710c82e-kube-api-access-z4r66\") pod \"26968216-f646-4245-91a5-eb8f0710c82e\" (UID: \"26968216-f646-4245-91a5-eb8f0710c82e\") "
Sep 11 13:50:07.167567 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:50:07.167333    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26968216-f646-4245-91a5-eb8f0710c82e-metrics-certs\") pod \"26968216-f646-4245-91a5-eb8f0710c82e\" (UID: \"26968216-f646-4245-91a5-eb8f0710c82e\") "
Sep 11 13:50:07.167567 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:50:07.167499    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/26968216-f646-4245-91a5-eb8f0710c82e-openshift-service-ca\") pod \"26968216-f646-4245-91a5-eb8f0710c82e\" (UID: \"26968216-f646-4245-91a5-eb8f0710c82e\") "
Sep 11 13:50:07.167712 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:50:07.167566    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/26968216-f646-4245-91a5-eb8f0710c82e-maas-api-tls\") pod \"26968216-f646-4245-91a5-eb8f0710c82e\" (UID: \"26968216-f646-4245-91a5-eb8f0710c82e\") "
Sep 11 13:50:07.171396 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:50:07.171341    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26968216-f646-4245-91a5-eb8f0710c82e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "26968216-f646-4245-91a5-eb8f0710c82e" (UID: "26968216-f646-4245-91a5-eb8f0710c82e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 13:50:07.172317 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:50:07.172281    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26968216-f646-4245-91a5-eb8f0710c82e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "26968216-f646-4245-91a5-eb8f0710c82e" (UID: "26968216-f646-4245-91a5-eb8f0710c82e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 13:50:07.175407 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:50:07.175341    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26968216-f646-4245-91a5-eb8f0710c82e-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "26968216-f646-4245-91a5-eb8f0710c82e" (UID: "26968216-f646-4245-91a5-eb8f0710c82e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 13:50:07.184287 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:50:07.183780    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26968216-f646-4245-91a5-eb8f0710c82e-kube-api-access-z4r66" (OuterVolumeSpecName: "kube-api-access-z4r66") pod "26968216-f646-4245-91a5-eb8f0710c82e" (UID: "26968216-f646-4245-91a5-eb8f0710c82e"). InnerVolumeSpecName "kube-api-access-z4r66". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 13:50:07.269240 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:50:07.269184    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/26968216-f646-4245-91a5-eb8f0710c82e-openshift-service-ca\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:50:07.269240 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:50:07.269230    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/26968216-f646-4245-91a5-eb8f0710c82e-maas-api-tls\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:50:07.269240 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:50:07.269241    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z4r66\" (UniqueName: \"kubernetes.io/projected/26968216-f646-4245-91a5-eb8f0710c82e-kube-api-access-z4r66\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:50:07.269240 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:50:07.269251    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26968216-f646-4245-91a5-eb8f0710c82e-metrics-certs\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:50:07.565787 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:50:07.565749    2578 generic.go:356] "Generic (PLEG): container finished" podID="26968216-f646-4245-91a5-eb8f0710c82e" containerID="d46891ffb9a23d46a9a2de0825fbed504ac3c51c91870f17734cbe9e20f5e412" exitCode=0
Sep 11 13:50:07.566041 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:50:07.565797    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c8d2f6-996c49f78-xnw4d" event={"ID":"26968216-f646-4245-91a5-eb8f0710c82e","Type":"ContainerDied","Data":"d46891ffb9a23d46a9a2de0825fbed504ac3c51c91870f17734cbe9e20f5e412"}
Sep 11 13:50:07.566041 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:50:07.565881    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c8d2f6-996c49f78-xnw4d"
Sep 11 13:50:07.566041 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:50:07.565910    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c8d2f6-996c49f78-xnw4d" event={"ID":"26968216-f646-4245-91a5-eb8f0710c82e","Type":"ContainerDied","Data":"550fcf34bca23001e92f3b5047affd8d24cd4bbdc2984a9ff53f979d95fc4636"}
Sep 11 13:50:07.566041 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:50:07.565941    2578 scope.go:117] "RemoveContainer" containerID="d46891ffb9a23d46a9a2de0825fbed504ac3c51c91870f17734cbe9e20f5e412"
Sep 11 13:50:07.578200 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:50:07.578175    2578 scope.go:117] "RemoveContainer" containerID="d46891ffb9a23d46a9a2de0825fbed504ac3c51c91870f17734cbe9e20f5e412"
Sep 11 13:50:07.578676 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:50:07.578645    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d46891ffb9a23d46a9a2de0825fbed504ac3c51c91870f17734cbe9e20f5e412\": container with ID starting with d46891ffb9a23d46a9a2de0825fbed504ac3c51c91870f17734cbe9e20f5e412 not found: ID does not exist" containerID="d46891ffb9a23d46a9a2de0825fbed504ac3c51c91870f17734cbe9e20f5e412"
Sep 11 13:50:07.578752 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:50:07.578693    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d46891ffb9a23d46a9a2de0825fbed504ac3c51c91870f17734cbe9e20f5e412"} err="failed to get container status \"d46891ffb9a23d46a9a2de0825fbed504ac3c51c91870f17734cbe9e20f5e412\": rpc error: code = NotFound desc = could not find container \"d46891ffb9a23d46a9a2de0825fbed504ac3c51c91870f17734cbe9e20f5e412\": container with ID starting with d46891ffb9a23d46a9a2de0825fbed504ac3c51c91870f17734cbe9e20f5e412 not found: ID does not exist"
Sep 11 13:50:07.620061 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:50:07.620005    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c8d2f6-996c49f78-xnw4d"]
Sep 11 13:50:07.629446 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:50:07.629389    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c8d2f6-996c49f78-xnw4d"]
Sep 11 13:50:09.117404 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:50:09.117342    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="26968216-f646-4245-91a5-eb8f0710c82e" path="/var/lib/kubelet/pods/26968216-f646-4245-91a5-eb8f0710c82e/volumes"
Sep 11 13:50:15.657060 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:50:15.657016    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-c8d2f6-openshift-default-77cccb6ccb-2cdcx"]
Sep 11 13:50:15.657565 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:50:15.657268    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-c8d2f6-openshift-default-77cccb6ccb-2cdcx" podUID="14cd118e-eb64-4b60-a329-0964cc60faf8" containerName="istio-proxy" containerID="cri-o://5b5ed2a6fa340d03eb7e84016817c7cb7b5ab96874bfcdcfea3afb97035a3e8a" gracePeriod=30
Sep 11 13:50:17.857858 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:50:17.857751    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-c8d2f6-openshift-default-77cccb6ccb-2cdcx"
Sep 11 13:50:17.919243 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:50:17.919192    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zb8jm\" (UniqueName: \"kubernetes.io/projected/14cd118e-eb64-4b60-a329-0964cc60faf8-kube-api-access-zb8jm\") pod \"14cd118e-eb64-4b60-a329-0964cc60faf8\" (UID: \"14cd118e-eb64-4b60-a329-0964cc60faf8\") "
Sep 11 13:50:17.919243 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:50:17.919250    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14cd118e-eb64-4b60-a329-0964cc60faf8-istio-envoy\") pod \"14cd118e-eb64-4b60-a329-0964cc60faf8\" (UID: \"14cd118e-eb64-4b60-a329-0964cc60faf8\") "
Sep 11 13:50:17.919593 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:50:17.919278    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14cd118e-eb64-4b60-a329-0964cc60faf8-istio-token\") pod \"14cd118e-eb64-4b60-a329-0964cc60faf8\" (UID: \"14cd118e-eb64-4b60-a329-0964cc60faf8\") "
Sep 11 13:50:17.919593 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:50:17.919310    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14cd118e-eb64-4b60-a329-0964cc60faf8-credential-socket\") pod \"14cd118e-eb64-4b60-a329-0964cc60faf8\" (UID: \"14cd118e-eb64-4b60-a329-0964cc60faf8\") "
Sep 11 13:50:17.919593 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:50:17.919395    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14cd118e-eb64-4b60-a329-0964cc60faf8-workload-certs\") pod \"14cd118e-eb64-4b60-a329-0964cc60faf8\" (UID: \"14cd118e-eb64-4b60-a329-0964cc60faf8\") "
Sep 11 13:50:17.919593 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:50:17.919494    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14cd118e-eb64-4b60-a329-0964cc60faf8-istiod-ca-cert\") pod \"14cd118e-eb64-4b60-a329-0964cc60faf8\" (UID: \"14cd118e-eb64-4b60-a329-0964cc60faf8\") "
Sep 11 13:50:17.919593 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:50:17.919542    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14cd118e-eb64-4b60-a329-0964cc60faf8-istio-podinfo\") pod \"14cd118e-eb64-4b60-a329-0964cc60faf8\" (UID: \"14cd118e-eb64-4b60-a329-0964cc60faf8\") "
Sep 11 13:50:17.919593 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:50:17.919580    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14cd118e-eb64-4b60-a329-0964cc60faf8-workload-socket\") pod \"14cd118e-eb64-4b60-a329-0964cc60faf8\" (UID: \"14cd118e-eb64-4b60-a329-0964cc60faf8\") "
Sep 11 13:50:17.919995 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:50:17.919638    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14cd118e-eb64-4b60-a329-0964cc60faf8-istio-data\") pod \"14cd118e-eb64-4b60-a329-0964cc60faf8\" (UID: \"14cd118e-eb64-4b60-a329-0964cc60faf8\") "
Sep 11 13:50:17.920449 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:50:17.920096    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14cd118e-eb64-4b60-a329-0964cc60faf8-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "14cd118e-eb64-4b60-a329-0964cc60faf8" (UID: "14cd118e-eb64-4b60-a329-0964cc60faf8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 13:50:17.920449 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:50:17.920186    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14cd118e-eb64-4b60-a329-0964cc60faf8-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "14cd118e-eb64-4b60-a329-0964cc60faf8" (UID: "14cd118e-eb64-4b60-a329-0964cc60faf8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 13:50:17.920449 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:50:17.920354    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14cd118e-eb64-4b60-a329-0964cc60faf8-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "14cd118e-eb64-4b60-a329-0964cc60faf8" (UID: "14cd118e-eb64-4b60-a329-0964cc60faf8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 13:50:17.920449 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:50:17.920392    2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14cd118e-eb64-4b60-a329-0964cc60faf8-credential-socket\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:50:17.920449 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:50:17.920416    2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14cd118e-eb64-4b60-a329-0964cc60faf8-workload-certs\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:50:17.920449 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:50:17.920381    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14cd118e-eb64-4b60-a329-0964cc60faf8-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "14cd118e-eb64-4b60-a329-0964cc60faf8" (UID: "14cd118e-eb64-4b60-a329-0964cc60faf8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 13:50:17.921009 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:50:17.920455    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14cd118e-eb64-4b60-a329-0964cc60faf8-istio-data" (OuterVolumeSpecName: "istio-data") pod "14cd118e-eb64-4b60-a329-0964cc60faf8" (UID: "14cd118e-eb64-4b60-a329-0964cc60faf8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 13:50:17.923248 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:50:17.923213    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/14cd118e-eb64-4b60-a329-0964cc60faf8-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "14cd118e-eb64-4b60-a329-0964cc60faf8" (UID: "14cd118e-eb64-4b60-a329-0964cc60faf8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 13:50:17.923394 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:50:17.923243    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14cd118e-eb64-4b60-a329-0964cc60faf8-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "14cd118e-eb64-4b60-a329-0964cc60faf8" (UID: "14cd118e-eb64-4b60-a329-0964cc60faf8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 13:50:17.923615 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:50:17.923585    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14cd118e-eb64-4b60-a329-0964cc60faf8-istio-token" (OuterVolumeSpecName: "istio-token") pod "14cd118e-eb64-4b60-a329-0964cc60faf8" (UID: "14cd118e-eb64-4b60-a329-0964cc60faf8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 13:50:17.924006 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:50:17.923976    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14cd118e-eb64-4b60-a329-0964cc60faf8-kube-api-access-zb8jm" (OuterVolumeSpecName: "kube-api-access-zb8jm") pod "14cd118e-eb64-4b60-a329-0964cc60faf8" (UID: "14cd118e-eb64-4b60-a329-0964cc60faf8"). InnerVolumeSpecName "kube-api-access-zb8jm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 13:50:18.021809 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:50:18.021750    2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14cd118e-eb64-4b60-a329-0964cc60faf8-istio-envoy\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:50:18.021809 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:50:18.021804    2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14cd118e-eb64-4b60-a329-0964cc60faf8-istio-token\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:50:18.021809 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:50:18.021859    2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14cd118e-eb64-4b60-a329-0964cc60faf8-istiod-ca-cert\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:50:18.022188 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:50:18.021872    2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14cd118e-eb64-4b60-a329-0964cc60faf8-istio-podinfo\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:50:18.022188 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:50:18.021886    2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14cd118e-eb64-4b60-a329-0964cc60faf8-workload-socket\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:50:18.022188 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:50:18.021897    2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14cd118e-eb64-4b60-a329-0964cc60faf8-istio-data\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:50:18.022188 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:50:18.021910    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zb8jm\" (UniqueName: \"kubernetes.io/projected/14cd118e-eb64-4b60-a329-0964cc60faf8-kube-api-access-zb8jm\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:50:18.679702 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:50:18.679657    2578 generic.go:356] "Generic (PLEG): container finished" podID="14cd118e-eb64-4b60-a329-0964cc60faf8" containerID="5b5ed2a6fa340d03eb7e84016817c7cb7b5ab96874bfcdcfea3afb97035a3e8a" exitCode=0
Sep 11 13:50:18.679972 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:50:18.679735    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-c8d2f6-openshift-default-77cccb6ccb-2cdcx"
Sep 11 13:50:18.679972 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:50:18.679749    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c8d2f6-openshift-default-77cccb6ccb-2cdcx" event={"ID":"14cd118e-eb64-4b60-a329-0964cc60faf8","Type":"ContainerDied","Data":"5b5ed2a6fa340d03eb7e84016817c7cb7b5ab96874bfcdcfea3afb97035a3e8a"}
Sep 11 13:50:18.679972 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:50:18.679864    2578 scope.go:117] "RemoveContainer" containerID="5b5ed2a6fa340d03eb7e84016817c7cb7b5ab96874bfcdcfea3afb97035a3e8a"
Sep 11 13:50:18.679972 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:50:18.679874    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c8d2f6-openshift-default-77cccb6ccb-2cdcx" event={"ID":"14cd118e-eb64-4b60-a329-0964cc60faf8","Type":"ContainerDied","Data":"3352ff10c1014c15f9dbd197076a04b8bd87b3a49d0d2bc3376f41a5f9b032a6"}
Sep 11 13:50:18.693939 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:50:18.693816    2578 scope.go:117] "RemoveContainer" containerID="5b5ed2a6fa340d03eb7e84016817c7cb7b5ab96874bfcdcfea3afb97035a3e8a"
Sep 11 13:50:18.694560 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:50:18.694523    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b5ed2a6fa340d03eb7e84016817c7cb7b5ab96874bfcdcfea3afb97035a3e8a\": container with ID starting with 5b5ed2a6fa340d03eb7e84016817c7cb7b5ab96874bfcdcfea3afb97035a3e8a not found: ID does not exist" containerID="5b5ed2a6fa340d03eb7e84016817c7cb7b5ab96874bfcdcfea3afb97035a3e8a"
Sep 11 13:50:18.694721 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:50:18.694571    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b5ed2a6fa340d03eb7e84016817c7cb7b5ab96874bfcdcfea3afb97035a3e8a"} err="failed to get container status \"5b5ed2a6fa340d03eb7e84016817c7cb7b5ab96874bfcdcfea3afb97035a3e8a\": rpc error: code = NotFound desc = could not find container \"5b5ed2a6fa340d03eb7e84016817c7cb7b5ab96874bfcdcfea3afb97035a3e8a\": container with ID starting with 5b5ed2a6fa340d03eb7e84016817c7cb7b5ab96874bfcdcfea3afb97035a3e8a not found: ID does not exist"
Sep 11 13:50:18.734428 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:50:18.734389    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-c8d2f6-openshift-default-77cccb6ccb-2cdcx"]
Sep 11 13:50:18.740291 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:50:18.740250    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-c8d2f6-openshift-default-77cccb6ccb-2cdcx"]
Sep 11 13:50:19.109303 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:50:19.109255    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="14cd118e-eb64-4b60-a329-0964cc60faf8" path="/var/lib/kubelet/pods/14cd118e-eb64-4b60-a329-0964cc60faf8/volumes"
Sep 11 13:50:27.072374 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:50:27.072303    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-470eeb-6d66876586-4mrvl"]
Sep 11 13:50:27.073052 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:50:27.073005    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-470eeb-6d66876586-4mrvl" podUID="e62b849e-6379-4858-aeb3-14eea3fc2f52" containerName="maas-api" containerID="cri-o://7bfcd81265fc739f3a58a9a773804872f393521c5e3bd9543d59a970371e5578" gracePeriod=30
Sep 11 13:50:27.385502 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:50:27.385465    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-470eeb-6d66876586-4mrvl"
Sep 11 13:50:27.441177 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:50:27.441127    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e62b849e-6379-4858-aeb3-14eea3fc2f52-openshift-service-ca\") pod \"e62b849e-6379-4858-aeb3-14eea3fc2f52\" (UID: \"e62b849e-6379-4858-aeb3-14eea3fc2f52\") "
Sep 11 13:50:27.441435 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:50:27.441234    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e62b849e-6379-4858-aeb3-14eea3fc2f52-metrics-certs\") pod \"e62b849e-6379-4858-aeb3-14eea3fc2f52\" (UID: \"e62b849e-6379-4858-aeb3-14eea3fc2f52\") "
Sep 11 13:50:27.441435 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:50:27.441301    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e62b849e-6379-4858-aeb3-14eea3fc2f52-maas-api-tls\") pod \"e62b849e-6379-4858-aeb3-14eea3fc2f52\" (UID: \"e62b849e-6379-4858-aeb3-14eea3fc2f52\") "
Sep 11 13:50:27.441435 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:50:27.441349    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vwvpw\" (UniqueName: \"kubernetes.io/projected/e62b849e-6379-4858-aeb3-14eea3fc2f52-kube-api-access-vwvpw\") pod \"e62b849e-6379-4858-aeb3-14eea3fc2f52\" (UID: \"e62b849e-6379-4858-aeb3-14eea3fc2f52\") "
Sep 11 13:50:27.441693 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:50:27.441642    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e62b849e-6379-4858-aeb3-14eea3fc2f52-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e62b849e-6379-4858-aeb3-14eea3fc2f52" (UID: "e62b849e-6379-4858-aeb3-14eea3fc2f52"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 13:50:27.441957 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:50:27.441928    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e62b849e-6379-4858-aeb3-14eea3fc2f52-openshift-service-ca\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:50:27.444207 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:50:27.444168    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e62b849e-6379-4858-aeb3-14eea3fc2f52-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e62b849e-6379-4858-aeb3-14eea3fc2f52" (UID: "e62b849e-6379-4858-aeb3-14eea3fc2f52"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 13:50:27.444385 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:50:27.444168    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e62b849e-6379-4858-aeb3-14eea3fc2f52-kube-api-access-vwvpw" (OuterVolumeSpecName: "kube-api-access-vwvpw") pod "e62b849e-6379-4858-aeb3-14eea3fc2f52" (UID: "e62b849e-6379-4858-aeb3-14eea3fc2f52"). InnerVolumeSpecName "kube-api-access-vwvpw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 13:50:27.444385 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:50:27.444244    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e62b849e-6379-4858-aeb3-14eea3fc2f52-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "e62b849e-6379-4858-aeb3-14eea3fc2f52" (UID: "e62b849e-6379-4858-aeb3-14eea3fc2f52"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 13:50:27.542772 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:50:27.542726    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e62b849e-6379-4858-aeb3-14eea3fc2f52-metrics-certs\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:50:27.542772 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:50:27.542766    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e62b849e-6379-4858-aeb3-14eea3fc2f52-maas-api-tls\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:50:27.542772 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:50:27.542776    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vwvpw\" (UniqueName: \"kubernetes.io/projected/e62b849e-6379-4858-aeb3-14eea3fc2f52-kube-api-access-vwvpw\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:50:27.754448 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:50:27.754408    2578 generic.go:356] "Generic (PLEG): container finished" podID="e62b849e-6379-4858-aeb3-14eea3fc2f52" containerID="7bfcd81265fc739f3a58a9a773804872f393521c5e3bd9543d59a970371e5578" exitCode=0
Sep 11 13:50:27.754712 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:50:27.754503    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-470eeb-6d66876586-4mrvl"
Sep 11 13:50:27.754712 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:50:27.754538    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-470eeb-6d66876586-4mrvl" event={"ID":"e62b849e-6379-4858-aeb3-14eea3fc2f52","Type":"ContainerDied","Data":"7bfcd81265fc739f3a58a9a773804872f393521c5e3bd9543d59a970371e5578"}
Sep 11 13:50:27.754712 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:50:27.754616    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-470eeb-6d66876586-4mrvl" event={"ID":"e62b849e-6379-4858-aeb3-14eea3fc2f52","Type":"ContainerDied","Data":"c8be9aaf69acd4e819b36892a197024391ba74e7ac309984eeb7ae24e8cce78e"}
Sep 11 13:50:27.754712 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:50:27.754652    2578 scope.go:117] "RemoveContainer" containerID="7bfcd81265fc739f3a58a9a773804872f393521c5e3bd9543d59a970371e5578"
Sep 11 13:50:27.766612 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:50:27.766590    2578 scope.go:117] "RemoveContainer" containerID="7bfcd81265fc739f3a58a9a773804872f393521c5e3bd9543d59a970371e5578"
Sep 11 13:50:27.766938 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:50:27.766921    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7bfcd81265fc739f3a58a9a773804872f393521c5e3bd9543d59a970371e5578\": container with ID starting with 7bfcd81265fc739f3a58a9a773804872f393521c5e3bd9543d59a970371e5578 not found: ID does not exist" containerID="7bfcd81265fc739f3a58a9a773804872f393521c5e3bd9543d59a970371e5578"
Sep 11 13:50:27.766985 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:50:27.766951    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7bfcd81265fc739f3a58a9a773804872f393521c5e3bd9543d59a970371e5578"} err="failed to get container status \"7bfcd81265fc739f3a58a9a773804872f393521c5e3bd9543d59a970371e5578\": rpc error: code = NotFound desc = could not find container \"7bfcd81265fc739f3a58a9a773804872f393521c5e3bd9543d59a970371e5578\": container with ID starting with 7bfcd81265fc739f3a58a9a773804872f393521c5e3bd9543d59a970371e5578 not found: ID does not exist"
Sep 11 13:50:27.798224 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:50:27.797945    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-470eeb-6d66876586-4mrvl"]
Sep 11 13:50:27.816576 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:50:27.816527    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-470eeb-6d66876586-4mrvl"]
Sep 11 13:50:29.110629 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:50:29.110557    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e62b849e-6379-4858-aeb3-14eea3fc2f52" path="/var/lib/kubelet/pods/e62b849e-6379-4858-aeb3-14eea3fc2f52/volumes"
Sep 11 13:50:48.749416 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:50:48.749346    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e52aaa-68484dc9c-68kbs"]
Sep 11 13:50:48.750522 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:50:48.749751    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e52aaa-68484dc9c-68kbs" podUID="7dc1eb52-2060-473a-a0d5-4fe00eca2972" containerName="maas-api" containerID="cri-o://cacfb4df4b2719431ca7423d1cd4fd62bf02c7f11c254639033a3048abbd9883" gracePeriod=30
Sep 11 13:50:48.928987 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:50:48.928940    2578 generic.go:356] "Generic (PLEG): container finished" podID="7dc1eb52-2060-473a-a0d5-4fe00eca2972" containerID="cacfb4df4b2719431ca7423d1cd4fd62bf02c7f11c254639033a3048abbd9883" exitCode=0
Sep 11 13:50:48.929264 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:50:48.929009    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e52aaa-68484dc9c-68kbs" event={"ID":"7dc1eb52-2060-473a-a0d5-4fe00eca2972","Type":"ContainerDied","Data":"cacfb4df4b2719431ca7423d1cd4fd62bf02c7f11c254639033a3048abbd9883"}
Sep 11 13:50:49.062721 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:50:49.062683    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e52aaa-68484dc9c-68kbs"
Sep 11 13:50:49.118808 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:50:49.118765    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7dc1eb52-2060-473a-a0d5-4fe00eca2972-openshift-service-ca\") pod \"7dc1eb52-2060-473a-a0d5-4fe00eca2972\" (UID: \"7dc1eb52-2060-473a-a0d5-4fe00eca2972\") "
Sep 11 13:50:49.119063 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:50:49.118873    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7dc1eb52-2060-473a-a0d5-4fe00eca2972-maas-api-tls\") pod \"7dc1eb52-2060-473a-a0d5-4fe00eca2972\" (UID: \"7dc1eb52-2060-473a-a0d5-4fe00eca2972\") "
Sep 11 13:50:49.119063 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:50:49.118990    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7dc1eb52-2060-473a-a0d5-4fe00eca2972-metrics-certs\") pod \"7dc1eb52-2060-473a-a0d5-4fe00eca2972\" (UID: \"7dc1eb52-2060-473a-a0d5-4fe00eca2972\") "
Sep 11 13:50:49.119063 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:50:49.119013    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m2vkk\" (UniqueName: \"kubernetes.io/projected/7dc1eb52-2060-473a-a0d5-4fe00eca2972-kube-api-access-m2vkk\") pod \"7dc1eb52-2060-473a-a0d5-4fe00eca2972\" (UID: \"7dc1eb52-2060-473a-a0d5-4fe00eca2972\") "
Sep 11 13:50:49.119293 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:50:49.119259    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7dc1eb52-2060-473a-a0d5-4fe00eca2972-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7dc1eb52-2060-473a-a0d5-4fe00eca2972" (UID: "7dc1eb52-2060-473a-a0d5-4fe00eca2972"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 13:50:49.119539 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:50:49.119519    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7dc1eb52-2060-473a-a0d5-4fe00eca2972-openshift-service-ca\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:50:49.121655 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:50:49.121623    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7dc1eb52-2060-473a-a0d5-4fe00eca2972-kube-api-access-m2vkk" (OuterVolumeSpecName: "kube-api-access-m2vkk") pod "7dc1eb52-2060-473a-a0d5-4fe00eca2972" (UID: "7dc1eb52-2060-473a-a0d5-4fe00eca2972"). InnerVolumeSpecName "kube-api-access-m2vkk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 13:50:49.121898 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:50:49.121877    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7dc1eb52-2060-473a-a0d5-4fe00eca2972-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7dc1eb52-2060-473a-a0d5-4fe00eca2972" (UID: "7dc1eb52-2060-473a-a0d5-4fe00eca2972"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 13:50:49.122527 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:50:49.122505    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7dc1eb52-2060-473a-a0d5-4fe00eca2972-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "7dc1eb52-2060-473a-a0d5-4fe00eca2972" (UID: "7dc1eb52-2060-473a-a0d5-4fe00eca2972"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 13:50:49.220610 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:50:49.220562    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7dc1eb52-2060-473a-a0d5-4fe00eca2972-maas-api-tls\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:50:49.220610 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:50:49.220605    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7dc1eb52-2060-473a-a0d5-4fe00eca2972-metrics-certs\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:50:49.220610 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:50:49.220619    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m2vkk\" (UniqueName: \"kubernetes.io/projected/7dc1eb52-2060-473a-a0d5-4fe00eca2972-kube-api-access-m2vkk\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:50:49.938390 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:50:49.938335    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e52aaa-68484dc9c-68kbs" event={"ID":"7dc1eb52-2060-473a-a0d5-4fe00eca2972","Type":"ContainerDied","Data":"be2470d58cb9d2c295afdb0c19bd5c02322ef779581695993d15ea921722be0d"}
Sep 11 13:50:49.938390 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:50:49.938393    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e52aaa-68484dc9c-68kbs"
Sep 11 13:50:49.939143 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:50:49.938426    2578 scope.go:117] "RemoveContainer" containerID="cacfb4df4b2719431ca7423d1cd4fd62bf02c7f11c254639033a3048abbd9883"
Sep 11 13:50:49.961951 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:50:49.961903    2578 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7dc1eb52_2060_473a_a0d5_4fe00eca2972.slice/crio-be2470d58cb9d2c295afdb0c19bd5c02322ef779581695993d15ea921722be0d\": RecentStats: unable to find data in memory cache]"
Sep 11 13:50:49.964594 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:50:49.964547    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e52aaa-68484dc9c-68kbs"]
Sep 11 13:50:49.971585 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:50:49.971543    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e52aaa-68484dc9c-68kbs"]
Sep 11 13:50:51.109167 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:50:51.109125    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7dc1eb52-2060-473a-a0d5-4fe00eca2972" path="/var/lib/kubelet/pods/7dc1eb52-2060-473a-a0d5-4fe00eca2972/volumes"
Sep 11 13:50:53.246181 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:50:53.246121    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-082f62-5967dc89bd-clbhs"]
Sep 11 13:50:53.247271 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:50:53.247214    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-082f62-5967dc89bd-clbhs" podUID="bc25932e-1541-4894-848c-b1a86210d91f" containerName="maas-api" containerID="cri-o://adbcf86ad6dffaac8b46a96e6e82cee6217e1ddf28ccfbf460f47285f5b15d69" gracePeriod=30
Sep 11 13:50:53.596411 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:50:53.596373    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-082f62-5967dc89bd-clbhs"
Sep 11 13:50:53.699364 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:50:53.699317    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc25932e-1541-4894-848c-b1a86210d91f-maas-api-tls\") pod \"bc25932e-1541-4894-848c-b1a86210d91f\" (UID: \"bc25932e-1541-4894-848c-b1a86210d91f\") "
Sep 11 13:50:53.699570 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:50:53.699429    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lhncz\" (UniqueName: \"kubernetes.io/projected/bc25932e-1541-4894-848c-b1a86210d91f-kube-api-access-lhncz\") pod \"bc25932e-1541-4894-848c-b1a86210d91f\" (UID: \"bc25932e-1541-4894-848c-b1a86210d91f\") "
Sep 11 13:50:53.699570 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:50:53.699475    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc25932e-1541-4894-848c-b1a86210d91f-metrics-certs\") pod \"bc25932e-1541-4894-848c-b1a86210d91f\" (UID: \"bc25932e-1541-4894-848c-b1a86210d91f\") "
Sep 11 13:50:53.699570 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:50:53.699521    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc25932e-1541-4894-848c-b1a86210d91f-openshift-service-ca\") pod \"bc25932e-1541-4894-848c-b1a86210d91f\" (UID: \"bc25932e-1541-4894-848c-b1a86210d91f\") "
Sep 11 13:50:53.700009 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:50:53.699974    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc25932e-1541-4894-848c-b1a86210d91f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "bc25932e-1541-4894-848c-b1a86210d91f" (UID: "bc25932e-1541-4894-848c-b1a86210d91f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 13:50:53.702149 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:50:53.702117    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc25932e-1541-4894-848c-b1a86210d91f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "bc25932e-1541-4894-848c-b1a86210d91f" (UID: "bc25932e-1541-4894-848c-b1a86210d91f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 13:50:53.702418 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:50:53.702379    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc25932e-1541-4894-848c-b1a86210d91f-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "bc25932e-1541-4894-848c-b1a86210d91f" (UID: "bc25932e-1541-4894-848c-b1a86210d91f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 13:50:53.702418 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:50:53.702408    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc25932e-1541-4894-848c-b1a86210d91f-kube-api-access-lhncz" (OuterVolumeSpecName: "kube-api-access-lhncz") pod "bc25932e-1541-4894-848c-b1a86210d91f" (UID: "bc25932e-1541-4894-848c-b1a86210d91f"). InnerVolumeSpecName "kube-api-access-lhncz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 13:50:53.801477 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:50:53.801433    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc25932e-1541-4894-848c-b1a86210d91f-metrics-certs\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:50:53.801477 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:50:53.801472    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc25932e-1541-4894-848c-b1a86210d91f-openshift-service-ca\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:50:53.801477 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:50:53.801483    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc25932e-1541-4894-848c-b1a86210d91f-maas-api-tls\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:50:53.801477 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:50:53.801492    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lhncz\" (UniqueName: \"kubernetes.io/projected/bc25932e-1541-4894-848c-b1a86210d91f-kube-api-access-lhncz\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:50:53.981418 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:50:53.981368    2578 generic.go:356] "Generic (PLEG): container finished" podID="bc25932e-1541-4894-848c-b1a86210d91f" containerID="adbcf86ad6dffaac8b46a96e6e82cee6217e1ddf28ccfbf460f47285f5b15d69" exitCode=0
Sep 11 13:50:53.981656 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:50:53.981435    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-082f62-5967dc89bd-clbhs"
Sep 11 13:50:53.981656 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:50:53.981436    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-082f62-5967dc89bd-clbhs" event={"ID":"bc25932e-1541-4894-848c-b1a86210d91f","Type":"ContainerDied","Data":"adbcf86ad6dffaac8b46a96e6e82cee6217e1ddf28ccfbf460f47285f5b15d69"}
Sep 11 13:50:53.981656 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:50:53.981523    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-082f62-5967dc89bd-clbhs" event={"ID":"bc25932e-1541-4894-848c-b1a86210d91f","Type":"ContainerDied","Data":"e5cf563d6b555a9357fac72e145f1d6ba5a096df51b655b791062008583ae68b"}
Sep 11 13:50:53.981656 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:50:53.981558    2578 scope.go:117] "RemoveContainer" containerID="adbcf86ad6dffaac8b46a96e6e82cee6217e1ddf28ccfbf460f47285f5b15d69"
Sep 11 13:50:53.997352 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:50:53.997322    2578 scope.go:117] "RemoveContainer" containerID="adbcf86ad6dffaac8b46a96e6e82cee6217e1ddf28ccfbf460f47285f5b15d69"
Sep 11 13:50:53.998041 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:50:53.998001    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"adbcf86ad6dffaac8b46a96e6e82cee6217e1ddf28ccfbf460f47285f5b15d69\": container with ID starting with adbcf86ad6dffaac8b46a96e6e82cee6217e1ddf28ccfbf460f47285f5b15d69 not found: ID does not exist" containerID="adbcf86ad6dffaac8b46a96e6e82cee6217e1ddf28ccfbf460f47285f5b15d69"
Sep 11 13:50:53.998193 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:50:53.998057    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"adbcf86ad6dffaac8b46a96e6e82cee6217e1ddf28ccfbf460f47285f5b15d69"} err="failed to get container status \"adbcf86ad6dffaac8b46a96e6e82cee6217e1ddf28ccfbf460f47285f5b15d69\": rpc error: code = NotFound desc = could not find container \"adbcf86ad6dffaac8b46a96e6e82cee6217e1ddf28ccfbf460f47285f5b15d69\": container with ID starting with adbcf86ad6dffaac8b46a96e6e82cee6217e1ddf28ccfbf460f47285f5b15d69 not found: ID does not exist"
Sep 11 13:50:54.012037 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:50:54.011980    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-082f62-5967dc89bd-clbhs"]
Sep 11 13:50:54.017100 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:50:54.017051    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-082f62-5967dc89bd-clbhs"]
Sep 11 13:50:55.108448 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:50:55.108403    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bc25932e-1541-4894-848c-b1a86210d91f" path="/var/lib/kubelet/pods/bc25932e-1541-4894-848c-b1a86210d91f/volumes"
Sep 11 13:51:01.188145 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:51:01.188081    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-082f62-openshift-default-7d446b6f4d-cvpwk"]
Sep 11 13:51:01.188780 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:51:01.188664    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-082f62-openshift-default-7d446b6f4d-cvpwk" podUID="f5eb98d8-6523-411d-bcfb-ad46743c0c72" containerName="istio-proxy" containerID="cri-o://1fb9677dad6b50981cb02f1a14e4532e4bc2f632f20a4e13875b29f1dba28066" gracePeriod=30
Sep 11 13:51:02.939278 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:51:02.939231    2578 patch_prober.go:28] interesting pod/e2e-worker-w0-082f62-openshift-default-7d446b6f4d-cvpwk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 11 13:51:02.939691 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:51:02.939315    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-082f62-openshift-default-7d446b6f4d-cvpwk" podUID="f5eb98d8-6523-411d-bcfb-ad46743c0c72" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 13:51:06.480985 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:51:06.480944    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-082f62-openshift-default-7d446b6f4d-cvpwk"
Sep 11 13:51:06.559237 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:51:06.559179    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5eb98d8-6523-411d-bcfb-ad46743c0c72-workload-certs\") pod \"f5eb98d8-6523-411d-bcfb-ad46743c0c72\" (UID: \"f5eb98d8-6523-411d-bcfb-ad46743c0c72\") "
Sep 11 13:51:06.559237 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:51:06.559235    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ltkwk\" (UniqueName: \"kubernetes.io/projected/f5eb98d8-6523-411d-bcfb-ad46743c0c72-kube-api-access-ltkwk\") pod \"f5eb98d8-6523-411d-bcfb-ad46743c0c72\" (UID: \"f5eb98d8-6523-411d-bcfb-ad46743c0c72\") "
Sep 11 13:51:06.559492 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:51:06.559287    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5eb98d8-6523-411d-bcfb-ad46743c0c72-istio-token\") pod \"f5eb98d8-6523-411d-bcfb-ad46743c0c72\" (UID: \"f5eb98d8-6523-411d-bcfb-ad46743c0c72\") "
Sep 11 13:51:06.559492 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:51:06.559360    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5eb98d8-6523-411d-bcfb-ad46743c0c72-workload-socket\") pod \"f5eb98d8-6523-411d-bcfb-ad46743c0c72\" (UID: \"f5eb98d8-6523-411d-bcfb-ad46743c0c72\") "
Sep 11 13:51:06.559492 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:51:06.559384    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5eb98d8-6523-411d-bcfb-ad46743c0c72-istiod-ca-cert\") pod \"f5eb98d8-6523-411d-bcfb-ad46743c0c72\" (UID: \"f5eb98d8-6523-411d-bcfb-ad46743c0c72\") "
Sep 11 13:51:06.559492 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:51:06.559411    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5eb98d8-6523-411d-bcfb-ad46743c0c72-credential-socket\") pod \"f5eb98d8-6523-411d-bcfb-ad46743c0c72\" (UID: \"f5eb98d8-6523-411d-bcfb-ad46743c0c72\") "
Sep 11 13:51:06.559492 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:51:06.559466    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5eb98d8-6523-411d-bcfb-ad46743c0c72-istio-podinfo\") pod \"f5eb98d8-6523-411d-bcfb-ad46743c0c72\" (UID: \"f5eb98d8-6523-411d-bcfb-ad46743c0c72\") "
Sep 11 13:51:06.559759 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:51:06.559494    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5eb98d8-6523-411d-bcfb-ad46743c0c72-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f5eb98d8-6523-411d-bcfb-ad46743c0c72" (UID: "f5eb98d8-6523-411d-bcfb-ad46743c0c72"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 13:51:06.559759 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:51:06.559511    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5eb98d8-6523-411d-bcfb-ad46743c0c72-istio-envoy\") pod \"f5eb98d8-6523-411d-bcfb-ad46743c0c72\" (UID: \"f5eb98d8-6523-411d-bcfb-ad46743c0c72\") "
Sep 11 13:51:06.559759 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:51:06.559586    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5eb98d8-6523-411d-bcfb-ad46743c0c72-istio-data\") pod \"f5eb98d8-6523-411d-bcfb-ad46743c0c72\" (UID: \"f5eb98d8-6523-411d-bcfb-ad46743c0c72\") "
Sep 11 13:51:06.559759 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:51:06.559654    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5eb98d8-6523-411d-bcfb-ad46743c0c72-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f5eb98d8-6523-411d-bcfb-ad46743c0c72" (UID: "f5eb98d8-6523-411d-bcfb-ad46743c0c72"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 13:51:06.559759 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:51:06.559674    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5eb98d8-6523-411d-bcfb-ad46743c0c72-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f5eb98d8-6523-411d-bcfb-ad46743c0c72" (UID: "f5eb98d8-6523-411d-bcfb-ad46743c0c72"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 13:51:06.560081 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:51:06.559955    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5eb98d8-6523-411d-bcfb-ad46743c0c72-istio-data" (OuterVolumeSpecName: "istio-data") pod "f5eb98d8-6523-411d-bcfb-ad46743c0c72" (UID: "f5eb98d8-6523-411d-bcfb-ad46743c0c72"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 13:51:06.560081 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:51:06.560006    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5eb98d8-6523-411d-bcfb-ad46743c0c72-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f5eb98d8-6523-411d-bcfb-ad46743c0c72" (UID: "f5eb98d8-6523-411d-bcfb-ad46743c0c72"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 13:51:06.560194 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:51:06.560107    2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5eb98d8-6523-411d-bcfb-ad46743c0c72-workload-socket\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:51:06.560194 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:51:06.560125    2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5eb98d8-6523-411d-bcfb-ad46743c0c72-credential-socket\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:51:06.560194 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:51:06.560139    2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5eb98d8-6523-411d-bcfb-ad46743c0c72-workload-certs\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:51:06.562074 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:51:06.562044    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f5eb98d8-6523-411d-bcfb-ad46743c0c72-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f5eb98d8-6523-411d-bcfb-ad46743c0c72" (UID: "f5eb98d8-6523-411d-bcfb-ad46743c0c72"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 13:51:06.562074 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:51:06.562047    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5eb98d8-6523-411d-bcfb-ad46743c0c72-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f5eb98d8-6523-411d-bcfb-ad46743c0c72" (UID: "f5eb98d8-6523-411d-bcfb-ad46743c0c72"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 13:51:06.562277 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:51:06.562245    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5eb98d8-6523-411d-bcfb-ad46743c0c72-istio-token" (OuterVolumeSpecName: "istio-token") pod "f5eb98d8-6523-411d-bcfb-ad46743c0c72" (UID: "f5eb98d8-6523-411d-bcfb-ad46743c0c72"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 13:51:06.562277 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:51:06.562259    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5eb98d8-6523-411d-bcfb-ad46743c0c72-kube-api-access-ltkwk" (OuterVolumeSpecName: "kube-api-access-ltkwk") pod "f5eb98d8-6523-411d-bcfb-ad46743c0c72" (UID: "f5eb98d8-6523-411d-bcfb-ad46743c0c72"). InnerVolumeSpecName "kube-api-access-ltkwk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 13:51:06.661226 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:51:06.661149    2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5eb98d8-6523-411d-bcfb-ad46743c0c72-istio-token\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:51:06.661226 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:51:06.661213    2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5eb98d8-6523-411d-bcfb-ad46743c0c72-istiod-ca-cert\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:51:06.661226 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:51:06.661223    2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5eb98d8-6523-411d-bcfb-ad46743c0c72-istio-podinfo\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:51:06.661226 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:51:06.661232    2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5eb98d8-6523-411d-bcfb-ad46743c0c72-istio-envoy\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:51:06.661226 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:51:06.661240    2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5eb98d8-6523-411d-bcfb-ad46743c0c72-istio-data\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:51:06.661226 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:51:06.661252    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ltkwk\" (UniqueName: \"kubernetes.io/projected/f5eb98d8-6523-411d-bcfb-ad46743c0c72-kube-api-access-ltkwk\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:51:07.082057 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:51:07.082010    2578 generic.go:356] "Generic (PLEG): container finished" podID="f5eb98d8-6523-411d-bcfb-ad46743c0c72" containerID="1fb9677dad6b50981cb02f1a14e4532e4bc2f632f20a4e13875b29f1dba28066" exitCode=0
Sep 11 13:51:07.082304 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:51:07.082068    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-082f62-openshift-default-7d446b6f4d-cvpwk"
Sep 11 13:51:07.082304 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:51:07.082095    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-082f62-openshift-default-7d446b6f4d-cvpwk" event={"ID":"f5eb98d8-6523-411d-bcfb-ad46743c0c72","Type":"ContainerDied","Data":"1fb9677dad6b50981cb02f1a14e4532e4bc2f632f20a4e13875b29f1dba28066"}
Sep 11 13:51:07.082304 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:51:07.082160    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-082f62-openshift-default-7d446b6f4d-cvpwk" event={"ID":"f5eb98d8-6523-411d-bcfb-ad46743c0c72","Type":"ContainerDied","Data":"be33f874ae891833b7fbf87dc2ce820740a53f9eef5ca47ff186e18f9e4506bb"}
Sep 11 13:51:07.082304 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:51:07.082184    2578 scope.go:117] "RemoveContainer" containerID="1fb9677dad6b50981cb02f1a14e4532e4bc2f632f20a4e13875b29f1dba28066"
Sep 11 13:51:07.096337 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:51:07.096306    2578 scope.go:117] "RemoveContainer" containerID="1248566e39bbb57e4c0d11f4b0d9dbc25bb8ec98668d6c379f57521d66a53bea"
Sep 11 13:51:07.108679 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:51:07.108650    2578 scope.go:117] "RemoveContainer" containerID="1fb9677dad6b50981cb02f1a14e4532e4bc2f632f20a4e13875b29f1dba28066"
Sep 11 13:51:07.109494 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:51:07.109454    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1fb9677dad6b50981cb02f1a14e4532e4bc2f632f20a4e13875b29f1dba28066\": container with ID starting with 1fb9677dad6b50981cb02f1a14e4532e4bc2f632f20a4e13875b29f1dba28066 not found: ID does not exist" containerID="1fb9677dad6b50981cb02f1a14e4532e4bc2f632f20a4e13875b29f1dba28066"
Sep 11 13:51:07.109659 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:51:07.109507    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1fb9677dad6b50981cb02f1a14e4532e4bc2f632f20a4e13875b29f1dba28066"} err="failed to get container status \"1fb9677dad6b50981cb02f1a14e4532e4bc2f632f20a4e13875b29f1dba28066\": rpc error: code = NotFound desc = could not find container \"1fb9677dad6b50981cb02f1a14e4532e4bc2f632f20a4e13875b29f1dba28066\": container with ID starting with 1fb9677dad6b50981cb02f1a14e4532e4bc2f632f20a4e13875b29f1dba28066 not found: ID does not exist"
Sep 11 13:51:07.109659 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:51:07.109541    2578 scope.go:117] "RemoveContainer" containerID="1248566e39bbb57e4c0d11f4b0d9dbc25bb8ec98668d6c379f57521d66a53bea"
Sep 11 13:51:07.110026 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:51:07.109995    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1248566e39bbb57e4c0d11f4b0d9dbc25bb8ec98668d6c379f57521d66a53bea\": container with ID starting with 1248566e39bbb57e4c0d11f4b0d9dbc25bb8ec98668d6c379f57521d66a53bea not found: ID does not exist" containerID="1248566e39bbb57e4c0d11f4b0d9dbc25bb8ec98668d6c379f57521d66a53bea"
Sep 11 13:51:07.110135 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:51:07.110042    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1248566e39bbb57e4c0d11f4b0d9dbc25bb8ec98668d6c379f57521d66a53bea"} err="failed to get container status \"1248566e39bbb57e4c0d11f4b0d9dbc25bb8ec98668d6c379f57521d66a53bea\": rpc error: code = NotFound desc = could not find container \"1248566e39bbb57e4c0d11f4b0d9dbc25bb8ec98668d6c379f57521d66a53bea\": container with ID starting with 1248566e39bbb57e4c0d11f4b0d9dbc25bb8ec98668d6c379f57521d66a53bea not found: ID does not exist"
Sep 11 13:51:07.111022 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:51:07.110997    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-082f62-openshift-default-7d446b6f4d-cvpwk"]
Sep 11 13:51:07.113907 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:51:07.113882    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-082f62-openshift-default-7d446b6f4d-cvpwk"]
Sep 11 13:51:09.110769 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:51:09.110729    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f5eb98d8-6523-411d-bcfb-ad46743c0c72" path="/var/lib/kubelet/pods/f5eb98d8-6523-411d-bcfb-ad46743c0c72/volumes"
Sep 11 13:51:33.258734 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:51:33.258698    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6gltj_e384caae-3cb1-4264-8a25-55b40ac8ac45/ovn-acl-logging/0.log"
Sep 11 13:51:33.263059 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:51:33.263029    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6gltj_e384caae-3cb1-4264-8a25-55b40ac8ac45/ovn-acl-logging/0.log"
Sep 11 13:51:59.385795 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:51:59.385751    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-371c38/premium-w4-371c38-kserve-5d8c655487-qw6kh"]
Sep 11 13:52:10.394370 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:10.394318    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-371c38-859578c76c-fgtss"]
Sep 11 13:52:10.395160 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:10.394664    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-371c38-859578c76c-fgtss" podUID="dbb78093-57b6-4adf-9b31-7d2bf633100c" containerName="maas-api" containerID="cri-o://5f062c962eda648c1103b4f85eaafa6e0c63e84b901155db04e4fe18b5f9eec9" gracePeriod=30
Sep 11 13:52:10.527032 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:10.526989    2578 generic.go:356] "Generic (PLEG): container finished" podID="dbb78093-57b6-4adf-9b31-7d2bf633100c" containerID="5f062c962eda648c1103b4f85eaafa6e0c63e84b901155db04e4fe18b5f9eec9" exitCode=0
Sep 11 13:52:10.527262 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:10.527059    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-371c38-859578c76c-fgtss" event={"ID":"dbb78093-57b6-4adf-9b31-7d2bf633100c","Type":"ContainerDied","Data":"5f062c962eda648c1103b4f85eaafa6e0c63e84b901155db04e4fe18b5f9eec9"}
Sep 11 13:52:10.704542 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:10.704502    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-371c38-859578c76c-fgtss"
Sep 11 13:52:10.861404 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:10.861347    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dbb78093-57b6-4adf-9b31-7d2bf633100c-maas-api-tls\") pod \"dbb78093-57b6-4adf-9b31-7d2bf633100c\" (UID: \"dbb78093-57b6-4adf-9b31-7d2bf633100c\") "
Sep 11 13:52:10.861404 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:10.861394    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dbb78093-57b6-4adf-9b31-7d2bf633100c-metrics-certs\") pod \"dbb78093-57b6-4adf-9b31-7d2bf633100c\" (UID: \"dbb78093-57b6-4adf-9b31-7d2bf633100c\") "
Sep 11 13:52:10.861790 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:10.861427    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5czw8\" (UniqueName: \"kubernetes.io/projected/dbb78093-57b6-4adf-9b31-7d2bf633100c-kube-api-access-5czw8\") pod \"dbb78093-57b6-4adf-9b31-7d2bf633100c\" (UID: \"dbb78093-57b6-4adf-9b31-7d2bf633100c\") "
Sep 11 13:52:10.861790 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:10.861476    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbb78093-57b6-4adf-9b31-7d2bf633100c-openshift-service-ca\") pod \"dbb78093-57b6-4adf-9b31-7d2bf633100c\" (UID: \"dbb78093-57b6-4adf-9b31-7d2bf633100c\") "
Sep 11 13:52:10.861958 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:10.861904    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbb78093-57b6-4adf-9b31-7d2bf633100c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "dbb78093-57b6-4adf-9b31-7d2bf633100c" (UID: "dbb78093-57b6-4adf-9b31-7d2bf633100c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 13:52:10.864060 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:10.864032    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dbb78093-57b6-4adf-9b31-7d2bf633100c-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "dbb78093-57b6-4adf-9b31-7d2bf633100c" (UID: "dbb78093-57b6-4adf-9b31-7d2bf633100c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 13:52:10.864170 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:10.864061    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbb78093-57b6-4adf-9b31-7d2bf633100c-kube-api-access-5czw8" (OuterVolumeSpecName: "kube-api-access-5czw8") pod "dbb78093-57b6-4adf-9b31-7d2bf633100c" (UID: "dbb78093-57b6-4adf-9b31-7d2bf633100c"). InnerVolumeSpecName "kube-api-access-5czw8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 13:52:10.864170 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:10.864061    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dbb78093-57b6-4adf-9b31-7d2bf633100c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "dbb78093-57b6-4adf-9b31-7d2bf633100c" (UID: "dbb78093-57b6-4adf-9b31-7d2bf633100c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 13:52:10.963383 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:10.963239    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dbb78093-57b6-4adf-9b31-7d2bf633100c-maas-api-tls\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:52:10.963383 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:10.963294    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dbb78093-57b6-4adf-9b31-7d2bf633100c-metrics-certs\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:52:10.963383 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:10.963312    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5czw8\" (UniqueName: \"kubernetes.io/projected/dbb78093-57b6-4adf-9b31-7d2bf633100c-kube-api-access-5czw8\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:52:10.963383 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:10.963329    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbb78093-57b6-4adf-9b31-7d2bf633100c-openshift-service-ca\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:52:11.535500 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:11.535466    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-371c38-859578c76c-fgtss"
Sep 11 13:52:11.536115 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:11.535506    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-371c38-859578c76c-fgtss" event={"ID":"dbb78093-57b6-4adf-9b31-7d2bf633100c","Type":"ContainerDied","Data":"dfabaea28ab28d79d8ee6bc9a660a24387b15cc9e9f29c63ec9ba4386a3784ea"}
Sep 11 13:52:11.536115 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:11.535595    2578 scope.go:117] "RemoveContainer" containerID="5f062c962eda648c1103b4f85eaafa6e0c63e84b901155db04e4fe18b5f9eec9"
Sep 11 13:52:11.564644 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:11.564599    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-371c38-859578c76c-fgtss"]
Sep 11 13:52:11.573030 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:11.572991    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-371c38-859578c76c-fgtss"]
Sep 11 13:52:13.108106 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:13.108068    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dbb78093-57b6-4adf-9b31-7d2bf633100c" path="/var/lib/kubelet/pods/dbb78093-57b6-4adf-9b31-7d2bf633100c/volumes"
Sep 11 13:52:14.413538 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:14.413477    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-371c38/premium-w4-371c38-kserve-5d8c655487-qw6kh" podUID="848ec50c-75af-4c07-be1a-cb8bf34c0c08" containerName="main" containerID="cri-o://9f44f8e81aebc5e94280961db301fab0ec54644de1ac42c545ebe58eac075f13" gracePeriod=45
Sep 11 13:52:14.560118 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:14.560064    2578 generic.go:356] "Generic (PLEG): container finished" podID="848ec50c-75af-4c07-be1a-cb8bf34c0c08" containerID="9f44f8e81aebc5e94280961db301fab0ec54644de1ac42c545ebe58eac075f13" exitCode=0
Sep 11 13:52:14.560423 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:14.560142    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-371c38/premium-w4-371c38-kserve-5d8c655487-qw6kh" event={"ID":"848ec50c-75af-4c07-be1a-cb8bf34c0c08","Type":"ContainerDied","Data":"9f44f8e81aebc5e94280961db301fab0ec54644de1ac42c545ebe58eac075f13"}
Sep 11 13:52:14.687134 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:14.687088    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-371c38/premium-w4-371c38-kserve-5d8c655487-qw6kh"
Sep 11 13:52:14.709128 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:14.709076    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/848ec50c-75af-4c07-be1a-cb8bf34c0c08-model-cache\") pod \"848ec50c-75af-4c07-be1a-cb8bf34c0c08\" (UID: \"848ec50c-75af-4c07-be1a-cb8bf34c0c08\") "
Sep 11 13:52:14.709347 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:14.709196    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/848ec50c-75af-4c07-be1a-cb8bf34c0c08-home\") pod \"848ec50c-75af-4c07-be1a-cb8bf34c0c08\" (UID: \"848ec50c-75af-4c07-be1a-cb8bf34c0c08\") "
Sep 11 13:52:14.709347 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:14.709245    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/848ec50c-75af-4c07-be1a-cb8bf34c0c08-tmp-dir\") pod \"848ec50c-75af-4c07-be1a-cb8bf34c0c08\" (UID: \"848ec50c-75af-4c07-be1a-cb8bf34c0c08\") "
Sep 11 13:52:14.709482 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:14.709346    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/848ec50c-75af-4c07-be1a-cb8bf34c0c08-dshm\") pod \"848ec50c-75af-4c07-be1a-cb8bf34c0c08\" (UID: \"848ec50c-75af-4c07-be1a-cb8bf34c0c08\") "
Sep 11 13:52:14.709482 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:14.709417    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/848ec50c-75af-4c07-be1a-cb8bf34c0c08-model-cache" (OuterVolumeSpecName: "model-cache") pod "848ec50c-75af-4c07-be1a-cb8bf34c0c08" (UID: "848ec50c-75af-4c07-be1a-cb8bf34c0c08"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 13:52:14.709482 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:14.709423    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-66ccf\" (UniqueName: \"kubernetes.io/projected/848ec50c-75af-4c07-be1a-cb8bf34c0c08-kube-api-access-66ccf\") pod \"848ec50c-75af-4c07-be1a-cb8bf34c0c08\" (UID: \"848ec50c-75af-4c07-be1a-cb8bf34c0c08\") "
Sep 11 13:52:14.709642 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:14.709485    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/848ec50c-75af-4c07-be1a-cb8bf34c0c08-home" (OuterVolumeSpecName: "home") pod "848ec50c-75af-4c07-be1a-cb8bf34c0c08" (UID: "848ec50c-75af-4c07-be1a-cb8bf34c0c08"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 13:52:14.709642 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:14.709512    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/848ec50c-75af-4c07-be1a-cb8bf34c0c08-tls-certs\") pod \"848ec50c-75af-4c07-be1a-cb8bf34c0c08\" (UID: \"848ec50c-75af-4c07-be1a-cb8bf34c0c08\") "
Sep 11 13:52:14.709642 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:14.709583    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/848ec50c-75af-4c07-be1a-cb8bf34c0c08-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "848ec50c-75af-4c07-be1a-cb8bf34c0c08" (UID: "848ec50c-75af-4c07-be1a-cb8bf34c0c08"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 13:52:14.710022 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:14.710003    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/848ec50c-75af-4c07-be1a-cb8bf34c0c08-model-cache\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:52:14.710022 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:14.710022    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/848ec50c-75af-4c07-be1a-cb8bf34c0c08-home\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:52:14.710358 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:14.710031    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/848ec50c-75af-4c07-be1a-cb8bf34c0c08-tmp-dir\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:52:14.711995 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:14.711970    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/848ec50c-75af-4c07-be1a-cb8bf34c0c08-dshm" (OuterVolumeSpecName: "dshm") pod "848ec50c-75af-4c07-be1a-cb8bf34c0c08" (UID: "848ec50c-75af-4c07-be1a-cb8bf34c0c08"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 13:52:14.712123 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:14.712002    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/848ec50c-75af-4c07-be1a-cb8bf34c0c08-kube-api-access-66ccf" (OuterVolumeSpecName: "kube-api-access-66ccf") pod "848ec50c-75af-4c07-be1a-cb8bf34c0c08" (UID: "848ec50c-75af-4c07-be1a-cb8bf34c0c08"). InnerVolumeSpecName "kube-api-access-66ccf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 13:52:14.712123 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:14.712063    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/848ec50c-75af-4c07-be1a-cb8bf34c0c08-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "848ec50c-75af-4c07-be1a-cb8bf34c0c08" (UID: "848ec50c-75af-4c07-be1a-cb8bf34c0c08"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 13:52:14.811754 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:14.811698    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/848ec50c-75af-4c07-be1a-cb8bf34c0c08-dshm\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:52:14.811754 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:14.811750    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-66ccf\" (UniqueName: \"kubernetes.io/projected/848ec50c-75af-4c07-be1a-cb8bf34c0c08-kube-api-access-66ccf\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:52:14.811754 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:14.811765    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/848ec50c-75af-4c07-be1a-cb8bf34c0c08-tls-certs\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:52:15.568199 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:15.568152    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-371c38/premium-w4-371c38-kserve-5d8c655487-qw6kh"
Sep 11 13:52:15.568199 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:15.568164    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-371c38/premium-w4-371c38-kserve-5d8c655487-qw6kh" event={"ID":"848ec50c-75af-4c07-be1a-cb8bf34c0c08","Type":"ContainerDied","Data":"81622ba7e088298ca00efcf36e0330a69ea9408a299bd08b2116767b2d318d6c"}
Sep 11 13:52:15.568971 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:15.568255    2578 scope.go:117] "RemoveContainer" containerID="9f44f8e81aebc5e94280961db301fab0ec54644de1ac42c545ebe58eac075f13"
Sep 11 13:52:15.593332 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:15.593277    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-371c38/premium-w4-371c38-kserve-5d8c655487-qw6kh"]
Sep 11 13:52:15.596524 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:15.596459    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-371c38/premium-w4-371c38-kserve-5d8c655487-qw6kh"]
Sep 11 13:52:17.108133 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:17.108080    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="848ec50c-75af-4c07-be1a-cb8bf34c0c08" path="/var/lib/kubelet/pods/848ec50c-75af-4c07-be1a-cb8bf34c0c08/volumes"
Sep 11 13:52:19.302321 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:19.302254    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-371c38-openshift-default-668769478-66p8n"]
Sep 11 13:52:19.302962 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:19.302600    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-371c38-openshift-default-668769478-66p8n" podUID="301e0569-db5c-45be-865c-491221d4883c" containerName="istio-proxy" containerID="cri-o://0aa2c4f8407c4a21a81095171f51f7843d615426f87544c5f99ca0e5d7f02483" gracePeriod=30
Sep 11 13:52:20.746784 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:20.746732    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-9edb53/auth-test-model-9edb53-kserve-5d97c5f9c9-nmzxg"]
Sep 11 13:52:24.293733 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:24.293681    2578 patch_prober.go:28] interesting pod/e2e-worker-w4-371c38-openshift-default-668769478-66p8n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 11 13:52:24.294209 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:24.293778    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-371c38-openshift-default-668769478-66p8n" podUID="301e0569-db5c-45be-865c-491221d4883c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 13:52:24.574981 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:24.574944    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-371c38-openshift-default-668769478-66p8n"
Sep 11 13:52:24.638959 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:24.638909    2578 generic.go:356] "Generic (PLEG): container finished" podID="301e0569-db5c-45be-865c-491221d4883c" containerID="0aa2c4f8407c4a21a81095171f51f7843d615426f87544c5f99ca0e5d7f02483" exitCode=0
Sep 11 13:52:24.638959 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:24.638965    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-371c38-openshift-default-668769478-66p8n" event={"ID":"301e0569-db5c-45be-865c-491221d4883c","Type":"ContainerDied","Data":"0aa2c4f8407c4a21a81095171f51f7843d615426f87544c5f99ca0e5d7f02483"}
Sep 11 13:52:24.639268 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:24.639007    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-371c38-openshift-default-668769478-66p8n"
Sep 11 13:52:24.639268 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:24.639023    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-371c38-openshift-default-668769478-66p8n" event={"ID":"301e0569-db5c-45be-865c-491221d4883c","Type":"ContainerDied","Data":"46e8d25ea9afac6f6d67aadfb6cdf773480a44955679882094fb42460861ddc9"}
Sep 11 13:52:24.639268 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:24.639049    2578 scope.go:117] "RemoveContainer" containerID="0aa2c4f8407c4a21a81095171f51f7843d615426f87544c5f99ca0e5d7f02483"
Sep 11 13:52:24.642902 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:24.642793    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/301e0569-db5c-45be-865c-491221d4883c-istio-data\") pod \"301e0569-db5c-45be-865c-491221d4883c\" (UID: \"301e0569-db5c-45be-865c-491221d4883c\") "
Sep 11 13:52:24.643096 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:24.642903    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/301e0569-db5c-45be-865c-491221d4883c-workload-socket\") pod \"301e0569-db5c-45be-865c-491221d4883c\" (UID: \"301e0569-db5c-45be-865c-491221d4883c\") "
Sep 11 13:52:24.643096 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:24.642950    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/301e0569-db5c-45be-865c-491221d4883c-istio-envoy\") pod \"301e0569-db5c-45be-865c-491221d4883c\" (UID: \"301e0569-db5c-45be-865c-491221d4883c\") "
Sep 11 13:52:24.643096 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:24.643031    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/301e0569-db5c-45be-865c-491221d4883c-istiod-ca-cert\") pod \"301e0569-db5c-45be-865c-491221d4883c\" (UID: \"301e0569-db5c-45be-865c-491221d4883c\") "
Sep 11 13:52:24.643096 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:24.643095    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/301e0569-db5c-45be-865c-491221d4883c-workload-certs\") pod \"301e0569-db5c-45be-865c-491221d4883c\" (UID: \"301e0569-db5c-45be-865c-491221d4883c\") "
Sep 11 13:52:24.643313 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:24.643137    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/301e0569-db5c-45be-865c-491221d4883c-istio-data" (OuterVolumeSpecName: "istio-data") pod "301e0569-db5c-45be-865c-491221d4883c" (UID: "301e0569-db5c-45be-865c-491221d4883c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 13:52:24.643313 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:24.643159    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/301e0569-db5c-45be-865c-491221d4883c-istio-token\") pod \"301e0569-db5c-45be-865c-491221d4883c\" (UID: \"301e0569-db5c-45be-865c-491221d4883c\") "
Sep 11 13:52:24.643313 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:24.643242    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/301e0569-db5c-45be-865c-491221d4883c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "301e0569-db5c-45be-865c-491221d4883c" (UID: "301e0569-db5c-45be-865c-491221d4883c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 13:52:24.643313 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:24.643265    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/301e0569-db5c-45be-865c-491221d4883c-istio-podinfo\") pod \"301e0569-db5c-45be-865c-491221d4883c\" (UID: \"301e0569-db5c-45be-865c-491221d4883c\") "
Sep 11 13:52:24.643513 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:24.643328    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zd7vf\" (UniqueName: \"kubernetes.io/projected/301e0569-db5c-45be-865c-491221d4883c-kube-api-access-zd7vf\") pod \"301e0569-db5c-45be-865c-491221d4883c\" (UID: \"301e0569-db5c-45be-865c-491221d4883c\") "
Sep 11 13:52:24.643513 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:24.643421    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/301e0569-db5c-45be-865c-491221d4883c-credential-socket\") pod \"301e0569-db5c-45be-865c-491221d4883c\" (UID: \"301e0569-db5c-45be-865c-491221d4883c\") "
Sep 11 13:52:24.643513 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:24.643461    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/301e0569-db5c-45be-865c-491221d4883c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "301e0569-db5c-45be-865c-491221d4883c" (UID: "301e0569-db5c-45be-865c-491221d4883c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 13:52:24.643974 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:24.643927    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/301e0569-db5c-45be-865c-491221d4883c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "301e0569-db5c-45be-865c-491221d4883c" (UID: "301e0569-db5c-45be-865c-491221d4883c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 13:52:24.643974 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:24.643969    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/301e0569-db5c-45be-865c-491221d4883c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "301e0569-db5c-45be-865c-491221d4883c" (UID: "301e0569-db5c-45be-865c-491221d4883c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 13:52:24.644423 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:24.644282    2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/301e0569-db5c-45be-865c-491221d4883c-credential-socket\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:52:24.644423 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:24.644330    2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/301e0569-db5c-45be-865c-491221d4883c-istio-data\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:52:24.644423 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:24.644368    2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/301e0569-db5c-45be-865c-491221d4883c-workload-socket\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:52:24.644423 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:24.644383    2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/301e0569-db5c-45be-865c-491221d4883c-istiod-ca-cert\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:52:24.644423 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:24.644397    2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/301e0569-db5c-45be-865c-491221d4883c-workload-certs\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:52:24.647109 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:24.647052    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/301e0569-db5c-45be-865c-491221d4883c-istio-token" (OuterVolumeSpecName: "istio-token") pod "301e0569-db5c-45be-865c-491221d4883c" (UID: "301e0569-db5c-45be-865c-491221d4883c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 13:52:24.647243 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:24.647142    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/301e0569-db5c-45be-865c-491221d4883c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "301e0569-db5c-45be-865c-491221d4883c" (UID: "301e0569-db5c-45be-865c-491221d4883c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 13:52:24.647292 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:24.647250    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/301e0569-db5c-45be-865c-491221d4883c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "301e0569-db5c-45be-865c-491221d4883c" (UID: "301e0569-db5c-45be-865c-491221d4883c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 13:52:24.651337 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:24.651288    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/301e0569-db5c-45be-865c-491221d4883c-kube-api-access-zd7vf" (OuterVolumeSpecName: "kube-api-access-zd7vf") pod "301e0569-db5c-45be-865c-491221d4883c" (UID: "301e0569-db5c-45be-865c-491221d4883c"). InnerVolumeSpecName "kube-api-access-zd7vf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 13:52:24.653648 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:24.653618    2578 scope.go:117] "RemoveContainer" containerID="3769238e4b8be4d079ed73c23768099625c3d7aee61eca86e402877ba483b4f1"
Sep 11 13:52:24.676219 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:24.676183    2578 scope.go:117] "RemoveContainer" containerID="0aa2c4f8407c4a21a81095171f51f7843d615426f87544c5f99ca0e5d7f02483"
Sep 11 13:52:24.676677 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:52:24.676654    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0aa2c4f8407c4a21a81095171f51f7843d615426f87544c5f99ca0e5d7f02483\": container with ID starting with 0aa2c4f8407c4a21a81095171f51f7843d615426f87544c5f99ca0e5d7f02483 not found: ID does not exist" containerID="0aa2c4f8407c4a21a81095171f51f7843d615426f87544c5f99ca0e5d7f02483"
Sep 11 13:52:24.676764 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:24.676691    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0aa2c4f8407c4a21a81095171f51f7843d615426f87544c5f99ca0e5d7f02483"} err="failed to get container status \"0aa2c4f8407c4a21a81095171f51f7843d615426f87544c5f99ca0e5d7f02483\": rpc error: code = NotFound desc = could not find container \"0aa2c4f8407c4a21a81095171f51f7843d615426f87544c5f99ca0e5d7f02483\": container with ID starting with 0aa2c4f8407c4a21a81095171f51f7843d615426f87544c5f99ca0e5d7f02483 not found: ID does not exist"
Sep 11 13:52:24.676764 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:24.676717    2578 scope.go:117] "RemoveContainer" containerID="3769238e4b8be4d079ed73c23768099625c3d7aee61eca86e402877ba483b4f1"
Sep 11 13:52:24.677328 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:52:24.677177    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3769238e4b8be4d079ed73c23768099625c3d7aee61eca86e402877ba483b4f1\": container with ID starting with 3769238e4b8be4d079ed73c23768099625c3d7aee61eca86e402877ba483b4f1 not found: ID does not exist" containerID="3769238e4b8be4d079ed73c23768099625c3d7aee61eca86e402877ba483b4f1"
Sep 11 13:52:24.677328 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:24.677206    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3769238e4b8be4d079ed73c23768099625c3d7aee61eca86e402877ba483b4f1"} err="failed to get container status \"3769238e4b8be4d079ed73c23768099625c3d7aee61eca86e402877ba483b4f1\": rpc error: code = NotFound desc = could not find container \"3769238e4b8be4d079ed73c23768099625c3d7aee61eca86e402877ba483b4f1\": container with ID starting with 3769238e4b8be4d079ed73c23768099625c3d7aee61eca86e402877ba483b4f1 not found: ID does not exist"
Sep 11 13:52:24.746271 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:24.746227    2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/301e0569-db5c-45be-865c-491221d4883c-istio-token\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:52:24.746271 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:24.746262    2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/301e0569-db5c-45be-865c-491221d4883c-istio-podinfo\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:52:24.746271 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:24.746272    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zd7vf\" (UniqueName: \"kubernetes.io/projected/301e0569-db5c-45be-865c-491221d4883c-kube-api-access-zd7vf\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:52:24.746271 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:24.746282    2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/301e0569-db5c-45be-865c-491221d4883c-istio-envoy\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:52:24.972376 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:24.972331    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-371c38-openshift-default-668769478-66p8n"]
Sep 11 13:52:24.981772 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:24.981735    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-371c38-openshift-default-668769478-66p8n"]
Sep 11 13:52:25.108643 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:25.108608    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="301e0569-db5c-45be-865c-491221d4883c" path="/var/lib/kubelet/pods/301e0569-db5c-45be-865c-491221d4883c/volumes"
Sep 11 13:52:35.768952 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:35.768903    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-9edb53/auth-test-model-9edb53-kserve-5d97c5f9c9-nmzxg" podUID="c0d24dd1-a65c-4ce4-92e4-42e8781ee8e6" containerName="main" containerID="cri-o://a81ae7d957c456608512d87ebd9443c5e7a3e8b386687468c914903d8531f2da" gracePeriod=45
Sep 11 13:52:36.056956 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:36.056917    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-9edb53/auth-test-model-9edb53-kserve-5d97c5f9c9-nmzxg"
Sep 11 13:52:36.075888 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:36.075826    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0d24dd1-a65c-4ce4-92e4-42e8781ee8e6-model-cache\") pod \"c0d24dd1-a65c-4ce4-92e4-42e8781ee8e6\" (UID: \"c0d24dd1-a65c-4ce4-92e4-42e8781ee8e6\") "
Sep 11 13:52:36.076123 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:36.075930    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zm5w9\" (UniqueName: \"kubernetes.io/projected/c0d24dd1-a65c-4ce4-92e4-42e8781ee8e6-kube-api-access-zm5w9\") pod \"c0d24dd1-a65c-4ce4-92e4-42e8781ee8e6\" (UID: \"c0d24dd1-a65c-4ce4-92e4-42e8781ee8e6\") "
Sep 11 13:52:36.076123 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:36.076006    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c0d24dd1-a65c-4ce4-92e4-42e8781ee8e6-home\") pod \"c0d24dd1-a65c-4ce4-92e4-42e8781ee8e6\" (UID: \"c0d24dd1-a65c-4ce4-92e4-42e8781ee8e6\") "
Sep 11 13:52:36.076123 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:36.076058    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0d24dd1-a65c-4ce4-92e4-42e8781ee8e6-dshm\") pod \"c0d24dd1-a65c-4ce4-92e4-42e8781ee8e6\" (UID: \"c0d24dd1-a65c-4ce4-92e4-42e8781ee8e6\") "
Sep 11 13:52:36.076123 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:36.076105    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0d24dd1-a65c-4ce4-92e4-42e8781ee8e6-tmp-dir\") pod \"c0d24dd1-a65c-4ce4-92e4-42e8781ee8e6\" (UID: \"c0d24dd1-a65c-4ce4-92e4-42e8781ee8e6\") "
Sep 11 13:52:36.076354 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:36.076149    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0d24dd1-a65c-4ce4-92e4-42e8781ee8e6-tls-certs\") pod \"c0d24dd1-a65c-4ce4-92e4-42e8781ee8e6\" (UID: \"c0d24dd1-a65c-4ce4-92e4-42e8781ee8e6\") "
Sep 11 13:52:36.076354 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:36.076235    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0d24dd1-a65c-4ce4-92e4-42e8781ee8e6-model-cache" (OuterVolumeSpecName: "model-cache") pod "c0d24dd1-a65c-4ce4-92e4-42e8781ee8e6" (UID: "c0d24dd1-a65c-4ce4-92e4-42e8781ee8e6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 13:52:36.076354 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:36.076321    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0d24dd1-a65c-4ce4-92e4-42e8781ee8e6-home" (OuterVolumeSpecName: "home") pod "c0d24dd1-a65c-4ce4-92e4-42e8781ee8e6" (UID: "c0d24dd1-a65c-4ce4-92e4-42e8781ee8e6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 13:52:36.076488 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:36.076361    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0d24dd1-a65c-4ce4-92e4-42e8781ee8e6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c0d24dd1-a65c-4ce4-92e4-42e8781ee8e6" (UID: "c0d24dd1-a65c-4ce4-92e4-42e8781ee8e6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 13:52:36.076661 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:36.076641    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0d24dd1-a65c-4ce4-92e4-42e8781ee8e6-model-cache\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:52:36.076661 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:36.076660    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c0d24dd1-a65c-4ce4-92e4-42e8781ee8e6-home\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:52:36.076820 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:36.076670    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0d24dd1-a65c-4ce4-92e4-42e8781ee8e6-tmp-dir\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:52:36.078871 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:36.078797    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0d24dd1-a65c-4ce4-92e4-42e8781ee8e6-dshm" (OuterVolumeSpecName: "dshm") pod "c0d24dd1-a65c-4ce4-92e4-42e8781ee8e6" (UID: "c0d24dd1-a65c-4ce4-92e4-42e8781ee8e6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 13:52:36.079174 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:36.079143    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0d24dd1-a65c-4ce4-92e4-42e8781ee8e6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c0d24dd1-a65c-4ce4-92e4-42e8781ee8e6" (UID: "c0d24dd1-a65c-4ce4-92e4-42e8781ee8e6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 13:52:36.079174 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:36.079158    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0d24dd1-a65c-4ce4-92e4-42e8781ee8e6-kube-api-access-zm5w9" (OuterVolumeSpecName: "kube-api-access-zm5w9") pod "c0d24dd1-a65c-4ce4-92e4-42e8781ee8e6" (UID: "c0d24dd1-a65c-4ce4-92e4-42e8781ee8e6"). InnerVolumeSpecName "kube-api-access-zm5w9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 13:52:36.177538 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:36.177490    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0d24dd1-a65c-4ce4-92e4-42e8781ee8e6-tls-certs\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:52:36.177538 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:36.177531    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zm5w9\" (UniqueName: \"kubernetes.io/projected/c0d24dd1-a65c-4ce4-92e4-42e8781ee8e6-kube-api-access-zm5w9\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:52:36.177538 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:36.177542    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0d24dd1-a65c-4ce4-92e4-42e8781ee8e6-dshm\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:52:36.722279 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:36.722045    2578 generic.go:356] "Generic (PLEG): container finished" podID="c0d24dd1-a65c-4ce4-92e4-42e8781ee8e6" containerID="a81ae7d957c456608512d87ebd9443c5e7a3e8b386687468c914903d8531f2da" exitCode=0
Sep 11 13:52:36.722279 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:36.722115    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-9edb53/auth-test-model-9edb53-kserve-5d97c5f9c9-nmzxg"
Sep 11 13:52:36.722279 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:36.722118    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-9edb53/auth-test-model-9edb53-kserve-5d97c5f9c9-nmzxg" event={"ID":"c0d24dd1-a65c-4ce4-92e4-42e8781ee8e6","Type":"ContainerDied","Data":"a81ae7d957c456608512d87ebd9443c5e7a3e8b386687468c914903d8531f2da"}
Sep 11 13:52:36.722279 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:36.722228    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-9edb53/auth-test-model-9edb53-kserve-5d97c5f9c9-nmzxg" event={"ID":"c0d24dd1-a65c-4ce4-92e4-42e8781ee8e6","Type":"ContainerDied","Data":"ccefda09fe8a2f26996b31e6244086347c99e17ada1224d99f549cc62e0590d4"}
Sep 11 13:52:36.722279 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:36.722284    2578 scope.go:117] "RemoveContainer" containerID="a81ae7d957c456608512d87ebd9443c5e7a3e8b386687468c914903d8531f2da"
Sep 11 13:52:36.735858 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:36.735806    2578 scope.go:117] "RemoveContainer" containerID="a81ae7d957c456608512d87ebd9443c5e7a3e8b386687468c914903d8531f2da"
Sep 11 13:52:36.736307 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:52:36.736282    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a81ae7d957c456608512d87ebd9443c5e7a3e8b386687468c914903d8531f2da\": container with ID starting with a81ae7d957c456608512d87ebd9443c5e7a3e8b386687468c914903d8531f2da not found: ID does not exist" containerID="a81ae7d957c456608512d87ebd9443c5e7a3e8b386687468c914903d8531f2da"
Sep 11 13:52:36.736390 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:36.736323    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a81ae7d957c456608512d87ebd9443c5e7a3e8b386687468c914903d8531f2da"} err="failed to get container status \"a81ae7d957c456608512d87ebd9443c5e7a3e8b386687468c914903d8531f2da\": rpc error: code = NotFound desc = could not find container \"a81ae7d957c456608512d87ebd9443c5e7a3e8b386687468c914903d8531f2da\": container with ID starting with a81ae7d957c456608512d87ebd9443c5e7a3e8b386687468c914903d8531f2da not found: ID does not exist"
Sep 11 13:52:36.758082 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:36.758000    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-9edb53/auth-test-model-9edb53-kserve-5d97c5f9c9-nmzxg"]
Sep 11 13:52:36.762001 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:36.761958    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-9edb53/auth-test-model-9edb53-kserve-5d97c5f9c9-nmzxg"]
Sep 11 13:52:37.109895 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:37.109777    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c0d24dd1-a65c-4ce4-92e4-42e8781ee8e6" path="/var/lib/kubelet/pods/c0d24dd1-a65c-4ce4-92e4-42e8781ee8e6/volumes"
Sep 11 13:52:53.460395 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:53.460257    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-9edb53/sub-test-model-9edb53-kserve-5787bf7dd7-49jpr"]
Sep 11 13:52:53.461242 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:53.461211    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="301e0569-db5c-45be-865c-491221d4883c" containerName="istio-proxy"
Sep 11 13:52:53.461361 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:53.461245    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="301e0569-db5c-45be-865c-491221d4883c" containerName="istio-proxy"
Sep 11 13:52:53.461361 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:53.461269    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc25932e-1541-4894-848c-b1a86210d91f" containerName="maas-api"
Sep 11 13:52:53.461361 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:53.461279    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc25932e-1541-4894-848c-b1a86210d91f" containerName="maas-api"
Sep 11 13:52:53.461361 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:53.461295    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dbb78093-57b6-4adf-9b31-7d2bf633100c" containerName="maas-api"
Sep 11 13:52:53.461361 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:53.461304    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="dbb78093-57b6-4adf-9b31-7d2bf633100c" containerName="maas-api"
Sep 11 13:52:53.461361 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:53.461314    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e62b849e-6379-4858-aeb3-14eea3fc2f52" containerName="maas-api"
Sep 11 13:52:53.461361 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:53.461324    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="e62b849e-6379-4858-aeb3-14eea3fc2f52" containerName="maas-api"
Sep 11 13:52:53.461361 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:53.461349    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="848ec50c-75af-4c07-be1a-cb8bf34c0c08" containerName="main"
Sep 11 13:52:53.461361 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:53.461357    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="848ec50c-75af-4c07-be1a-cb8bf34c0c08" containerName="main"
Sep 11 13:52:53.461860 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:53.461369    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5eb98d8-6523-411d-bcfb-ad46743c0c72" containerName="istio-proxy"
Sep 11 13:52:53.461860 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:53.461378    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5eb98d8-6523-411d-bcfb-ad46743c0c72" containerName="istio-proxy"
Sep 11 13:52:53.461860 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:53.461388    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7dc1eb52-2060-473a-a0d5-4fe00eca2972" containerName="maas-api"
Sep 11 13:52:53.461860 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:53.461396    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="7dc1eb52-2060-473a-a0d5-4fe00eca2972" containerName="maas-api"
Sep 11 13:52:53.461860 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:53.461430    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83e64ec1-0915-4ff5-8be0-2bfb03873398" containerName="maas-api"
Sep 11 13:52:53.461860 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:53.461439    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="83e64ec1-0915-4ff5-8be0-2bfb03873398" containerName="maas-api"
Sep 11 13:52:53.461860 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:53.461451    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26968216-f646-4245-91a5-eb8f0710c82e" containerName="maas-api"
Sep 11 13:52:53.461860 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:53.461460    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="26968216-f646-4245-91a5-eb8f0710c82e" containerName="maas-api"
Sep 11 13:52:53.461860 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:53.461475    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0d24dd1-a65c-4ce4-92e4-42e8781ee8e6" containerName="main"
Sep 11 13:52:53.461860 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:53.461484    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0d24dd1-a65c-4ce4-92e4-42e8781ee8e6" containerName="main"
Sep 11 13:52:53.461860 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:53.461492    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14cd118e-eb64-4b60-a329-0964cc60faf8" containerName="istio-proxy"
Sep 11 13:52:53.461860 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:53.461500    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="14cd118e-eb64-4b60-a329-0964cc60faf8" containerName="istio-proxy"
Sep 11 13:52:53.461860 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:53.461512    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5eb98d8-6523-411d-bcfb-ad46743c0c72" containerName="istio-proxy"
Sep 11 13:52:53.461860 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:53.461521    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5eb98d8-6523-411d-bcfb-ad46743c0c72" containerName="istio-proxy"
Sep 11 13:52:53.461860 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:53.461655    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0d24dd1-a65c-4ce4-92e4-42e8781ee8e6" containerName="main"
Sep 11 13:52:53.461860 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:53.461668    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="301e0569-db5c-45be-865c-491221d4883c" containerName="istio-proxy"
Sep 11 13:52:53.461860 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:53.461681    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5eb98d8-6523-411d-bcfb-ad46743c0c72" containerName="istio-proxy"
Sep 11 13:52:53.461860 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:53.461689    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="301e0569-db5c-45be-865c-491221d4883c" containerName="istio-proxy"
Sep 11 13:52:53.461860 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:53.461699    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="83e64ec1-0915-4ff5-8be0-2bfb03873398" containerName="maas-api"
Sep 11 13:52:53.461860 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:53.461712    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="14cd118e-eb64-4b60-a329-0964cc60faf8" containerName="istio-proxy"
Sep 11 13:52:53.461860 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:53.461721    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="26968216-f646-4245-91a5-eb8f0710c82e" containerName="maas-api"
Sep 11 13:52:53.461860 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:53.461732    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc25932e-1541-4894-848c-b1a86210d91f" containerName="maas-api"
Sep 11 13:52:53.461860 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:53.461752    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="dbb78093-57b6-4adf-9b31-7d2bf633100c" containerName="maas-api"
Sep 11 13:52:53.461860 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:53.461762    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="848ec50c-75af-4c07-be1a-cb8bf34c0c08" containerName="main"
Sep 11 13:52:53.461860 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:53.461771    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5eb98d8-6523-411d-bcfb-ad46743c0c72" containerName="istio-proxy"
Sep 11 13:52:53.461860 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:53.461782    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="e62b849e-6379-4858-aeb3-14eea3fc2f52" containerName="maas-api"
Sep 11 13:52:53.461860 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:53.461791    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="7dc1eb52-2060-473a-a0d5-4fe00eca2972" containerName="maas-api"
Sep 11 13:52:53.467425 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:53.467390    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-9edb53/sub-test-model-9edb53-kserve-5787bf7dd7-49jpr"
Sep 11 13:52:53.469859 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:53.469814    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-9edb53\"/\"default-dockercfg-4shkp\""
Sep 11 13:52:53.472317 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:53.472279    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-9edb53\"/\"sub-test-model-9edb53-kserve-self-signed-certs\""
Sep 11 13:52:53.472317 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:53.472279    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-9edb53\"/\"kube-root-ca.crt\""
Sep 11 13:52:53.472542 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:53.472310    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-9edb53\"/\"openshift-service-ca.crt\""
Sep 11 13:52:53.487959 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:53.487913    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-9edb53/sub-test-model-9edb53-kserve-5787bf7dd7-49jpr"]
Sep 11 13:52:53.587040 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:53.586979    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c30f72e-6bf3-40d6-9619-a35f2c16105d-model-cache\") pod \"sub-test-model-9edb53-kserve-5787bf7dd7-49jpr\" (UID: \"0c30f72e-6bf3-40d6-9619-a35f2c16105d\") " pod="ai-tenant-e2e-shared-b-w3-9edb53/sub-test-model-9edb53-kserve-5787bf7dd7-49jpr"
Sep 11 13:52:53.587317 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:53.587062    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c30f72e-6bf3-40d6-9619-a35f2c16105d-dshm\") pod \"sub-test-model-9edb53-kserve-5787bf7dd7-49jpr\" (UID: \"0c30f72e-6bf3-40d6-9619-a35f2c16105d\") " pod="ai-tenant-e2e-shared-b-w3-9edb53/sub-test-model-9edb53-kserve-5787bf7dd7-49jpr"
Sep 11 13:52:53.587317 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:53.587095    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c30f72e-6bf3-40d6-9619-a35f2c16105d-home\") pod \"sub-test-model-9edb53-kserve-5787bf7dd7-49jpr\" (UID: \"0c30f72e-6bf3-40d6-9619-a35f2c16105d\") " pod="ai-tenant-e2e-shared-b-w3-9edb53/sub-test-model-9edb53-kserve-5787bf7dd7-49jpr"
Sep 11 13:52:53.587317 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:53.587122    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5csj\" (UniqueName: \"kubernetes.io/projected/0c30f72e-6bf3-40d6-9619-a35f2c16105d-kube-api-access-l5csj\") pod \"sub-test-model-9edb53-kserve-5787bf7dd7-49jpr\" (UID: \"0c30f72e-6bf3-40d6-9619-a35f2c16105d\") " pod="ai-tenant-e2e-shared-b-w3-9edb53/sub-test-model-9edb53-kserve-5787bf7dd7-49jpr"
Sep 11 13:52:53.587317 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:53.587183    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c30f72e-6bf3-40d6-9619-a35f2c16105d-tls-certs\") pod \"sub-test-model-9edb53-kserve-5787bf7dd7-49jpr\" (UID: \"0c30f72e-6bf3-40d6-9619-a35f2c16105d\") " pod="ai-tenant-e2e-shared-b-w3-9edb53/sub-test-model-9edb53-kserve-5787bf7dd7-49jpr"
Sep 11 13:52:53.587317 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:53.587240    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c30f72e-6bf3-40d6-9619-a35f2c16105d-tmp-dir\") pod \"sub-test-model-9edb53-kserve-5787bf7dd7-49jpr\" (UID: \"0c30f72e-6bf3-40d6-9619-a35f2c16105d\") " pod="ai-tenant-e2e-shared-b-w3-9edb53/sub-test-model-9edb53-kserve-5787bf7dd7-49jpr"
Sep 11 13:52:53.688612 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:53.688548    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c30f72e-6bf3-40d6-9619-a35f2c16105d-model-cache\") pod \"sub-test-model-9edb53-kserve-5787bf7dd7-49jpr\" (UID: \"0c30f72e-6bf3-40d6-9619-a35f2c16105d\") " pod="ai-tenant-e2e-shared-b-w3-9edb53/sub-test-model-9edb53-kserve-5787bf7dd7-49jpr"
Sep 11 13:52:53.688941 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:53.688655    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c30f72e-6bf3-40d6-9619-a35f2c16105d-dshm\") pod \"sub-test-model-9edb53-kserve-5787bf7dd7-49jpr\" (UID: \"0c30f72e-6bf3-40d6-9619-a35f2c16105d\") " pod="ai-tenant-e2e-shared-b-w3-9edb53/sub-test-model-9edb53-kserve-5787bf7dd7-49jpr"
Sep 11 13:52:53.688941 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:53.688701    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c30f72e-6bf3-40d6-9619-a35f2c16105d-home\") pod \"sub-test-model-9edb53-kserve-5787bf7dd7-49jpr\" (UID: \"0c30f72e-6bf3-40d6-9619-a35f2c16105d\") " pod="ai-tenant-e2e-shared-b-w3-9edb53/sub-test-model-9edb53-kserve-5787bf7dd7-49jpr"
Sep 11 13:52:53.688941 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:53.688741    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5csj\" (UniqueName: \"kubernetes.io/projected/0c30f72e-6bf3-40d6-9619-a35f2c16105d-kube-api-access-l5csj\") pod \"sub-test-model-9edb53-kserve-5787bf7dd7-49jpr\" (UID: \"0c30f72e-6bf3-40d6-9619-a35f2c16105d\") " pod="ai-tenant-e2e-shared-b-w3-9edb53/sub-test-model-9edb53-kserve-5787bf7dd7-49jpr"
Sep 11 13:52:53.688941 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:53.688887    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c30f72e-6bf3-40d6-9619-a35f2c16105d-tls-certs\") pod \"sub-test-model-9edb53-kserve-5787bf7dd7-49jpr\" (UID: \"0c30f72e-6bf3-40d6-9619-a35f2c16105d\") " pod="ai-tenant-e2e-shared-b-w3-9edb53/sub-test-model-9edb53-kserve-5787bf7dd7-49jpr"
Sep 11 13:52:53.689190 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:53.688954    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c30f72e-6bf3-40d6-9619-a35f2c16105d-tmp-dir\") pod \"sub-test-model-9edb53-kserve-5787bf7dd7-49jpr\" (UID: \"0c30f72e-6bf3-40d6-9619-a35f2c16105d\") " pod="ai-tenant-e2e-shared-b-w3-9edb53/sub-test-model-9edb53-kserve-5787bf7dd7-49jpr"
Sep 11 13:52:53.689250 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:53.689199    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c30f72e-6bf3-40d6-9619-a35f2c16105d-model-cache\") pod \"sub-test-model-9edb53-kserve-5787bf7dd7-49jpr\" (UID: \"0c30f72e-6bf3-40d6-9619-a35f2c16105d\") " pod="ai-tenant-e2e-shared-b-w3-9edb53/sub-test-model-9edb53-kserve-5787bf7dd7-49jpr"
Sep 11 13:52:53.689305 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:53.689259    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c30f72e-6bf3-40d6-9619-a35f2c16105d-home\") pod \"sub-test-model-9edb53-kserve-5787bf7dd7-49jpr\" (UID: \"0c30f72e-6bf3-40d6-9619-a35f2c16105d\") " pod="ai-tenant-e2e-shared-b-w3-9edb53/sub-test-model-9edb53-kserve-5787bf7dd7-49jpr"
Sep 11 13:52:53.689395 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:53.689363    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c30f72e-6bf3-40d6-9619-a35f2c16105d-tmp-dir\") pod \"sub-test-model-9edb53-kserve-5787bf7dd7-49jpr\" (UID: \"0c30f72e-6bf3-40d6-9619-a35f2c16105d\") " pod="ai-tenant-e2e-shared-b-w3-9edb53/sub-test-model-9edb53-kserve-5787bf7dd7-49jpr"
Sep 11 13:52:53.691951 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:53.691910    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c30f72e-6bf3-40d6-9619-a35f2c16105d-dshm\") pod \"sub-test-model-9edb53-kserve-5787bf7dd7-49jpr\" (UID: \"0c30f72e-6bf3-40d6-9619-a35f2c16105d\") " pod="ai-tenant-e2e-shared-b-w3-9edb53/sub-test-model-9edb53-kserve-5787bf7dd7-49jpr"
Sep 11 13:52:53.692187 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:53.692169    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c30f72e-6bf3-40d6-9619-a35f2c16105d-tls-certs\") pod \"sub-test-model-9edb53-kserve-5787bf7dd7-49jpr\" (UID: \"0c30f72e-6bf3-40d6-9619-a35f2c16105d\") " pod="ai-tenant-e2e-shared-b-w3-9edb53/sub-test-model-9edb53-kserve-5787bf7dd7-49jpr"
Sep 11 13:52:53.701153 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:53.701110    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5csj\" (UniqueName: \"kubernetes.io/projected/0c30f72e-6bf3-40d6-9619-a35f2c16105d-kube-api-access-l5csj\") pod \"sub-test-model-9edb53-kserve-5787bf7dd7-49jpr\" (UID: \"0c30f72e-6bf3-40d6-9619-a35f2c16105d\") " pod="ai-tenant-e2e-shared-b-w3-9edb53/sub-test-model-9edb53-kserve-5787bf7dd7-49jpr"
Sep 11 13:52:53.779750 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:53.779684    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-9edb53/sub-test-model-9edb53-kserve-5787bf7dd7-49jpr"
Sep 11 13:52:54.074762 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:54.074611    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-9edb53/sub-test-model-9edb53-kserve-5787bf7dd7-49jpr"]
Sep 11 13:52:54.081187 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:52:54.081144    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0c30f72e_6bf3_40d6_9619_a35f2c16105d.slice/crio-41b2e5508980c01e37f92cf028e5d9a2006eebaaa2dec8f22b7b2ed93e0f01cb WatchSource:0}: Error finding container 41b2e5508980c01e37f92cf028e5d9a2006eebaaa2dec8f22b7b2ed93e0f01cb: Status 404 returned error can't find the container with id 41b2e5508980c01e37f92cf028e5d9a2006eebaaa2dec8f22b7b2ed93e0f01cb
Sep 11 13:52:54.841962 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:54.841908    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-9edb53/sub-test-model-9edb53-kserve-5787bf7dd7-49jpr" event={"ID":"0c30f72e-6bf3-40d6-9619-a35f2c16105d","Type":"ContainerStarted","Data":"9eaaf1dc7aab76a43e2df555b56f07cee542afe23c7ebc7522f8cd6b51f02bc0"}
Sep 11 13:52:54.841962 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:54.841972    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-9edb53/sub-test-model-9edb53-kserve-5787bf7dd7-49jpr" event={"ID":"0c30f72e-6bf3-40d6-9619-a35f2c16105d","Type":"ContainerStarted","Data":"41b2e5508980c01e37f92cf028e5d9a2006eebaaa2dec8f22b7b2ed93e0f01cb"}
Sep 11 13:52:54.872138 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:52:54.872076    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-9edb53/sub-test-model-9edb53-kserve-5787bf7dd7-49jpr" podStartSLOduration=1.872059064 podStartE2EDuration="1.872059064s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:52:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:52:54.868599899 +0000 UTC m=+1882.254703081" watchObservedRunningTime="2026-09-11 13:52:54.872059064 +0000 UTC m=+1882.258162228"
Sep 11 13:53:03.780471 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:53:03.780416    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-9edb53/sub-test-model-9edb53-kserve-5787bf7dd7-49jpr"
Sep 11 13:53:03.780943 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:53:03.780489    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-9edb53/sub-test-model-9edb53-kserve-5787bf7dd7-49jpr"
Sep 11 13:53:03.794122 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:53:03.794080    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-9edb53/sub-test-model-9edb53-kserve-5787bf7dd7-49jpr"
Sep 11 13:53:03.911566 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:53:03.911529    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-9edb53/sub-test-model-9edb53-kserve-5787bf7dd7-49jpr"
Sep 11 13:53:36.139777 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:53:36.139733    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-9edb53/sub-test-model-9edb53-kserve-5787bf7dd7-49jpr"]
Sep 11 13:53:51.162239 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:53:51.162182    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-9edb53/sub-test-model-9edb53-kserve-5787bf7dd7-49jpr" podUID="0c30f72e-6bf3-40d6-9619-a35f2c16105d" containerName="main" containerID="cri-o://9eaaf1dc7aab76a43e2df555b56f07cee542afe23c7ebc7522f8cd6b51f02bc0" gracePeriod=45
Sep 11 13:53:51.450166 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:53:51.450117    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-9edb53/sub-test-model-9edb53-kserve-5787bf7dd7-49jpr"
Sep 11 13:53:51.475278 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:53:51.475232    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l5csj\" (UniqueName: \"kubernetes.io/projected/0c30f72e-6bf3-40d6-9619-a35f2c16105d-kube-api-access-l5csj\") pod \"0c30f72e-6bf3-40d6-9619-a35f2c16105d\" (UID: \"0c30f72e-6bf3-40d6-9619-a35f2c16105d\") "
Sep 11 13:53:51.475588 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:53:51.475315    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c30f72e-6bf3-40d6-9619-a35f2c16105d-model-cache\") pod \"0c30f72e-6bf3-40d6-9619-a35f2c16105d\" (UID: \"0c30f72e-6bf3-40d6-9619-a35f2c16105d\") "
Sep 11 13:53:51.475588 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:53:51.475454    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c30f72e-6bf3-40d6-9619-a35f2c16105d-home\") pod \"0c30f72e-6bf3-40d6-9619-a35f2c16105d\" (UID: \"0c30f72e-6bf3-40d6-9619-a35f2c16105d\") "
Sep 11 13:53:51.475588 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:53:51.475516    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c30f72e-6bf3-40d6-9619-a35f2c16105d-dshm\") pod \"0c30f72e-6bf3-40d6-9619-a35f2c16105d\" (UID: \"0c30f72e-6bf3-40d6-9619-a35f2c16105d\") "
Sep 11 13:53:51.475588 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:53:51.475553    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c30f72e-6bf3-40d6-9619-a35f2c16105d-tls-certs\") pod \"0c30f72e-6bf3-40d6-9619-a35f2c16105d\" (UID: \"0c30f72e-6bf3-40d6-9619-a35f2c16105d\") "
Sep 11 13:53:51.475588 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:53:51.475587    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c30f72e-6bf3-40d6-9619-a35f2c16105d-tmp-dir\") pod \"0c30f72e-6bf3-40d6-9619-a35f2c16105d\" (UID: \"0c30f72e-6bf3-40d6-9619-a35f2c16105d\") "
Sep 11 13:53:51.475917 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:53:51.475675    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c30f72e-6bf3-40d6-9619-a35f2c16105d-model-cache" (OuterVolumeSpecName: "model-cache") pod "0c30f72e-6bf3-40d6-9619-a35f2c16105d" (UID: "0c30f72e-6bf3-40d6-9619-a35f2c16105d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 13:53:51.476397 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:53:51.476058    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c30f72e-6bf3-40d6-9619-a35f2c16105d-model-cache\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:53:51.477739 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:53:51.476087    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c30f72e-6bf3-40d6-9619-a35f2c16105d-home" (OuterVolumeSpecName: "home") pod "0c30f72e-6bf3-40d6-9619-a35f2c16105d" (UID: "0c30f72e-6bf3-40d6-9619-a35f2c16105d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 13:53:51.477915 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:53:51.476425    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c30f72e-6bf3-40d6-9619-a35f2c16105d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0c30f72e-6bf3-40d6-9619-a35f2c16105d" (UID: "0c30f72e-6bf3-40d6-9619-a35f2c16105d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 13:53:51.479123 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:53:51.479092    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c30f72e-6bf3-40d6-9619-a35f2c16105d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0c30f72e-6bf3-40d6-9619-a35f2c16105d" (UID: "0c30f72e-6bf3-40d6-9619-a35f2c16105d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 13:53:51.479306 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:53:51.479288    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c30f72e-6bf3-40d6-9619-a35f2c16105d-dshm" (OuterVolumeSpecName: "dshm") pod "0c30f72e-6bf3-40d6-9619-a35f2c16105d" (UID: "0c30f72e-6bf3-40d6-9619-a35f2c16105d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 13:53:51.479709 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:53:51.479671    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c30f72e-6bf3-40d6-9619-a35f2c16105d-kube-api-access-l5csj" (OuterVolumeSpecName: "kube-api-access-l5csj") pod "0c30f72e-6bf3-40d6-9619-a35f2c16105d" (UID: "0c30f72e-6bf3-40d6-9619-a35f2c16105d"). InnerVolumeSpecName "kube-api-access-l5csj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 13:53:51.578889 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:53:51.578784    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c30f72e-6bf3-40d6-9619-a35f2c16105d-dshm\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:53:51.578889 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:53:51.578825    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c30f72e-6bf3-40d6-9619-a35f2c16105d-tls-certs\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:53:51.578889 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:53:51.578853    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c30f72e-6bf3-40d6-9619-a35f2c16105d-tmp-dir\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:53:51.578889 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:53:51.578862    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l5csj\" (UniqueName: \"kubernetes.io/projected/0c30f72e-6bf3-40d6-9619-a35f2c16105d-kube-api-access-l5csj\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:53:51.578889 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:53:51.578871    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c30f72e-6bf3-40d6-9619-a35f2c16105d-home\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:53:52.220398 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:53:52.220356    2578 generic.go:356] "Generic (PLEG): container finished" podID="0c30f72e-6bf3-40d6-9619-a35f2c16105d" containerID="9eaaf1dc7aab76a43e2df555b56f07cee542afe23c7ebc7522f8cd6b51f02bc0" exitCode=0
Sep 11 13:53:52.220398 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:53:52.220398    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-9edb53/sub-test-model-9edb53-kserve-5787bf7dd7-49jpr"
Sep 11 13:53:52.220986 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:53:52.220434    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-9edb53/sub-test-model-9edb53-kserve-5787bf7dd7-49jpr" event={"ID":"0c30f72e-6bf3-40d6-9619-a35f2c16105d","Type":"ContainerDied","Data":"9eaaf1dc7aab76a43e2df555b56f07cee542afe23c7ebc7522f8cd6b51f02bc0"}
Sep 11 13:53:52.220986 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:53:52.220487    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-9edb53/sub-test-model-9edb53-kserve-5787bf7dd7-49jpr" event={"ID":"0c30f72e-6bf3-40d6-9619-a35f2c16105d","Type":"ContainerDied","Data":"41b2e5508980c01e37f92cf028e5d9a2006eebaaa2dec8f22b7b2ed93e0f01cb"}
Sep 11 13:53:52.220986 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:53:52.220518    2578 scope.go:117] "RemoveContainer" containerID="9eaaf1dc7aab76a43e2df555b56f07cee542afe23c7ebc7522f8cd6b51f02bc0"
Sep 11 13:53:52.234792 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:53:52.234753    2578 scope.go:117] "RemoveContainer" containerID="9eaaf1dc7aab76a43e2df555b56f07cee542afe23c7ebc7522f8cd6b51f02bc0"
Sep 11 13:53:52.235481 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:53:52.235442    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9eaaf1dc7aab76a43e2df555b56f07cee542afe23c7ebc7522f8cd6b51f02bc0\": container with ID starting with 9eaaf1dc7aab76a43e2df555b56f07cee542afe23c7ebc7522f8cd6b51f02bc0 not found: ID does not exist" containerID="9eaaf1dc7aab76a43e2df555b56f07cee542afe23c7ebc7522f8cd6b51f02bc0"
Sep 11 13:53:52.235637 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:53:52.235496    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9eaaf1dc7aab76a43e2df555b56f07cee542afe23c7ebc7522f8cd6b51f02bc0"} err="failed to get container status \"9eaaf1dc7aab76a43e2df555b56f07cee542afe23c7ebc7522f8cd6b51f02bc0\": rpc error: code = NotFound desc = could not find container \"9eaaf1dc7aab76a43e2df555b56f07cee542afe23c7ebc7522f8cd6b51f02bc0\": container with ID starting with 9eaaf1dc7aab76a43e2df555b56f07cee542afe23c7ebc7522f8cd6b51f02bc0 not found: ID does not exist"
Sep 11 13:53:52.246380 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:53:52.246333    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-9edb53/sub-test-model-9edb53-kserve-5787bf7dd7-49jpr"]
Sep 11 13:53:52.257250 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:53:52.257201    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-9edb53/sub-test-model-9edb53-kserve-5787bf7dd7-49jpr"]
Sep 11 13:53:53.108651 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:53:53.108591    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0c30f72e-6bf3-40d6-9619-a35f2c16105d" path="/var/lib/kubelet/pods/0c30f72e-6bf3-40d6-9619-a35f2c16105d/volumes"
Sep 11 13:54:30.468559 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:54:30.468444    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-9edb53/rate-test-model-9edb53-kserve-69d4d55b55-2g2xb"]
Sep 11 13:54:30.469671 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:54:30.469082    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c30f72e-6bf3-40d6-9619-a35f2c16105d" containerName="main"
Sep 11 13:54:30.469671 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:54:30.469100    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c30f72e-6bf3-40d6-9619-a35f2c16105d" containerName="main"
Sep 11 13:54:30.469671 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:54:30.469111    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="301e0569-db5c-45be-865c-491221d4883c" containerName="istio-proxy"
Sep 11 13:54:30.469671 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:54:30.469117    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="301e0569-db5c-45be-865c-491221d4883c" containerName="istio-proxy"
Sep 11 13:54:30.469671 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:54:30.469209    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="0c30f72e-6bf3-40d6-9619-a35f2c16105d" containerName="main"
Sep 11 13:54:30.473005 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:54:30.472984    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-9edb53/rate-test-model-9edb53-kserve-69d4d55b55-2g2xb"
Sep 11 13:54:30.475682 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:54:30.475652    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-9edb53\"/\"default-dockercfg-4shkp\""
Sep 11 13:54:30.476392 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:54:30.476363    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-9edb53\"/\"kube-root-ca.crt\""
Sep 11 13:54:30.476528 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:54:30.476363    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-9edb53\"/\"rate-test-model-9edb53-kserve-self-signed-certs\""
Sep 11 13:54:30.476528 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:54:30.476414    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-9edb53\"/\"openshift-service-ca.crt\""
Sep 11 13:54:30.483538 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:54:30.483506    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-9edb53/rate-test-model-9edb53-kserve-69d4d55b55-2g2xb"]
Sep 11 13:54:30.545637 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:54:30.545587    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mjdq\" (UniqueName: \"kubernetes.io/projected/66f65f8b-07ac-4751-b4a8-929b4e132f26-kube-api-access-9mjdq\") pod \"rate-test-model-9edb53-kserve-69d4d55b55-2g2xb\" (UID: \"66f65f8b-07ac-4751-b4a8-929b4e132f26\") " pod="ai-tenant-e2e-shared-b-w3-9edb53/rate-test-model-9edb53-kserve-69d4d55b55-2g2xb"
Sep 11 13:54:30.545637 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:54:30.545639    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66f65f8b-07ac-4751-b4a8-929b4e132f26-dshm\") pod \"rate-test-model-9edb53-kserve-69d4d55b55-2g2xb\" (UID: \"66f65f8b-07ac-4751-b4a8-929b4e132f26\") " pod="ai-tenant-e2e-shared-b-w3-9edb53/rate-test-model-9edb53-kserve-69d4d55b55-2g2xb"
Sep 11 13:54:30.545972 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:54:30.545709    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66f65f8b-07ac-4751-b4a8-929b4e132f26-home\") pod \"rate-test-model-9edb53-kserve-69d4d55b55-2g2xb\" (UID: \"66f65f8b-07ac-4751-b4a8-929b4e132f26\") " pod="ai-tenant-e2e-shared-b-w3-9edb53/rate-test-model-9edb53-kserve-69d4d55b55-2g2xb"
Sep 11 13:54:30.545972 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:54:30.545738    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66f65f8b-07ac-4751-b4a8-929b4e132f26-model-cache\") pod \"rate-test-model-9edb53-kserve-69d4d55b55-2g2xb\" (UID: \"66f65f8b-07ac-4751-b4a8-929b4e132f26\") " pod="ai-tenant-e2e-shared-b-w3-9edb53/rate-test-model-9edb53-kserve-69d4d55b55-2g2xb"
Sep 11 13:54:30.545972 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:54:30.545774    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66f65f8b-07ac-4751-b4a8-929b4e132f26-tls-certs\") pod \"rate-test-model-9edb53-kserve-69d4d55b55-2g2xb\" (UID: \"66f65f8b-07ac-4751-b4a8-929b4e132f26\") " pod="ai-tenant-e2e-shared-b-w3-9edb53/rate-test-model-9edb53-kserve-69d4d55b55-2g2xb"
Sep 11 13:54:30.545972 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:54:30.545799    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66f65f8b-07ac-4751-b4a8-929b4e132f26-tmp-dir\") pod \"rate-test-model-9edb53-kserve-69d4d55b55-2g2xb\" (UID: \"66f65f8b-07ac-4751-b4a8-929b4e132f26\") " pod="ai-tenant-e2e-shared-b-w3-9edb53/rate-test-model-9edb53-kserve-69d4d55b55-2g2xb"
Sep 11 13:54:30.647219 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:54:30.647172    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66f65f8b-07ac-4751-b4a8-929b4e132f26-tmp-dir\") pod \"rate-test-model-9edb53-kserve-69d4d55b55-2g2xb\" (UID: \"66f65f8b-07ac-4751-b4a8-929b4e132f26\") " pod="ai-tenant-e2e-shared-b-w3-9edb53/rate-test-model-9edb53-kserve-69d4d55b55-2g2xb"
Sep 11 13:54:30.647511 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:54:30.647363    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mjdq\" (UniqueName: \"kubernetes.io/projected/66f65f8b-07ac-4751-b4a8-929b4e132f26-kube-api-access-9mjdq\") pod \"rate-test-model-9edb53-kserve-69d4d55b55-2g2xb\" (UID: \"66f65f8b-07ac-4751-b4a8-929b4e132f26\") " pod="ai-tenant-e2e-shared-b-w3-9edb53/rate-test-model-9edb53-kserve-69d4d55b55-2g2xb"
Sep 11 13:54:30.647511 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:54:30.647391    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66f65f8b-07ac-4751-b4a8-929b4e132f26-dshm\") pod \"rate-test-model-9edb53-kserve-69d4d55b55-2g2xb\" (UID: \"66f65f8b-07ac-4751-b4a8-929b4e132f26\") " pod="ai-tenant-e2e-shared-b-w3-9edb53/rate-test-model-9edb53-kserve-69d4d55b55-2g2xb"
Sep 11 13:54:30.647511 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:54:30.647460    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66f65f8b-07ac-4751-b4a8-929b4e132f26-home\") pod \"rate-test-model-9edb53-kserve-69d4d55b55-2g2xb\" (UID: \"66f65f8b-07ac-4751-b4a8-929b4e132f26\") " pod="ai-tenant-e2e-shared-b-w3-9edb53/rate-test-model-9edb53-kserve-69d4d55b55-2g2xb"
Sep 11 13:54:30.647511 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:54:30.647500    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66f65f8b-07ac-4751-b4a8-929b4e132f26-model-cache\") pod \"rate-test-model-9edb53-kserve-69d4d55b55-2g2xb\" (UID: \"66f65f8b-07ac-4751-b4a8-929b4e132f26\") " pod="ai-tenant-e2e-shared-b-w3-9edb53/rate-test-model-9edb53-kserve-69d4d55b55-2g2xb"
Sep 11 13:54:30.647697 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:54:30.647542    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66f65f8b-07ac-4751-b4a8-929b4e132f26-tls-certs\") pod \"rate-test-model-9edb53-kserve-69d4d55b55-2g2xb\" (UID: \"66f65f8b-07ac-4751-b4a8-929b4e132f26\") " pod="ai-tenant-e2e-shared-b-w3-9edb53/rate-test-model-9edb53-kserve-69d4d55b55-2g2xb"
Sep 11 13:54:30.647758 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:54:30.647732    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66f65f8b-07ac-4751-b4a8-929b4e132f26-tmp-dir\") pod \"rate-test-model-9edb53-kserve-69d4d55b55-2g2xb\" (UID: \"66f65f8b-07ac-4751-b4a8-929b4e132f26\") " pod="ai-tenant-e2e-shared-b-w3-9edb53/rate-test-model-9edb53-kserve-69d4d55b55-2g2xb"
Sep 11 13:54:30.647823 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:54:30.647808    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66f65f8b-07ac-4751-b4a8-929b4e132f26-home\") pod \"rate-test-model-9edb53-kserve-69d4d55b55-2g2xb\" (UID: \"66f65f8b-07ac-4751-b4a8-929b4e132f26\") " pod="ai-tenant-e2e-shared-b-w3-9edb53/rate-test-model-9edb53-kserve-69d4d55b55-2g2xb"
Sep 11 13:54:30.647984 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:54:30.647963    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66f65f8b-07ac-4751-b4a8-929b4e132f26-model-cache\") pod \"rate-test-model-9edb53-kserve-69d4d55b55-2g2xb\" (UID: \"66f65f8b-07ac-4751-b4a8-929b4e132f26\") " pod="ai-tenant-e2e-shared-b-w3-9edb53/rate-test-model-9edb53-kserve-69d4d55b55-2g2xb"
Sep 11 13:54:30.650504 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:54:30.650460    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66f65f8b-07ac-4751-b4a8-929b4e132f26-dshm\") pod \"rate-test-model-9edb53-kserve-69d4d55b55-2g2xb\" (UID: \"66f65f8b-07ac-4751-b4a8-929b4e132f26\") " pod="ai-tenant-e2e-shared-b-w3-9edb53/rate-test-model-9edb53-kserve-69d4d55b55-2g2xb"
Sep 11 13:54:30.650745 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:54:30.650724    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66f65f8b-07ac-4751-b4a8-929b4e132f26-tls-certs\") pod \"rate-test-model-9edb53-kserve-69d4d55b55-2g2xb\" (UID: \"66f65f8b-07ac-4751-b4a8-929b4e132f26\") " pod="ai-tenant-e2e-shared-b-w3-9edb53/rate-test-model-9edb53-kserve-69d4d55b55-2g2xb"
Sep 11 13:54:30.657191 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:54:30.657144    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mjdq\" (UniqueName: \"kubernetes.io/projected/66f65f8b-07ac-4751-b4a8-929b4e132f26-kube-api-access-9mjdq\") pod \"rate-test-model-9edb53-kserve-69d4d55b55-2g2xb\" (UID: \"66f65f8b-07ac-4751-b4a8-929b4e132f26\") " pod="ai-tenant-e2e-shared-b-w3-9edb53/rate-test-model-9edb53-kserve-69d4d55b55-2g2xb"
Sep 11 13:54:30.785495 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:54:30.785324    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-9edb53/rate-test-model-9edb53-kserve-69d4d55b55-2g2xb"
Sep 11 13:54:31.076791 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:54:31.076748    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-9edb53/rate-test-model-9edb53-kserve-69d4d55b55-2g2xb"]
Sep 11 13:54:31.079196 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:54:31.079157    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod66f65f8b_07ac_4751_b4a8_929b4e132f26.slice/crio-1558dac89de6a629452192c67a3d4782b2c7db9ecd8eaacd41c526ccaca903cd WatchSource:0}: Error finding container 1558dac89de6a629452192c67a3d4782b2c7db9ecd8eaacd41c526ccaca903cd: Status 404 returned error can't find the container with id 1558dac89de6a629452192c67a3d4782b2c7db9ecd8eaacd41c526ccaca903cd
Sep 11 13:54:31.477054 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:54:31.477000    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-9edb53/rate-test-model-9edb53-kserve-69d4d55b55-2g2xb" event={"ID":"66f65f8b-07ac-4751-b4a8-929b4e132f26","Type":"ContainerStarted","Data":"0c61907b5d2424ef5776f5fcb19e8aa6ec7614cc8224fcd44ee9a730ee462387"}
Sep 11 13:54:31.477054 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:54:31.477060    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-9edb53/rate-test-model-9edb53-kserve-69d4d55b55-2g2xb" event={"ID":"66f65f8b-07ac-4751-b4a8-929b4e132f26","Type":"ContainerStarted","Data":"1558dac89de6a629452192c67a3d4782b2c7db9ecd8eaacd41c526ccaca903cd"}
Sep 11 13:54:31.497633 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:54:31.497560    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-9edb53/rate-test-model-9edb53-kserve-69d4d55b55-2g2xb" podStartSLOduration=1.497536399 podStartE2EDuration="1.497536399s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:54:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:54:31.496465642 +0000 UTC m=+1978.882568846" watchObservedRunningTime="2026-09-11 13:54:31.497536399 +0000 UTC m=+1978.883639563"
Sep 11 13:54:40.785980 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:54:40.785918    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-9edb53/rate-test-model-9edb53-kserve-69d4d55b55-2g2xb"
Sep 11 13:54:40.786546 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:54:40.786001    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-9edb53/rate-test-model-9edb53-kserve-69d4d55b55-2g2xb"
Sep 11 13:54:40.800045 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:54:40.800000    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-9edb53/rate-test-model-9edb53-kserve-69d4d55b55-2g2xb"
Sep 11 13:54:41.550531 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:54:41.550487    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-9edb53/rate-test-model-9edb53-kserve-69d4d55b55-2g2xb"
Sep 11 13:55:12.465515 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:55:12.465449    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-9edb53/rate-test-model-9edb53-kserve-69d4d55b55-2g2xb"]
Sep 11 13:55:13.257522 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:55:13.257472    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-264b08-openshift-default-5f4c68fb78-hw2m9"]
Sep 11 13:55:13.261541 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:55:13.261511    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-264b08-openshift-default-5f4c68fb78-hw2m9"
Sep 11 13:55:13.264124 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:55:13.264091    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-264b08-openshift-default-dockercfg-cjqdb\""
Sep 11 13:55:13.275462 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:55:13.275430    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-264b08-openshift-default-5f4c68fb78-hw2m9"]
Sep 11 13:55:13.350041 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:55:13.349996    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42a374b5-a779-416f-9f2c-bb1cf1d6f405-istio-podinfo\") pod \"e2e-ipp-a-264b08-openshift-default-5f4c68fb78-hw2m9\" (UID: \"42a374b5-a779-416f-9f2c-bb1cf1d6f405\") " pod="openshift-ingress/e2e-ipp-a-264b08-openshift-default-5f4c68fb78-hw2m9"
Sep 11 13:55:13.350310 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:55:13.350140    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42a374b5-a779-416f-9f2c-bb1cf1d6f405-istio-data\") pod \"e2e-ipp-a-264b08-openshift-default-5f4c68fb78-hw2m9\" (UID: \"42a374b5-a779-416f-9f2c-bb1cf1d6f405\") " pod="openshift-ingress/e2e-ipp-a-264b08-openshift-default-5f4c68fb78-hw2m9"
Sep 11 13:55:13.350310 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:55:13.350186    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42a374b5-a779-416f-9f2c-bb1cf1d6f405-workload-certs\") pod \"e2e-ipp-a-264b08-openshift-default-5f4c68fb78-hw2m9\" (UID: \"42a374b5-a779-416f-9f2c-bb1cf1d6f405\") " pod="openshift-ingress/e2e-ipp-a-264b08-openshift-default-5f4c68fb78-hw2m9"
Sep 11 13:55:13.350310 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:55:13.350223    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42a374b5-a779-416f-9f2c-bb1cf1d6f405-istio-token\") pod \"e2e-ipp-a-264b08-openshift-default-5f4c68fb78-hw2m9\" (UID: \"42a374b5-a779-416f-9f2c-bb1cf1d6f405\") " pod="openshift-ingress/e2e-ipp-a-264b08-openshift-default-5f4c68fb78-hw2m9"
Sep 11 13:55:13.350310 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:55:13.350245    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42a374b5-a779-416f-9f2c-bb1cf1d6f405-istiod-ca-cert\") pod \"e2e-ipp-a-264b08-openshift-default-5f4c68fb78-hw2m9\" (UID: \"42a374b5-a779-416f-9f2c-bb1cf1d6f405\") " pod="openshift-ingress/e2e-ipp-a-264b08-openshift-default-5f4c68fb78-hw2m9"
Sep 11 13:55:13.350310 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:55:13.350295    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42a374b5-a779-416f-9f2c-bb1cf1d6f405-istio-envoy\") pod \"e2e-ipp-a-264b08-openshift-default-5f4c68fb78-hw2m9\" (UID: \"42a374b5-a779-416f-9f2c-bb1cf1d6f405\") " pod="openshift-ingress/e2e-ipp-a-264b08-openshift-default-5f4c68fb78-hw2m9"
Sep 11 13:55:13.350544 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:55:13.350351    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42a374b5-a779-416f-9f2c-bb1cf1d6f405-workload-socket\") pod \"e2e-ipp-a-264b08-openshift-default-5f4c68fb78-hw2m9\" (UID: \"42a374b5-a779-416f-9f2c-bb1cf1d6f405\") " pod="openshift-ingress/e2e-ipp-a-264b08-openshift-default-5f4c68fb78-hw2m9"
Sep 11 13:55:13.350544 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:55:13.350454    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42a374b5-a779-416f-9f2c-bb1cf1d6f405-credential-socket\") pod \"e2e-ipp-a-264b08-openshift-default-5f4c68fb78-hw2m9\" (UID: \"42a374b5-a779-416f-9f2c-bb1cf1d6f405\") " pod="openshift-ingress/e2e-ipp-a-264b08-openshift-default-5f4c68fb78-hw2m9"
Sep 11 13:55:13.350620 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:55:13.350545    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxkm2\" (UniqueName: \"kubernetes.io/projected/42a374b5-a779-416f-9f2c-bb1cf1d6f405-kube-api-access-rxkm2\") pod \"e2e-ipp-a-264b08-openshift-default-5f4c68fb78-hw2m9\" (UID: \"42a374b5-a779-416f-9f2c-bb1cf1d6f405\") " pod="openshift-ingress/e2e-ipp-a-264b08-openshift-default-5f4c68fb78-hw2m9"
Sep 11 13:55:13.451483 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:55:13.451393    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42a374b5-a779-416f-9f2c-bb1cf1d6f405-credential-socket\") pod \"e2e-ipp-a-264b08-openshift-default-5f4c68fb78-hw2m9\" (UID: \"42a374b5-a779-416f-9f2c-bb1cf1d6f405\") " pod="openshift-ingress/e2e-ipp-a-264b08-openshift-default-5f4c68fb78-hw2m9"
Sep 11 13:55:13.451765 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:55:13.451531    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxkm2\" (UniqueName: \"kubernetes.io/projected/42a374b5-a779-416f-9f2c-bb1cf1d6f405-kube-api-access-rxkm2\") pod \"e2e-ipp-a-264b08-openshift-default-5f4c68fb78-hw2m9\" (UID: \"42a374b5-a779-416f-9f2c-bb1cf1d6f405\") " pod="openshift-ingress/e2e-ipp-a-264b08-openshift-default-5f4c68fb78-hw2m9"
Sep 11 13:55:13.451765 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:55:13.451659    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42a374b5-a779-416f-9f2c-bb1cf1d6f405-istio-podinfo\") pod \"e2e-ipp-a-264b08-openshift-default-5f4c68fb78-hw2m9\" (UID: \"42a374b5-a779-416f-9f2c-bb1cf1d6f405\") " pod="openshift-ingress/e2e-ipp-a-264b08-openshift-default-5f4c68fb78-hw2m9"
Sep 11 13:55:13.451765 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:55:13.451690    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42a374b5-a779-416f-9f2c-bb1cf1d6f405-istio-data\") pod \"e2e-ipp-a-264b08-openshift-default-5f4c68fb78-hw2m9\" (UID: \"42a374b5-a779-416f-9f2c-bb1cf1d6f405\") " pod="openshift-ingress/e2e-ipp-a-264b08-openshift-default-5f4c68fb78-hw2m9"
Sep 11 13:55:13.451765 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:55:13.451721    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42a374b5-a779-416f-9f2c-bb1cf1d6f405-workload-certs\") pod \"e2e-ipp-a-264b08-openshift-default-5f4c68fb78-hw2m9\" (UID: \"42a374b5-a779-416f-9f2c-bb1cf1d6f405\") " pod="openshift-ingress/e2e-ipp-a-264b08-openshift-default-5f4c68fb78-hw2m9"
Sep 11 13:55:13.451765 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:55:13.451761    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42a374b5-a779-416f-9f2c-bb1cf1d6f405-istio-token\") pod \"e2e-ipp-a-264b08-openshift-default-5f4c68fb78-hw2m9\" (UID: \"42a374b5-a779-416f-9f2c-bb1cf1d6f405\") " pod="openshift-ingress/e2e-ipp-a-264b08-openshift-default-5f4c68fb78-hw2m9"
Sep 11 13:55:13.452069 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:55:13.451790    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42a374b5-a779-416f-9f2c-bb1cf1d6f405-istiod-ca-cert\") pod \"e2e-ipp-a-264b08-openshift-default-5f4c68fb78-hw2m9\" (UID: \"42a374b5-a779-416f-9f2c-bb1cf1d6f405\") " pod="openshift-ingress/e2e-ipp-a-264b08-openshift-default-5f4c68fb78-hw2m9"
Sep 11 13:55:13.452069 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:55:13.451881    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42a374b5-a779-416f-9f2c-bb1cf1d6f405-istio-envoy\") pod \"e2e-ipp-a-264b08-openshift-default-5f4c68fb78-hw2m9\" (UID: \"42a374b5-a779-416f-9f2c-bb1cf1d6f405\") " pod="openshift-ingress/e2e-ipp-a-264b08-openshift-default-5f4c68fb78-hw2m9"
Sep 11 13:55:13.452069 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:55:13.451935    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42a374b5-a779-416f-9f2c-bb1cf1d6f405-workload-socket\") pod \"e2e-ipp-a-264b08-openshift-default-5f4c68fb78-hw2m9\" (UID: \"42a374b5-a779-416f-9f2c-bb1cf1d6f405\") " pod="openshift-ingress/e2e-ipp-a-264b08-openshift-default-5f4c68fb78-hw2m9"
Sep 11 13:55:13.452069 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:55:13.451940    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42a374b5-a779-416f-9f2c-bb1cf1d6f405-credential-socket\") pod \"e2e-ipp-a-264b08-openshift-default-5f4c68fb78-hw2m9\" (UID: \"42a374b5-a779-416f-9f2c-bb1cf1d6f405\") " pod="openshift-ingress/e2e-ipp-a-264b08-openshift-default-5f4c68fb78-hw2m9"
Sep 11 13:55:13.452273 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:55:13.452101    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42a374b5-a779-416f-9f2c-bb1cf1d6f405-istio-data\") pod \"e2e-ipp-a-264b08-openshift-default-5f4c68fb78-hw2m9\" (UID: \"42a374b5-a779-416f-9f2c-bb1cf1d6f405\") " pod="openshift-ingress/e2e-ipp-a-264b08-openshift-default-5f4c68fb78-hw2m9"
Sep 11 13:55:13.452273 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:55:13.452170    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42a374b5-a779-416f-9f2c-bb1cf1d6f405-workload-certs\") pod \"e2e-ipp-a-264b08-openshift-default-5f4c68fb78-hw2m9\" (UID: \"42a374b5-a779-416f-9f2c-bb1cf1d6f405\") " pod="openshift-ingress/e2e-ipp-a-264b08-openshift-default-5f4c68fb78-hw2m9"
Sep 11 13:55:13.452383 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:55:13.452324    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42a374b5-a779-416f-9f2c-bb1cf1d6f405-workload-socket\") pod \"e2e-ipp-a-264b08-openshift-default-5f4c68fb78-hw2m9\" (UID: \"42a374b5-a779-416f-9f2c-bb1cf1d6f405\") " pod="openshift-ingress/e2e-ipp-a-264b08-openshift-default-5f4c68fb78-hw2m9"
Sep 11 13:55:13.452734 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:55:13.452714    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42a374b5-a779-416f-9f2c-bb1cf1d6f405-istiod-ca-cert\") pod \"e2e-ipp-a-264b08-openshift-default-5f4c68fb78-hw2m9\" (UID: \"42a374b5-a779-416f-9f2c-bb1cf1d6f405\") " pod="openshift-ingress/e2e-ipp-a-264b08-openshift-default-5f4c68fb78-hw2m9"
Sep 11 13:55:13.454501 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:55:13.454478    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42a374b5-a779-416f-9f2c-bb1cf1d6f405-istio-envoy\") pod \"e2e-ipp-a-264b08-openshift-default-5f4c68fb78-hw2m9\" (UID: \"42a374b5-a779-416f-9f2c-bb1cf1d6f405\") " pod="openshift-ingress/e2e-ipp-a-264b08-openshift-default-5f4c68fb78-hw2m9"
Sep 11 13:55:13.454714 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:55:13.454697    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42a374b5-a779-416f-9f2c-bb1cf1d6f405-istio-podinfo\") pod \"e2e-ipp-a-264b08-openshift-default-5f4c68fb78-hw2m9\" (UID: \"42a374b5-a779-416f-9f2c-bb1cf1d6f405\") " pod="openshift-ingress/e2e-ipp-a-264b08-openshift-default-5f4c68fb78-hw2m9"
Sep 11 13:55:13.462692 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:55:13.462656    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42a374b5-a779-416f-9f2c-bb1cf1d6f405-istio-token\") pod \"e2e-ipp-a-264b08-openshift-default-5f4c68fb78-hw2m9\" (UID: \"42a374b5-a779-416f-9f2c-bb1cf1d6f405\") " pod="openshift-ingress/e2e-ipp-a-264b08-openshift-default-5f4c68fb78-hw2m9"
Sep 11 13:55:13.462818 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:55:13.462768    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxkm2\" (UniqueName: \"kubernetes.io/projected/42a374b5-a779-416f-9f2c-bb1cf1d6f405-kube-api-access-rxkm2\") pod \"e2e-ipp-a-264b08-openshift-default-5f4c68fb78-hw2m9\" (UID: \"42a374b5-a779-416f-9f2c-bb1cf1d6f405\") " pod="openshift-ingress/e2e-ipp-a-264b08-openshift-default-5f4c68fb78-hw2m9"
Sep 11 13:55:13.576460 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:55:13.576301    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-264b08-openshift-default-5f4c68fb78-hw2m9"
Sep 11 13:55:13.872204 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:55:13.872173    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-264b08-openshift-default-5f4c68fb78-hw2m9"]
Sep 11 13:55:13.875466 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:55:13.875416    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42a374b5_a779_416f_9f2c_bb1cf1d6f405.slice/crio-5bcbec075a8bd493f642561e7ca05d02d2d9ee37ee0dcbd29306a58c56b25f1e WatchSource:0}: Error finding container 5bcbec075a8bd493f642561e7ca05d02d2d9ee37ee0dcbd29306a58c56b25f1e: Status 404 returned error can't find the container with id 5bcbec075a8bd493f642561e7ca05d02d2d9ee37ee0dcbd29306a58c56b25f1e
Sep 11 13:55:13.878348 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:55:13.878310    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 11 13:55:13.878461 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:55:13.878394    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 11 13:55:13.878461 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:55:13.878430    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 11 13:55:14.758959 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:55:14.758912    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-264b08-openshift-default-5f4c68fb78-hw2m9" event={"ID":"42a374b5-a779-416f-9f2c-bb1cf1d6f405","Type":"ContainerStarted","Data":"a213fffc77d16b9c7535d58c1a471c37e31c00a18a25be53bf3e4114f99d31e7"}
Sep 11 13:55:14.759466 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:55:14.758980    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-264b08-openshift-default-5f4c68fb78-hw2m9" event={"ID":"42a374b5-a779-416f-9f2c-bb1cf1d6f405","Type":"ContainerStarted","Data":"5bcbec075a8bd493f642561e7ca05d02d2d9ee37ee0dcbd29306a58c56b25f1e"}
Sep 11 13:55:14.783266 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:55:14.783188    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-264b08-openshift-default-5f4c68fb78-hw2m9" podStartSLOduration=1.783169386 podStartE2EDuration="1.783169386s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:55:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:55:14.779574305 +0000 UTC m=+2022.165677501" watchObservedRunningTime="2026-09-11 13:55:14.783169386 +0000 UTC m=+2022.169272550"
Sep 11 13:55:15.577325 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:55:15.577283    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-264b08-openshift-default-5f4c68fb78-hw2m9"
Sep 11 13:55:15.583922 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:55:15.583876    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-264b08-openshift-default-5f4c68fb78-hw2m9"
Sep 11 13:55:15.765288 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:55:15.765247    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-264b08-openshift-default-5f4c68fb78-hw2m9"
Sep 11 13:55:15.766521 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:55:15.766502    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-264b08-openshift-default-5f4c68fb78-hw2m9"
Sep 11 13:55:19.476527 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:55:19.476454    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-264b08-5767fc97-4pv4s"]
Sep 11 13:55:19.482119 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:55:19.482088    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-264b08-5767fc97-4pv4s"
Sep 11 13:55:19.485104 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:55:19.485073    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-264b08-serving-cert\""
Sep 11 13:55:19.493033 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:55:19.492997    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-264b08-5767fc97-4pv4s"]
Sep 11 13:55:19.529683 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:55:19.529628    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9204f93d-4753-4663-aed0-fba9565a8df9-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-264b08-5767fc97-4pv4s\" (UID: \"9204f93d-4753-4663-aed0-fba9565a8df9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-264b08-5767fc97-4pv4s"
Sep 11 13:55:19.529927 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:55:19.529723    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9204f93d-4753-4663-aed0-fba9565a8df9-maas-api-tls\") pod \"maas-api-e2e-ipp-a-264b08-5767fc97-4pv4s\" (UID: \"9204f93d-4753-4663-aed0-fba9565a8df9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-264b08-5767fc97-4pv4s"
Sep 11 13:55:19.529927 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:55:19.529823    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9204f93d-4753-4663-aed0-fba9565a8df9-metrics-certs\") pod \"maas-api-e2e-ipp-a-264b08-5767fc97-4pv4s\" (UID: \"9204f93d-4753-4663-aed0-fba9565a8df9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-264b08-5767fc97-4pv4s"
Sep 11 13:55:19.530019 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:55:19.529917    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57wlz\" (UniqueName: \"kubernetes.io/projected/9204f93d-4753-4663-aed0-fba9565a8df9-kube-api-access-57wlz\") pod \"maas-api-e2e-ipp-a-264b08-5767fc97-4pv4s\" (UID: \"9204f93d-4753-4663-aed0-fba9565a8df9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-264b08-5767fc97-4pv4s"
Sep 11 13:55:19.631028 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:55:19.630982    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9204f93d-4753-4663-aed0-fba9565a8df9-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-264b08-5767fc97-4pv4s\" (UID: \"9204f93d-4753-4663-aed0-fba9565a8df9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-264b08-5767fc97-4pv4s"
Sep 11 13:55:19.631302 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:55:19.631078    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9204f93d-4753-4663-aed0-fba9565a8df9-maas-api-tls\") pod \"maas-api-e2e-ipp-a-264b08-5767fc97-4pv4s\" (UID: \"9204f93d-4753-4663-aed0-fba9565a8df9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-264b08-5767fc97-4pv4s"
Sep 11 13:55:19.631302 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:55:19.631166    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9204f93d-4753-4663-aed0-fba9565a8df9-metrics-certs\") pod \"maas-api-e2e-ipp-a-264b08-5767fc97-4pv4s\" (UID: \"9204f93d-4753-4663-aed0-fba9565a8df9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-264b08-5767fc97-4pv4s"
Sep 11 13:55:19.631302 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:55:19.631206    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57wlz\" (UniqueName: \"kubernetes.io/projected/9204f93d-4753-4663-aed0-fba9565a8df9-kube-api-access-57wlz\") pod \"maas-api-e2e-ipp-a-264b08-5767fc97-4pv4s\" (UID: \"9204f93d-4753-4663-aed0-fba9565a8df9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-264b08-5767fc97-4pv4s"
Sep 11 13:55:19.631902 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:55:19.631870    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9204f93d-4753-4663-aed0-fba9565a8df9-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-264b08-5767fc97-4pv4s\" (UID: \"9204f93d-4753-4663-aed0-fba9565a8df9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-264b08-5767fc97-4pv4s"
Sep 11 13:55:19.634287 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:55:19.634242    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9204f93d-4753-4663-aed0-fba9565a8df9-metrics-certs\") pod \"maas-api-e2e-ipp-a-264b08-5767fc97-4pv4s\" (UID: \"9204f93d-4753-4663-aed0-fba9565a8df9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-264b08-5767fc97-4pv4s"
Sep 11 13:55:19.634442 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:55:19.634420    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9204f93d-4753-4663-aed0-fba9565a8df9-maas-api-tls\") pod \"maas-api-e2e-ipp-a-264b08-5767fc97-4pv4s\" (UID: \"9204f93d-4753-4663-aed0-fba9565a8df9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-264b08-5767fc97-4pv4s"
Sep 11 13:55:19.641943 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:55:19.641803    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-57wlz\" (UniqueName: \"kubernetes.io/projected/9204f93d-4753-4663-aed0-fba9565a8df9-kube-api-access-57wlz\") pod \"maas-api-e2e-ipp-a-264b08-5767fc97-4pv4s\" (UID: \"9204f93d-4753-4663-aed0-fba9565a8df9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-264b08-5767fc97-4pv4s"
Sep 11 13:55:19.800095 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:55:19.799994    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-264b08-5767fc97-4pv4s"
Sep 11 13:55:20.110217 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:55:20.110176    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-264b08-5767fc97-4pv4s"]
Sep 11 13:55:20.113974 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:55:20.113928    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9204f93d_4753_4663_aed0_fba9565a8df9.slice/crio-d62a0fd37d271c10326ab65d5a8fbcefbb6e828295ca24b59ae02e74f0fcf4e4 WatchSource:0}: Error finding container d62a0fd37d271c10326ab65d5a8fbcefbb6e828295ca24b59ae02e74f0fcf4e4: Status 404 returned error can't find the container with id d62a0fd37d271c10326ab65d5a8fbcefbb6e828295ca24b59ae02e74f0fcf4e4
Sep 11 13:55:20.115699 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:55:20.115664    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 13:55:20.802716 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:55:20.802666    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-264b08-5767fc97-4pv4s" event={"ID":"9204f93d-4753-4663-aed0-fba9565a8df9","Type":"ContainerStarted","Data":"da3fa71cd700c61fcf24818a72bccef0f8b1d1dea1ef6e1ba0dc44221998affb"}
Sep 11 13:55:20.803238 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:55:20.802730    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-264b08-5767fc97-4pv4s" event={"ID":"9204f93d-4753-4663-aed0-fba9565a8df9","Type":"ContainerStarted","Data":"d62a0fd37d271c10326ab65d5a8fbcefbb6e828295ca24b59ae02e74f0fcf4e4"}
Sep 11 13:55:20.803238 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:55:20.802765    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-264b08-5767fc97-4pv4s"
Sep 11 13:55:20.823115 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:55:20.823021    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-264b08-5767fc97-4pv4s" podStartSLOduration=1.350457665 podStartE2EDuration="1.822988465s" totalImagesPullingTime="472.5308ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:55:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:55:20.821958862 +0000 UTC m=+2028.208062052" watchObservedRunningTime="2026-09-11 13:55:20.822988465 +0000 UTC m=+2028.209091630"
Sep 11 13:55:26.816729 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:55:26.816689    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-264b08-5767fc97-4pv4s"
Sep 11 13:55:27.510318 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:55:27.510228    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-9edb53/rate-test-model-9edb53-kserve-69d4d55b55-2g2xb" podUID="66f65f8b-07ac-4751-b4a8-929b4e132f26" containerName="main" containerID="cri-o://0c61907b5d2424ef5776f5fcb19e8aa6ec7614cc8224fcd44ee9a730ee462387" gracePeriod=45
Sep 11 13:55:27.836859 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:55:27.836806    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-9edb53/rate-test-model-9edb53-kserve-69d4d55b55-2g2xb"
Sep 11 13:55:27.849679 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:55:27.849632    2578 generic.go:356] "Generic (PLEG): container finished" podID="66f65f8b-07ac-4751-b4a8-929b4e132f26" containerID="0c61907b5d2424ef5776f5fcb19e8aa6ec7614cc8224fcd44ee9a730ee462387" exitCode=0
Sep 11 13:55:27.849939 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:55:27.849691    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-9edb53/rate-test-model-9edb53-kserve-69d4d55b55-2g2xb"
Sep 11 13:55:27.849939 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:55:27.849725    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-9edb53/rate-test-model-9edb53-kserve-69d4d55b55-2g2xb" event={"ID":"66f65f8b-07ac-4751-b4a8-929b4e132f26","Type":"ContainerDied","Data":"0c61907b5d2424ef5776f5fcb19e8aa6ec7614cc8224fcd44ee9a730ee462387"}
Sep 11 13:55:27.849939 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:55:27.849792    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-9edb53/rate-test-model-9edb53-kserve-69d4d55b55-2g2xb" event={"ID":"66f65f8b-07ac-4751-b4a8-929b4e132f26","Type":"ContainerDied","Data":"1558dac89de6a629452192c67a3d4782b2c7db9ecd8eaacd41c526ccaca903cd"}
Sep 11 13:55:27.849939 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:55:27.849820    2578 scope.go:117] "RemoveContainer" containerID="0c61907b5d2424ef5776f5fcb19e8aa6ec7614cc8224fcd44ee9a730ee462387"
Sep 11 13:55:27.862149 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:55:27.862123    2578 scope.go:117] "RemoveContainer" containerID="0c61907b5d2424ef5776f5fcb19e8aa6ec7614cc8224fcd44ee9a730ee462387"
Sep 11 13:55:27.862663 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:55:27.862637    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c61907b5d2424ef5776f5fcb19e8aa6ec7614cc8224fcd44ee9a730ee462387\": container with ID starting with 0c61907b5d2424ef5776f5fcb19e8aa6ec7614cc8224fcd44ee9a730ee462387 not found: ID does not exist" containerID="0c61907b5d2424ef5776f5fcb19e8aa6ec7614cc8224fcd44ee9a730ee462387"
Sep 11 13:55:27.862727 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:55:27.862675    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c61907b5d2424ef5776f5fcb19e8aa6ec7614cc8224fcd44ee9a730ee462387"} err="failed to get container status \"0c61907b5d2424ef5776f5fcb19e8aa6ec7614cc8224fcd44ee9a730ee462387\": rpc error: code = NotFound desc = could not find container \"0c61907b5d2424ef5776f5fcb19e8aa6ec7614cc8224fcd44ee9a730ee462387\": container with ID starting with 0c61907b5d2424ef5776f5fcb19e8aa6ec7614cc8224fcd44ee9a730ee462387 not found: ID does not exist"
Sep 11 13:55:27.942787 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:55:27.942747    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66f65f8b-07ac-4751-b4a8-929b4e132f26-model-cache\") pod \"66f65f8b-07ac-4751-b4a8-929b4e132f26\" (UID: \"66f65f8b-07ac-4751-b4a8-929b4e132f26\") "
Sep 11 13:55:27.942787 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:55:27.942812    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66f65f8b-07ac-4751-b4a8-929b4e132f26-dshm\") pod \"66f65f8b-07ac-4751-b4a8-929b4e132f26\" (UID: \"66f65f8b-07ac-4751-b4a8-929b4e132f26\") "
Sep 11 13:55:27.943076 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:55:27.942858    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66f65f8b-07ac-4751-b4a8-929b4e132f26-home\") pod \"66f65f8b-07ac-4751-b4a8-929b4e132f26\" (UID: \"66f65f8b-07ac-4751-b4a8-929b4e132f26\") "
Sep 11 13:55:27.943076 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:55:27.942909    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66f65f8b-07ac-4751-b4a8-929b4e132f26-tmp-dir\") pod \"66f65f8b-07ac-4751-b4a8-929b4e132f26\" (UID: \"66f65f8b-07ac-4751-b4a8-929b4e132f26\") "
Sep 11 13:55:27.943076 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:55:27.942950    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9mjdq\" (UniqueName: \"kubernetes.io/projected/66f65f8b-07ac-4751-b4a8-929b4e132f26-kube-api-access-9mjdq\") pod \"66f65f8b-07ac-4751-b4a8-929b4e132f26\" (UID: \"66f65f8b-07ac-4751-b4a8-929b4e132f26\") "
Sep 11 13:55:27.943076 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:55:27.943010    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66f65f8b-07ac-4751-b4a8-929b4e132f26-tls-certs\") pod \"66f65f8b-07ac-4751-b4a8-929b4e132f26\" (UID: \"66f65f8b-07ac-4751-b4a8-929b4e132f26\") "
Sep 11 13:55:27.943300 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:55:27.943093    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66f65f8b-07ac-4751-b4a8-929b4e132f26-model-cache" (OuterVolumeSpecName: "model-cache") pod "66f65f8b-07ac-4751-b4a8-929b4e132f26" (UID: "66f65f8b-07ac-4751-b4a8-929b4e132f26"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 13:55:27.943300 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:55:27.943121    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66f65f8b-07ac-4751-b4a8-929b4e132f26-home" (OuterVolumeSpecName: "home") pod "66f65f8b-07ac-4751-b4a8-929b4e132f26" (UID: "66f65f8b-07ac-4751-b4a8-929b4e132f26"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 13:55:27.943300 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:55:27.943184    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66f65f8b-07ac-4751-b4a8-929b4e132f26-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "66f65f8b-07ac-4751-b4a8-929b4e132f26" (UID: "66f65f8b-07ac-4751-b4a8-929b4e132f26"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 13:55:27.943487 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:55:27.943466    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66f65f8b-07ac-4751-b4a8-929b4e132f26-model-cache\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:55:27.943487 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:55:27.943488    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66f65f8b-07ac-4751-b4a8-929b4e132f26-home\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:55:27.943627 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:55:27.943499    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66f65f8b-07ac-4751-b4a8-929b4e132f26-tmp-dir\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:55:27.945351 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:55:27.945326    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66f65f8b-07ac-4751-b4a8-929b4e132f26-dshm" (OuterVolumeSpecName: "dshm") pod "66f65f8b-07ac-4751-b4a8-929b4e132f26" (UID: "66f65f8b-07ac-4751-b4a8-929b4e132f26"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 13:55:27.945351 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:55:27.945338    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66f65f8b-07ac-4751-b4a8-929b4e132f26-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "66f65f8b-07ac-4751-b4a8-929b4e132f26" (UID: "66f65f8b-07ac-4751-b4a8-929b4e132f26"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 13:55:27.945519 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:55:27.945342    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66f65f8b-07ac-4751-b4a8-929b4e132f26-kube-api-access-9mjdq" (OuterVolumeSpecName: "kube-api-access-9mjdq") pod "66f65f8b-07ac-4751-b4a8-929b4e132f26" (UID: "66f65f8b-07ac-4751-b4a8-929b4e132f26"). InnerVolumeSpecName "kube-api-access-9mjdq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 13:55:28.045058 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:55:28.044937    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9mjdq\" (UniqueName: \"kubernetes.io/projected/66f65f8b-07ac-4751-b4a8-929b4e132f26-kube-api-access-9mjdq\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:55:28.045058 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:55:28.044993    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66f65f8b-07ac-4751-b4a8-929b4e132f26-tls-certs\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:55:28.045058 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:55:28.045009    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66f65f8b-07ac-4751-b4a8-929b4e132f26-dshm\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:55:28.173909 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:55:28.173848    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-9edb53/rate-test-model-9edb53-kserve-69d4d55b55-2g2xb"]
Sep 11 13:55:28.178585 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:55:28.178542    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-9edb53/rate-test-model-9edb53-kserve-69d4d55b55-2g2xb"]
Sep 11 13:55:29.114374 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:55:29.114304    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="66f65f8b-07ac-4751-b4a8-929b4e132f26" path="/var/lib/kubelet/pods/66f65f8b-07ac-4751-b4a8-929b4e132f26/volumes"
Sep 11 13:55:34.902362 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:55:34.902308    2578 generic.go:356] "Generic (PLEG): container finished" podID="42a374b5-a779-416f-9f2c-bb1cf1d6f405" containerID="a213fffc77d16b9c7535d58c1a471c37e31c00a18a25be53bf3e4114f99d31e7" exitCode=0
Sep 11 13:55:34.902957 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:55:34.902385    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-264b08-openshift-default-5f4c68fb78-hw2m9" event={"ID":"42a374b5-a779-416f-9f2c-bb1cf1d6f405","Type":"ContainerDied","Data":"a213fffc77d16b9c7535d58c1a471c37e31c00a18a25be53bf3e4114f99d31e7"}
Sep 11 13:55:34.903063 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:55:34.903044    2578 scope.go:117] "RemoveContainer" containerID="a213fffc77d16b9c7535d58c1a471c37e31c00a18a25be53bf3e4114f99d31e7"
Sep 11 13:55:34.904629 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:55:34.904594    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 11 13:55:34.904753 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:55:34.904676    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 11 13:55:34.904753 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:55:34.904708    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 11 13:55:35.917921 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:55:35.917872    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-264b08-openshift-default-5f4c68fb78-hw2m9" event={"ID":"42a374b5-a779-416f-9f2c-bb1cf1d6f405","Type":"ContainerStarted","Data":"12b086021f1e495fbbd9625109a6e29ba52f5326419b1e90589f214e6e2a1c09"}
Sep 11 13:55:36.577442 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:55:36.577385    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-264b08-openshift-default-5f4c68fb78-hw2m9"
Sep 11 13:55:36.577756 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:55:36.577729    2578 patch_prober.go:28] interesting pod/e2e-ipp-a-264b08-openshift-default-5f4c68fb78-hw2m9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Sep 11 13:55:36.577813 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:55:36.577791    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-264b08-openshift-default-5f4c68fb78-hw2m9" podUID="42a374b5-a779-416f-9f2c-bb1cf1d6f405" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Sep 11 13:55:37.576784 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:55:37.576733    2578 patch_prober.go:28] interesting pod/e2e-ipp-a-264b08-openshift-default-5f4c68fb78-hw2m9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Sep 11 13:55:37.577336 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:55:37.576804    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-264b08-openshift-default-5f4c68fb78-hw2m9" podUID="42a374b5-a779-416f-9f2c-bb1cf1d6f405" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Sep 11 13:55:38.577296 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:55:38.577248    2578 patch_prober.go:28] interesting pod/e2e-ipp-a-264b08-openshift-default-5f4c68fb78-hw2m9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Sep 11 13:55:38.577791 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:55:38.577322    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-264b08-openshift-default-5f4c68fb78-hw2m9" podUID="42a374b5-a779-416f-9f2c-bb1cf1d6f405" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Sep 11 13:55:39.577565 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:55:39.577509    2578 patch_prober.go:28] interesting pod/e2e-ipp-a-264b08-openshift-default-5f4c68fb78-hw2m9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Sep 11 13:55:39.578123 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:55:39.577587    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-264b08-openshift-default-5f4c68fb78-hw2m9" podUID="42a374b5-a779-416f-9f2c-bb1cf1d6f405" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Sep 11 13:55:40.577079 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:55:40.577035    2578 patch_prober.go:28] interesting pod/e2e-ipp-a-264b08-openshift-default-5f4c68fb78-hw2m9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Sep 11 13:55:40.577301 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:55:40.577127    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-264b08-openshift-default-5f4c68fb78-hw2m9" podUID="42a374b5-a779-416f-9f2c-bb1cf1d6f405" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Sep 11 13:55:41.582038 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:55:41.581993    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-264b08-openshift-default-5f4c68fb78-hw2m9"
Sep 11 13:55:41.582629 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:55:41.582577    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-264b08-openshift-default-5f4c68fb78-hw2m9"
Sep 11 13:55:41.584156 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:55:41.584127    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-264b08-openshift-default-5f4c68fb78-hw2m9"
Sep 11 13:55:49.441354 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:55:49.441310    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a403bc-597d56745f-nfn6d"]
Sep 11 13:55:49.441962 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:55:49.441919    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66f65f8b-07ac-4751-b4a8-929b4e132f26" containerName="main"
Sep 11 13:55:49.441962 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:55:49.441934    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="66f65f8b-07ac-4751-b4a8-929b4e132f26" containerName="main"
Sep 11 13:55:49.442049 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:55:49.442038    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="66f65f8b-07ac-4751-b4a8-929b4e132f26" containerName="main"
Sep 11 13:55:49.445149 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:55:49.445111    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a403bc-597d56745f-nfn6d"
Sep 11 13:55:49.447657 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:55:49.447627    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-a403bc-serving-cert\""
Sep 11 13:55:49.458545 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:55:49.458500    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a403bc-597d56745f-nfn6d"]
Sep 11 13:55:49.612479 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:55:49.612426    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/88ee4ae2-29d7-4611-ada9-efa119d09d1d-maas-api-tls\") pod \"maas-api-e2e-ipp-b-a403bc-597d56745f-nfn6d\" (UID: \"88ee4ae2-29d7-4611-ada9-efa119d09d1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a403bc-597d56745f-nfn6d"
Sep 11 13:55:49.612817 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:55:49.612542    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/88ee4ae2-29d7-4611-ada9-efa119d09d1d-metrics-certs\") pod \"maas-api-e2e-ipp-b-a403bc-597d56745f-nfn6d\" (UID: \"88ee4ae2-29d7-4611-ada9-efa119d09d1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a403bc-597d56745f-nfn6d"
Sep 11 13:55:49.612817 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:55:49.612576    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zx77\" (UniqueName: \"kubernetes.io/projected/88ee4ae2-29d7-4611-ada9-efa119d09d1d-kube-api-access-9zx77\") pod \"maas-api-e2e-ipp-b-a403bc-597d56745f-nfn6d\" (UID: \"88ee4ae2-29d7-4611-ada9-efa119d09d1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a403bc-597d56745f-nfn6d"
Sep 11 13:55:49.612817 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:55:49.612632    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/88ee4ae2-29d7-4611-ada9-efa119d09d1d-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-a403bc-597d56745f-nfn6d\" (UID: \"88ee4ae2-29d7-4611-ada9-efa119d09d1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a403bc-597d56745f-nfn6d"
Sep 11 13:55:49.713791 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:55:49.713639    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/88ee4ae2-29d7-4611-ada9-efa119d09d1d-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-a403bc-597d56745f-nfn6d\" (UID: \"88ee4ae2-29d7-4611-ada9-efa119d09d1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a403bc-597d56745f-nfn6d"
Sep 11 13:55:49.714124 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:55:49.713870    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/88ee4ae2-29d7-4611-ada9-efa119d09d1d-maas-api-tls\") pod \"maas-api-e2e-ipp-b-a403bc-597d56745f-nfn6d\" (UID: \"88ee4ae2-29d7-4611-ada9-efa119d09d1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a403bc-597d56745f-nfn6d"
Sep 11 13:55:49.714124 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:55:49.713933    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/88ee4ae2-29d7-4611-ada9-efa119d09d1d-metrics-certs\") pod \"maas-api-e2e-ipp-b-a403bc-597d56745f-nfn6d\" (UID: \"88ee4ae2-29d7-4611-ada9-efa119d09d1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a403bc-597d56745f-nfn6d"
Sep 11 13:55:49.714292 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:55:49.714132    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zx77\" (UniqueName: \"kubernetes.io/projected/88ee4ae2-29d7-4611-ada9-efa119d09d1d-kube-api-access-9zx77\") pod \"maas-api-e2e-ipp-b-a403bc-597d56745f-nfn6d\" (UID: \"88ee4ae2-29d7-4611-ada9-efa119d09d1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a403bc-597d56745f-nfn6d"
Sep 11 13:55:49.714634 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:55:49.714602    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/88ee4ae2-29d7-4611-ada9-efa119d09d1d-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-a403bc-597d56745f-nfn6d\" (UID: \"88ee4ae2-29d7-4611-ada9-efa119d09d1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a403bc-597d56745f-nfn6d"
Sep 11 13:55:49.717089 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:55:49.717061    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/88ee4ae2-29d7-4611-ada9-efa119d09d1d-metrics-certs\") pod \"maas-api-e2e-ipp-b-a403bc-597d56745f-nfn6d\" (UID: \"88ee4ae2-29d7-4611-ada9-efa119d09d1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a403bc-597d56745f-nfn6d"
Sep 11 13:55:49.717089 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:55:49.717081    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/88ee4ae2-29d7-4611-ada9-efa119d09d1d-maas-api-tls\") pod \"maas-api-e2e-ipp-b-a403bc-597d56745f-nfn6d\" (UID: \"88ee4ae2-29d7-4611-ada9-efa119d09d1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a403bc-597d56745f-nfn6d"
Sep 11 13:55:49.723560 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:55:49.723523    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zx77\" (UniqueName: \"kubernetes.io/projected/88ee4ae2-29d7-4611-ada9-efa119d09d1d-kube-api-access-9zx77\") pod \"maas-api-e2e-ipp-b-a403bc-597d56745f-nfn6d\" (UID: \"88ee4ae2-29d7-4611-ada9-efa119d09d1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a403bc-597d56745f-nfn6d"
Sep 11 13:55:49.761637 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:55:49.761585    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a403bc-597d56745f-nfn6d"
Sep 11 13:55:50.049530 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:55:50.049494    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a403bc-597d56745f-nfn6d"]
Sep 11 13:55:50.052642 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:55:50.052605    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88ee4ae2_29d7_4611_ada9_efa119d09d1d.slice/crio-8102a83272ebdac0876805218d3fc2de10e609e3f82ce8c7460505ae49c7af23 WatchSource:0}: Error finding container 8102a83272ebdac0876805218d3fc2de10e609e3f82ce8c7460505ae49c7af23: Status 404 returned error can't find the container with id 8102a83272ebdac0876805218d3fc2de10e609e3f82ce8c7460505ae49c7af23
Sep 11 13:55:51.024068 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:55:51.024011    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a403bc-597d56745f-nfn6d" event={"ID":"88ee4ae2-29d7-4611-ada9-efa119d09d1d","Type":"ContainerStarted","Data":"e587dea977c526ed902ffcb81e5361d89a13697988efb6660ce81feb090a7e15"}
Sep 11 13:55:51.024068 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:55:51.024086    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a403bc-597d56745f-nfn6d" event={"ID":"88ee4ae2-29d7-4611-ada9-efa119d09d1d","Type":"ContainerStarted","Data":"8102a83272ebdac0876805218d3fc2de10e609e3f82ce8c7460505ae49c7af23"}
Sep 11 13:55:51.024614 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:55:51.024121    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a403bc-597d56745f-nfn6d"
Sep 11 13:55:51.044251 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:55:51.044187    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a403bc-597d56745f-nfn6d" podStartSLOduration=1.64184355 podStartE2EDuration="2.044164313s" totalImagesPullingTime="402.320763ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:55:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:55:51.041621837 +0000 UTC m=+2058.427725023" watchObservedRunningTime="2026-09-11 13:55:51.044164313 +0000 UTC m=+2058.430267476"
Sep 11 13:55:57.037389 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:55:57.037286    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a403bc-597d56745f-nfn6d"
Sep 11 13:56:21.173977 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:56:21.173931    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-4cgf5_773e0144-50b8-4d59-8fed-57d39bb74da2/payload-processing/0.log"
Sep 11 13:56:33.312498 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:56:33.312457    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6gltj_e384caae-3cb1-4264-8a25-55b40ac8ac45/ovn-acl-logging/0.log"
Sep 11 13:56:33.318361 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:56:33.318335    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6gltj_e384caae-3cb1-4264-8a25-55b40ac8ac45/ovn-acl-logging/0.log"
Sep 11 13:56:57.468344 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:56:57.468301    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-4cgf5_773e0144-50b8-4d59-8fed-57d39bb74da2/payload-processing/0.log"
Sep 11 13:57:05.609860 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:57:05.606432    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-7b7819-7cc47bb6fb-gtss2"]
Sep 11 13:57:05.614775 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:57:05.614734    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-7b7819-7cc47bb6fb-gtss2"
Sep 11 13:57:05.621123 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:57:05.621076    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-7b7819-serving-cert\""
Sep 11 13:57:05.625276 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:57:05.625238    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-7b7819-7cc47bb6fb-gtss2"]
Sep 11 13:57:05.741072 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:57:05.741017    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f231a7a-dad4-4a47-9727-49f2afbdcd80-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-7b7819-7cc47bb6fb-gtss2\" (UID: \"1f231a7a-dad4-4a47-9727-49f2afbdcd80\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-7b7819-7cc47bb6fb-gtss2"
Sep 11 13:57:05.741317 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:57:05.741115    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7w5rg\" (UniqueName: \"kubernetes.io/projected/1f231a7a-dad4-4a47-9727-49f2afbdcd80-kube-api-access-7w5rg\") pod \"maas-api-e2e-ipp-cleanup-7b7819-7cc47bb6fb-gtss2\" (UID: \"1f231a7a-dad4-4a47-9727-49f2afbdcd80\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-7b7819-7cc47bb6fb-gtss2"
Sep 11 13:57:05.741317 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:57:05.741156    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f231a7a-dad4-4a47-9727-49f2afbdcd80-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-7b7819-7cc47bb6fb-gtss2\" (UID: \"1f231a7a-dad4-4a47-9727-49f2afbdcd80\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-7b7819-7cc47bb6fb-gtss2"
Sep 11 13:57:05.741399 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:57:05.741302    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f231a7a-dad4-4a47-9727-49f2afbdcd80-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-7b7819-7cc47bb6fb-gtss2\" (UID: \"1f231a7a-dad4-4a47-9727-49f2afbdcd80\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-7b7819-7cc47bb6fb-gtss2"
Sep 11 13:57:05.842707 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:57:05.842648    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f231a7a-dad4-4a47-9727-49f2afbdcd80-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-7b7819-7cc47bb6fb-gtss2\" (UID: \"1f231a7a-dad4-4a47-9727-49f2afbdcd80\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-7b7819-7cc47bb6fb-gtss2"
Sep 11 13:57:05.843034 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:57:05.842810    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7w5rg\" (UniqueName: \"kubernetes.io/projected/1f231a7a-dad4-4a47-9727-49f2afbdcd80-kube-api-access-7w5rg\") pod \"maas-api-e2e-ipp-cleanup-7b7819-7cc47bb6fb-gtss2\" (UID: \"1f231a7a-dad4-4a47-9727-49f2afbdcd80\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-7b7819-7cc47bb6fb-gtss2"
Sep 11 13:57:05.843034 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:57:05.842911    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f231a7a-dad4-4a47-9727-49f2afbdcd80-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-7b7819-7cc47bb6fb-gtss2\" (UID: \"1f231a7a-dad4-4a47-9727-49f2afbdcd80\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-7b7819-7cc47bb6fb-gtss2"
Sep 11 13:57:05.843168 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:57:05.843069    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f231a7a-dad4-4a47-9727-49f2afbdcd80-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-7b7819-7cc47bb6fb-gtss2\" (UID: \"1f231a7a-dad4-4a47-9727-49f2afbdcd80\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-7b7819-7cc47bb6fb-gtss2"
Sep 11 13:57:05.843864 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:57:05.843790    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f231a7a-dad4-4a47-9727-49f2afbdcd80-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-7b7819-7cc47bb6fb-gtss2\" (UID: \"1f231a7a-dad4-4a47-9727-49f2afbdcd80\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-7b7819-7cc47bb6fb-gtss2"
Sep 11 13:57:05.845884 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:57:05.845861    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f231a7a-dad4-4a47-9727-49f2afbdcd80-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-7b7819-7cc47bb6fb-gtss2\" (UID: \"1f231a7a-dad4-4a47-9727-49f2afbdcd80\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-7b7819-7cc47bb6fb-gtss2"
Sep 11 13:57:05.845884 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:57:05.845883    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f231a7a-dad4-4a47-9727-49f2afbdcd80-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-7b7819-7cc47bb6fb-gtss2\" (UID: \"1f231a7a-dad4-4a47-9727-49f2afbdcd80\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-7b7819-7cc47bb6fb-gtss2"
Sep 11 13:57:05.852965 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:57:05.852935    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7w5rg\" (UniqueName: \"kubernetes.io/projected/1f231a7a-dad4-4a47-9727-49f2afbdcd80-kube-api-access-7w5rg\") pod \"maas-api-e2e-ipp-cleanup-7b7819-7cc47bb6fb-gtss2\" (UID: \"1f231a7a-dad4-4a47-9727-49f2afbdcd80\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-7b7819-7cc47bb6fb-gtss2"
Sep 11 13:57:05.933525 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:57:05.933360    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-7b7819-7cc47bb6fb-gtss2"
Sep 11 13:57:06.232669 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:57:06.232594    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-7b7819-7cc47bb6fb-gtss2"]
Sep 11 13:57:06.236195 ip-10-0-131-86 kubenswrapper[2578]: W0911 13:57:06.236156    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f231a7a_dad4_4a47_9727_49f2afbdcd80.slice/crio-1382777395c9c278a5ef67ed76baf074f4dbc4ef868391011972c1f4f5e2e131 WatchSource:0}: Error finding container 1382777395c9c278a5ef67ed76baf074f4dbc4ef868391011972c1f4f5e2e131: Status 404 returned error can't find the container with id 1382777395c9c278a5ef67ed76baf074f4dbc4ef868391011972c1f4f5e2e131
Sep 11 13:57:06.525430 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:57:06.525309    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-7b7819-7cc47bb6fb-gtss2" event={"ID":"1f231a7a-dad4-4a47-9727-49f2afbdcd80","Type":"ContainerStarted","Data":"1382777395c9c278a5ef67ed76baf074f4dbc4ef868391011972c1f4f5e2e131"}
Sep 11 13:57:07.534556 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:57:07.534508    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-7b7819-7cc47bb6fb-gtss2" event={"ID":"1f231a7a-dad4-4a47-9727-49f2afbdcd80","Type":"ContainerStarted","Data":"ebd82d58b339db4f12329860296243deeaa2adcdb9538c70e1dfd7a6b36d1b0d"}
Sep 11 13:57:07.535105 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:57:07.534738    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-7b7819-7cc47bb6fb-gtss2"
Sep 11 13:57:07.568171 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:57:07.568088    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-7b7819-7cc47bb6fb-gtss2" podStartSLOduration=2.170211512 podStartE2EDuration="2.5680587s" totalImagesPullingTime="397.847188ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:57:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:57:07.567080982 +0000 UTC m=+2134.953184166" watchObservedRunningTime="2026-09-11 13:57:07.5680587 +0000 UTC m=+2134.954161865"
Sep 11 13:57:13.547259 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:57:13.547207    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-7b7819-7cc47bb6fb-gtss2"
Sep 11 13:57:43.021765 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:57:43.021644    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-7b7819-7cc47bb6fb-gtss2"]
Sep 11 13:57:43.022455 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:57:43.022082    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-7b7819-7cc47bb6fb-gtss2" podUID="1f231a7a-dad4-4a47-9727-49f2afbdcd80" containerName="maas-api" containerID="cri-o://ebd82d58b339db4f12329860296243deeaa2adcdb9538c70e1dfd7a6b36d1b0d" gracePeriod=30
Sep 11 13:57:43.308586 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:57:43.308545    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-7b7819-7cc47bb6fb-gtss2"
Sep 11 13:57:43.380490 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:57:43.380450    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f231a7a-dad4-4a47-9727-49f2afbdcd80-metrics-certs\") pod \"1f231a7a-dad4-4a47-9727-49f2afbdcd80\" (UID: \"1f231a7a-dad4-4a47-9727-49f2afbdcd80\") "
Sep 11 13:57:43.380707 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:57:43.380654    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f231a7a-dad4-4a47-9727-49f2afbdcd80-openshift-service-ca\") pod \"1f231a7a-dad4-4a47-9727-49f2afbdcd80\" (UID: \"1f231a7a-dad4-4a47-9727-49f2afbdcd80\") "
Sep 11 13:57:43.380707 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:57:43.380690    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7w5rg\" (UniqueName: \"kubernetes.io/projected/1f231a7a-dad4-4a47-9727-49f2afbdcd80-kube-api-access-7w5rg\") pod \"1f231a7a-dad4-4a47-9727-49f2afbdcd80\" (UID: \"1f231a7a-dad4-4a47-9727-49f2afbdcd80\") "
Sep 11 13:57:43.380825 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:57:43.380721    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f231a7a-dad4-4a47-9727-49f2afbdcd80-maas-api-tls\") pod \"1f231a7a-dad4-4a47-9727-49f2afbdcd80\" (UID: \"1f231a7a-dad4-4a47-9727-49f2afbdcd80\") "
Sep 11 13:57:43.381331 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:57:43.381198    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f231a7a-dad4-4a47-9727-49f2afbdcd80-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1f231a7a-dad4-4a47-9727-49f2afbdcd80" (UID: "1f231a7a-dad4-4a47-9727-49f2afbdcd80"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 13:57:43.383890 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:57:43.383823    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f231a7a-dad4-4a47-9727-49f2afbdcd80-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1f231a7a-dad4-4a47-9727-49f2afbdcd80" (UID: "1f231a7a-dad4-4a47-9727-49f2afbdcd80"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 13:57:43.383890 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:57:43.383878    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f231a7a-dad4-4a47-9727-49f2afbdcd80-kube-api-access-7w5rg" (OuterVolumeSpecName: "kube-api-access-7w5rg") pod "1f231a7a-dad4-4a47-9727-49f2afbdcd80" (UID: "1f231a7a-dad4-4a47-9727-49f2afbdcd80"). InnerVolumeSpecName "kube-api-access-7w5rg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 13:57:43.384066 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:57:43.383983    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f231a7a-dad4-4a47-9727-49f2afbdcd80-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "1f231a7a-dad4-4a47-9727-49f2afbdcd80" (UID: "1f231a7a-dad4-4a47-9727-49f2afbdcd80"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 13:57:43.482681 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:57:43.482627    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f231a7a-dad4-4a47-9727-49f2afbdcd80-metrics-certs\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:57:43.482681 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:57:43.482672    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f231a7a-dad4-4a47-9727-49f2afbdcd80-openshift-service-ca\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:57:43.482681 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:57:43.482683    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7w5rg\" (UniqueName: \"kubernetes.io/projected/1f231a7a-dad4-4a47-9727-49f2afbdcd80-kube-api-access-7w5rg\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:57:43.482681 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:57:43.482694    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f231a7a-dad4-4a47-9727-49f2afbdcd80-maas-api-tls\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:57:43.774460 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:57:43.774418    2578 generic.go:356] "Generic (PLEG): container finished" podID="1f231a7a-dad4-4a47-9727-49f2afbdcd80" containerID="ebd82d58b339db4f12329860296243deeaa2adcdb9538c70e1dfd7a6b36d1b0d" exitCode=0
Sep 11 13:57:43.774705 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:57:43.774472    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-7b7819-7cc47bb6fb-gtss2"
Sep 11 13:57:43.774705 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:57:43.774472    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-7b7819-7cc47bb6fb-gtss2" event={"ID":"1f231a7a-dad4-4a47-9727-49f2afbdcd80","Type":"ContainerDied","Data":"ebd82d58b339db4f12329860296243deeaa2adcdb9538c70e1dfd7a6b36d1b0d"}
Sep 11 13:57:43.774705 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:57:43.774606    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-7b7819-7cc47bb6fb-gtss2" event={"ID":"1f231a7a-dad4-4a47-9727-49f2afbdcd80","Type":"ContainerDied","Data":"1382777395c9c278a5ef67ed76baf074f4dbc4ef868391011972c1f4f5e2e131"}
Sep 11 13:57:43.774705 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:57:43.774643    2578 scope.go:117] "RemoveContainer" containerID="ebd82d58b339db4f12329860296243deeaa2adcdb9538c70e1dfd7a6b36d1b0d"
Sep 11 13:57:43.788559 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:57:43.788527    2578 scope.go:117] "RemoveContainer" containerID="ebd82d58b339db4f12329860296243deeaa2adcdb9538c70e1dfd7a6b36d1b0d"
Sep 11 13:57:43.789155 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:57:43.789115    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebd82d58b339db4f12329860296243deeaa2adcdb9538c70e1dfd7a6b36d1b0d\": container with ID starting with ebd82d58b339db4f12329860296243deeaa2adcdb9538c70e1dfd7a6b36d1b0d not found: ID does not exist" containerID="ebd82d58b339db4f12329860296243deeaa2adcdb9538c70e1dfd7a6b36d1b0d"
Sep 11 13:57:43.789281 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:57:43.789170    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebd82d58b339db4f12329860296243deeaa2adcdb9538c70e1dfd7a6b36d1b0d"} err="failed to get container status \"ebd82d58b339db4f12329860296243deeaa2adcdb9538c70e1dfd7a6b36d1b0d\": rpc error: code = NotFound desc = could not find container \"ebd82d58b339db4f12329860296243deeaa2adcdb9538c70e1dfd7a6b36d1b0d\": container with ID starting with ebd82d58b339db4f12329860296243deeaa2adcdb9538c70e1dfd7a6b36d1b0d not found: ID does not exist"
Sep 11 13:57:43.803683 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:57:43.803639    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-7b7819-7cc47bb6fb-gtss2"]
Sep 11 13:57:43.811109 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:57:43.811074    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-7b7819-7cc47bb6fb-gtss2"]
Sep 11 13:57:45.107363 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:57:45.107322    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1f231a7a-dad4-4a47-9727-49f2afbdcd80" path="/var/lib/kubelet/pods/1f231a7a-dad4-4a47-9727-49f2afbdcd80/volumes"
Sep 11 13:58:09.900167 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:58:09.900133    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-264b08-5767fc97-4pv4s"]
Sep 11 13:58:09.900619 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:58:09.900379    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-264b08-5767fc97-4pv4s" podUID="9204f93d-4753-4663-aed0-fba9565a8df9" containerName="maas-api" containerID="cri-o://da3fa71cd700c61fcf24818a72bccef0f8b1d1dea1ef6e1ba0dc44221998affb" gracePeriod=30
Sep 11 13:58:10.157860 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:58:10.157749    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-264b08-5767fc97-4pv4s"
Sep 11 13:58:10.284948 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:58:10.284913    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9204f93d-4753-4663-aed0-fba9565a8df9-metrics-certs\") pod \"9204f93d-4753-4663-aed0-fba9565a8df9\" (UID: \"9204f93d-4753-4663-aed0-fba9565a8df9\") "
Sep 11 13:58:10.285134 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:58:10.285047    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9204f93d-4753-4663-aed0-fba9565a8df9-maas-api-tls\") pod \"9204f93d-4753-4663-aed0-fba9565a8df9\" (UID: \"9204f93d-4753-4663-aed0-fba9565a8df9\") "
Sep 11 13:58:10.285134 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:58:10.285086    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9204f93d-4753-4663-aed0-fba9565a8df9-openshift-service-ca\") pod \"9204f93d-4753-4663-aed0-fba9565a8df9\" (UID: \"9204f93d-4753-4663-aed0-fba9565a8df9\") "
Sep 11 13:58:10.285134 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:58:10.285118    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-57wlz\" (UniqueName: \"kubernetes.io/projected/9204f93d-4753-4663-aed0-fba9565a8df9-kube-api-access-57wlz\") pod \"9204f93d-4753-4663-aed0-fba9565a8df9\" (UID: \"9204f93d-4753-4663-aed0-fba9565a8df9\") "
Sep 11 13:58:10.285498 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:58:10.285453    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9204f93d-4753-4663-aed0-fba9565a8df9-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9204f93d-4753-4663-aed0-fba9565a8df9" (UID: "9204f93d-4753-4663-aed0-fba9565a8df9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 13:58:10.289711 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:58:10.289639    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9204f93d-4753-4663-aed0-fba9565a8df9-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "9204f93d-4753-4663-aed0-fba9565a8df9" (UID: "9204f93d-4753-4663-aed0-fba9565a8df9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 13:58:10.292368 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:58:10.292337    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9204f93d-4753-4663-aed0-fba9565a8df9-kube-api-access-57wlz" (OuterVolumeSpecName: "kube-api-access-57wlz") pod "9204f93d-4753-4663-aed0-fba9565a8df9" (UID: "9204f93d-4753-4663-aed0-fba9565a8df9"). InnerVolumeSpecName "kube-api-access-57wlz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 13:58:10.297867 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:58:10.297786    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9204f93d-4753-4663-aed0-fba9565a8df9-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9204f93d-4753-4663-aed0-fba9565a8df9" (UID: "9204f93d-4753-4663-aed0-fba9565a8df9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 13:58:10.387170 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:58:10.387131    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9204f93d-4753-4663-aed0-fba9565a8df9-metrics-certs\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:58:10.387170 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:58:10.387165    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9204f93d-4753-4663-aed0-fba9565a8df9-maas-api-tls\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:58:10.387170 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:58:10.387178    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9204f93d-4753-4663-aed0-fba9565a8df9-openshift-service-ca\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:58:10.387422 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:58:10.387189    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-57wlz\" (UniqueName: \"kubernetes.io/projected/9204f93d-4753-4663-aed0-fba9565a8df9-kube-api-access-57wlz\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:58:10.952313 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:58:10.952280    2578 generic.go:356] "Generic (PLEG): container finished" podID="9204f93d-4753-4663-aed0-fba9565a8df9" containerID="da3fa71cd700c61fcf24818a72bccef0f8b1d1dea1ef6e1ba0dc44221998affb" exitCode=0
Sep 11 13:58:10.952756 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:58:10.952334    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-264b08-5767fc97-4pv4s"
Sep 11 13:58:10.952756 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:58:10.952362    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-264b08-5767fc97-4pv4s" event={"ID":"9204f93d-4753-4663-aed0-fba9565a8df9","Type":"ContainerDied","Data":"da3fa71cd700c61fcf24818a72bccef0f8b1d1dea1ef6e1ba0dc44221998affb"}
Sep 11 13:58:10.952756 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:58:10.952423    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-264b08-5767fc97-4pv4s" event={"ID":"9204f93d-4753-4663-aed0-fba9565a8df9","Type":"ContainerDied","Data":"d62a0fd37d271c10326ab65d5a8fbcefbb6e828295ca24b59ae02e74f0fcf4e4"}
Sep 11 13:58:10.952756 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:58:10.952447    2578 scope.go:117] "RemoveContainer" containerID="da3fa71cd700c61fcf24818a72bccef0f8b1d1dea1ef6e1ba0dc44221998affb"
Sep 11 13:58:10.965864 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:58:10.964023    2578 scope.go:117] "RemoveContainer" containerID="da3fa71cd700c61fcf24818a72bccef0f8b1d1dea1ef6e1ba0dc44221998affb"
Sep 11 13:58:10.966307 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:58:10.966285    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da3fa71cd700c61fcf24818a72bccef0f8b1d1dea1ef6e1ba0dc44221998affb\": container with ID starting with da3fa71cd700c61fcf24818a72bccef0f8b1d1dea1ef6e1ba0dc44221998affb not found: ID does not exist" containerID="da3fa71cd700c61fcf24818a72bccef0f8b1d1dea1ef6e1ba0dc44221998affb"
Sep 11 13:58:10.966382 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:58:10.966318    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da3fa71cd700c61fcf24818a72bccef0f8b1d1dea1ef6e1ba0dc44221998affb"} err="failed to get container status \"da3fa71cd700c61fcf24818a72bccef0f8b1d1dea1ef6e1ba0dc44221998affb\": rpc error: code = NotFound desc = could not find container \"da3fa71cd700c61fcf24818a72bccef0f8b1d1dea1ef6e1ba0dc44221998affb\": container with ID starting with da3fa71cd700c61fcf24818a72bccef0f8b1d1dea1ef6e1ba0dc44221998affb not found: ID does not exist"
Sep 11 13:58:10.979501 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:58:10.979471    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-264b08-5767fc97-4pv4s"]
Sep 11 13:58:10.986855 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:58:10.986772    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-264b08-5767fc97-4pv4s"]
Sep 11 13:58:11.108180 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:58:11.108149    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9204f93d-4753-4663-aed0-fba9565a8df9" path="/var/lib/kubelet/pods/9204f93d-4753-4663-aed0-fba9565a8df9/volumes"
Sep 11 13:58:55.456862 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:58:55.456759    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-264b08-openshift-default-5f4c68fb78-hw2m9"]
Sep 11 13:58:55.457354 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:58:55.457043    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-264b08-openshift-default-5f4c68fb78-hw2m9" podUID="42a374b5-a779-416f-9f2c-bb1cf1d6f405" containerName="istio-proxy" containerID="cri-o://12b086021f1e495fbbd9625109a6e29ba52f5326419b1e90589f214e6e2a1c09" gracePeriod=30
Sep 11 13:58:56.585586 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:58:56.585547    2578 patch_prober.go:28] interesting pod/e2e-ipp-a-264b08-openshift-default-5f4c68fb78-hw2m9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 11 13:58:56.586024 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:58:56.585610    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-264b08-openshift-default-5f4c68fb78-hw2m9" podUID="42a374b5-a779-416f-9f2c-bb1cf1d6f405" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 13:59:00.731347 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:59:00.731279    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-264b08-openshift-default-5f4c68fb78-hw2m9"
Sep 11 13:59:00.781458 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:59:00.781423    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42a374b5-a779-416f-9f2c-bb1cf1d6f405-istio-podinfo\") pod \"42a374b5-a779-416f-9f2c-bb1cf1d6f405\" (UID: \"42a374b5-a779-416f-9f2c-bb1cf1d6f405\") "
Sep 11 13:59:00.781458 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:59:00.781461    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42a374b5-a779-416f-9f2c-bb1cf1d6f405-istio-envoy\") pod \"42a374b5-a779-416f-9f2c-bb1cf1d6f405\" (UID: \"42a374b5-a779-416f-9f2c-bb1cf1d6f405\") "
Sep 11 13:59:00.781458 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:59:00.781477    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42a374b5-a779-416f-9f2c-bb1cf1d6f405-istio-token\") pod \"42a374b5-a779-416f-9f2c-bb1cf1d6f405\" (UID: \"42a374b5-a779-416f-9f2c-bb1cf1d6f405\") "
Sep 11 13:59:00.781792 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:59:00.781518    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42a374b5-a779-416f-9f2c-bb1cf1d6f405-istio-data\") pod \"42a374b5-a779-416f-9f2c-bb1cf1d6f405\" (UID: \"42a374b5-a779-416f-9f2c-bb1cf1d6f405\") "
Sep 11 13:59:00.781792 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:59:00.781631    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42a374b5-a779-416f-9f2c-bb1cf1d6f405-workload-certs\") pod \"42a374b5-a779-416f-9f2c-bb1cf1d6f405\" (UID: \"42a374b5-a779-416f-9f2c-bb1cf1d6f405\") "
Sep 11 13:59:00.781792 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:59:00.781699    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42a374b5-a779-416f-9f2c-bb1cf1d6f405-credential-socket\") pod \"42a374b5-a779-416f-9f2c-bb1cf1d6f405\" (UID: \"42a374b5-a779-416f-9f2c-bb1cf1d6f405\") "
Sep 11 13:59:00.781792 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:59:00.781759    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42a374b5-a779-416f-9f2c-bb1cf1d6f405-istiod-ca-cert\") pod \"42a374b5-a779-416f-9f2c-bb1cf1d6f405\" (UID: \"42a374b5-a779-416f-9f2c-bb1cf1d6f405\") "
Sep 11 13:59:00.782109 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:59:00.781802    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42a374b5-a779-416f-9f2c-bb1cf1d6f405-istio-data" (OuterVolumeSpecName: "istio-data") pod "42a374b5-a779-416f-9f2c-bb1cf1d6f405" (UID: "42a374b5-a779-416f-9f2c-bb1cf1d6f405"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 13:59:00.782109 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:59:00.781879    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42a374b5-a779-416f-9f2c-bb1cf1d6f405-workload-socket\") pod \"42a374b5-a779-416f-9f2c-bb1cf1d6f405\" (UID: \"42a374b5-a779-416f-9f2c-bb1cf1d6f405\") "
Sep 11 13:59:00.782109 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:59:00.781910    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rxkm2\" (UniqueName: \"kubernetes.io/projected/42a374b5-a779-416f-9f2c-bb1cf1d6f405-kube-api-access-rxkm2\") pod \"42a374b5-a779-416f-9f2c-bb1cf1d6f405\" (UID: \"42a374b5-a779-416f-9f2c-bb1cf1d6f405\") "
Sep 11 13:59:00.782109 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:59:00.782052    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42a374b5-a779-416f-9f2c-bb1cf1d6f405-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "42a374b5-a779-416f-9f2c-bb1cf1d6f405" (UID: "42a374b5-a779-416f-9f2c-bb1cf1d6f405"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 13:59:00.782348 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:59:00.782164    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42a374b5-a779-416f-9f2c-bb1cf1d6f405-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "42a374b5-a779-416f-9f2c-bb1cf1d6f405" (UID: "42a374b5-a779-416f-9f2c-bb1cf1d6f405"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 13:59:00.782348 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:59:00.782269    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42a374b5-a779-416f-9f2c-bb1cf1d6f405-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "42a374b5-a779-416f-9f2c-bb1cf1d6f405" (UID: "42a374b5-a779-416f-9f2c-bb1cf1d6f405"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 13:59:00.782459 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:59:00.782406    2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42a374b5-a779-416f-9f2c-bb1cf1d6f405-workload-socket\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:59:00.782459 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:59:00.782425    2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42a374b5-a779-416f-9f2c-bb1cf1d6f405-istio-data\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:59:00.782459 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:59:00.782438    2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42a374b5-a779-416f-9f2c-bb1cf1d6f405-workload-certs\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:59:00.782459 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:59:00.782451    2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42a374b5-a779-416f-9f2c-bb1cf1d6f405-istiod-ca-cert\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:59:00.782667 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:59:00.782629    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42a374b5-a779-416f-9f2c-bb1cf1d6f405-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "42a374b5-a779-416f-9f2c-bb1cf1d6f405" (UID: "42a374b5-a779-416f-9f2c-bb1cf1d6f405"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 13:59:00.784084 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:59:00.784056    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/42a374b5-a779-416f-9f2c-bb1cf1d6f405-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "42a374b5-a779-416f-9f2c-bb1cf1d6f405" (UID: "42a374b5-a779-416f-9f2c-bb1cf1d6f405"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 13:59:00.784549 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:59:00.784527    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42a374b5-a779-416f-9f2c-bb1cf1d6f405-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "42a374b5-a779-416f-9f2c-bb1cf1d6f405" (UID: "42a374b5-a779-416f-9f2c-bb1cf1d6f405"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 13:59:00.784685 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:59:00.784669    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42a374b5-a779-416f-9f2c-bb1cf1d6f405-istio-token" (OuterVolumeSpecName: "istio-token") pod "42a374b5-a779-416f-9f2c-bb1cf1d6f405" (UID: "42a374b5-a779-416f-9f2c-bb1cf1d6f405"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 13:59:00.784967 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:59:00.784952    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42a374b5-a779-416f-9f2c-bb1cf1d6f405-kube-api-access-rxkm2" (OuterVolumeSpecName: "kube-api-access-rxkm2") pod "42a374b5-a779-416f-9f2c-bb1cf1d6f405" (UID: "42a374b5-a779-416f-9f2c-bb1cf1d6f405"). InnerVolumeSpecName "kube-api-access-rxkm2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 13:59:00.883265 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:59:00.883229    2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42a374b5-a779-416f-9f2c-bb1cf1d6f405-credential-socket\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:59:00.883265 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:59:00.883256    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxkm2\" (UniqueName: \"kubernetes.io/projected/42a374b5-a779-416f-9f2c-bb1cf1d6f405-kube-api-access-rxkm2\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:59:00.883265 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:59:00.883265    2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42a374b5-a779-416f-9f2c-bb1cf1d6f405-istio-podinfo\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:59:00.883265 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:59:00.883276    2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42a374b5-a779-416f-9f2c-bb1cf1d6f405-istio-envoy\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:59:00.883535 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:59:00.883284    2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42a374b5-a779-416f-9f2c-bb1cf1d6f405-istio-token\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:59:01.230558 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:59:01.230525    2578 generic.go:356] "Generic (PLEG): container finished" podID="42a374b5-a779-416f-9f2c-bb1cf1d6f405" containerID="12b086021f1e495fbbd9625109a6e29ba52f5326419b1e90589f214e6e2a1c09" exitCode=0
Sep 11 13:59:01.230742 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:59:01.230577    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-264b08-openshift-default-5f4c68fb78-hw2m9"
Sep 11 13:59:01.230742 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:59:01.230615    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-264b08-openshift-default-5f4c68fb78-hw2m9" event={"ID":"42a374b5-a779-416f-9f2c-bb1cf1d6f405","Type":"ContainerDied","Data":"12b086021f1e495fbbd9625109a6e29ba52f5326419b1e90589f214e6e2a1c09"}
Sep 11 13:59:01.230742 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:59:01.230673    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-264b08-openshift-default-5f4c68fb78-hw2m9" event={"ID":"42a374b5-a779-416f-9f2c-bb1cf1d6f405","Type":"ContainerDied","Data":"5bcbec075a8bd493f642561e7ca05d02d2d9ee37ee0dcbd29306a58c56b25f1e"}
Sep 11 13:59:01.230742 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:59:01.230704    2578 scope.go:117] "RemoveContainer" containerID="12b086021f1e495fbbd9625109a6e29ba52f5326419b1e90589f214e6e2a1c09"
Sep 11 13:59:01.240598 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:59:01.240579    2578 scope.go:117] "RemoveContainer" containerID="a213fffc77d16b9c7535d58c1a471c37e31c00a18a25be53bf3e4114f99d31e7"
Sep 11 13:59:01.249026 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:59:01.249008    2578 scope.go:117] "RemoveContainer" containerID="12b086021f1e495fbbd9625109a6e29ba52f5326419b1e90589f214e6e2a1c09"
Sep 11 13:59:01.249279 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:59:01.249259    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12b086021f1e495fbbd9625109a6e29ba52f5326419b1e90589f214e6e2a1c09\": container with ID starting with 12b086021f1e495fbbd9625109a6e29ba52f5326419b1e90589f214e6e2a1c09 not found: ID does not exist" containerID="12b086021f1e495fbbd9625109a6e29ba52f5326419b1e90589f214e6e2a1c09"
Sep 11 13:59:01.249324 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:59:01.249290    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12b086021f1e495fbbd9625109a6e29ba52f5326419b1e90589f214e6e2a1c09"} err="failed to get container status \"12b086021f1e495fbbd9625109a6e29ba52f5326419b1e90589f214e6e2a1c09\": rpc error: code = NotFound desc = could not find container \"12b086021f1e495fbbd9625109a6e29ba52f5326419b1e90589f214e6e2a1c09\": container with ID starting with 12b086021f1e495fbbd9625109a6e29ba52f5326419b1e90589f214e6e2a1c09 not found: ID does not exist"
Sep 11 13:59:01.249324 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:59:01.249307    2578 scope.go:117] "RemoveContainer" containerID="a213fffc77d16b9c7535d58c1a471c37e31c00a18a25be53bf3e4114f99d31e7"
Sep 11 13:59:01.249557 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:59:01.249539    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a213fffc77d16b9c7535d58c1a471c37e31c00a18a25be53bf3e4114f99d31e7\": container with ID starting with a213fffc77d16b9c7535d58c1a471c37e31c00a18a25be53bf3e4114f99d31e7 not found: ID does not exist" containerID="a213fffc77d16b9c7535d58c1a471c37e31c00a18a25be53bf3e4114f99d31e7"
Sep 11 13:59:01.249603 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:59:01.249568    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a213fffc77d16b9c7535d58c1a471c37e31c00a18a25be53bf3e4114f99d31e7"} err="failed to get container status \"a213fffc77d16b9c7535d58c1a471c37e31c00a18a25be53bf3e4114f99d31e7\": rpc error: code = NotFound desc = could not find container \"a213fffc77d16b9c7535d58c1a471c37e31c00a18a25be53bf3e4114f99d31e7\": container with ID starting with a213fffc77d16b9c7535d58c1a471c37e31c00a18a25be53bf3e4114f99d31e7 not found: ID does not exist"
Sep 11 13:59:01.265379 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:59:01.265356    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-264b08-openshift-default-5f4c68fb78-hw2m9"]
Sep 11 13:59:01.268423 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:59:01.268405    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-264b08-openshift-default-5f4c68fb78-hw2m9"]
Sep 11 13:59:03.107781 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:59:03.107744    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="42a374b5-a779-416f-9f2c-bb1cf1d6f405" path="/var/lib/kubelet/pods/42a374b5-a779-416f-9f2c-bb1cf1d6f405/volumes"
Sep 11 13:59:06.730733 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:59:06.730696    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a403bc-597d56745f-nfn6d"]
Sep 11 13:59:06.731316 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:59:06.730939    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a403bc-597d56745f-nfn6d" podUID="88ee4ae2-29d7-4611-ada9-efa119d09d1d" containerName="maas-api" containerID="cri-o://e587dea977c526ed902ffcb81e5361d89a13697988efb6660ce81feb090a7e15" gracePeriod=30
Sep 11 13:59:07.009081 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:59:07.009054    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a403bc-597d56745f-nfn6d"
Sep 11 13:59:07.145003 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:59:07.144977    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/88ee4ae2-29d7-4611-ada9-efa119d09d1d-openshift-service-ca\") pod \"88ee4ae2-29d7-4611-ada9-efa119d09d1d\" (UID: \"88ee4ae2-29d7-4611-ada9-efa119d09d1d\") "
Sep 11 13:59:07.145189 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:59:07.145045    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/88ee4ae2-29d7-4611-ada9-efa119d09d1d-maas-api-tls\") pod \"88ee4ae2-29d7-4611-ada9-efa119d09d1d\" (UID: \"88ee4ae2-29d7-4611-ada9-efa119d09d1d\") "
Sep 11 13:59:07.145189 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:59:07.145069    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9zx77\" (UniqueName: \"kubernetes.io/projected/88ee4ae2-29d7-4611-ada9-efa119d09d1d-kube-api-access-9zx77\") pod \"88ee4ae2-29d7-4611-ada9-efa119d09d1d\" (UID: \"88ee4ae2-29d7-4611-ada9-efa119d09d1d\") "
Sep 11 13:59:07.145189 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:59:07.145116    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/88ee4ae2-29d7-4611-ada9-efa119d09d1d-metrics-certs\") pod \"88ee4ae2-29d7-4611-ada9-efa119d09d1d\" (UID: \"88ee4ae2-29d7-4611-ada9-efa119d09d1d\") "
Sep 11 13:59:07.145358 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:59:07.145329    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88ee4ae2-29d7-4611-ada9-efa119d09d1d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "88ee4ae2-29d7-4611-ada9-efa119d09d1d" (UID: "88ee4ae2-29d7-4611-ada9-efa119d09d1d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 13:59:07.145950 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:59:07.145545    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/88ee4ae2-29d7-4611-ada9-efa119d09d1d-openshift-service-ca\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:59:07.147344 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:59:07.147316    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88ee4ae2-29d7-4611-ada9-efa119d09d1d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "88ee4ae2-29d7-4611-ada9-efa119d09d1d" (UID: "88ee4ae2-29d7-4611-ada9-efa119d09d1d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 13:59:07.147344 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:59:07.147333    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88ee4ae2-29d7-4611-ada9-efa119d09d1d-kube-api-access-9zx77" (OuterVolumeSpecName: "kube-api-access-9zx77") pod "88ee4ae2-29d7-4611-ada9-efa119d09d1d" (UID: "88ee4ae2-29d7-4611-ada9-efa119d09d1d"). InnerVolumeSpecName "kube-api-access-9zx77". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 13:59:07.147512 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:59:07.147348    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88ee4ae2-29d7-4611-ada9-efa119d09d1d-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "88ee4ae2-29d7-4611-ada9-efa119d09d1d" (UID: "88ee4ae2-29d7-4611-ada9-efa119d09d1d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 13:59:07.246736 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:59:07.246652    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/88ee4ae2-29d7-4611-ada9-efa119d09d1d-maas-api-tls\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:59:07.246736 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:59:07.246682    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9zx77\" (UniqueName: \"kubernetes.io/projected/88ee4ae2-29d7-4611-ada9-efa119d09d1d-kube-api-access-9zx77\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:59:07.246736 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:59:07.246692    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/88ee4ae2-29d7-4611-ada9-efa119d09d1d-metrics-certs\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:59:07.265790 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:59:07.265757    2578 generic.go:356] "Generic (PLEG): container finished" podID="88ee4ae2-29d7-4611-ada9-efa119d09d1d" containerID="e587dea977c526ed902ffcb81e5361d89a13697988efb6660ce81feb090a7e15" exitCode=0
Sep 11 13:59:07.265978 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:59:07.265790    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a403bc-597d56745f-nfn6d" event={"ID":"88ee4ae2-29d7-4611-ada9-efa119d09d1d","Type":"ContainerDied","Data":"e587dea977c526ed902ffcb81e5361d89a13697988efb6660ce81feb090a7e15"}
Sep 11 13:59:07.265978 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:59:07.265811    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a403bc-597d56745f-nfn6d"
Sep 11 13:59:07.265978 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:59:07.265890    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a403bc-597d56745f-nfn6d" event={"ID":"88ee4ae2-29d7-4611-ada9-efa119d09d1d","Type":"ContainerDied","Data":"8102a83272ebdac0876805218d3fc2de10e609e3f82ce8c7460505ae49c7af23"}
Sep 11 13:59:07.265978 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:59:07.265924    2578 scope.go:117] "RemoveContainer" containerID="e587dea977c526ed902ffcb81e5361d89a13697988efb6660ce81feb090a7e15"
Sep 11 13:59:07.280798 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:59:07.280776    2578 scope.go:117] "RemoveContainer" containerID="e587dea977c526ed902ffcb81e5361d89a13697988efb6660ce81feb090a7e15"
Sep 11 13:59:07.281638 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:59:07.281618    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e587dea977c526ed902ffcb81e5361d89a13697988efb6660ce81feb090a7e15\": container with ID starting with e587dea977c526ed902ffcb81e5361d89a13697988efb6660ce81feb090a7e15 not found: ID does not exist" containerID="e587dea977c526ed902ffcb81e5361d89a13697988efb6660ce81feb090a7e15"
Sep 11 13:59:07.281711 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:59:07.281652    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e587dea977c526ed902ffcb81e5361d89a13697988efb6660ce81feb090a7e15"} err="failed to get container status \"e587dea977c526ed902ffcb81e5361d89a13697988efb6660ce81feb090a7e15\": rpc error: code = NotFound desc = could not find container \"e587dea977c526ed902ffcb81e5361d89a13697988efb6660ce81feb090a7e15\": container with ID starting with e587dea977c526ed902ffcb81e5361d89a13697988efb6660ce81feb090a7e15 not found: ID does not exist"
Sep 11 13:59:07.302317 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:59:07.302291    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a403bc-597d56745f-nfn6d"]
Sep 11 13:59:07.310728 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:59:07.310706    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a403bc-597d56745f-nfn6d"]
Sep 11 13:59:09.108808 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:59:09.108767    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="88ee4ae2-29d7-4611-ada9-efa119d09d1d" path="/var/lib/kubelet/pods/88ee4ae2-29d7-4611-ada9-efa119d09d1d/volumes"
Sep 11 13:59:25.808131 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:59:25.808096    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0317c3-5544cf86cf-ps7g5"]
Sep 11 13:59:25.808603 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:59:25.808379    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0317c3-5544cf86cf-ps7g5" podUID="99c8676f-2468-4393-ac09-bd0ebe7e185b" containerName="maas-api" containerID="cri-o://9b996073196cb74bd5f8f0b6825ce43d8b868f242edfcb9ad96af5f4bd59cd8b" gracePeriod=30
Sep 11 13:59:26.066434 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:59:26.066369    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0317c3-5544cf86cf-ps7g5"
Sep 11 13:59:26.235723 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:59:26.235684    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99c8676f-2468-4393-ac09-bd0ebe7e185b-metrics-certs\") pod \"99c8676f-2468-4393-ac09-bd0ebe7e185b\" (UID: \"99c8676f-2468-4393-ac09-bd0ebe7e185b\") "
Sep 11 13:59:26.235960 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:59:26.235860    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p29rl\" (UniqueName: \"kubernetes.io/projected/99c8676f-2468-4393-ac09-bd0ebe7e185b-kube-api-access-p29rl\") pod \"99c8676f-2468-4393-ac09-bd0ebe7e185b\" (UID: \"99c8676f-2468-4393-ac09-bd0ebe7e185b\") "
Sep 11 13:59:26.235960 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:59:26.235949    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/99c8676f-2468-4393-ac09-bd0ebe7e185b-maas-api-tls\") pod \"99c8676f-2468-4393-ac09-bd0ebe7e185b\" (UID: \"99c8676f-2468-4393-ac09-bd0ebe7e185b\") "
Sep 11 13:59:26.236103 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:59:26.235990    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/99c8676f-2468-4393-ac09-bd0ebe7e185b-openshift-service-ca\") pod \"99c8676f-2468-4393-ac09-bd0ebe7e185b\" (UID: \"99c8676f-2468-4393-ac09-bd0ebe7e185b\") "
Sep 11 13:59:26.236477 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:59:26.236445    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99c8676f-2468-4393-ac09-bd0ebe7e185b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "99c8676f-2468-4393-ac09-bd0ebe7e185b" (UID: "99c8676f-2468-4393-ac09-bd0ebe7e185b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 13:59:26.238335 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:59:26.238298    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99c8676f-2468-4393-ac09-bd0ebe7e185b-kube-api-access-p29rl" (OuterVolumeSpecName: "kube-api-access-p29rl") pod "99c8676f-2468-4393-ac09-bd0ebe7e185b" (UID: "99c8676f-2468-4393-ac09-bd0ebe7e185b"). InnerVolumeSpecName "kube-api-access-p29rl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 13:59:26.238335 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:59:26.238324    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99c8676f-2468-4393-ac09-bd0ebe7e185b-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "99c8676f-2468-4393-ac09-bd0ebe7e185b" (UID: "99c8676f-2468-4393-ac09-bd0ebe7e185b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 13:59:26.238468 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:59:26.238411    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99c8676f-2468-4393-ac09-bd0ebe7e185b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "99c8676f-2468-4393-ac09-bd0ebe7e185b" (UID: "99c8676f-2468-4393-ac09-bd0ebe7e185b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 13:59:26.337382 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:59:26.337303    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99c8676f-2468-4393-ac09-bd0ebe7e185b-metrics-certs\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:59:26.337382 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:59:26.337330    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p29rl\" (UniqueName: \"kubernetes.io/projected/99c8676f-2468-4393-ac09-bd0ebe7e185b-kube-api-access-p29rl\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:59:26.337382 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:59:26.337339    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/99c8676f-2468-4393-ac09-bd0ebe7e185b-maas-api-tls\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:59:26.337382 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:59:26.337348    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/99c8676f-2468-4393-ac09-bd0ebe7e185b-openshift-service-ca\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:59:26.378695 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:59:26.378663    2578 generic.go:356] "Generic (PLEG): container finished" podID="99c8676f-2468-4393-ac09-bd0ebe7e185b" containerID="9b996073196cb74bd5f8f0b6825ce43d8b868f242edfcb9ad96af5f4bd59cd8b" exitCode=0
Sep 11 13:59:26.378883 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:59:26.378707    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0317c3-5544cf86cf-ps7g5"
Sep 11 13:59:26.378883 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:59:26.378745    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0317c3-5544cf86cf-ps7g5" event={"ID":"99c8676f-2468-4393-ac09-bd0ebe7e185b","Type":"ContainerDied","Data":"9b996073196cb74bd5f8f0b6825ce43d8b868f242edfcb9ad96af5f4bd59cd8b"}
Sep 11 13:59:26.378883 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:59:26.378803    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0317c3-5544cf86cf-ps7g5" event={"ID":"99c8676f-2468-4393-ac09-bd0ebe7e185b","Type":"ContainerDied","Data":"d80b7ccf6f4f1ddb7731f71f51f9167b8906824d331f5e6013861bc6c2b2e410"}
Sep 11 13:59:26.378883 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:59:26.378841    2578 scope.go:117] "RemoveContainer" containerID="9b996073196cb74bd5f8f0b6825ce43d8b868f242edfcb9ad96af5f4bd59cd8b"
Sep 11 13:59:26.389909 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:59:26.389866    2578 scope.go:117] "RemoveContainer" containerID="9b996073196cb74bd5f8f0b6825ce43d8b868f242edfcb9ad96af5f4bd59cd8b"
Sep 11 13:59:26.390148 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:59:26.390126    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b996073196cb74bd5f8f0b6825ce43d8b868f242edfcb9ad96af5f4bd59cd8b\": container with ID starting with 9b996073196cb74bd5f8f0b6825ce43d8b868f242edfcb9ad96af5f4bd59cd8b not found: ID does not exist" containerID="9b996073196cb74bd5f8f0b6825ce43d8b868f242edfcb9ad96af5f4bd59cd8b"
Sep 11 13:59:26.390249 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:59:26.390158    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b996073196cb74bd5f8f0b6825ce43d8b868f242edfcb9ad96af5f4bd59cd8b"} err="failed to get container status \"9b996073196cb74bd5f8f0b6825ce43d8b868f242edfcb9ad96af5f4bd59cd8b\": rpc error: code = NotFound desc = could not find container \"9b996073196cb74bd5f8f0b6825ce43d8b868f242edfcb9ad96af5f4bd59cd8b\": container with ID starting with 9b996073196cb74bd5f8f0b6825ce43d8b868f242edfcb9ad96af5f4bd59cd8b not found: ID does not exist"
Sep 11 13:59:26.404586 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:59:26.404562    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0317c3-5544cf86cf-ps7g5"]
Sep 11 13:59:26.406746 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:59:26.406727    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0317c3-5544cf86cf-ps7g5"]
Sep 11 13:59:27.107355 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:59:27.107321    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="99c8676f-2468-4393-ac09-bd0ebe7e185b" path="/var/lib/kubelet/pods/99c8676f-2468-4393-ac09-bd0ebe7e185b/volumes"
Sep 11 13:59:44.872855 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:59:44.872748    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-9edb53-69dbb74897-9w4g8"]
Sep 11 13:59:44.873474 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:59:44.873048    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-9edb53-69dbb74897-9w4g8" podUID="06530615-4817-4218-9591-7e2eb7dea6a0" containerName="maas-api" containerID="cri-o://c3caa65a0e473877bcd3247f6c4bf7eb3aa6fa4074c6a98878aeb3356c78badb" gracePeriod=30
Sep 11 13:59:45.132797 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:59:45.132735    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-9edb53-69dbb74897-9w4g8"
Sep 11 13:59:45.220973 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:59:45.220941    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06530615-4817-4218-9591-7e2eb7dea6a0-metrics-certs\") pod \"06530615-4817-4218-9591-7e2eb7dea6a0\" (UID: \"06530615-4817-4218-9591-7e2eb7dea6a0\") "
Sep 11 13:59:45.221161 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:59:45.221025    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-87fbl\" (UniqueName: \"kubernetes.io/projected/06530615-4817-4218-9591-7e2eb7dea6a0-kube-api-access-87fbl\") pod \"06530615-4817-4218-9591-7e2eb7dea6a0\" (UID: \"06530615-4817-4218-9591-7e2eb7dea6a0\") "
Sep 11 13:59:45.221161 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:59:45.221065    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06530615-4817-4218-9591-7e2eb7dea6a0-maas-api-tls\") pod \"06530615-4817-4218-9591-7e2eb7dea6a0\" (UID: \"06530615-4817-4218-9591-7e2eb7dea6a0\") "
Sep 11 13:59:45.221161 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:59:45.221092    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06530615-4817-4218-9591-7e2eb7dea6a0-openshift-service-ca\") pod \"06530615-4817-4218-9591-7e2eb7dea6a0\" (UID: \"06530615-4817-4218-9591-7e2eb7dea6a0\") "
Sep 11 13:59:45.221538 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:59:45.221509    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06530615-4817-4218-9591-7e2eb7dea6a0-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "06530615-4817-4218-9591-7e2eb7dea6a0" (UID: "06530615-4817-4218-9591-7e2eb7dea6a0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 13:59:45.223333 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:59:45.223306    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06530615-4817-4218-9591-7e2eb7dea6a0-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "06530615-4817-4218-9591-7e2eb7dea6a0" (UID: "06530615-4817-4218-9591-7e2eb7dea6a0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 13:59:45.223451 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:59:45.223342    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06530615-4817-4218-9591-7e2eb7dea6a0-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "06530615-4817-4218-9591-7e2eb7dea6a0" (UID: "06530615-4817-4218-9591-7e2eb7dea6a0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 13:59:45.223451 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:59:45.223382    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06530615-4817-4218-9591-7e2eb7dea6a0-kube-api-access-87fbl" (OuterVolumeSpecName: "kube-api-access-87fbl") pod "06530615-4817-4218-9591-7e2eb7dea6a0" (UID: "06530615-4817-4218-9591-7e2eb7dea6a0"). InnerVolumeSpecName "kube-api-access-87fbl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 13:59:45.322758 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:59:45.322728    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-87fbl\" (UniqueName: \"kubernetes.io/projected/06530615-4817-4218-9591-7e2eb7dea6a0-kube-api-access-87fbl\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:59:45.322758 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:59:45.322756    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06530615-4817-4218-9591-7e2eb7dea6a0-maas-api-tls\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:59:45.322758 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:59:45.322765    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06530615-4817-4218-9591-7e2eb7dea6a0-openshift-service-ca\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:59:45.323024 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:59:45.322773    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06530615-4817-4218-9591-7e2eb7dea6a0-metrics-certs\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:59:45.487292 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:59:45.487260    2578 generic.go:356] "Generic (PLEG): container finished" podID="06530615-4817-4218-9591-7e2eb7dea6a0" containerID="c3caa65a0e473877bcd3247f6c4bf7eb3aa6fa4074c6a98878aeb3356c78badb" exitCode=0
Sep 11 13:59:45.487496 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:59:45.487311    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-9edb53-69dbb74897-9w4g8"
Sep 11 13:59:45.487496 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:59:45.487438    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-9edb53-69dbb74897-9w4g8" event={"ID":"06530615-4817-4218-9591-7e2eb7dea6a0","Type":"ContainerDied","Data":"c3caa65a0e473877bcd3247f6c4bf7eb3aa6fa4074c6a98878aeb3356c78badb"}
Sep 11 13:59:45.487496 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:59:45.487488    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-9edb53-69dbb74897-9w4g8" event={"ID":"06530615-4817-4218-9591-7e2eb7dea6a0","Type":"ContainerDied","Data":"d6c565feb7cd2cf285905df1f08e3f12f4625021d49e50a513cfc85354ab392f"}
Sep 11 13:59:45.487668 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:59:45.487533    2578 scope.go:117] "RemoveContainer" containerID="c3caa65a0e473877bcd3247f6c4bf7eb3aa6fa4074c6a98878aeb3356c78badb"
Sep 11 13:59:45.499966 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:59:45.499934    2578 scope.go:117] "RemoveContainer" containerID="c3caa65a0e473877bcd3247f6c4bf7eb3aa6fa4074c6a98878aeb3356c78badb"
Sep 11 13:59:45.500306 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:59:45.500279    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3caa65a0e473877bcd3247f6c4bf7eb3aa6fa4074c6a98878aeb3356c78badb\": container with ID starting with c3caa65a0e473877bcd3247f6c4bf7eb3aa6fa4074c6a98878aeb3356c78badb not found: ID does not exist" containerID="c3caa65a0e473877bcd3247f6c4bf7eb3aa6fa4074c6a98878aeb3356c78badb"
Sep 11 13:59:45.500394 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:59:45.500319    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3caa65a0e473877bcd3247f6c4bf7eb3aa6fa4074c6a98878aeb3356c78badb"} err="failed to get container status \"c3caa65a0e473877bcd3247f6c4bf7eb3aa6fa4074c6a98878aeb3356c78badb\": rpc error: code = NotFound desc = could not find container \"c3caa65a0e473877bcd3247f6c4bf7eb3aa6fa4074c6a98878aeb3356c78badb\": container with ID starting with c3caa65a0e473877bcd3247f6c4bf7eb3aa6fa4074c6a98878aeb3356c78badb not found: ID does not exist"
Sep 11 13:59:45.515340 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:59:45.515318    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-9edb53-69dbb74897-9w4g8"]
Sep 11 13:59:45.523031 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:59:45.523011    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-9edb53-69dbb74897-9w4g8"]
Sep 11 13:59:47.106463 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:59:47.106427    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="06530615-4817-4218-9591-7e2eb7dea6a0" path="/var/lib/kubelet/pods/06530615-4817-4218-9591-7e2eb7dea6a0/volumes"
Sep 11 13:59:55.837761 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:59:55.837724    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6b447f7bf9-7cs9k"]
Sep 11 13:59:55.838289 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:59:55.837959    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-6b447f7bf9-7cs9k" podUID="babc7d58-1418-46b3-ace0-4341db9351ee" containerName="manager" containerID="cri-o://bb9eb29982700e680ff48ba65ba83420267edcc244437a847e429ec93b034979" gracePeriod=30
Sep 11 13:59:56.082617 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:59:56.082592    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6b447f7bf9-7cs9k"
Sep 11 13:59:56.229996 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:59:56.229967    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cz7lb\" (UniqueName: \"kubernetes.io/projected/babc7d58-1418-46b3-ace0-4341db9351ee-kube-api-access-cz7lb\") pod \"babc7d58-1418-46b3-ace0-4341db9351ee\" (UID: \"babc7d58-1418-46b3-ace0-4341db9351ee\") "
Sep 11 13:59:56.230176 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:59:56.230044    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/babc7d58-1418-46b3-ace0-4341db9351ee-metrics-certs\") pod \"babc7d58-1418-46b3-ace0-4341db9351ee\" (UID: \"babc7d58-1418-46b3-ace0-4341db9351ee\") "
Sep 11 13:59:56.230176 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:59:56.230080    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/babc7d58-1418-46b3-ace0-4341db9351ee-webhook-cert\") pod \"babc7d58-1418-46b3-ace0-4341db9351ee\" (UID: \"babc7d58-1418-46b3-ace0-4341db9351ee\") "
Sep 11 13:59:56.232285 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:59:56.232259    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/babc7d58-1418-46b3-ace0-4341db9351ee-kube-api-access-cz7lb" (OuterVolumeSpecName: "kube-api-access-cz7lb") pod "babc7d58-1418-46b3-ace0-4341db9351ee" (UID: "babc7d58-1418-46b3-ace0-4341db9351ee"). InnerVolumeSpecName "kube-api-access-cz7lb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 13:59:56.232784 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:59:56.232764    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/babc7d58-1418-46b3-ace0-4341db9351ee-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "babc7d58-1418-46b3-ace0-4341db9351ee" (UID: "babc7d58-1418-46b3-ace0-4341db9351ee"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 13:59:56.232919 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:59:56.232786    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/babc7d58-1418-46b3-ace0-4341db9351ee-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "babc7d58-1418-46b3-ace0-4341db9351ee" (UID: "babc7d58-1418-46b3-ace0-4341db9351ee"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 13:59:56.331455 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:59:56.331405    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cz7lb\" (UniqueName: \"kubernetes.io/projected/babc7d58-1418-46b3-ace0-4341db9351ee-kube-api-access-cz7lb\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:59:56.331455 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:59:56.331448    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/babc7d58-1418-46b3-ace0-4341db9351ee-metrics-certs\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:59:56.331455 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:59:56.331457    2578 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/babc7d58-1418-46b3-ace0-4341db9351ee-webhook-cert\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 13:59:56.559685 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:59:56.559600    2578 generic.go:356] "Generic (PLEG): container finished" podID="babc7d58-1418-46b3-ace0-4341db9351ee" containerID="bb9eb29982700e680ff48ba65ba83420267edcc244437a847e429ec93b034979" exitCode=0
Sep 11 13:59:56.559685 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:59:56.559654    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6b447f7bf9-7cs9k"
Sep 11 13:59:56.559884 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:59:56.559685    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6b447f7bf9-7cs9k" event={"ID":"babc7d58-1418-46b3-ace0-4341db9351ee","Type":"ContainerDied","Data":"bb9eb29982700e680ff48ba65ba83420267edcc244437a847e429ec93b034979"}
Sep 11 13:59:56.559884 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:59:56.559761    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6b447f7bf9-7cs9k" event={"ID":"babc7d58-1418-46b3-ace0-4341db9351ee","Type":"ContainerDied","Data":"64319b088b7822816db1604278e5d71314a074de25fd091afa4c3c4a76e45633"}
Sep 11 13:59:56.559884 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:59:56.559782    2578 scope.go:117] "RemoveContainer" containerID="bb9eb29982700e680ff48ba65ba83420267edcc244437a847e429ec93b034979"
Sep 11 13:59:56.570070 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:59:56.570052    2578 scope.go:117] "RemoveContainer" containerID="bb9eb29982700e680ff48ba65ba83420267edcc244437a847e429ec93b034979"
Sep 11 13:59:56.570348 ip-10-0-131-86 kubenswrapper[2578]: E0911 13:59:56.570330    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb9eb29982700e680ff48ba65ba83420267edcc244437a847e429ec93b034979\": container with ID starting with bb9eb29982700e680ff48ba65ba83420267edcc244437a847e429ec93b034979 not found: ID does not exist" containerID="bb9eb29982700e680ff48ba65ba83420267edcc244437a847e429ec93b034979"
Sep 11 13:59:56.570411 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:59:56.570357    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb9eb29982700e680ff48ba65ba83420267edcc244437a847e429ec93b034979"} err="failed to get container status \"bb9eb29982700e680ff48ba65ba83420267edcc244437a847e429ec93b034979\": rpc error: code = NotFound desc = could not find container \"bb9eb29982700e680ff48ba65ba83420267edcc244437a847e429ec93b034979\": container with ID starting with bb9eb29982700e680ff48ba65ba83420267edcc244437a847e429ec93b034979 not found: ID does not exist"
Sep 11 13:59:56.584533 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:59:56.584512    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6b447f7bf9-7cs9k"]
Sep 11 13:59:56.589521 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:59:56.589499    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-6b447f7bf9-7cs9k"]
Sep 11 13:59:57.107134 ip-10-0-131-86 kubenswrapper[2578]: I0911 13:59:57.107100    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="babc7d58-1418-46b3-ace0-4341db9351ee" path="/var/lib/kubelet/pods/babc7d58-1418-46b3-ace0-4341db9351ee/volumes"
Sep 11 14:00:57.509120 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:00:57.509086    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6b447f7bf9-hv7km"]
Sep 11 14:00:57.511661 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:00:57.509509    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42a374b5-a779-416f-9f2c-bb1cf1d6f405" containerName="istio-proxy"
Sep 11 14:00:57.511661 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:00:57.509519    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="42a374b5-a779-416f-9f2c-bb1cf1d6f405" containerName="istio-proxy"
Sep 11 14:00:57.511661 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:00:57.509532    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99c8676f-2468-4393-ac09-bd0ebe7e185b" containerName="maas-api"
Sep 11 14:00:57.511661 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:00:57.509537    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="99c8676f-2468-4393-ac09-bd0ebe7e185b" containerName="maas-api"
Sep 11 14:00:57.511661 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:00:57.509544    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88ee4ae2-29d7-4611-ada9-efa119d09d1d" containerName="maas-api"
Sep 11 14:00:57.511661 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:00:57.509550    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="88ee4ae2-29d7-4611-ada9-efa119d09d1d" containerName="maas-api"
Sep 11 14:00:57.511661 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:00:57.509560    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06530615-4817-4218-9591-7e2eb7dea6a0" containerName="maas-api"
Sep 11 14:00:57.511661 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:00:57.509565    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="06530615-4817-4218-9591-7e2eb7dea6a0" containerName="maas-api"
Sep 11 14:00:57.511661 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:00:57.509571    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42a374b5-a779-416f-9f2c-bb1cf1d6f405" containerName="istio-proxy"
Sep 11 14:00:57.511661 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:00:57.509576    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="42a374b5-a779-416f-9f2c-bb1cf1d6f405" containerName="istio-proxy"
Sep 11 14:00:57.511661 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:00:57.509590    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f231a7a-dad4-4a47-9727-49f2afbdcd80" containerName="maas-api"
Sep 11 14:00:57.511661 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:00:57.509595    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f231a7a-dad4-4a47-9727-49f2afbdcd80" containerName="maas-api"
Sep 11 14:00:57.511661 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:00:57.509610    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="babc7d58-1418-46b3-ace0-4341db9351ee" containerName="manager"
Sep 11 14:00:57.511661 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:00:57.509614    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="babc7d58-1418-46b3-ace0-4341db9351ee" containerName="manager"
Sep 11 14:00:57.511661 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:00:57.509619    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9204f93d-4753-4663-aed0-fba9565a8df9" containerName="maas-api"
Sep 11 14:00:57.511661 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:00:57.509623    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="9204f93d-4753-4663-aed0-fba9565a8df9" containerName="maas-api"
Sep 11 14:00:57.511661 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:00:57.509680    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="99c8676f-2468-4393-ac09-bd0ebe7e185b" containerName="maas-api"
Sep 11 14:00:57.511661 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:00:57.509691    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="babc7d58-1418-46b3-ace0-4341db9351ee" containerName="manager"
Sep 11 14:00:57.511661 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:00:57.509697    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f231a7a-dad4-4a47-9727-49f2afbdcd80" containerName="maas-api"
Sep 11 14:00:57.511661 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:00:57.509703    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="42a374b5-a779-416f-9f2c-bb1cf1d6f405" containerName="istio-proxy"
Sep 11 14:00:57.511661 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:00:57.509708    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="9204f93d-4753-4663-aed0-fba9565a8df9" containerName="maas-api"
Sep 11 14:00:57.511661 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:00:57.509713    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="42a374b5-a779-416f-9f2c-bb1cf1d6f405" containerName="istio-proxy"
Sep 11 14:00:57.511661 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:00:57.509718    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="88ee4ae2-29d7-4611-ada9-efa119d09d1d" containerName="maas-api"
Sep 11 14:00:57.511661 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:00:57.509723    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="06530615-4817-4218-9591-7e2eb7dea6a0" containerName="maas-api"
Sep 11 14:00:57.512740 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:00:57.512724    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6b447f7bf9-hv7km"
Sep 11 14:00:57.524911 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:00:57.524883    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 11 14:00:57.525040 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:00:57.524921    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 11 14:00:57.525040 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:00:57.524963    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-dpcsj\""
Sep 11 14:00:57.525040 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:00:57.524934    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 11 14:00:57.536767 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:00:57.536740    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6b447f7bf9-hv7km"]
Sep 11 14:00:57.615638 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:00:57.615608    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8602d572-e70e-4ee1-9f56-d7ebf8e0f0a1-metrics-certs\") pod \"maas-controller-6b447f7bf9-hv7km\" (UID: \"8602d572-e70e-4ee1-9f56-d7ebf8e0f0a1\") " pod="opendatahub/maas-controller-6b447f7bf9-hv7km"
Sep 11 14:00:57.615848 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:00:57.615680    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghndq\" (UniqueName: \"kubernetes.io/projected/8602d572-e70e-4ee1-9f56-d7ebf8e0f0a1-kube-api-access-ghndq\") pod \"maas-controller-6b447f7bf9-hv7km\" (UID: \"8602d572-e70e-4ee1-9f56-d7ebf8e0f0a1\") " pod="opendatahub/maas-controller-6b447f7bf9-hv7km"
Sep 11 14:00:57.615848 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:00:57.615699    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8602d572-e70e-4ee1-9f56-d7ebf8e0f0a1-webhook-cert\") pod \"maas-controller-6b447f7bf9-hv7km\" (UID: \"8602d572-e70e-4ee1-9f56-d7ebf8e0f0a1\") " pod="opendatahub/maas-controller-6b447f7bf9-hv7km"
Sep 11 14:00:57.716887 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:00:57.716845    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghndq\" (UniqueName: \"kubernetes.io/projected/8602d572-e70e-4ee1-9f56-d7ebf8e0f0a1-kube-api-access-ghndq\") pod \"maas-controller-6b447f7bf9-hv7km\" (UID: \"8602d572-e70e-4ee1-9f56-d7ebf8e0f0a1\") " pod="opendatahub/maas-controller-6b447f7bf9-hv7km"
Sep 11 14:00:57.717067 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:00:57.716897    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8602d572-e70e-4ee1-9f56-d7ebf8e0f0a1-webhook-cert\") pod \"maas-controller-6b447f7bf9-hv7km\" (UID: \"8602d572-e70e-4ee1-9f56-d7ebf8e0f0a1\") " pod="opendatahub/maas-controller-6b447f7bf9-hv7km"
Sep 11 14:00:57.717067 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:00:57.716986    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8602d572-e70e-4ee1-9f56-d7ebf8e0f0a1-metrics-certs\") pod \"maas-controller-6b447f7bf9-hv7km\" (UID: \"8602d572-e70e-4ee1-9f56-d7ebf8e0f0a1\") " pod="opendatahub/maas-controller-6b447f7bf9-hv7km"
Sep 11 14:00:57.719502 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:00:57.719469    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8602d572-e70e-4ee1-9f56-d7ebf8e0f0a1-metrics-certs\") pod \"maas-controller-6b447f7bf9-hv7km\" (UID: \"8602d572-e70e-4ee1-9f56-d7ebf8e0f0a1\") " pod="opendatahub/maas-controller-6b447f7bf9-hv7km"
Sep 11 14:00:57.719609 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:00:57.719517    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8602d572-e70e-4ee1-9f56-d7ebf8e0f0a1-webhook-cert\") pod \"maas-controller-6b447f7bf9-hv7km\" (UID: \"8602d572-e70e-4ee1-9f56-d7ebf8e0f0a1\") " pod="opendatahub/maas-controller-6b447f7bf9-hv7km"
Sep 11 14:00:57.725748 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:00:57.725727    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghndq\" (UniqueName: \"kubernetes.io/projected/8602d572-e70e-4ee1-9f56-d7ebf8e0f0a1-kube-api-access-ghndq\") pod \"maas-controller-6b447f7bf9-hv7km\" (UID: \"8602d572-e70e-4ee1-9f56-d7ebf8e0f0a1\") " pod="opendatahub/maas-controller-6b447f7bf9-hv7km"
Sep 11 14:00:57.822445 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:00:57.822339    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6b447f7bf9-hv7km"
Sep 11 14:00:58.097625 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:00:58.097591    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6b447f7bf9-hv7km"]
Sep 11 14:00:58.099243 ip-10-0-131-86 kubenswrapper[2578]: W0911 14:00:58.099218    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8602d572_e70e_4ee1_9f56_d7ebf8e0f0a1.slice/crio-59933bc344aa0c6f67a7e15ae2e1992750ffecd8372c003cc22685ed3878b577 WatchSource:0}: Error finding container 59933bc344aa0c6f67a7e15ae2e1992750ffecd8372c003cc22685ed3878b577: Status 404 returned error can't find the container with id 59933bc344aa0c6f67a7e15ae2e1992750ffecd8372c003cc22685ed3878b577
Sep 11 14:00:58.100640 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:00:58.100620    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 14:00:58.898478 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:00:58.898383    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6b447f7bf9-hv7km" event={"ID":"8602d572-e70e-4ee1-9f56-d7ebf8e0f0a1","Type":"ContainerStarted","Data":"d01dd60db70e342398de5db0e4843a4401551bbc0698b7320a62c5c5192a239f"}
Sep 11 14:00:58.898478 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:00:58.898440    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6b447f7bf9-hv7km" event={"ID":"8602d572-e70e-4ee1-9f56-d7ebf8e0f0a1","Type":"ContainerStarted","Data":"59933bc344aa0c6f67a7e15ae2e1992750ffecd8372c003cc22685ed3878b577"}
Sep 11 14:00:58.898902 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:00:58.898538    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6b447f7bf9-hv7km"
Sep 11 14:01:09.906551 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:01:09.906521    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6b447f7bf9-hv7km"
Sep 11 14:01:09.927308 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:01:09.927261    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-6b447f7bf9-hv7km" podStartSLOduration=12.485002276 podStartE2EDuration="12.927246059s" totalImagesPullingTime="442.243783ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:00:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:00:58.920031029 +0000 UTC m=+2366.306134209" watchObservedRunningTime="2026-09-11 14:01:09.927246059 +0000 UTC m=+2377.313349221"
Sep 11 14:01:17.461604 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:01:17.461568    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6b447f7bf9-hv7km"]
Sep 11 14:01:17.462109 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:01:17.461777    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-6b447f7bf9-hv7km" podUID="8602d572-e70e-4ee1-9f56-d7ebf8e0f0a1" containerName="manager" containerID="cri-o://d01dd60db70e342398de5db0e4843a4401551bbc0698b7320a62c5c5192a239f" gracePeriod=30
Sep 11 14:01:17.730669 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:01:17.730643    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6b447f7bf9-hv7km"
Sep 11 14:01:17.824504 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:01:17.824456    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ghndq\" (UniqueName: \"kubernetes.io/projected/8602d572-e70e-4ee1-9f56-d7ebf8e0f0a1-kube-api-access-ghndq\") pod \"8602d572-e70e-4ee1-9f56-d7ebf8e0f0a1\" (UID: \"8602d572-e70e-4ee1-9f56-d7ebf8e0f0a1\") "
Sep 11 14:01:17.824692 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:01:17.824592    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8602d572-e70e-4ee1-9f56-d7ebf8e0f0a1-metrics-certs\") pod \"8602d572-e70e-4ee1-9f56-d7ebf8e0f0a1\" (UID: \"8602d572-e70e-4ee1-9f56-d7ebf8e0f0a1\") "
Sep 11 14:01:17.824692 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:01:17.824646    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8602d572-e70e-4ee1-9f56-d7ebf8e0f0a1-webhook-cert\") pod \"8602d572-e70e-4ee1-9f56-d7ebf8e0f0a1\" (UID: \"8602d572-e70e-4ee1-9f56-d7ebf8e0f0a1\") "
Sep 11 14:01:17.827274 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:01:17.827239    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8602d572-e70e-4ee1-9f56-d7ebf8e0f0a1-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "8602d572-e70e-4ee1-9f56-d7ebf8e0f0a1" (UID: "8602d572-e70e-4ee1-9f56-d7ebf8e0f0a1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 14:01:17.827428 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:01:17.827243    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8602d572-e70e-4ee1-9f56-d7ebf8e0f0a1-kube-api-access-ghndq" (OuterVolumeSpecName: "kube-api-access-ghndq") pod "8602d572-e70e-4ee1-9f56-d7ebf8e0f0a1" (UID: "8602d572-e70e-4ee1-9f56-d7ebf8e0f0a1"). InnerVolumeSpecName "kube-api-access-ghndq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:01:17.827491 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:01:17.827418    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8602d572-e70e-4ee1-9f56-d7ebf8e0f0a1-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "8602d572-e70e-4ee1-9f56-d7ebf8e0f0a1" (UID: "8602d572-e70e-4ee1-9f56-d7ebf8e0f0a1"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 14:01:17.925466 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:01:17.925425    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ghndq\" (UniqueName: \"kubernetes.io/projected/8602d572-e70e-4ee1-9f56-d7ebf8e0f0a1-kube-api-access-ghndq\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 14:01:17.925466 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:01:17.925464    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8602d572-e70e-4ee1-9f56-d7ebf8e0f0a1-metrics-certs\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 14:01:17.925466 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:01:17.925476    2578 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8602d572-e70e-4ee1-9f56-d7ebf8e0f0a1-webhook-cert\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 14:01:18.002073 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:01:18.002039    2578 generic.go:356] "Generic (PLEG): container finished" podID="8602d572-e70e-4ee1-9f56-d7ebf8e0f0a1" containerID="d01dd60db70e342398de5db0e4843a4401551bbc0698b7320a62c5c5192a239f" exitCode=0
Sep 11 14:01:18.002260 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:01:18.002087    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6b447f7bf9-hv7km"
Sep 11 14:01:18.002260 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:01:18.002125    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6b447f7bf9-hv7km" event={"ID":"8602d572-e70e-4ee1-9f56-d7ebf8e0f0a1","Type":"ContainerDied","Data":"d01dd60db70e342398de5db0e4843a4401551bbc0698b7320a62c5c5192a239f"}
Sep 11 14:01:18.002260 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:01:18.002180    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6b447f7bf9-hv7km" event={"ID":"8602d572-e70e-4ee1-9f56-d7ebf8e0f0a1","Type":"ContainerDied","Data":"59933bc344aa0c6f67a7e15ae2e1992750ffecd8372c003cc22685ed3878b577"}
Sep 11 14:01:18.002260 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:01:18.002208    2578 scope.go:117] "RemoveContainer" containerID="d01dd60db70e342398de5db0e4843a4401551bbc0698b7320a62c5c5192a239f"
Sep 11 14:01:18.011739 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:01:18.011722    2578 scope.go:117] "RemoveContainer" containerID="d01dd60db70e342398de5db0e4843a4401551bbc0698b7320a62c5c5192a239f"
Sep 11 14:01:18.012036 ip-10-0-131-86 kubenswrapper[2578]: E0911 14:01:18.012015    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d01dd60db70e342398de5db0e4843a4401551bbc0698b7320a62c5c5192a239f\": container with ID starting with d01dd60db70e342398de5db0e4843a4401551bbc0698b7320a62c5c5192a239f not found: ID does not exist" containerID="d01dd60db70e342398de5db0e4843a4401551bbc0698b7320a62c5c5192a239f"
Sep 11 14:01:18.012115 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:01:18.012048    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d01dd60db70e342398de5db0e4843a4401551bbc0698b7320a62c5c5192a239f"} err="failed to get container status \"d01dd60db70e342398de5db0e4843a4401551bbc0698b7320a62c5c5192a239f\": rpc error: code = NotFound desc = could not find container \"d01dd60db70e342398de5db0e4843a4401551bbc0698b7320a62c5c5192a239f\": container with ID starting with d01dd60db70e342398de5db0e4843a4401551bbc0698b7320a62c5c5192a239f not found: ID does not exist"
Sep 11 14:01:18.026254 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:01:18.026230    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6b447f7bf9-hv7km"]
Sep 11 14:01:18.030405 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:01:18.030385    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-6b447f7bf9-hv7km"]
Sep 11 14:01:19.108316 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:01:19.108284    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8602d572-e70e-4ee1-9f56-d7ebf8e0f0a1" path="/var/lib/kubelet/pods/8602d572-e70e-4ee1-9f56-d7ebf8e0f0a1/volumes"
Sep 11 14:01:19.282457 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:01:19.282424    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6b447f7bf9-pj4gf"]
Sep 11 14:01:19.282885 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:01:19.282868    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8602d572-e70e-4ee1-9f56-d7ebf8e0f0a1" containerName="manager"
Sep 11 14:01:19.282955 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:01:19.282887    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="8602d572-e70e-4ee1-9f56-d7ebf8e0f0a1" containerName="manager"
Sep 11 14:01:19.282991 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:01:19.282980    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="8602d572-e70e-4ee1-9f56-d7ebf8e0f0a1" containerName="manager"
Sep 11 14:01:19.291037 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:01:19.291007    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6b447f7bf9-pj4gf"
Sep 11 14:01:19.293963 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:01:19.293939    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 11 14:01:19.294104 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:01:19.294072    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 11 14:01:19.294189 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:01:19.293939    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-dpcsj\""
Sep 11 14:01:19.294490 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:01:19.294433    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 11 14:01:19.297529 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:01:19.297506    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6b447f7bf9-pj4gf"]
Sep 11 14:01:19.440534 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:01:19.440442    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v579t\" (UniqueName: \"kubernetes.io/projected/34b3897d-3f58-43bc-953f-3e7b3b6396e6-kube-api-access-v579t\") pod \"maas-controller-6b447f7bf9-pj4gf\" (UID: \"34b3897d-3f58-43bc-953f-3e7b3b6396e6\") " pod="opendatahub/maas-controller-6b447f7bf9-pj4gf"
Sep 11 14:01:19.440534 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:01:19.440484    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/34b3897d-3f58-43bc-953f-3e7b3b6396e6-webhook-cert\") pod \"maas-controller-6b447f7bf9-pj4gf\" (UID: \"34b3897d-3f58-43bc-953f-3e7b3b6396e6\") " pod="opendatahub/maas-controller-6b447f7bf9-pj4gf"
Sep 11 14:01:19.440730 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:01:19.440559    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/34b3897d-3f58-43bc-953f-3e7b3b6396e6-metrics-certs\") pod \"maas-controller-6b447f7bf9-pj4gf\" (UID: \"34b3897d-3f58-43bc-953f-3e7b3b6396e6\") " pod="opendatahub/maas-controller-6b447f7bf9-pj4gf"
Sep 11 14:01:19.542059 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:01:19.542013    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v579t\" (UniqueName: \"kubernetes.io/projected/34b3897d-3f58-43bc-953f-3e7b3b6396e6-kube-api-access-v579t\") pod \"maas-controller-6b447f7bf9-pj4gf\" (UID: \"34b3897d-3f58-43bc-953f-3e7b3b6396e6\") " pod="opendatahub/maas-controller-6b447f7bf9-pj4gf"
Sep 11 14:01:19.542059 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:01:19.542062    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/34b3897d-3f58-43bc-953f-3e7b3b6396e6-webhook-cert\") pod \"maas-controller-6b447f7bf9-pj4gf\" (UID: \"34b3897d-3f58-43bc-953f-3e7b3b6396e6\") " pod="opendatahub/maas-controller-6b447f7bf9-pj4gf"
Sep 11 14:01:19.542282 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:01:19.542125    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/34b3897d-3f58-43bc-953f-3e7b3b6396e6-metrics-certs\") pod \"maas-controller-6b447f7bf9-pj4gf\" (UID: \"34b3897d-3f58-43bc-953f-3e7b3b6396e6\") " pod="opendatahub/maas-controller-6b447f7bf9-pj4gf"
Sep 11 14:01:19.544674 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:01:19.544644    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/34b3897d-3f58-43bc-953f-3e7b3b6396e6-metrics-certs\") pod \"maas-controller-6b447f7bf9-pj4gf\" (UID: \"34b3897d-3f58-43bc-953f-3e7b3b6396e6\") " pod="opendatahub/maas-controller-6b447f7bf9-pj4gf"
Sep 11 14:01:19.544786 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:01:19.544703    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/34b3897d-3f58-43bc-953f-3e7b3b6396e6-webhook-cert\") pod \"maas-controller-6b447f7bf9-pj4gf\" (UID: \"34b3897d-3f58-43bc-953f-3e7b3b6396e6\") " pod="opendatahub/maas-controller-6b447f7bf9-pj4gf"
Sep 11 14:01:19.550357 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:01:19.550328    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v579t\" (UniqueName: \"kubernetes.io/projected/34b3897d-3f58-43bc-953f-3e7b3b6396e6-kube-api-access-v579t\") pod \"maas-controller-6b447f7bf9-pj4gf\" (UID: \"34b3897d-3f58-43bc-953f-3e7b3b6396e6\") " pod="opendatahub/maas-controller-6b447f7bf9-pj4gf"
Sep 11 14:01:19.605197 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:01:19.605159    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6b447f7bf9-pj4gf"
Sep 11 14:01:19.877904 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:01:19.877880    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6b447f7bf9-pj4gf"]
Sep 11 14:01:19.880019 ip-10-0-131-86 kubenswrapper[2578]: W0911 14:01:19.879983    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34b3897d_3f58_43bc_953f_3e7b3b6396e6.slice/crio-3f8142c8822474827b56c0458d1a0d1fdac6587ca56973ad7bd8e61b9fd794b0 WatchSource:0}: Error finding container 3f8142c8822474827b56c0458d1a0d1fdac6587ca56973ad7bd8e61b9fd794b0: Status 404 returned error can't find the container with id 3f8142c8822474827b56c0458d1a0d1fdac6587ca56973ad7bd8e61b9fd794b0
Sep 11 14:01:20.016236 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:01:20.016200    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6b447f7bf9-pj4gf" event={"ID":"34b3897d-3f58-43bc-953f-3e7b3b6396e6","Type":"ContainerStarted","Data":"3f8142c8822474827b56c0458d1a0d1fdac6587ca56973ad7bd8e61b9fd794b0"}
Sep 11 14:01:21.022914 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:01:21.022878    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6b447f7bf9-pj4gf" event={"ID":"34b3897d-3f58-43bc-953f-3e7b3b6396e6","Type":"ContainerStarted","Data":"33681c83b74f3ad872b50b640dc3eef2bad70095ff2cb27ef21c90ed164e200d"}
Sep 11 14:01:21.023347 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:01:21.022940    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6b447f7bf9-pj4gf"
Sep 11 14:01:21.042161 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:01:21.042117    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-6b447f7bf9-pj4gf" podStartSLOduration=1.651036692 podStartE2EDuration="2.042099473s" totalImagesPullingTime="391.062781ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:01:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:01:21.03988725 +0000 UTC m=+2388.425990428" watchObservedRunningTime="2026-09-11 14:01:21.042099473 +0000 UTC m=+2388.428202637"
Sep 11 14:01:32.031032 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:01:32.030999    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6b447f7bf9-pj4gf"
Sep 11 14:01:33.347066 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:01:33.347038    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6gltj_e384caae-3cb1-4264-8a25-55b40ac8ac45/ovn-acl-logging/0.log"
Sep 11 14:01:33.353779 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:01:33.353760    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6gltj_e384caae-3cb1-4264-8a25-55b40ac8ac45/ovn-acl-logging/0.log"
Sep 11 14:02:33.725939 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:02:33.725900    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-d8lgv"]
Sep 11 14:02:33.726427 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:02:33.726212    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-d8lgv" podUID="944e63d2-014c-40e1-9b2c-caa5f1259b21" containerName="manager" containerID="cri-o://147696462aa3448ab971010e2e89a5b4f17388693b83170659305cbede8d7f0e" gracePeriod=10
Sep 11 14:02:36.184536 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:02:36.184511    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-d8lgv"
Sep 11 14:02:36.298823 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:02:36.298726    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ldj5v\" (UniqueName: \"kubernetes.io/projected/944e63d2-014c-40e1-9b2c-caa5f1259b21-kube-api-access-ldj5v\") pod \"944e63d2-014c-40e1-9b2c-caa5f1259b21\" (UID: \"944e63d2-014c-40e1-9b2c-caa5f1259b21\") "
Sep 11 14:02:36.298992 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:02:36.298937    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/944e63d2-014c-40e1-9b2c-caa5f1259b21-extensions-socket-volume\") pod \"944e63d2-014c-40e1-9b2c-caa5f1259b21\" (UID: \"944e63d2-014c-40e1-9b2c-caa5f1259b21\") "
Sep 11 14:02:36.299272 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:02:36.299252    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/944e63d2-014c-40e1-9b2c-caa5f1259b21-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "944e63d2-014c-40e1-9b2c-caa5f1259b21" (UID: "944e63d2-014c-40e1-9b2c-caa5f1259b21"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:02:36.300909 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:02:36.300884    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/944e63d2-014c-40e1-9b2c-caa5f1259b21-kube-api-access-ldj5v" (OuterVolumeSpecName: "kube-api-access-ldj5v") pod "944e63d2-014c-40e1-9b2c-caa5f1259b21" (UID: "944e63d2-014c-40e1-9b2c-caa5f1259b21"). InnerVolumeSpecName "kube-api-access-ldj5v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:02:36.400013 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:02:36.399978    2578 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/944e63d2-014c-40e1-9b2c-caa5f1259b21-extensions-socket-volume\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 14:02:36.400013 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:02:36.400007    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ldj5v\" (UniqueName: \"kubernetes.io/projected/944e63d2-014c-40e1-9b2c-caa5f1259b21-kube-api-access-ldj5v\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 14:02:36.427237 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:02:36.427209    2578 generic.go:356] "Generic (PLEG): container finished" podID="944e63d2-014c-40e1-9b2c-caa5f1259b21" containerID="147696462aa3448ab971010e2e89a5b4f17388693b83170659305cbede8d7f0e" exitCode=0
Sep 11 14:02:36.427379 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:02:36.427259    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-d8lgv"
Sep 11 14:02:36.427379 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:02:36.427300    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-d8lgv" event={"ID":"944e63d2-014c-40e1-9b2c-caa5f1259b21","Type":"ContainerDied","Data":"147696462aa3448ab971010e2e89a5b4f17388693b83170659305cbede8d7f0e"}
Sep 11 14:02:36.427379 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:02:36.427348    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-d8lgv" event={"ID":"944e63d2-014c-40e1-9b2c-caa5f1259b21","Type":"ContainerDied","Data":"b419ae587d2f0e18d71c04d22123aa31b381bd23c5a48d4482f8ee9662432f9a"}
Sep 11 14:02:36.427379 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:02:36.427370    2578 scope.go:117] "RemoveContainer" containerID="147696462aa3448ab971010e2e89a5b4f17388693b83170659305cbede8d7f0e"
Sep 11 14:02:36.436717 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:02:36.436702    2578 scope.go:117] "RemoveContainer" containerID="716f452cecdef11943e418b4748eec2e32cd57c43fc5161d7eff1d3f3683dd88"
Sep 11 14:02:36.444738 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:02:36.444713    2578 scope.go:117] "RemoveContainer" containerID="147696462aa3448ab971010e2e89a5b4f17388693b83170659305cbede8d7f0e"
Sep 11 14:02:36.444995 ip-10-0-131-86 kubenswrapper[2578]: E0911 14:02:36.444979    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"147696462aa3448ab971010e2e89a5b4f17388693b83170659305cbede8d7f0e\": container with ID starting with 147696462aa3448ab971010e2e89a5b4f17388693b83170659305cbede8d7f0e not found: ID does not exist" containerID="147696462aa3448ab971010e2e89a5b4f17388693b83170659305cbede8d7f0e"
Sep 11 14:02:36.445055 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:02:36.445003    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"147696462aa3448ab971010e2e89a5b4f17388693b83170659305cbede8d7f0e"} err="failed to get container status \"147696462aa3448ab971010e2e89a5b4f17388693b83170659305cbede8d7f0e\": rpc error: code = NotFound desc = could not find container \"147696462aa3448ab971010e2e89a5b4f17388693b83170659305cbede8d7f0e\": container with ID starting with 147696462aa3448ab971010e2e89a5b4f17388693b83170659305cbede8d7f0e not found: ID does not exist"
Sep 11 14:02:36.445055 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:02:36.445019    2578 scope.go:117] "RemoveContainer" containerID="716f452cecdef11943e418b4748eec2e32cd57c43fc5161d7eff1d3f3683dd88"
Sep 11 14:02:36.445234 ip-10-0-131-86 kubenswrapper[2578]: E0911 14:02:36.445220    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"716f452cecdef11943e418b4748eec2e32cd57c43fc5161d7eff1d3f3683dd88\": container with ID starting with 716f452cecdef11943e418b4748eec2e32cd57c43fc5161d7eff1d3f3683dd88 not found: ID does not exist" containerID="716f452cecdef11943e418b4748eec2e32cd57c43fc5161d7eff1d3f3683dd88"
Sep 11 14:02:36.445276 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:02:36.445236    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"716f452cecdef11943e418b4748eec2e32cd57c43fc5161d7eff1d3f3683dd88"} err="failed to get container status \"716f452cecdef11943e418b4748eec2e32cd57c43fc5161d7eff1d3f3683dd88\": rpc error: code = NotFound desc = could not find container \"716f452cecdef11943e418b4748eec2e32cd57c43fc5161d7eff1d3f3683dd88\": container with ID starting with 716f452cecdef11943e418b4748eec2e32cd57c43fc5161d7eff1d3f3683dd88 not found: ID does not exist"
Sep 11 14:02:36.448906 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:02:36.448883    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-d8lgv"]
Sep 11 14:02:36.455462 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:02:36.455440    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-d8lgv"]
Sep 11 14:02:37.106553 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:02:37.106520    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="944e63d2-014c-40e1-9b2c-caa5f1259b21" path="/var/lib/kubelet/pods/944e63d2-014c-40e1-9b2c-caa5f1259b21/volumes"
Sep 11 14:03:34.569921 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:03:34.569881    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-v227b"]
Sep 11 14:03:34.570540 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:03:34.570520    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="944e63d2-014c-40e1-9b2c-caa5f1259b21" containerName="manager"
Sep 11 14:03:34.570628 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:03:34.570544    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="944e63d2-014c-40e1-9b2c-caa5f1259b21" containerName="manager"
Sep 11 14:03:34.570628 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:03:34.570558    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="944e63d2-014c-40e1-9b2c-caa5f1259b21" containerName="manager"
Sep 11 14:03:34.570628 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:03:34.570566    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="944e63d2-014c-40e1-9b2c-caa5f1259b21" containerName="manager"
Sep 11 14:03:34.570782 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:03:34.570697    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="944e63d2-014c-40e1-9b2c-caa5f1259b21" containerName="manager"
Sep 11 14:03:34.574958 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:03:34.574933    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-v227b"
Sep 11 14:03:34.577082 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:03:34.577063    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-tws25\""
Sep 11 14:03:34.585590 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:03:34.585569    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-v227b"]
Sep 11 14:03:34.645022 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:03:34.644991    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9gzt\" (UniqueName: \"kubernetes.io/projected/3350e6f5-a27b-43e2-ac61-ff4b1173c78d-kube-api-access-v9gzt\") pod \"kuadrant-operator-controller-manager-5769f99d9f-v227b\" (UID: \"3350e6f5-a27b-43e2-ac61-ff4b1173c78d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-v227b"
Sep 11 14:03:34.645177 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:03:34.645061    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3350e6f5-a27b-43e2-ac61-ff4b1173c78d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-v227b\" (UID: \"3350e6f5-a27b-43e2-ac61-ff4b1173c78d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-v227b"
Sep 11 14:03:34.746064 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:03:34.746026    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9gzt\" (UniqueName: \"kubernetes.io/projected/3350e6f5-a27b-43e2-ac61-ff4b1173c78d-kube-api-access-v9gzt\") pod \"kuadrant-operator-controller-manager-5769f99d9f-v227b\" (UID: \"3350e6f5-a27b-43e2-ac61-ff4b1173c78d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-v227b"
Sep 11 14:03:34.746246 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:03:34.746104    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3350e6f5-a27b-43e2-ac61-ff4b1173c78d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-v227b\" (UID: \"3350e6f5-a27b-43e2-ac61-ff4b1173c78d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-v227b"
Sep 11 14:03:34.746474 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:03:34.746457    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3350e6f5-a27b-43e2-ac61-ff4b1173c78d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-v227b\" (UID: \"3350e6f5-a27b-43e2-ac61-ff4b1173c78d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-v227b"
Sep 11 14:03:34.754622 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:03:34.754597    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9gzt\" (UniqueName: \"kubernetes.io/projected/3350e6f5-a27b-43e2-ac61-ff4b1173c78d-kube-api-access-v9gzt\") pod \"kuadrant-operator-controller-manager-5769f99d9f-v227b\" (UID: \"3350e6f5-a27b-43e2-ac61-ff4b1173c78d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-v227b"
Sep 11 14:03:34.885313 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:03:34.885211    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-v227b"
Sep 11 14:03:35.161910 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:03:35.161882    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-v227b"]
Sep 11 14:03:35.164002 ip-10-0-131-86 kubenswrapper[2578]: W0911 14:03:35.163973    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3350e6f5_a27b_43e2_ac61_ff4b1173c78d.slice/crio-165c0a34d41b9508f36e3da73c629c7656fcfa85beefcc4ff302e75c949954eb WatchSource:0}: Error finding container 165c0a34d41b9508f36e3da73c629c7656fcfa85beefcc4ff302e75c949954eb: Status 404 returned error can't find the container with id 165c0a34d41b9508f36e3da73c629c7656fcfa85beefcc4ff302e75c949954eb
Sep 11 14:03:35.742804 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:03:35.742770    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-v227b" event={"ID":"3350e6f5-a27b-43e2-ac61-ff4b1173c78d","Type":"ContainerStarted","Data":"a2617a1bf2fd468f1a12ff8656548d1b04728f932c8be032fc88641bfc16f9be"}
Sep 11 14:03:35.743209 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:03:35.742820    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-v227b" event={"ID":"3350e6f5-a27b-43e2-ac61-ff4b1173c78d","Type":"ContainerStarted","Data":"165c0a34d41b9508f36e3da73c629c7656fcfa85beefcc4ff302e75c949954eb"}
Sep 11 14:03:35.743209 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:03:35.742874    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-v227b"
Sep 11 14:03:35.766848 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:03:35.766794    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-v227b" podStartSLOduration=1.7667793330000001 podStartE2EDuration="1.766779333s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:03:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:03:35.763947629 +0000 UTC m=+2523.150050802" watchObservedRunningTime="2026-09-11 14:03:35.766779333 +0000 UTC m=+2523.152882493"
Sep 11 14:03:46.750363 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:03:46.750333    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-v227b"
Sep 11 14:05:02.607316 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:05:02.607221    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-f8bb4c8cc-xtlrq"]
Sep 11 14:05:02.608071 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:05:02.608043    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="944e63d2-014c-40e1-9b2c-caa5f1259b21" containerName="manager"
Sep 11 14:05:02.611093 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:05:02.611076    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-f8bb4c8cc-xtlrq"
Sep 11 14:05:02.628111 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:05:02.628083    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-f8bb4c8cc-xtlrq"]
Sep 11 14:05:02.703630 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:05:02.703598    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ca8c1d3-a6f9-4e9a-bcf3-f0079b3f4a48-openshift-service-ca\") pod \"maas-api-f8bb4c8cc-xtlrq\" (UID: \"9ca8c1d3-a6f9-4e9a-bcf3-f0079b3f4a48\") " pod="odh-ai-gateway-infra/maas-api-f8bb4c8cc-xtlrq"
Sep 11 14:05:02.703813 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:05:02.703648    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wwh6\" (UniqueName: \"kubernetes.io/projected/9ca8c1d3-a6f9-4e9a-bcf3-f0079b3f4a48-kube-api-access-2wwh6\") pod \"maas-api-f8bb4c8cc-xtlrq\" (UID: \"9ca8c1d3-a6f9-4e9a-bcf3-f0079b3f4a48\") " pod="odh-ai-gateway-infra/maas-api-f8bb4c8cc-xtlrq"
Sep 11 14:05:02.703813 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:05:02.703724    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ca8c1d3-a6f9-4e9a-bcf3-f0079b3f4a48-metrics-certs\") pod \"maas-api-f8bb4c8cc-xtlrq\" (UID: \"9ca8c1d3-a6f9-4e9a-bcf3-f0079b3f4a48\") " pod="odh-ai-gateway-infra/maas-api-f8bb4c8cc-xtlrq"
Sep 11 14:05:02.703813 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:05:02.703780    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9ca8c1d3-a6f9-4e9a-bcf3-f0079b3f4a48-maas-api-tls\") pod \"maas-api-f8bb4c8cc-xtlrq\" (UID: \"9ca8c1d3-a6f9-4e9a-bcf3-f0079b3f4a48\") " pod="odh-ai-gateway-infra/maas-api-f8bb4c8cc-xtlrq"
Sep 11 14:05:02.805266 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:05:02.805227    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ca8c1d3-a6f9-4e9a-bcf3-f0079b3f4a48-metrics-certs\") pod \"maas-api-f8bb4c8cc-xtlrq\" (UID: \"9ca8c1d3-a6f9-4e9a-bcf3-f0079b3f4a48\") " pod="odh-ai-gateway-infra/maas-api-f8bb4c8cc-xtlrq"
Sep 11 14:05:02.805470 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:05:02.805311    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9ca8c1d3-a6f9-4e9a-bcf3-f0079b3f4a48-maas-api-tls\") pod \"maas-api-f8bb4c8cc-xtlrq\" (UID: \"9ca8c1d3-a6f9-4e9a-bcf3-f0079b3f4a48\") " pod="odh-ai-gateway-infra/maas-api-f8bb4c8cc-xtlrq"
Sep 11 14:05:02.805470 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:05:02.805337    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ca8c1d3-a6f9-4e9a-bcf3-f0079b3f4a48-openshift-service-ca\") pod \"maas-api-f8bb4c8cc-xtlrq\" (UID: \"9ca8c1d3-a6f9-4e9a-bcf3-f0079b3f4a48\") " pod="odh-ai-gateway-infra/maas-api-f8bb4c8cc-xtlrq"
Sep 11 14:05:02.805470 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:05:02.805370    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wwh6\" (UniqueName: \"kubernetes.io/projected/9ca8c1d3-a6f9-4e9a-bcf3-f0079b3f4a48-kube-api-access-2wwh6\") pod \"maas-api-f8bb4c8cc-xtlrq\" (UID: \"9ca8c1d3-a6f9-4e9a-bcf3-f0079b3f4a48\") " pod="odh-ai-gateway-infra/maas-api-f8bb4c8cc-xtlrq"
Sep 11 14:05:02.806007 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:05:02.805987    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ca8c1d3-a6f9-4e9a-bcf3-f0079b3f4a48-openshift-service-ca\") pod \"maas-api-f8bb4c8cc-xtlrq\" (UID: \"9ca8c1d3-a6f9-4e9a-bcf3-f0079b3f4a48\") " pod="odh-ai-gateway-infra/maas-api-f8bb4c8cc-xtlrq"
Sep 11 14:05:02.807823 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:05:02.807800    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ca8c1d3-a6f9-4e9a-bcf3-f0079b3f4a48-metrics-certs\") pod \"maas-api-f8bb4c8cc-xtlrq\" (UID: \"9ca8c1d3-a6f9-4e9a-bcf3-f0079b3f4a48\") " pod="odh-ai-gateway-infra/maas-api-f8bb4c8cc-xtlrq"
Sep 11 14:05:02.808009 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:05:02.807991    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9ca8c1d3-a6f9-4e9a-bcf3-f0079b3f4a48-maas-api-tls\") pod \"maas-api-f8bb4c8cc-xtlrq\" (UID: \"9ca8c1d3-a6f9-4e9a-bcf3-f0079b3f4a48\") " pod="odh-ai-gateway-infra/maas-api-f8bb4c8cc-xtlrq"
Sep 11 14:05:02.814597 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:05:02.814571    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wwh6\" (UniqueName: \"kubernetes.io/projected/9ca8c1d3-a6f9-4e9a-bcf3-f0079b3f4a48-kube-api-access-2wwh6\") pod \"maas-api-f8bb4c8cc-xtlrq\" (UID: \"9ca8c1d3-a6f9-4e9a-bcf3-f0079b3f4a48\") " pod="odh-ai-gateway-infra/maas-api-f8bb4c8cc-xtlrq"
Sep 11 14:05:02.923262 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:05:02.923171    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-f8bb4c8cc-xtlrq"
Sep 11 14:05:03.207018 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:05:03.206990    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-f8bb4c8cc-xtlrq"]
Sep 11 14:05:03.210037 ip-10-0-131-86 kubenswrapper[2578]: W0911 14:05:03.210004    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ca8c1d3_a6f9_4e9a_bcf3_f0079b3f4a48.slice/crio-671f507a369e3a3f9b4f4275383027532910ee7b4a0f896f11c831a2945a93f5 WatchSource:0}: Error finding container 671f507a369e3a3f9b4f4275383027532910ee7b4a0f896f11c831a2945a93f5: Status 404 returned error can't find the container with id 671f507a369e3a3f9b4f4275383027532910ee7b4a0f896f11c831a2945a93f5
Sep 11 14:05:03.226431 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:05:03.226404    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-f8bb4c8cc-xtlrq" event={"ID":"9ca8c1d3-a6f9-4e9a-bcf3-f0079b3f4a48","Type":"ContainerStarted","Data":"671f507a369e3a3f9b4f4275383027532910ee7b4a0f896f11c831a2945a93f5"}
Sep 11 14:05:04.235123 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:05:04.235080    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-f8bb4c8cc-xtlrq" event={"ID":"9ca8c1d3-a6f9-4e9a-bcf3-f0079b3f4a48","Type":"ContainerStarted","Data":"a8b38be8114ebdc4df0c56a31113ebd4dca33a3416b81ccb6aabdd1b5dae75a5"}
Sep 11 14:05:04.235590 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:05:04.235149    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-f8bb4c8cc-xtlrq"
Sep 11 14:05:04.255303 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:05:04.255247    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-f8bb4c8cc-xtlrq" podStartSLOduration=1.597423979 podStartE2EDuration="2.255225427s" totalImagesPullingTime="657.801448ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:05:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:05:04.253762808 +0000 UTC m=+2611.639865997" watchObservedRunningTime="2026-09-11 14:05:04.255225427 +0000 UTC m=+2611.641328592"
Sep 11 14:05:10.246336 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:05:10.246305    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-f8bb4c8cc-xtlrq"
Sep 11 14:05:20.430888 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:05:20.430848    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5d987ff8b5-pvm5s"]
Sep 11 14:05:20.431285 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:05:20.431059    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-5d987ff8b5-pvm5s" podUID="3015eac6-2320-4916-82ab-fd3b873bbeae" containerName="maas-api" containerID="cri-o://05099fdfb20539272138a0265fcefcb57d969ada3d37272678d8ed236c3ceac5" gracePeriod=30
Sep 11 14:05:21.688763 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:05:21.688734    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5d987ff8b5-pvm5s"
Sep 11 14:05:21.790282 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:05:21.790185    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t6k6s\" (UniqueName: \"kubernetes.io/projected/3015eac6-2320-4916-82ab-fd3b873bbeae-kube-api-access-t6k6s\") pod \"3015eac6-2320-4916-82ab-fd3b873bbeae\" (UID: \"3015eac6-2320-4916-82ab-fd3b873bbeae\") "
Sep 11 14:05:21.790444 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:05:21.790313    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3015eac6-2320-4916-82ab-fd3b873bbeae-metrics-certs\") pod \"3015eac6-2320-4916-82ab-fd3b873bbeae\" (UID: \"3015eac6-2320-4916-82ab-fd3b873bbeae\") "
Sep 11 14:05:21.790444 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:05:21.790352    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3015eac6-2320-4916-82ab-fd3b873bbeae-openshift-service-ca\") pod \"3015eac6-2320-4916-82ab-fd3b873bbeae\" (UID: \"3015eac6-2320-4916-82ab-fd3b873bbeae\") "
Sep 11 14:05:21.790569 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:05:21.790446    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3015eac6-2320-4916-82ab-fd3b873bbeae-maas-api-tls\") pod \"3015eac6-2320-4916-82ab-fd3b873bbeae\" (UID: \"3015eac6-2320-4916-82ab-fd3b873bbeae\") "
Sep 11 14:05:21.790814 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:05:21.790775    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3015eac6-2320-4916-82ab-fd3b873bbeae-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "3015eac6-2320-4916-82ab-fd3b873bbeae" (UID: "3015eac6-2320-4916-82ab-fd3b873bbeae"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 14:05:21.792491 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:05:21.792462    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3015eac6-2320-4916-82ab-fd3b873bbeae-kube-api-access-t6k6s" (OuterVolumeSpecName: "kube-api-access-t6k6s") pod "3015eac6-2320-4916-82ab-fd3b873bbeae" (UID: "3015eac6-2320-4916-82ab-fd3b873bbeae"). InnerVolumeSpecName "kube-api-access-t6k6s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:05:21.792617 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:05:21.792595    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3015eac6-2320-4916-82ab-fd3b873bbeae-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "3015eac6-2320-4916-82ab-fd3b873bbeae" (UID: "3015eac6-2320-4916-82ab-fd3b873bbeae"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 14:05:21.792666 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:05:21.792602    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3015eac6-2320-4916-82ab-fd3b873bbeae-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "3015eac6-2320-4916-82ab-fd3b873bbeae" (UID: "3015eac6-2320-4916-82ab-fd3b873bbeae"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 14:05:21.892026 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:05:21.891983    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3015eac6-2320-4916-82ab-fd3b873bbeae-maas-api-tls\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 14:05:21.892026 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:05:21.892020    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t6k6s\" (UniqueName: \"kubernetes.io/projected/3015eac6-2320-4916-82ab-fd3b873bbeae-kube-api-access-t6k6s\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 14:05:21.892026 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:05:21.892033    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3015eac6-2320-4916-82ab-fd3b873bbeae-metrics-certs\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 14:05:21.892273 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:05:21.892045    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3015eac6-2320-4916-82ab-fd3b873bbeae-openshift-service-ca\") on node \"ip-10-0-131-86.ec2.internal\" DevicePath \"\""
Sep 11 14:05:22.336257 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:05:22.336218    2578 generic.go:356] "Generic (PLEG): container finished" podID="3015eac6-2320-4916-82ab-fd3b873bbeae" containerID="05099fdfb20539272138a0265fcefcb57d969ada3d37272678d8ed236c3ceac5" exitCode=0
Sep 11 14:05:22.336450 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:05:22.336272    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5d987ff8b5-pvm5s"
Sep 11 14:05:22.336450 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:05:22.336305    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5d987ff8b5-pvm5s" event={"ID":"3015eac6-2320-4916-82ab-fd3b873bbeae","Type":"ContainerDied","Data":"05099fdfb20539272138a0265fcefcb57d969ada3d37272678d8ed236c3ceac5"}
Sep 11 14:05:22.336450 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:05:22.336359    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5d987ff8b5-pvm5s" event={"ID":"3015eac6-2320-4916-82ab-fd3b873bbeae","Type":"ContainerDied","Data":"f5af346e061981dd656c4583684ac7eac6fdc27acd4088ad2116fe53993ff4df"}
Sep 11 14:05:22.336450 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:05:22.336381    2578 scope.go:117] "RemoveContainer" containerID="05099fdfb20539272138a0265fcefcb57d969ada3d37272678d8ed236c3ceac5"
Sep 11 14:05:22.347712 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:05:22.347685    2578 scope.go:117] "RemoveContainer" containerID="05099fdfb20539272138a0265fcefcb57d969ada3d37272678d8ed236c3ceac5"
Sep 11 14:05:22.348039 ip-10-0-131-86 kubenswrapper[2578]: E0911 14:05:22.348011    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05099fdfb20539272138a0265fcefcb57d969ada3d37272678d8ed236c3ceac5\": container with ID starting with 05099fdfb20539272138a0265fcefcb57d969ada3d37272678d8ed236c3ceac5 not found: ID does not exist" containerID="05099fdfb20539272138a0265fcefcb57d969ada3d37272678d8ed236c3ceac5"
Sep 11 14:05:22.348147 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:05:22.348048    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05099fdfb20539272138a0265fcefcb57d969ada3d37272678d8ed236c3ceac5"} err="failed to get container status \"05099fdfb20539272138a0265fcefcb57d969ada3d37272678d8ed236c3ceac5\": rpc error: code = NotFound desc = could not find container \"05099fdfb20539272138a0265fcefcb57d969ada3d37272678d8ed236c3ceac5\": container with ID starting with 05099fdfb20539272138a0265fcefcb57d969ada3d37272678d8ed236c3ceac5 not found: ID does not exist"
Sep 11 14:05:22.362867 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:05:22.362817    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5d987ff8b5-pvm5s"]
Sep 11 14:05:22.370479 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:05:22.370441    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-5d987ff8b5-pvm5s"]
Sep 11 14:05:23.107469 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:05:23.107434    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3015eac6-2320-4916-82ab-fd3b873bbeae" path="/var/lib/kubelet/pods/3015eac6-2320-4916-82ab-fd3b873bbeae/volumes"
Sep 11 14:05:39.811768 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:05:39.811734    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-5clkn_723fc010-ed76-4913-8dcc-61e0a16e063a/manager/0.log"
Sep 11 14:05:39.970435 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:05:39.970394    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-42rpr_d97fd737-e7b3-4ffa-a705-a800fa9090fb/manager/0.log"
Sep 11 14:05:40.116655 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:05:40.116568    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6b447f7bf9-pj4gf_34b3897d-3f58-43bc-953f-3e7b3b6396e6/manager/0.log"
Sep 11 14:05:40.808781 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:05:40.808742    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-9576ccb85-jz224_5b7f1cc4-b656-46e4-bfa0-2248d0e988be/manager/0.log"
Sep 11 14:05:40.816131 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:05:40.816110    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-9576ccb85-jz224_5b7f1cc4-b656-46e4-bfa0-2248d0e988be/copy-manifests/0.log"
Sep 11 14:05:41.369283 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:05:41.369253    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-f8bb4c8cc-xtlrq_9ca8c1d3-a6f9-4e9a-bcf3-f0079b3f4a48/maas-api/0.log"
Sep 11 14:05:41.776330 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:05:41.776271    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-jscrc_0220ee9d-6757-41c7-a654-bb7f56272bcc/postgres/0.log"
Sep 11 14:05:42.852942 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:05:42.852911    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-8tfmj_ce37c0ef-3e90-415d-bf21-ec4eef68401e/manager/0.log"
Sep 11 14:05:42.974136 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:05:42.974096    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-g7qtw_adcd135f-bce8-410d-a231-0ae0660f8e32/manager/0.log"
Sep 11 14:05:43.235093 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:05:43.235063    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-v227b_3350e6f5-a27b-43e2-ac61-ff4b1173c78d/manager/0.log"
Sep 11 14:05:43.529118 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:05:43.529045    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-xxs4x_a5d2f9a4-004d-4efe-b282-6ba0a4621aa8/manager/0.log"
Sep 11 14:05:44.195357 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:05:44.195328    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7c9bdb7954-fwnlt_d1d8740e-7e95-42db-9247-3613ec035a2d/kube-auth-proxy/0.log"
Sep 11 14:05:44.466664 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:05:44.466636    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-6kwrp_b1f2ae7a-43d1-4043-a1fc-9d59b34d1b90/istio-proxy/1.log"
Sep 11 14:05:44.617449 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:05:44.617416    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-c8r4s_cfdc9b69-bbd2-4073-8010-782f82019e75/payload-pre-processing/0.log"
Sep 11 14:05:44.766075 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:05:44.765996    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-4cgf5_773e0144-50b8-4d59-8fed-57d39bb74da2/payload-processing/0.log"
Sep 11 14:05:44.940728 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:05:44.940698    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7798cf8dd4-5hzkk_a93d3879-2973-4fa1-bd7d-bc855a257ddf/router/0.log"
Sep 11 14:05:45.646763 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:05:45.646737    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-bcc9ff8d8-csbw2_afbfd468-29d1-4f10-9703-52cfa77cf44d/main/0.log"
Sep 11 14:05:46.042445 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:05:46.042413    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-h7xf4_375e4146-1062-42bd-83e7-c243c0ce730f/main/0.log"
Sep 11 14:05:56.380026 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:05:56.379994    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-m6d9g_be2cb948-420f-4f91-b56b-a292efb25181/global-pull-secret-syncer/0.log"
Sep 11 14:05:56.481967 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:05:56.481922    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-v874k_61643763-a1e1-4b2f-8e7c-548d0618d90c/konnectivity-agent/0.log"
Sep 11 14:05:56.557998 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:05:56.557969    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-131-86.ec2.internal_ac822255a7ab3b2619823b5af72a01ec/haproxy/0.log"
Sep 11 14:06:01.421591 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:06:01.421542    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-8tfmj_ce37c0ef-3e90-415d-bf21-ec4eef68401e/manager/0.log"
Sep 11 14:06:01.446672 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:06:01.446639    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-g7qtw_adcd135f-bce8-410d-a231-0ae0660f8e32/manager/0.log"
Sep 11 14:06:01.529312 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:06:01.529276    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-v227b_3350e6f5-a27b-43e2-ac61-ff4b1173c78d/manager/0.log"
Sep 11 14:06:01.644234 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:06:01.644200    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-xxs4x_a5d2f9a4-004d-4efe-b282-6ba0a4621aa8/manager/0.log"
Sep 11 14:06:02.938816 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:06:02.938743    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_2ee7ae7e-c851-4356-899c-b7ad526d2c1f/alertmanager/0.log"
Sep 11 14:06:02.961643 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:06:02.961618    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_2ee7ae7e-c851-4356-899c-b7ad526d2c1f/config-reloader/0.log"
Sep 11 14:06:02.983708 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:06:02.983688    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_2ee7ae7e-c851-4356-899c-b7ad526d2c1f/kube-rbac-proxy-web/0.log"
Sep 11 14:06:03.008924 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:06:03.008807    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_2ee7ae7e-c851-4356-899c-b7ad526d2c1f/kube-rbac-proxy/0.log"
Sep 11 14:06:03.036114 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:06:03.036091    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_2ee7ae7e-c851-4356-899c-b7ad526d2c1f/kube-rbac-proxy-metric/0.log"
Sep 11 14:06:03.064233 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:06:03.064211    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_2ee7ae7e-c851-4356-899c-b7ad526d2c1f/prom-label-proxy/0.log"
Sep 11 14:06:03.089247 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:06:03.089226    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_2ee7ae7e-c851-4356-899c-b7ad526d2c1f/init-config-reloader/0.log"
Sep 11 14:06:03.178084 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:06:03.178049    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-ldgml_eb050379-a425-4c99-81d2-4f07256eb6dc/kube-state-metrics/0.log"
Sep 11 14:06:03.201599 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:06:03.201507    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-ldgml_eb050379-a425-4c99-81d2-4f07256eb6dc/kube-rbac-proxy-main/0.log"
Sep 11 14:06:03.233776 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:06:03.233747    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-ldgml_eb050379-a425-4c99-81d2-4f07256eb6dc/kube-rbac-proxy-self/0.log"
Sep 11 14:06:03.266300 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:06:03.266249    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-6c44ff6d57-l66j4_8e49bb1c-288a-429e-8210-3f38ec6f9b43/metrics-server/0.log"
Sep 11 14:06:03.348080 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:06:03.348050    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-gscfs_7b15b37c-f15c-454a-a7d6-871e4b808b66/node-exporter/0.log"
Sep 11 14:06:03.373192 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:06:03.373156    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-gscfs_7b15b37c-f15c-454a-a7d6-871e4b808b66/kube-rbac-proxy/0.log"
Sep 11 14:06:03.403147 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:06:03.403116    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-gscfs_7b15b37c-f15c-454a-a7d6-871e4b808b66/init-textfile/0.log"
Sep 11 14:06:03.612875 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:06:03.612819    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-jnwg7_4f231d67-fe86-4633-b1d9-050a1b9935d9/kube-rbac-proxy-main/0.log"
Sep 11 14:06:03.635470 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:06:03.635435    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-jnwg7_4f231d67-fe86-4633-b1d9-050a1b9935d9/kube-rbac-proxy-self/0.log"
Sep 11 14:06:03.658568 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:06:03.658549    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-jnwg7_4f231d67-fe86-4633-b1d9-050a1b9935d9/openshift-state-metrics/0.log"
Sep 11 14:06:03.694863 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:06:03.694844    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_8a7d435a-339c-4f49-9218-dc6812e9479e/prometheus/0.log"
Sep 11 14:06:03.721470 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:06:03.721454    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_8a7d435a-339c-4f49-9218-dc6812e9479e/config-reloader/0.log"
Sep 11 14:06:03.744432 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:06:03.744410    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_8a7d435a-339c-4f49-9218-dc6812e9479e/thanos-sidecar/0.log"
Sep 11 14:06:03.767307 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:06:03.767286    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_8a7d435a-339c-4f49-9218-dc6812e9479e/kube-rbac-proxy-web/0.log"
Sep 11 14:06:03.796648 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:06:03.796627    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_8a7d435a-339c-4f49-9218-dc6812e9479e/kube-rbac-proxy/0.log"
Sep 11 14:06:03.821718 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:06:03.821673    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_8a7d435a-339c-4f49-9218-dc6812e9479e/kube-rbac-proxy-thanos/0.log"
Sep 11 14:06:03.845264 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:06:03.845242    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_8a7d435a-339c-4f49-9218-dc6812e9479e/init-config-reloader/0.log"
Sep 11 14:06:03.886279 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:06:03.886213    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-hqsnj_64ef96a4-569e-4a9c-9f3c-cc72a2db18eb/prometheus-operator/0.log"
Sep 11 14:06:03.905177 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:06:03.905152    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-hqsnj_64ef96a4-569e-4a9c-9f3c-cc72a2db18eb/kube-rbac-proxy/0.log"
Sep 11 14:06:03.932335 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:06:03.932309    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-5864cd99d7-vz6d4_cd48dfdc-fee8-46a7-b05f-b1180b7bec63/prometheus-operator-admission-webhook/0.log"
Sep 11 14:06:04.053099 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:06:04.053072    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-79dbffb978-qgmlk_b36694c5-a445-43e4-b3e1-16c5fa7a1694/thanos-query/0.log"
Sep 11 14:06:04.079299 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:06:04.079268    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-79dbffb978-qgmlk_b36694c5-a445-43e4-b3e1-16c5fa7a1694/kube-rbac-proxy-web/0.log"
Sep 11 14:06:04.103320 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:06:04.103297    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-79dbffb978-qgmlk_b36694c5-a445-43e4-b3e1-16c5fa7a1694/kube-rbac-proxy/0.log"
Sep 11 14:06:04.127404 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:06:04.127379    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-79dbffb978-qgmlk_b36694c5-a445-43e4-b3e1-16c5fa7a1694/prom-label-proxy/0.log"
Sep 11 14:06:04.155685 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:06:04.155602    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-79dbffb978-qgmlk_b36694c5-a445-43e4-b3e1-16c5fa7a1694/kube-rbac-proxy-rules/0.log"
Sep 11 14:06:04.190441 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:06:04.190418    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-79dbffb978-qgmlk_b36694c5-a445-43e4-b3e1-16c5fa7a1694/kube-rbac-proxy-metrics/0.log"
Sep 11 14:06:06.335170 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:06:06.335135    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-jwnld/perf-node-gather-daemonset-bklnq"]
Sep 11 14:06:06.335615 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:06:06.335602    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3015eac6-2320-4916-82ab-fd3b873bbeae" containerName="maas-api"
Sep 11 14:06:06.335661 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:06:06.335617    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="3015eac6-2320-4916-82ab-fd3b873bbeae" containerName="maas-api"
Sep 11 14:06:06.335703 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:06:06.335695    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="3015eac6-2320-4916-82ab-fd3b873bbeae" containerName="maas-api"
Sep 11 14:06:06.339342 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:06:06.339323    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jwnld/perf-node-gather-daemonset-bklnq"
Sep 11 14:06:06.344838 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:06:06.344815    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-jwnld\"/\"default-dockercfg-zb6z7\""
Sep 11 14:06:06.345564 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:06:06.345543    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-jwnld\"/\"kube-root-ca.crt\""
Sep 11 14:06:06.351611 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:06:06.351595    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-jwnld\"/\"openshift-service-ca.crt\""
Sep 11 14:06:06.360102 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:06:06.360080    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-jwnld/perf-node-gather-daemonset-bklnq"]
Sep 11 14:06:06.539317 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:06:06.539276    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2a6ff528-8daf-4507-9f49-a5cf0fcf89a1-proc\") pod \"perf-node-gather-daemonset-bklnq\" (UID: \"2a6ff528-8daf-4507-9f49-a5cf0fcf89a1\") " pod="openshift-must-gather-jwnld/perf-node-gather-daemonset-bklnq"
Sep 11 14:06:06.539504 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:06:06.539343    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2a6ff528-8daf-4507-9f49-a5cf0fcf89a1-lib-modules\") pod \"perf-node-gather-daemonset-bklnq\" (UID: \"2a6ff528-8daf-4507-9f49-a5cf0fcf89a1\") " pod="openshift-must-gather-jwnld/perf-node-gather-daemonset-bklnq"
Sep 11 14:06:06.539504 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:06:06.539395    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2a6ff528-8daf-4507-9f49-a5cf0fcf89a1-podres\") pod \"perf-node-gather-daemonset-bklnq\" (UID: \"2a6ff528-8daf-4507-9f49-a5cf0fcf89a1\") " pod="openshift-must-gather-jwnld/perf-node-gather-daemonset-bklnq"
Sep 11 14:06:06.539504 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:06:06.539413    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2a6ff528-8daf-4507-9f49-a5cf0fcf89a1-sys\") pod \"perf-node-gather-daemonset-bklnq\" (UID: \"2a6ff528-8daf-4507-9f49-a5cf0fcf89a1\") " pod="openshift-must-gather-jwnld/perf-node-gather-daemonset-bklnq"
Sep 11 14:06:06.539504 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:06:06.539492    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqp9f\" (UniqueName: \"kubernetes.io/projected/2a6ff528-8daf-4507-9f49-a5cf0fcf89a1-kube-api-access-tqp9f\") pod \"perf-node-gather-daemonset-bklnq\" (UID: \"2a6ff528-8daf-4507-9f49-a5cf0fcf89a1\") " pod="openshift-must-gather-jwnld/perf-node-gather-daemonset-bklnq"
Sep 11 14:06:06.640818 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:06:06.640713    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2a6ff528-8daf-4507-9f49-a5cf0fcf89a1-podres\") pod \"perf-node-gather-daemonset-bklnq\" (UID: \"2a6ff528-8daf-4507-9f49-a5cf0fcf89a1\") " pod="openshift-must-gather-jwnld/perf-node-gather-daemonset-bklnq"
Sep 11 14:06:06.640818 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:06:06.640756    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2a6ff528-8daf-4507-9f49-a5cf0fcf89a1-sys\") pod \"perf-node-gather-daemonset-bklnq\" (UID: \"2a6ff528-8daf-4507-9f49-a5cf0fcf89a1\") " pod="openshift-must-gather-jwnld/perf-node-gather-daemonset-bklnq"
Sep 11 14:06:06.640818 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:06:06.640798    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqp9f\" (UniqueName: \"kubernetes.io/projected/2a6ff528-8daf-4507-9f49-a5cf0fcf89a1-kube-api-access-tqp9f\") pod \"perf-node-gather-daemonset-bklnq\" (UID: \"2a6ff528-8daf-4507-9f49-a5cf0fcf89a1\") " pod="openshift-must-gather-jwnld/perf-node-gather-daemonset-bklnq"
Sep 11 14:06:06.641134 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:06:06.640889    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2a6ff528-8daf-4507-9f49-a5cf0fcf89a1-sys\") pod \"perf-node-gather-daemonset-bklnq\" (UID: \"2a6ff528-8daf-4507-9f49-a5cf0fcf89a1\") " pod="openshift-must-gather-jwnld/perf-node-gather-daemonset-bklnq"
Sep 11 14:06:06.641134 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:06:06.640928    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2a6ff528-8daf-4507-9f49-a5cf0fcf89a1-proc\") pod \"perf-node-gather-daemonset-bklnq\" (UID: \"2a6ff528-8daf-4507-9f49-a5cf0fcf89a1\") " pod="openshift-must-gather-jwnld/perf-node-gather-daemonset-bklnq"
Sep 11 14:06:06.641134 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:06:06.640936    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2a6ff528-8daf-4507-9f49-a5cf0fcf89a1-podres\") pod \"perf-node-gather-daemonset-bklnq\" (UID: \"2a6ff528-8daf-4507-9f49-a5cf0fcf89a1\") " pod="openshift-must-gather-jwnld/perf-node-gather-daemonset-bklnq"
Sep 11 14:06:06.641134 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:06:06.640989    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2a6ff528-8daf-4507-9f49-a5cf0fcf89a1-lib-modules\") pod \"perf-node-gather-daemonset-bklnq\" (UID: \"2a6ff528-8daf-4507-9f49-a5cf0fcf89a1\") " pod="openshift-must-gather-jwnld/perf-node-gather-daemonset-bklnq"
Sep 11 14:06:06.641134 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:06:06.640998    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2a6ff528-8daf-4507-9f49-a5cf0fcf89a1-proc\") pod \"perf-node-gather-daemonset-bklnq\" (UID: \"2a6ff528-8daf-4507-9f49-a5cf0fcf89a1\") " pod="openshift-must-gather-jwnld/perf-node-gather-daemonset-bklnq"
Sep 11 14:06:06.641134 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:06:06.641091    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2a6ff528-8daf-4507-9f49-a5cf0fcf89a1-lib-modules\") pod \"perf-node-gather-daemonset-bklnq\" (UID: \"2a6ff528-8daf-4507-9f49-a5cf0fcf89a1\") " pod="openshift-must-gather-jwnld/perf-node-gather-daemonset-bklnq"
Sep 11 14:06:06.661557 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:06:06.661519    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqp9f\" (UniqueName: \"kubernetes.io/projected/2a6ff528-8daf-4507-9f49-a5cf0fcf89a1-kube-api-access-tqp9f\") pod \"perf-node-gather-daemonset-bklnq\" (UID: \"2a6ff528-8daf-4507-9f49-a5cf0fcf89a1\") " pod="openshift-must-gather-jwnld/perf-node-gather-daemonset-bklnq"
Sep 11 14:06:06.949333 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:06:06.949224    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jwnld/perf-node-gather-daemonset-bklnq"
Sep 11 14:06:07.915326 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:06:07.915291    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-jwnld/perf-node-gather-daemonset-bklnq"]
Sep 11 14:06:07.922490 ip-10-0-131-86 kubenswrapper[2578]: W0911 14:06:07.922465    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod2a6ff528_8daf_4507_9f49_a5cf0fcf89a1.slice/crio-ce5229d1cbbcae1568985fb8238365d29631a92ffd9fd8735dbfb1a19cdf76ee WatchSource:0}: Error finding container ce5229d1cbbcae1568985fb8238365d29631a92ffd9fd8735dbfb1a19cdf76ee: Status 404 returned error can't find the container with id ce5229d1cbbcae1568985fb8238365d29631a92ffd9fd8735dbfb1a19cdf76ee
Sep 11 14:06:08.600725 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:06:08.600686    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jwnld/perf-node-gather-daemonset-bklnq" event={"ID":"2a6ff528-8daf-4507-9f49-a5cf0fcf89a1","Type":"ContainerStarted","Data":"38086726697bf58da1e5aed5177e566a3d6e17c050bca7a13a97c754e75e75f4"}
Sep 11 14:06:08.600954 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:06:08.600746    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jwnld/perf-node-gather-daemonset-bklnq" event={"ID":"2a6ff528-8daf-4507-9f49-a5cf0fcf89a1","Type":"ContainerStarted","Data":"ce5229d1cbbcae1568985fb8238365d29631a92ffd9fd8735dbfb1a19cdf76ee"}
Sep 11 14:06:08.600954 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:06:08.600789    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-jwnld/perf-node-gather-daemonset-bklnq"
Sep 11 14:06:09.098121 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:06:09.098065    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-jwnld/perf-node-gather-daemonset-bklnq" podStartSLOduration=3.098043431 podStartE2EDuration="3.098043431s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 14:06:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:06:09.094202175 +0000 UTC m=+2676.480305338" watchObservedRunningTime="2026-09-11 14:06:09.098043431 +0000 UTC m=+2676.484146596"
Sep 11 14:06:11.996259 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:06:11.996215    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-pfffb_5ab38ce4-40c2-4630-98f3-62d1b84532a5/dns/0.log"
Sep 11 14:06:12.040362 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:06:12.040330    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-pfffb_5ab38ce4-40c2-4630-98f3-62d1b84532a5/kube-rbac-proxy/0.log"
Sep 11 14:06:12.184571 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:06:12.184542    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-z8p25_f7ab88b5-7073-460e-b3d4-a9841f50b185/dns-node-resolver/0.log"
Sep 11 14:06:13.042593 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:06:13.042560    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-8jj8l_d9bf058c-d89b-4236-86ce-3dee6f389711/node-ca/0.log"
Sep 11 14:06:14.616295 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:06:14.616268    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-jwnld/perf-node-gather-daemonset-bklnq"
Sep 11 14:06:14.638341 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:06:14.638312    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7c9bdb7954-fwnlt_d1d8740e-7e95-42db-9247-3613ec035a2d/kube-auth-proxy/0.log"
Sep 11 14:06:14.725079 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:06:14.725050    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-6kwrp_b1f2ae7a-43d1-4043-a1fc-9d59b34d1b90/istio-proxy/0.log"
Sep 11 14:06:14.730243 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:06:14.730220    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-6kwrp_b1f2ae7a-43d1-4043-a1fc-9d59b34d1b90/istio-proxy/1.log"
Sep 11 14:06:14.779518 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:06:14.779493    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-c8r4s_cfdc9b69-bbd2-4073-8010-782f82019e75/payload-pre-processing/0.log"
Sep 11 14:06:14.827749 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:06:14.827717    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-4cgf5_773e0144-50b8-4d59-8fed-57d39bb74da2/payload-processing/0.log"
Sep 11 14:06:14.883958 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:06:14.883871    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7798cf8dd4-5hzkk_a93d3879-2973-4fa1-bd7d-bc855a257ddf/router/0.log"
Sep 11 14:06:16.148726 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:06:16.148701    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-dcgnn_42df0901-f4f9-4a7e-85da-4551acd80be7/serve-healthcheck-canary/0.log"
Sep 11 14:06:16.878307 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:06:16.878259    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-m686n_f833a58e-8d54-42fc-848c-bea07b176b0b/insights-operator/0.log"
Sep 11 14:06:16.880090 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:06:16.880070    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-m686n_f833a58e-8d54-42fc-848c-bea07b176b0b/insights-operator/1.log"
Sep 11 14:06:17.036106 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:06:17.036077    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-dsppd_9d1ee163-9447-4e3f-a7d5-90f8210d9c2f/kube-rbac-proxy/0.log"
Sep 11 14:06:17.098394 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:06:17.098369    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-dsppd_9d1ee163-9447-4e3f-a7d5-90f8210d9c2f/exporter/0.log"
Sep 11 14:06:17.153590 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:06:17.153559    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-dsppd_9d1ee163-9447-4e3f-a7d5-90f8210d9c2f/extractor/0.log"
Sep 11 14:06:18.866112 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:06:18.866025    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7c9bdb7954-fwnlt_d1d8740e-7e95-42db-9247-3613ec035a2d/kube-auth-proxy/0.log"
Sep 11 14:06:18.984140 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:06:18.984099    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-6kwrp_b1f2ae7a-43d1-4043-a1fc-9d59b34d1b90/istio-proxy/0.log"
Sep 11 14:06:18.989188 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:06:18.989164    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-6kwrp_b1f2ae7a-43d1-4043-a1fc-9d59b34d1b90/istio-proxy/1.log"
Sep 11 14:06:19.034349 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:06:19.034316    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-c8r4s_cfdc9b69-bbd2-4073-8010-782f82019e75/payload-pre-processing/0.log"
Sep 11 14:06:19.086452 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:06:19.086423    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-4cgf5_773e0144-50b8-4d59-8fed-57d39bb74da2/payload-processing/0.log"
Sep 11 14:06:19.121726 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:06:19.121644    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7798cf8dd4-5hzkk_a93d3879-2973-4fa1-bd7d-bc855a257ddf/router/0.log"
Sep 11 14:06:19.858197 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:06:19.858164    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-5clkn_723fc010-ed76-4913-8dcc-61e0a16e063a/manager/0.log"
Sep 11 14:06:20.359973 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:06:20.359937    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-42rpr_d97fd737-e7b3-4ffa-a705-a800fa9090fb/manager/0.log"
Sep 11 14:06:20.439661 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:06:20.439632    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6b447f7bf9-pj4gf_34b3897d-3f58-43bc-953f-3e7b3b6396e6/manager/0.log"
Sep 11 14:06:20.861292 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:06:20.861260    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-9576ccb85-jz224_5b7f1cc4-b656-46e4-bfa0-2248d0e988be/manager/0.log"
Sep 11 14:06:20.891531 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:06:20.891501    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-9576ccb85-jz224_5b7f1cc4-b656-46e4-bfa0-2248d0e988be/copy-manifests/0.log"
Sep 11 14:06:21.025696 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:06:21.025665    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7c9bdb7954-fwnlt_d1d8740e-7e95-42db-9247-3613ec035a2d/kube-auth-proxy/0.log"
Sep 11 14:06:21.066276 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:06:21.066238    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-6kwrp_b1f2ae7a-43d1-4043-a1fc-9d59b34d1b90/istio-proxy/0.log"
Sep 11 14:06:21.071210 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:06:21.071178    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-6kwrp_b1f2ae7a-43d1-4043-a1fc-9d59b34d1b90/istio-proxy/1.log"
Sep 11 14:06:21.104726 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:06:21.104696    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-c8r4s_cfdc9b69-bbd2-4073-8010-782f82019e75/payload-pre-processing/0.log"
Sep 11 14:06:21.142256 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:06:21.142152    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-4cgf5_773e0144-50b8-4d59-8fed-57d39bb74da2/payload-processing/0.log"
Sep 11 14:06:21.161549 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:06:21.161526    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7798cf8dd4-5hzkk_a93d3879-2973-4fa1-bd7d-bc855a257ddf/router/0.log"
Sep 11 14:06:22.249138 ip-10-0-131-86 kubenswrapper[2578]: I0911 14:06:22.249106    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-7pj9t_c08d81cf-415c-4971-b610-c4e94047eceb/openshift-lws-operator/0.log"